Today I Ran

Last entry left a real question hanging: the exact Podman number got killed under kind, twice. Did that mean the gap between the two runtimes was huge — needing something close to double the memory — or was the real ceiling only a little higher than 1663Mi, and I’d just clipped it? Only one way to find out. Redeployed with the limit bumped to 2048Mi — a deliberately big jump, so the answer would be unambiguous either way: ...

OOMKilled is what Kubernetes says when a container tries to use more memory than its limit allows and the kernel steps in and kills it. Going into this one, my assumption was straightforward: Entry 09 measured 1.663GB under Podman, so setting a Kubernetes limit to that exact number should be enough. Same model, same host, same measurement. That assumption was wrong, and finding out why turned into the most interesting result in the series so far. ...

Before testing whether last entry’s real Podman number (1.663GB) holds up as an actual Kubernetes resource limit, I wanted a baseline: what happens with nothing set at all. kind runs a full local Kubernetes cluster inside containers acting as nodes — Kubernetes on top of the same containers everything else in this series has been using. Spun up a fresh cluster and deployed Ollama with no resources section in the pod spec whatsoever: ...

On Linux, containers run natively — straight on the host kernel. On macOS, they can’t, because containers need a Linux kernel underneath and your Mac isn’t running one. So Podman and Docker Desktop quietly spin up a small Linux VM in the background and put every container inside that instead. Which means every container on a Mac is sharing a fixed slice of memory carved out for that VM — not your Mac’s actual RAM — and that distinction is about to matter a lot. ...

Everything up to this point — building the Chroma index, chunking, querying — was done by hand, one Python command at a time. An MCP server is what turns that into an actual tool: a small program exposing specific capabilities to an AI client, so it can call your stuff directly instead of you running queries yourself every time. What I actually wanted to prove here wasn’t that the server starts. It’s that a real client can reach it and get something useful back. ...

Chunking means splitting a long document into smaller overlapping pieces and embedding each one separately, instead of embedding the whole thing or lopping off the end like Entry 05’s truncation hack. It fixes the content-loss problem cleanly — nothing gets silently dropped. What it doesn’t fix, and what I didn’t see coming until I was staring at the results, is a structural bias: a document split into 17 pieces now has 17 separate shots at showing up in search results, while a short document still only gets one. More chunks means more chances to rank, whether or not any individual chunk is actually the most relevant thing in the store. ...

Building the vector store last time was only half the job. The actual point of embeddings is asking a question and getting back the right document, not just any document that happens to be sitting there. Chroma does this with distance: turn the question into a vector the same way you turned each document into one, then measure how close the question’s vector is to each stored vector. Lower distance, more similar in meaning — at least in theory. The real test isn’t whether it returns something, because it always will. It’s whether the number actually means anything. ...

An embedding model doesn’t generate text — it converts text into a list of numbers that represents what that text means. Two pieces of writing about similar topics end up with similar vectors even if they don’t share a single word, which is the whole trick behind semantic search: you can find “things that mean something like this,” not just “things that contain this exact word.” The detail that actually surprised me: the vector is always the same fixed length no matter how long the source text is. A one-sentence note and a ten-page article both come out as the same-sized list of numbers — a fingerprint, basically, summarizing something much bigger into a fixed shape. ...

A model’s weights are normally stored at high precision — FP16, 16-bit floating point. Quantization compresses those numbers down to fewer bits to shrink the file and memory footprint, trading some precision for space. Same idea as squashing a high-res photo down to a smaller JPEG: the file shrinks a lot, technically some detail is lost, and in practice you usually can’t tell. Before pulling anything new for this one, I ran ollama show llama3.2:1b out of curiosity — and immediately had to correct myself. The model I’d been calling “the default” in Entries 01 and 03 turned out to already be Q8_0, not some unspecified baseline. Worth owning that plainly rather than quietly fixing it: I’d been comparing against a number without actually knowing what it was. ...

The number in a model name — 1b, 3b — is parameter count, roughly the tunable values inside the model that encode whatever it’s learned. More parameters, generally, means better reasoning, at the cost of more memory and slower responses. I wanted an actual number for that tradeoff instead of just nodding along to the general idea, so I pulled the 3-billion-parameter sibling of Entry 01’s model and put them side by side. ...

A system prompt is basically a character brief you hand a model before the real conversation starts — instructions that shape how it behaves for the rest of the session, without you repeating yourself every message. I wanted to see how far that actually holds under pressure, so I tried to build the strictest version I could think of: a model that refuses to explain anything and only ever answers with a real oc command. ...

I wanted to start this series with the most boring possible first step — not because it’s impressive, but because everyone skips it. Every tutorial about running LLMs locally jumps straight to “here’s how you fine-tune it” without ever answering the question I actually had: what does this thing cost me in memory, for real, on my machine? So: installed Ollama, pulled llama3.2:1b (Meta’s smallest Llama 3.2 model — 1 billion parameters, small enough that it downloads in a couple of minutes instead of eating your afternoon), and loaded it with a throwaway prompt just to force it into memory. ...