Byte-size guides on DevOps, Cloud, and AI

Last entry left a real question hanging: the exact Podman number got killed under kind, twice. Did that mean the gap between the two runtimes was huge — needing something close to double the memory — or was the real ceiling only a little higher than 1663Mi, and I’d just clipped it? Only one way to find out. Redeployed with the limit bumped to 2048Mi — a deliberately big jump, so the answer would be unambiguous either way: ...

OOMKilled is what Kubernetes says when a container tries to use more memory than its limit allows and the kernel steps in and kills it. Going into this one, my assumption was straightforward: Entry 09 measured 1.663GB under Podman, so setting a Kubernetes limit to that exact number should be enough. Same model, same host, same measurement. That assumption was wrong, and finding out why turned into the most interesting result in the series so far. ...

Before testing whether last entry’s real Podman number (1.663GB) holds up as an actual Kubernetes resource limit, I wanted a baseline: what happens with nothing set at all. kind runs a full local Kubernetes cluster inside containers acting as nodes — Kubernetes on top of the same containers everything else in this series has been using. Spun up a fresh cluster and deployed Ollama with no resources section in the pod spec whatsoever: ...

On Linux, containers run natively — straight on the host kernel. On macOS, they can’t, because containers need a Linux kernel underneath and your Mac isn’t running one. So Podman and Docker Desktop quietly spin up a small Linux VM in the background and put every container inside that instead. Which means every container on a Mac is sharing a fixed slice of memory carved out for that VM — not your Mac’s actual RAM — and that distinction is about to matter a lot. ...

The Story Back in Article 1, I said we’d get back to this: once you’ve decided what goes in the cloud and who manages it, there’s a third question that decides whether any of it actually works — how does your data get there? A telecom customer I worked with was pushing sustained real-time Kafka streams past 850 Mbps between on-prem and the cloud, with big unpredictable spikes on top, over an AWS Site-to-Site VPN. A single tunnel is rated up to 1.25 Gbps. On paper they had headroom. ...

Everything up to this point — building the Chroma index, chunking, querying — was done by hand, one Python command at a time. An MCP server is what turns that into an actual tool: a small program exposing specific capabilities to an AI client, so it can call your stuff directly instead of you running queries yourself every time. What I actually wanted to prove here wasn’t that the server starts. It’s that a real client can reach it and get something useful back. ...

Chunking means splitting a long document into smaller overlapping pieces and embedding each one separately, instead of embedding the whole thing or lopping off the end like Entry 05’s truncation hack. It fixes the content-loss problem cleanly — nothing gets silently dropped. What it doesn’t fix, and what I didn’t see coming until I was staring at the results, is a structural bias: a document split into 17 pieces now has 17 separate shots at showing up in search results, while a short document still only gets one. More chunks means more chances to rank, whether or not any individual chunk is actually the most relevant thing in the store. ...

Building the vector store last time was only half the job. The actual point of embeddings is asking a question and getting back the right document, not just any document that happens to be sitting there. Chroma does this with distance: turn the question into a vector the same way you turned each document into one, then measure how close the question’s vector is to each stored vector. Lower distance, more similar in meaning — at least in theory. The real test isn’t whether it returns something, because it always will. It’s whether the number actually means anything. ...

An embedding model doesn’t generate text — it converts text into a list of numbers that represents what that text means. Two pieces of writing about similar topics end up with similar vectors even if they don’t share a single word, which is the whole trick behind semantic search: you can find “things that mean something like this,” not just “things that contain this exact word.” The detail that actually surprised me: the vector is always the same fixed length no matter how long the source text is. A one-sentence note and a ten-page article both come out as the same-sized list of numbers — a fingerprint, basically, summarizing something much bigger into a fixed shape. ...

Managed Services vs Self-Hosted: When to Hand Over the Keys ☁ Cloud Without the Chaos — Article 02 You’ve placed the workload in the cloud. That decision is done. Now comes the one nobody gives you a clean framework for: do you run the infrastructure yourself, or do you hand the keys to a managed service? I worked with a customer who answered this the hard way — twice. Once on their own platform, and once after they’d already made the switch. Both incidents involved the same underlying failure mode. Neither one turned out the way anyone in the room expected. ...

A model’s weights are normally stored at high precision — FP16, 16-bit floating point. Quantization compresses those numbers down to fewer bits to shrink the file and memory footprint, trading some precision for space. Same idea as squashing a high-res photo down to a smaller JPEG: the file shrinks a lot, technically some detail is lost, and in practice you usually can’t tell. Before pulling anything new for this one, I ran ollama show llama3.2:1b out of curiosity — and immediately had to correct myself. The model I’d been calling “the default” in Entries 01 and 03 turned out to already be Q8_0, not some unspecified baseline. Worth owning that plainly rather than quietly fixing it: I’d been comparing against a number without actually knowing what it was. ...

⚡ Byte Size Summary See why we shipped an OpenShift diagnostic MCP server as read-only by design, and the RBAC wall that made write access harder than it looks Walk through a real failed remediation test where an agent recommended a correct-looking fix built on stale, deprecated config — and what that failure mode actually is Get the maturity-gated approval architecture we designed for write access — and why it’s still sitting on paper, not in production The Story In Article 06 we wired an n8n workflow to MCP and RAG for automated incident triage. That article ended with a question: what happens when the agent gets a longer leash? ...

The number in a model name — 1b, 3b — is parameter count, roughly the tunable values inside the model that encode whatever it’s learned. More parameters, generally, means better reasoning, at the cost of more memory and slower responses. I wanted an actual number for that tradeoff instead of just nodding along to the general idea, so I pulled the 3-billion-parameter sibling of Entry 01’s model and put them side by side. ...

☁ Cloud Without the Chaos — Article 01 · ⏱ 10 min read ☁ Byte Size Summary Hybrid cloud isn’t a strategy — it’s the outcome of real architectural decisions Most teams default to cloud or on-prem for the wrong reasons (inertia, not intent) Five dimensions drive every defensible placement decision: latency, data sovereignty, cost profile, operational ownership, and blast radius A reusable decision framework for evaluating any workload — run it before the migration starts, not after Network design thinking from logistics applies directly to hybrid cloud architecture — and most platform engineers have never seen it framed this way I spent a few years working in supply chain and logistics before I ended up in cloud infrastructure. One of the first things you learn in that world is that the network is never fully yours. ...

A system prompt is basically a character brief you hand a model before the real conversation starts — instructions that shape how it behaves for the rest of the session, without you repeating yourself every message. I wanted to see how far that actually holds under pressure, so I tried to build the strictest version I could think of: a model that refuses to explain anything and only ever answers with a real oc command. ...

⚡ Byte Size Summary Implement the four-phase Expand/Contract pattern — nullable column, batched backfill, code pivot, then constraint — so schema changes never ship coupled to application code Understand why Argo CD’s “rollback” can revert your container image but cannot touch a database schema, and why that gap is the actual thesis of zero-downtime deployment failures See the enforcement gap that let a smaller version of the same incident happen twice — and why process-only guardrails (PR review, runbooks) fail under deploy pressure The Story In the previous article we shifted from managing Kubernetes to managing applications. This article shows what happens when the database layer doesn’t get the same treatment. ...

I wanted to start this series with the most boring possible first step — not because it’s impressive, but because everyone skips it. Every tutorial about running LLMs locally jumps straight to “here’s how you fine-tune it” without ever answering the question I actually had: what does this thing cost me in memory, for real, on my machine? So: installed Ollama, pulled llama3.2:1b (Meta’s smallest Llama 3.2 model — 1 billion parameters, small enough that it downloads in a couple of minutes instead of eating your afternoon), and loaded it with a throwaway prompt just to force it into memory. ...

⚡ Byte Size Summary Understand why a Helm sub-chart with a hardcoded deprecated API can pass CI, deploy cleanly, and still crash production ten minutes later See why admission-time governance and a managed control plane solve two completely different problems — and why conflating them is how teams get burned Walk away with a clearer sense of where platform ownership should actually sit as an organization scales The Story I was building an optimization model for scheduling — an internal Red Hat project, nothing exotic. Somewhere in the middle of it I noticed something uncomfortable: I wasn’t spending my time on the application anymore. I was spending it on the platform underneath it. Roughly half my working hours were going into upgrade cycles — chasing down which APIs had quietly disappeared and which of my dependencies would break the moment the Kubernetes cluster moved forward a minor version. ...

⚡ Byte Size Summary Wire n8n to an MCP (Model Context Protocol) server over Streamable HTTP and a RAG (Retrieval-Augmented Generation) pipeline to build an automated incident triage workflow — runbook lookup and live cluster diagnostics assembled before a human opens a single dashboard The moment you place an LLM agent node inside an n8n workflow, execution becomes non-deterministic — the model decides how many tool calls to make, so your timeout and call limits must assume worst case, not average case Start with hard execution limits (5-minute timeout, 2-3 max calls per run) and loosen as you validate — a runaway workflow with no bounds will exhaust API quotas and flood notification channels before you notice The Story I started using n8n because I watched a teammate build a workflow in twenty minutes that would have taken me a day to write in Python. The visual canvas, the drag-and-drop nodes, the instant execution feedback — it made automation feel fast. So I started experimenting. ...

⚡ Byte Size Summary Deploy a LiteLLM proxy on OpenShift to decouple your applications from individual LLM providers — switch or failover between models by changing a ConfigMap, not your code Inject cluster-specific context (API versions, cloud annotations, golden templates) before every prompt to prevent models from generating “plausible averages” that look correct but fail on your platform Validate every generated manifest with oc apply --dry-run=server before applying — the cluster’s API server catches structural errors that no amount of prompt engineering will prevent The Story I was building a GenAI data poisoning demo for a proof of concept on ROSA (Red Hat OpenShift on AWS). The demo needed OpenShift-native manifests — Routes with TLS termination, proper SecurityContextConstraints, current API versions. I built it using Claude, and it worked. The model understood OpenShift. The manifests deployed cleanly. The demo ran on the cluster. ...

Pipelines in the Wild #4 Byte Size Summary Red Hat OpenShift Service on AWS (ROSA), Azure Red Hat OpenShift (ARO), and OpenShift Dedicated on GCP (OSD) are mature, SRE-operated platforms — same OpenShift surface, three clouds, control plane work you don’t own. When prerequisites are met, installs complete in the documented window. Enterprise timelines stretch when governance approval tracks and Terraform state hygiene are treated as afterthoughts, not when the platform fails. This article separates the solid platform from the enterprise wrapper, then covers what actually prevents pain on the IaC side: remote state before the first resource, scheduled drift detection, and platform-specific recovery when partial applies leave residue behind. ...

⚡ Byte Size Summary Store prompts as versioned YAML manifests in Git and run them through a three-stage GitHub Actions gate — schema validation, secret scanning with gitleaks, and model policy enforcement — before any LLM call reaches your OpenShift environment A CI-gated prompt pipeline gives your enterprise auditors a traceable answer to “what prompt was active during the incident window” — without it, the forensic work is manual, billed, and slow Prompt versioning is necessary but not sufficient: you’re versioning one variable in a system with multiple unversioned dependencies, and this article shows you what to do about the rest of them The Story I was presenting a prototype at a conference. The demo was built over three weeks of late-night sessions — an AI-assisted operations assistant for OpenShift that could answer runbook-style questions against live cluster state. The architecture was solid. The underlying idea was good. ...

⚡ Byte Size Summary Secret management failures are invisible until they cause a production incident — start with RBAC and namespace isolation before the first workload goes live Storing secrets in a central vault solves the sprawl problem but introduces a new failure mode: rotation lag between the vault and the namespace-level Kubernetes secret The real unsolved problem isn’t technical — it’s knowing who owns the approval and escalation path when a credential rotates at 2 AM across a multi-timezone team The Story In Article 02 we built retry logic and tiered alerting so transient pipeline failures don’t wake anyone up. This article tackles the credential layer underneath — because a retry won’t help when the secret itself is missing. ...

Byte Size Summary Most pipeline failures are transient — a registry returning a 503, a smoke test catching a slow cold start, a network blip during an image push. Retrying them automatically, with exponential backoff, means engineers never see them. The failures that reach a human should be the ones that actually need one. This article builds a retry wrapper and a three-tier alerting system (transient → silent, degraded → Slack warning, critical → PagerDuty page) on top of a GitHub Actions blue/green deploy workflow. The demo application is Waybill — a FastAPI shipment tracking API backed by PostgreSQL, where the health endpoint checks real database connectivity rather than returning a static 200. That distinction matters: a smoke test that only checks HTTP status is a smoke test that passes while your database is unreachable. By the end you’ll have a working repo you can run locally with Docker Compose and test today. ...

⚡ Byte Size Summary MCP (Model Context Protocol) is the standard that lets AI agents interact with external systems — your cluster, your observability stack, your ticketing system — without bespoke integration code for every tool. MCP directly addresses AI hallucination and 2AM incident response by grounding AI answers in live system state. It doesn’t solve tribal knowledge alone — that needs RAG alongside it. This article covers the production-grade architecture: what MCP servers are, how to design them for platform engineering use cases, and what you need to get right before running them anywhere near production. In Article 01 we built the evaluation framework. In Article 02 we built a RAG pipeline for static runbook knowledge. Now we give AI agents access to live infrastructure. ...

⚡ Byte Size Summary RAG inserts a retrieval layer between your existing runbooks and an LLM — answers come from your documentation, not generic training data, with source citations included. This article builds a complete FastAPI service with /ingest, /query, and /health endpoints, using OpenAI embeddings and Chroma as the vector store. Everything is cloneable from GitHub. The goal isn’t to replace your runbooks. It’s to make them queryable at the moment an incident is happening. In Article 01 we built a framework for evaluating where AI actually changes outcomes in platform engineering. This article implements the first concrete use case: making your existing runbooks semantically searchable. ...

Byte size summary After reading this article, you will know how to implement a blue/green deployment pipeline on OpenShift that uses HAProxy-backed Route weight splitting for traffic control and Flagsmith for feature flag management — and more importantly, you will know where the implementation breaks silently. Specifically: the HAProxy propagation gap that lets your smoke tests lie to you, the partial rollout state that puts two versions in production simultaneously, and why the standard approach of patching a Route weight and immediately proceeding has cost teams I’ve worked with entire migrations. The implementation uses GitHub Actions for orchestration, oc commands for OpenShift-specific traffic control, and Flagsmith as the feature flag service. The patterns apply to AKS, EKS, and GKE with platform-specific variations called out. ...

Byte size summary After reading this article, you’ll have a framework for evaluating AI tools in platform engineering contexts — not by capability type, but by where in your workflow the tool actually changes the outcome. You’ll understand why the tools that sound most compelling are still hype, where genuine productivity gains exist today, and what governance infrastructure you need in place before any AI component gets near production. This article is the foundation for the series; subsequent articles implement each touch point against real OpenShift infrastructure. ...

I Still Remember the Sound Forklifts beeping in reverse. Conveyor belts humming. Cold warehouse air hitting my face as I stood on the floor of a Delphi plant in 2002. I was staring at a maze of pallets, racks, and production lines, trying to redesign the entire material movement system. I had a chemical engineering degree, a head full of equations, and absolutely no idea how this moment would shape the next 20 years of my career. ...

The Meeting That Changed How I Prepare I used to spend hours before important workshops and client meetings pulling together materials. Downloading presentation decks, saving links, reading through documents, trying to hold enough in my head to answer questions confidently in the room. Then I discovered NotebookLM. NotebookLM is a generative AI tool from Google that lets you upload your own documents, presentations, and links and then have a conversation with all of that material as if it were a knowledgeable colleague who had read everything perfectly. ...

I Was Selling AI Before Most People Knew What It Was A decade ago I was selling predictive and prescriptive analytics solutions to enterprise clients. Tools like SPSS Modeler — IBM’s data science platform for predicting future outcomes — and CPLEX, the optimisation engine we talked about in Article 6, which solved complex scheduling and logistics problems for supply chain and warehouse operations. Back then AI wasn’t a word that appeared in everyday conversation. It lived in university research departments, specialist software vendors, and the back offices of large corporations with data science teams. It was powerful, it was real, and almost nobody outside of those environments knew it existed. ...

The Problem With Doing It By Hand Early in my Cloud and Infrastructure career I watched a colleague spend three days manually building out a production environment on Azure. Clicking through dashboards, configuring virtual networks, setting up security groups, deploying OpenShift, installing operators. Three days of careful, methodical work. Two weeks later, we needed an identical environment for testing. Nobody could remember exactly what had been clicked, in what order, with what settings. The tribal knowledge lived entirely in one person’s head — and that person was on holiday. What followed was a painful reconstruction exercise involving guesswork, old notes, and a lot of “I think this is how we did it.” ...

From Supply Chain to Container Orchestration When IBM acquired Red Hat, I was working as a technical seller trying to position IBM’s data science platform to clients. Our internal team was containerising CPLEX — a powerful optimisation engine used in warehouse management and supply chain applications — and running it on OpenShift. I had seen CPLEX solve complex scheduling problems in the real world. I understood inputs, equations, constraints, and outputs. But pods? Nodes? Dockerfiles? It felt like a science project. I couldn’t connect what I was seeing on screen to anything that could work in real production. ...

The First Time I Saw a Pipeline Run I still remember the first time I watched a CI/CD pipeline run from start to finish. A developer pushed their code to GitHub. Within seconds, a series of automated steps fired off on their own — tests ran, the application was packaged into a container, and it was deployed to a live environment. Nobody pressed a button. Nobody sent an email saying “please deploy this.” It just happened. ...

The Moment Someone Finally Explained Containers to Me When IBM acquired Red Hat, my world changed overnight. Suddenly everyone around me was talking about containers. Kubernetes. Pods. Orchestration. I was nodding along in meetings while internally having absolutely no idea what any of it meant. My background was in supply chain and logistics. I understood how physical goods moved around the world — warehouses, pallets, shipping routes. But containers in software? That meant nothing to me. ...

The Day I Nearly Broke Production Early in my Cloud career, I was making what I thought was a small change to a configuration file. I’d been working on it locally, testing it, feeling confident. Then I committed and pushed — straight to the main branch. Within seconds, a CI/CD pipeline picked up the change and started deploying it to production. My phone buzzed with an alert. My colleague looked over and said, very calmly, “Did you just push to main?” ...

The Day I Realised Linux Was Everywhere When I first started working in Cloud and Infrastructure, I assumed most servers ran Windows — because that’s what I grew up using on my laptop. Then I got access to my first cloud environment and was greeted with a black screen, a blinking cursor, and absolutely no Start menu in sight. That was my introduction to Linux. I typed dir (the Windows command for listing files) and got an error. I tried clicking around and realised there was nothing to click. Just me, a terminal, and a lot to learn. ...

Ever Wondered How Netflix Never Seems to Go Down? Think about this for a second. Netflix has over 260 million subscribers worldwide. People are watching shows in Tokyo, London, Lagos, and New York — all at the same time. And yet, when was the last time Netflix crashed on you? Now think about your favourite food delivery app. You open it, order food, track your driver in real time, and get a notification the moment your burger arrives. All of that happens in seconds. ...