Build an AI Incident Triage Workflow in n8n with MCP
🤖 AI in the Stack #6 Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI ⚡ Byte Size Summary Wire n8n to an MCP (Model Context Protocol) server over Streamable HTTP and a RAG (Retrieval-Augmented Generation) pipeline to build an automated incident triage workflow — runbook lookup and live cluster diagnostics assembled before a human opens a single dashboard The moment you place an LLM agent node inside an n8n workflow, execution becomes non-deterministic — the model decides how many tool calls to make, so your timeout and call limits must assume worst case, not average case Start with hard execution limits (5-minute timeout, 2-3 max calls per run) and loosen as you validate — a runaway workflow with no bounds will exhaust API quotas and flood notification channels before you notice The Story I started using n8n because I watched a teammate build a workflow in twenty minutes that would have taken me a day to write in Python. The visual canvas, the drag-and-drop nodes, the instant execution feedback — it made automation feel fast. So I started experimenting. ...
Swapping LLM Providers Without Rewriting Your Stack
🤖 AI in the Stack #5 Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI ⚡ Byte Size Summary Deploy a LiteLLM proxy on OpenShift to decouple your applications from individual LLM providers — switch or failover between models by changing a ConfigMap, not your code Inject cluster-specific context (API versions, cloud annotations, golden templates) before every prompt to prevent models from generating “plausible averages” that look correct but fail on your platform Validate every generated manifest with oc apply --dry-run=server before applying — the cluster’s API server catches structural errors that no amount of prompt engineering will prevent The Story I was building a GenAI data poisoning demo for a proof of concept on ROSA (Red Hat OpenShift on AWS). The demo needed OpenShift-native manifests — Routes with TLS termination, proper SecurityContextConstraints, current API versions. I built it using Claude, and it worked. The model understood OpenShift. The manifests deployed cleanly. The demo ran on the cluster. ...
Managed OpenShift, Lost State, and Daily Drift Checks
🛠️ Pipelines in the Wild #4 Byte Size Summary Red Hat OpenShift Service on AWS (ROSA), Azure Red Hat OpenShift (ARO), and OpenShift Dedicated on GCP (OSD) are mature, SRE-operated platforms — same OpenShift surface, three clouds, control plane work you do not own. When prerequisites are met, installs complete in the documented window. Enterprise timelines stretch when governance approval tracks and Terraform state hygiene are treated as afterthoughts, not when the platform fails. This article separates the solid platform from the enterprise wrapper, then covers what actually prevents pain on the IaC side: remote state before the first resource, scheduled drift detection, and platform-specific recovery when partial applies leave residue behind. ...
Treat Prompts Like Code: A CI Gate for LLM Workflows on OpenShift
🤖 AI in the Stack #4 Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI ⚡ Byte Size Summary Store prompts as versioned YAML manifests in Git and run them through a three-stage GitHub Actions gate — schema validation, secret scanning with gitleaks, and model policy enforcement — before any LLM call reaches your OpenShift environment A CI-gated prompt pipeline gives your enterprise auditors a traceable answer to “what prompt was active during the incident window” — without it, the forensic work is manual, billed, and slow Prompt versioning is necessary but not sufficient: you’re versioning one variable in a system with multiple unversioned dependencies, and this article shows you what to do about the rest of them The Story I was presenting a prototype at a conference. The demo was built over three weeks of late-night sessions — an AI-assisted operations assistant for OpenShift that could answer runbook-style questions against live cluster state. The architecture was solid. The underlying idea was good. ...
Secrets Management Across Multi-Cloud Pipelines
🛠️ Pipelines in the Wild #3 Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI ⚡ Byte Size Summary Secret management failures are invisible until they cause a production incident — start with RBAC and namespace isolation before the first workload goes live Storing secrets in a central vault solves the sprawl problem but introduces a new failure mode: rotation lag between the vault and the namespace-level Kubernetes secret The real unsolved problem is not technical — it is knowing who owns the approval and escalation path when a credential rotates at 2 AM across a multi-timezone team The Story The deployment had been running fine in dev for two days. Same manifests, same pipeline, same container images. We promoted to production and the pods went straight into ImagePullBackOff. ...
Retry Logic and Tiered Alerting in GitHub Actions
🛠️ Pipelines in the Wild #2 Byte Size Summary Most pipeline failures are transient — a registry returning a 503, a smoke test catching a slow cold start, a network blip during an image push. Retrying them automatically, with exponential backoff, means engineers never see them. The failures that reach a human should be the ones that actually need one. This article builds a retry wrapper and a three-tier alerting system (transient → silent, degraded → Slack warning, critical → PagerDuty page) on top of a GitHub Actions blue/green deploy workflow. The demo application is Waybill — a FastAPI shipment tracking API backed by PostgreSQL, where the health endpoint checks real database connectivity rather than returning a static 200. That distinction matters: a smoke test that only checks HTTP status is a smoke test that passes while your database is unreachable. By the end you will have a working repo you can run locally with Docker Compose and test today. ...
MCP Server Architecture for Platform Teams — Giving AI Live Access to Your Infrastructure
Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI AI in the Stack #3 ⚡ Byte Size Summary MCP (Model Context Protocol) is the standard that lets AI agents interact with external systems — your cluster, your observability stack, your ticketing system — without bespoke integration code for every tool. MCP directly addresses AI hallucination and 2AM incident response by grounding AI answers in live system state. It does not solve tribal knowledge alone — that needs RAG alongside it. This article covers the production-grade architecture: what MCP servers are, how to design them for platform engineering use cases, and what you need to get right before running them anywhere near production. In logistics, the hardest problems rarely come from missing data. ...
Build a RAG Pipeline for Internal Runbooks with FastAPI and Chroma
Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI AI in the Stack #2 ⚡ Byte Size Summary RAG inserts a retrieval layer between your existing runbooks and an LLM — answers come from your documentation, not generic training data, with source citations included. This article builds a complete FastAPI service with /ingest, /query, and /health endpoints, using OpenAI embeddings and Chroma as the vector store. Everything is cloneable from GitHub. The goal is not to replace your runbooks. It is to make them queryable at the moment an incident is happening. I have never met a platform team with bad runbooks. ...
Zero-Downtime Deployments on OpenShift with GitHub Actions and Feature Flags
Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI Byte size summary After reading this article, you will know how to implement a blue/green deployment pipeline on OpenShift that uses HAProxy-backed Route weight splitting for traffic control and Flagsmith for feature flag management — and more importantly, you will know where the implementation breaks silently. Specifically: the HAProxy propagation gap that lets your smoke tests lie to you, the partial rollout state that puts two versions in production simultaneously, and why the standard approach of patching a Route weight and immediately proceeding has cost teams I’ve worked with entire migrations. The implementation uses GitHub Actions for orchestration, oc commands for OpenShift-specific traffic control, and Flagsmith as the feature flag service. The patterns apply to AKS, EKS, and GKE with platform-specific variations called out. ...
AI Tooling on OpenShift: A Practitioner's Evaluation Framework
Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI ** AI in the Stack #1** Byte size summary After reading this article, you’ll have a framework for evaluating AI tools in platform engineering contexts — not by capability type, but by where in your workflow the tool actually changes the outcome. You’ll understand why the tools that sound most compelling are still hype, where genuine productivity gains exist today, and what governance infrastructure you need in place before any AI component gets near production. This article is the foundation for the series; subsequent articles implement each touch point against real OpenShift infrastructure. ...