The case for boring AI infrastructure

There is a particular flavour of meeting that has become very common in 2026. A product team walks into a room with an AI feature that worked beautifully for two months and started misbehaving last week. They have tried three different models. They have rewritten the prompt seventeen times. They are wondering whether to fine-tune. They want a recommendation.

The honest recommendation, almost every time, is the unglamorous one. Stop tweaking the model. Build the evaluation harness you skipped. Wire up the observability you postponed. Put cost gates on the gateway. Add the fallback path you thought you would not need. The teams shipping reliable AI in 2026 are not winning on model choice. They are winning on the boring infrastructure around the model.

This post is an argument for that infrastructure, made specifically. We will cover evals, observability, cost control, prompt-injection defence, and fallback paths — in roughly the order they pay back the investment.

Evals are the difference between iteration and guessing

Hamel Husain, who has trained engineers and product managers at Anthropic, OpenAI, Google, and Meta on the topic, makes a point that lands harder the more production AI systems you have shipped: like software engineering, success with AI hinges on how fast you can iterate, and you must have processes and tools for evaluating quality, debugging issues, and changing system behaviour — many teams focus exclusively on the third, which prevents them from improving their LLM products beyond a demo.

That is the whole argument, in one sentence. Without evals, every change is a guess. Without evals, every regression is discovered by a user. Without evals, the model upgrade your vendor pushes at 3am quietly degrades a quarter of your product, and you find out via a support ticket.

The pattern of building a useful eval harness is not complicated, even if doing it well takes practice. It looks roughly like this:

Capture real traces from production. A trace is the complete record of all actions, messages, tool calls, and data retrievals from a single initial user query through to the final response. Log everything.
Do error analysis on the traces. Open the trace viewer. Look at the first trace. Note what went wrong in plain language. Categorise. Count. This is the step that separates teams shipping reliable AI from teams constantly firefighting bugs.
Build evaluators that catch the categories that matter. Some are code-based (deterministic checks — did the output contain a required field, is it valid JSON, is the math right). Some are LLM-as-judge (does the answer address the question, is it grounded in the retrieved context). A complete eval suite typically has two to three code-based evals and one to two LLM-based evals.
Run them on every change and on a schedule. In CI on every PR. Nightly against a held-out set. Quarterly review of the eval set itself to catch drift in what "good" means.

The thing most teams discover, doing this for the first time, is that the LLM-as-judge approach works far better when guided by examples from real production data than when bolted on as a generic check. A confident-sounding wrong answer that an off-the-shelf evaluator will rate as fine, gets caught immediately when the judge has seen ten examples of the specific failure mode your users encounter.

The investment is real — building a useful eval harness is a fortnight of work for a small team, not a weekend — and it pays back inside a quarter on every measure that matters: regressions caught earlier, prompts iterated more confidently, model upgrades done without holding your breath.

Observability turns "the AI is broken" into a debuggable system

If evals are the offline quality net, observability is the online one. The two are different things and you need both.

The pillars are not exotic. Trace every request end-to-end. Log inputs, outputs, metadata, user identifiers, and timestamps for every request — storage is cheap, missing data during incidents is not. Attribute cost per trace and per tool call. Capture user feedback (thumbs up / thumbs down, ratings, comments) and link it directly to the exact trace that produced it. Surface latency by request stage so you can see whether the bottleneck is retrieval, generation, or your own glue code.

Splunk's framing of LLM observability KPIs reads about right for a starting point: define KPIs that tie to business outcomes — trust (groundedness), cost (cost-per-answer), and user experience via p95 latency — and instrument prompts, retrieval, and generation before launch, not after.

What you should be able to do, at any time, within a couple of clicks:

Pull up the full trace of any specific user request.
See which model, which prompt version, and which retrieval result produced a given answer.
Filter traces by user, feature, time window, or evaluator score.
Drill into the slow ones. Drill into the expensive ones. Drill into the wrong ones.

The tooling has matured. LangSmith, Langfuse, Arize Phoenix, Helicone, LangWatch, Braintrust — pick one that fits your stack and budget; the differences matter less than picking one and instrumenting properly. The single most expensive choice is picking observability after the fact, because retrofitting trace context across a code base that did not anticipate it is its own quarter of work.

Cost control belongs in the code, not on the bill

The third boring layer is cost. Production AI teams who lose sleep are not the ones who picked an expensive model. They are the ones who do not know, in real time, where their money is going.

A few patterns we recommend by default:

Per-trace and per-tool-call cost attribution. Every LLM call, retrieval step, and tool invocation should carry token counts and estimated cost as its own span, so teams can identify the exact workflow step driving spend.
Budget alerts at 50%, 80%, and 100% of token thresholds. One runaway prompt can burn thousands overnight. Alerts catch this. The alternative is finding out from finance.
Tag-based cost grouping. Spend broken down by user, feature, model, or environment without requiring custom attribution logic.
Per-trace caps and per-feature rate limits. A hard ceiling on tokens per user request is the difference between an embarrassing answer and a $400 incident.

The economics of LLM features compound differently than traditional infrastructure costs. A loop bug in regular code wastes CPU cycles. A loop bug in agentic code wastes dollars in real time. Treating LLM spend as an engineering metric, with the same rigour as latency or error rate, is what separates teams who can confidently launch AI features from teams who quietly cap their usage to avoid surprises.

Prompt injection is a real attack surface, not a thought experiment

The fourth piece of boring infrastructure is security, and specifically the kind of security that did not exist for traditional applications.

Prompt injection sits at the top of the OWASP Top 10 for LLM Applications in 2025 — the second consecutive year. The technical reason is structural: LLMs do not distinguish between instructions and data, which means any text in the context window can act as an instruction. It is the new SQL injection in the sense that it is a class of vulnerability that emerges from how the platform processes input, not from any specific application's bug.

The threat is not abstract. In June 2025 researchers disclosed EchoLeak, a zero-click vulnerability in Microsoft 365 Copilot that allowed a remote attacker to steal confidential data simply by sending an email — Microsoft assigned CVE-2025-32711 and issued emergency patches, underscoring the severity and novelty of the threat. The attack required no user interaction. The model was tricked into helpful behaviour that served the attacker.

Mitigations are layered, no single one is sufficient, and all are mundane:

Treat retrieved content as untrusted input. Anything that came from the web, a customer upload, a third-party email, or an external API can contain instructions. Do not let it govern model behaviour without bounded scope.
Least privilege on tools. Give the model the smallest set of tools that lets it do the job. An agent that can read files, query a database, send emails, and make HTTP calls is a much larger blast radius than one that can do exactly the two things the feature requires.
Human approval on destructive or irreversible actions. Anything that spends money, modifies persistent state, or affects another party gets a human gate. Even a thin one.
Output-side detection. Watch for outbound URLs containing data that should never leave, for unusual response shapes, for sudden behavioural drift. The OWASP guidance is clear that guardrails are probabilistic, raising the cost of attack but not eliminating it — meaning infrastructure must contain the damage when a guardrail eventually fails.
Segment LLM infrastructure into isolated network zones. If the model gets compromised, you want it unable to reach the rest of the network.

The single most important mental model: defence in depth. The system prompt is not your security boundary. The guardrail model is not your security boundary. The network segmentation, the least-privilege tool scopes, and the human approval gates — together — are your security boundary.

Fallback paths are how you stay shipped during the incident

The last piece is the one most teams skip until they cannot. When the AI feature is broken — the model vendor is having an outage, your rate limit hit, the retrieval index is corrupt, the prompt regressed, the cost gate fired — what does the user see?

The answer should not be a 500 page. It should be a graceful degradation. The patterns:

Cached responses for common queries. Not always appropriate, but for read-heavy use cases (FAQs, common help requests, repeated structured extractions) a cache layer with a sensible TTL turns a 100% outage into a 90% normal experience.
Direct LLM response when RAG confidence is low. A direct model response, with an explicit disclaimer that the answer is not based on your knowledge base, beats a confidently-wrong RAG answer or a blank page.
Hand-off to humans. A "talk to support" button that the system surfaces when its own confidence is below threshold, when it has retried twice, or when the user explicitly asks for a person.
Read-only mode. When the writing path is broken (vendor outage, cost cap hit), the reading path can usually still work. Degrading to "we can answer your questions but cannot take new instructions" is a much better experience than the whole feature being down.

Designing the fallback explicitly means asking, for every AI-driven feature: what does this look like when the AI is unavailable, slow, or unreliable? Most features have a sensible answer. Most teams have not actually built it.

The shape of a reliable AI team

If you put it all together, the operating posture of a team shipping reliable AI in 2026 looks roughly like this:

They can show you an evaluation dashboard with metrics tied to business outcomes, plotted over time.
They can pull up the trace of any user complaint within thirty seconds.
They know what each feature costs per request, and they have alerts on the budget.
They have a written threat model for prompt injection, with specific mitigations mapped to specific data flows.
Every AI-driven feature has a documented fallback path.
The number of times they have changed the model in production this quarter is zero or one. The number of times they have iterated on prompts, evaluators, and retrieval configuration is significantly higher.

This is not what gets a team a write-up in TechCrunch. It is what gets a team a five-year customer base.

The deeper point is that AI engineering, eighteen months past the model novelty, is becoming engineering. The same disciplines that separated reliable software companies from unreliable ones in 2010 — tests, observability, incident response, security defence in depth, capacity planning — are the disciplines that separate reliable AI companies from unreliable ones in 2026. The label on the wrapper has changed. The work underneath is mostly the same work, applied to a new substrate. Teams that internalise this are calmer, ship more, lose fewer customers, and spend less time in incident channels. Teams that do not, find out the hard way.

References

Hamel Husain — Your AI Product Needs Evals. https://hamel.dev/blog/posts/evals/
Hamel Husain — LLM Evals: Everything You Need to Know (FAQ). https://hamel.dev/blog/posts/evals-faq/
Hamel Husain & Shreya Shankar (interview, Aakash Gupta) — How to Master AI Evals: A Step-by-Step Guide. https://www.aakashg.com/ai-evals-masterclass-with-hamel-shreya/
Vellum — A Guide to LLM Observability. https://www.vellum.ai/blog/a-guide-to-llm-observability
Splunk — LLM Observability Explained. https://www.splunk.com/en_us/blog/learn/llm-observability.html
LangWatch — 4 Best Tools for Monitoring LLM & Agent Applications in 2026. https://langwatch.ai/blog/4-best-tools-for-monitoring-llm-agentapplications-in-2026
Braintrust — Best Tools for Tracking LLM Costs in Production (2026). https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026
TrueFoundry — AI Cost Observability for LLM and Agent Workloads. https://www.truefoundry.com/blog/ai-cost-observability
Helicone — How to Monitor Your LLM API Costs. https://www.helicone.ai/blog/monitor-and-optimize-llm-costs
OWASP Generative AI Security Project — LLM01:2025 Prompt Injection. https://genai.owasp.org/llmrisk/llm01-prompt-injection/
Cisco Blogs — Prompt injection is the new SQL injection, and guardrails aren't enough. https://blogs.cisco.com/ai/prompt-injection-is-the-new-sql-injection-and-guardrails-arent-enough
arXiv — EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System. https://arxiv.org/abs/2509.10540
OWASP Cheat Sheet Series — LLM Prompt Injection Prevention. https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html
arXiv — Failure Modes in LLM Systems: A System-Level Taxonomy. https://arxiv.org/abs/2511.19933

Written by JP, Sixees Labs. Last reviewed May 2026.