AI Inference Architecture: Building Software Around Uncertainty

Most software systems are built around a comforting assumption: given the same valid input, a component should produce the same correct output.

Advanced AI features break that assumption.

A language model can produce a useful answer, a plausible answer, or a confidently wrong answer from the same prompt. Its behavior changes with model versions, decoding parameters, context length, retrieved documents, and even the order of examples in a prompt. That does not make AI unsuitable for production. It means the engineering model must change.

The most useful mental shift is this: an LLM is not an application layer. It is an inference engine operating inside an application layer.

Once that distinction is clear, many design decisions become easier. You stop asking how to make a prompt perfect and start designing how the surrounding system handles uncertainty, evidence, latency, cost, and failure.

Model output is a distribution, not a fact#

Traditional APIs usually expose a contract such as:

The function has defined inputs, a deterministic algorithm, and a result that can be independently verified. AI calls look superficially similar:

But the second call is fundamentally different. The returned text is a sample from a probability distribution conditioned on the model, prompt, context, and decoding settings.

That distinction has practical consequences:

• A retry can produce a different result.
• A new model version can change behavior without changing your code.
• A result may be fluent while unsupported by the supplied data.
• A test that asserts exact wording is usually brittle.
• A successful HTTP response says nothing about semantic correctness.

This is why production AI systems need a layer between model output and business action. That layer should validate structure, measure confidence through independent signals, and decide whether the result is safe enough to use.

For example, an AI system that extracts invoice fields should not directly write values into a PostgreSQL database. It should produce a typed candidate record, attach evidence, run deterministic validation, and route ambiguous cases for review.

The schema is necessary, but it is not sufficient. A valid JSON object can still contain an invented invoice number. The evidence requirement changes the task from “produce fields” to “produce fields that can be traced to source material.”

Separate generation from decision-making#

One of the most reliable AI architecture patterns is to divide the system into two planes:

1. The generative plane proposes interpretations, classifications, summaries, or actions.
2. The deterministic plane validates, authorizes, stores, and executes.

The generative plane can be creative and probabilistic. The deterministic plane should be constrained and auditable.

Consider a support-ticket triage workflow. A model may recommend a priority, identify a product area, and draft a response. But escalation policy should remain code:

The model may influence the input to that rule, but it should not silently replace the rule. For higher-risk domains, add a confidence gate or require explicit human approval before the deterministic plane executes a consequential action.

This pattern is useful because it narrows the blast radius of model mistakes. A bad classification becomes a review item, not an incorrect refund, deleted account, or misleading compliance statement.

Retrieval is an evidence pipeline, not a prompt accessory#

Retrieval-augmented generation is often presented as a simple recipe: split documents, create embeddings, fetch nearby chunks, paste them into a prompt.

That recipe works for demos. Production retrieval is closer to search infrastructure than prompt engineering.

A retrieval pipeline has at least five design problems:

1. Document quality — Is the source current, complete, and authoritative?
2. Segmentation — Does each chunk preserve the semantic unit needed to answer a question?
3. Candidate recall — Did the search find all potentially relevant evidence?
4. Ranking precision — Are the best candidates placed first?
5. Grounded synthesis — Does the final answer stay within the evidence provided?

Embedding similarity solves only part of candidate recall. It does not reliably handle exact product identifiers, dates, negation, or policy language. A hybrid approach commonly performs better: use vector search for semantic similarity, lexical search for exact terms, metadata filters for scope, and a reranker to refine the final context.

For example, a knowledge base query about “refund terms for annual plans in Germany” should filter by locale and product state before semantic ranking. Without that filter, a highly similar but obsolete United States policy may win.

Your retrieval record should also carry provenance:

When an answer reaches a user, retain the document IDs, chunk IDs, ranking scores, and model version used to generate it. That record is essential for debugging. Without it, every bad answer becomes an anecdote. With it, a bad answer becomes an inspectable pipeline failure: stale content, weak retrieval, missing filter, poor ranking, or unsupported generation.

Design for abstention, not forced completion#

Many AI failures occur because the system asks the model to answer even when the evidence is weak. A better interface gives the system permission to decline.

Instead of defining the result as Answer, define it as a discriminated union:

This turns uncertainty into an explicit product state.

Abstention should not depend on the model saying “I am unsure.” Self-reported confidence is often poorly calibrated. Combine several observable signals instead:

• Retrieval scores and score gaps between top candidates
• Whether cited text actually supports the generated claim
• Agreement across independent extraction passes
• Schema validation results
• Business-rule conflicts
• Detection of missing required fields
• Historical error rates for similar requests

For a classification feature, you can also compare the probabilities of the first and second candidate labels. A narrow margin often indicates ambiguity. For generative tasks, use a second constrained pass to verify whether each claim is entailed by the source context.

The goal is not to eliminate uncertainty. It is to ensure uncertainty changes system behavior.

Treat evaluation as executable product specification#

Teams often test AI features with a few memorable prompts. That is useful during development, but it is not an evaluation strategy.

An evaluation suite should resemble a compact, versioned specification of real user behavior. Include ordinary examples, adversarial examples, long-tail examples, multilingual inputs, outdated terminology, malformed documents, and cases where the correct answer is to abstain.

Each test case should capture more than the expected text. Depending on the feature, it may include:

• Required facts
• Forbidden claims
• Expected citations
• Structured-field accuracy
• Allowed latency budget
• Maximum token cost
• Expected safety or escalation outcome

For a document extraction system, field-level metrics are more informative than an overall “passed” result. Measure exact match for invoice numbers, numeric tolerance for totals, and precision and recall for optional fields. For a retrieval system, separately measure whether the correct source appears in the candidate set and whether the final answer uses it correctly.

Run these evaluations in CI/CD when prompts, retrieval logic, models, embeddings, or source corpora change. A prompt is production code in the operational sense: it has dependencies, behavior, regressions, and version history.

Observability must capture the inference path#

Standard application telemetry tells you that a request took 1.8 seconds and returned HTTP 200. AI telemetry needs to explain how the system arrived at its output.

A useful trace connects:

For every stage, capture identifiers, durations, token counts, model settings, and safe representations of input and output. Be deliberate about privacy: raw prompts and retrieved documents may contain personal or confidential data. Use redaction, access controls, retention policies, and hashed identifiers where appropriate.

The most important operational metric is not simply cost per request. It is cost per successful outcome. A cheaper model that causes more human reviews, customer corrections, or retry loops may be more expensive in the system that matters.

The durable advantage is orchestration discipline#

The strongest AI products are rarely built around a single exceptional prompt. They are built from familiar software engineering disciplines applied to an unfamiliar kind of component:

• typed boundaries
• validation layers
• evidence and provenance
• graceful degradation
• asynchronous workflows
• versioned evaluation sets
• observable pipelines
• explicit authorization

The model remains important, but it becomes one dependency among many. It contributes interpretation and language capability; the rest of the system supplies memory, policy, correctness checks, and operational control.

That is the mature shape of AI engineering. Not a chatbot bolted onto a database, and not an attempt to force probability into deterministic behavior. It is software designed to make useful decisions in the presence of uncertainty—and to know when it should not decide at all.