The Hidden Runtime of AI Systems: Context, State, and Evaluation

Large language models are often introduced as an API call: send text, receive text. That mental model is useful for a prototype, but it breaks down quickly in production.

A production AI feature is not primarily a prompt. It is a runtime system that assembles context, manages uncertainty, constrains outputs, evaluates behavior, and records enough evidence to explain what happened later. The model is important, but it is only one subsystem in a larger computational loop.

For software engineers, the most valuable advanced AI concepts are therefore not exotic model architectures. They are the concepts that help turn probabilistic inference into an operable product capability.

Context is a runtime dependency#

Traditional applications receive explicit inputs: HTTP request bodies, database rows, environment variables, and authenticated user claims. AI systems also receive inputs, but much of the important input is assembled dynamically as context.

Context can include:

• The current user request
• Conversation history
• Retrieved documentation or records
• System instructions and policy constraints
• Tool descriptions and schemas
• Tenant-specific configuration
• Recent application events
• The model's token budget

This makes context engineering analogous to dependency injection and request composition. A model response is not determined solely by a user prompt. It is determined by the complete context window, its ordering, its formatting, and what was excluded.

That last point matters. Context windows are finite, and adding more text can reduce quality rather than improve it. Irrelevant retrieval results compete with relevant evidence. Long conversation histories can preserve obsolete assumptions. Repeated instructions consume space and create conflicting priorities.

Treat context as a first-class artifact. Build a context object before rendering a prompt, and log a safe, redacted representation of it. For example:

This design makes context inspectable, testable, and versionable. It also creates a clean boundary between application logic and model invocation.

Retrieval is an information-retrieval problem, not a vector database feature#

Retrieval-augmented generation is frequently described as “put documents in embeddings, then ask questions.” That description hides the difficult part: deciding which evidence is relevant enough to place in front of the model.

Embeddings perform semantic similarity search. They are valuable, but similarity is not the same as usefulness. A query about a customer’s current subscription may retrieve a semantically similar pricing guide instead of the authoritative billing record. A query containing a product code may need lexical matching, not semantic search.

Robust retrieval commonly combines multiple signals:

1. Metadata filtering limits the search space by tenant, permission, document type, version, date, or region.
2. Hybrid search combines keyword scoring with vector similarity.
3. Reranking applies a more precise model to a small candidate set.
4. Diversity selection prevents five nearly identical chunks from occupying the entire context.
5. Citation requirements force the generated answer to remain anchored to supplied evidence.

The underlying principle is simple: retrieval quality places an upper bound on answer quality. A fluent model cannot reliably recover facts that were never retrieved.

For structured operational data, use the system of record directly where possible. PostgreSQL, SQL Server, or Redis may be more suitable than an embedding index when the question is fundamentally transactional: “Which invoices are overdue?” or “What deployment ran last Tuesday?” Semantic retrieval is strongest for unstructured knowledge, not as a replacement for every query interface.

Structured output turns generation into integration#

Natural language is an excellent user interface and a poor application contract. If downstream code needs to make a decision, call a service, populate a form, or write a database record, prefer validated structured output.

Instead of asking a model to “summarize the support issue,” define a schema:

The model can still reason over unstructured text, but its output must pass through a parser and schema validator before it reaches business logic. In TypeScript, this might be Zod; in .NET, JSON Schema validation or strongly typed deserialization with explicit checks.

Validation does not guarantee truth. It guarantees shape. A perfectly valid JSON object can contain an incorrect classification or invented evidence ID. Still, structured output removes an entire category of failure: fragile regular expressions, ambiguous prose parsing, and hidden formatting assumptions.

Think of the model as an untrusted upstream service. Its output deserves the same defensive handling as a webhook payload from an external provider.

Model calls are probabilistic distributed systems calls#

An AI request has familiar distributed-systems properties: latency, timeouts, retries, partial failures, rate limits, cost variability, and upstream behavior changes. It also adds nondeterminism.

Two identical requests can produce different answers. A provider can update a model alias. A safety filter can alter a response. A long request may exceed a context limit only for certain customers. These are not edge cases; they are operating conditions.

Design accordingly:

• Set explicit timeouts and cancellation propagation.
• Use idempotency keys for actions with side effects.
• Retry only transient failures, with bounded exponential backoff.
• Separate generation from execution when an output could change data or trigger workflows.
• Cache stable, low-risk results when appropriate.
• Pin model versions for workflows that require reproducibility.
• Measure token usage, latency, validation failures, and fallback rates.

A useful architecture often has two phases. First, the model proposes a structured plan or classification. Second, deterministic application code validates permissions, checks business rules, and performs the action. The model may recommend; the application remains responsible.

Evaluation is the missing test suite#

Conventional tests expect a precise result from a deterministic function. AI systems require a broader evaluation strategy because correctness can be semantic, partial, and context-dependent.

Start by creating a representative evaluation set: real anonymized requests, difficult edge cases, adversarial inputs, stale-data scenarios, and examples that previously failed. For each case, define the expected properties of a good response rather than only an exact string.

Useful evaluation dimensions include:

• Groundedness: Are claims supported by retrieved evidence?
• Task success: Did the response solve the user’s actual problem?
• Instruction adherence: Did it follow policy and output constraints?
• Safety: Did it avoid exposing protected or unauthorized data?
• Format validity: Can downstream code consume the response?
• Cost and latency: Is the result viable at production volume?

Some checks can be deterministic. A response either conforms to a JSON schema or it does not. A citation either refers to a retrieved document or it does not. Other checks need human review, rubric-based scoring, or a separate model acting as a judge.

Model-based judging is useful, but it should not become an unquestioned oracle. Calibrate it against human-labeled examples. Track disagreement. Keep a small set of high-value cases reviewed by people who understand the domain.

Most importantly, run evaluations in CI/CD. A prompt edit, retrieval change, model upgrade, or schema revision is a production change. Treat it with the same discipline as an API change.

State is not memory#

AI products often describe conversation history as memory. That language can be misleading. A transcript is merely a record of prior text, and blindly replaying it is expensive, noisy, and vulnerable to old mistakes.

Production systems need distinct forms of state:

• Conversation state: recent turns needed for local coherence.
• User profile state: durable preferences and permissions, stored explicitly.
• Task state: workflow progress, approvals, and external operation results.
• Knowledge state: indexed documents and authoritative source records.
• Audit state: what context, model, policy, and tools influenced an outcome.

Each state type has different retention, authorization, and consistency requirements. Store durable facts in ordinary databases. Use summaries for long-running conversations. Rehydrate only the context needed for the current task.

This separation prevents a common failure mode: allowing a model-generated summary to become an unverified source of truth about a user or business process.

Build the system around evidence#

The advanced engineering move is to stop treating AI as a magical text endpoint. Treat it as a probabilistic component inside an evidence-driven system.

Assemble context deliberately. Retrieve information with measurable relevance. Require structured outputs. Validate every boundary. Separate proposals from side effects. Evaluate changes continuously. Persist authoritative state outside the model.

When these foundations are in place, model quality becomes an accelerator rather than a single point of failure. That is the difference between a compelling demo and software that can survive real users, real data, and real operational pressure.