SKShad KhanAI ArchitectStart a conversation

AI Research

Evaluating an enterprise AI assistant beyond answer accuracy

A layered evaluation model covering retrieval, generation, tool use, policy, latency, cost, and real task outcomes.

Shad KhanJuly 202612 min read

Evaluating an enterprise AI assistant beyond answer accuracy

A layered evaluation model covering retrieval, generation, tool use, policy, latency, cost, and real task outcomes.

A single quality score cannot tell a team where an enterprise assistant is unsafe, expensive, slow, or simply unhelpful.

Start with a task contract

Evaluation begins before a test set. Define what the user is trying to accomplish, what evidence is permitted, what actions are allowed, what an acceptable response must contain, and what the assistant should do when the task cannot be completed safely.

This task contract prevents a familiar failure: optimizing the answer's wording while ignoring whether the assistant used the correct source, called the wrong tool, exceeded the latency budget, or created more work for the user.

System design: five independent evaluation layers

An incorrect answer can begin with a missing document, a poor chunk, a stale index, an unsuitable reranker, a generation error, a tool failure, or a policy mistake. Grading only the final text collapses all of those causes into one number. The system needs layer-specific tests so a failure has somewhere to live.

Architecture diagram

Retrieval layer

  • Relevance and coverage measured against the task contract, not top-k similarity alone
  • Freshness checked against the source system of record, not the index's last build time
  • Permission-aware ranking — a chunk the user cannot see must never outrank one they can
  • Citation correctness traced back to the exact source span, not the source document

Generation layer

  • Factual consistency against retrieved evidence, not the model's general world knowledge
  • Completeness against every required field in the task contract
  • Instruction following under conflicting, partial, or contradictory context
  • Calibrated uncertainty language when the evidence is thin or ambiguous

Tool layer

  • Correct tool selection for the stated intent, including near-miss tools with similar names
  • Argument validity checked against the tool's schema before execution, not after a failure
  • Authorization checked against the caller's identity and scope, not a shared service identity
  • Idempotency and normalized error handling on retry

Policy layer

  • Sensitive data handling for PII, PHI, and financial detail
  • Forbidden actions blocked before execution, not flagged after the fact
  • Escalation triggered automatically when confidence or evidence falls below threshold
  • Adversarial or manipulative input handled without silently complying

Production signal loop

  • Task completion and user correction rates
  • Escalation and abandonment, tracked by segment
  • Time saved against the manual baseline the assistant replaced
  • Downstream error and support burden created by the assistant's output

Use judges carefully

LLM judges are useful for criteria that require semantic comparison, especially when the rubric is narrow and examples are available. They are not a substitute for deterministic checks, domain review, or calibration. A judge should produce a reason and evidence, not only a number.

I compare judge decisions with human labels on a representative sample, monitor disagreement by failure category, and keep exact checks for citations, schemas, tool arguments, and policy invariants. The goal is scalable review without pretending the reviewer model is objective truth.

  • Sample judge decisions against human labels on every release, not only at launch
  • Track disagreement by category — a judge can be reliable on tone and unreliable on policy
  • Never let a judge be the only check on a deterministic invariant like a schema or a citation

Close the loop with production outcomes

Offline evaluation protects releases; production evaluation shows whether the assistant helps. Useful outcome signals include task completion, user correction, escalation, abandonment, repeat attempts, time saved, downstream error, and support burden.

Those signals must be segmented. An average can hide a dangerous result for one language, user role, document type, or tool. Production quality is a distribution, not a leaderboard position.

From production: where the average score lied

On the Bedrock knowledge-base platform, the blended quality score stayed flat release over release while one tenant's escalation rate crept upward in the same window. The blended number hid it because that tenant's traffic was a small share of total volume — exactly the failure mode a single score is built to disguise. The retrieval agent had started answering more of that tenant's queries from the semantic cache, and the cached content had drifted stale relative to a source system that tenant updated more often than most. Segmenting production outcomes by tenant, not only by feature, surfaced the regression before it reached a support escalation. The fix was not a bigger model — it was tightening the cache-freshness policy for that source class, consistent with the trade-off already recorded for that decision: cache invalidation has to respect source freshness and tenant boundaries, or it quietly buys latency at the cost of correctness.

On the Enterprise MCP Gateway, the same principle showed up at the tool layer instead of the retrieval layer. A tool-selection regression never moved the top-line satisfaction number, because the agent was choosing a similar-sounding tool that returned a plausible-looking result with different side effects. It only became visible in the audit stream — the trace that records tool inputs, outcomes, latency, cost, identity, and policy decisions for every call — because that trace let a reviewer compare intended tool against executed tool at the event level, not the conversation level. A layered evaluation model exists precisely so a regression like that has a place to be caught before a customer notices it.

What to carry forward

  • Define success at the user-task level
  • Isolate retrieval, generation, tool, and policy failures
  • Calibrate LLM judges with human review
  • Track latency and cost beside quality
  • Segment outcomes by tenant, language, and role before trusting an average

Applied in the portfolio