Define the intended task and consequence
Begin with a bounded task rather than an abstract ambition to make an assistant accurate. Finding a policy clause, explaining an inventory exception and proposing a contract action require different evidence. State what a successful answer enables a user to do, what authority remains with the user and which errors could cause material harm or unnecessary work.
The NIST AI Risk Management Framework provides a foundation for considering AI risk across the system lifecycle. Apply that perspective to the particular workflow. Identify affected stakeholders and foreseeable misuse alongside the desired benefit. An evaluation plan should reflect the consequence of failure, not only the convenience of measuring a response with a single automated score.
Build a representative and reviewable test set
Collect questions that reflect actual terminology, incomplete requests and the information sources a user can access. Include different business units, source formats and levels of complexity. For each case, record the expected evidence, acceptable answer boundaries and circumstances in which the system should ask for clarification or abstain. Keep a separate holdout set for assessing meaningful changes.
Protect sensitive information in the evaluation corpus and give examples the same permission discipline as production records. Label uncertain cases explicitly rather than forcing a false consensus. Have domain reviewers resolve disagreements about business meaning. An evaluation set is a maintained product: changes in policy, source structure or business definitions can make previously correct answers obsolete.
Evaluate the stages independently
A final answer can fail because retrieval missed the evidence, identity resolution selected the wrong entity or generation misrepresented an otherwise correct source. Measure these stages separately. Inspect whether the relevant authorised documents were retrieved, whether the context preserved their qualifications and whether the response’s statements can be supported by the cited passages.
For a workflow that proposes actions, evaluate action selection and parameter construction independently from explanation quality. A well written recommendation can still target the wrong record or omit a required approval. Preserve structured evidence from each stage so teams can identify the actual cause of regression. Improving a prompt cannot reliably repair an incorrect entity mapping or a missing access boundary.
Test conflicting evidence and restricted access
Test stale documents, conflicting sources, ambiguous identifiers and questions that exceed the available evidence. Include users with different permissions and verify that restricted content does not appear in retrieval results, explanations or citations. OWASP’s authorization guidance emphasises validating access on every request; evaluation should exercise those checks across the entire answer path rather than only at sign in.
Add instructions embedded in retrieved documents that attempt to redirect the assistant or request unauthorised actions. Evaluate how the workflow handles those inputs without treating document text as trusted operating instructions. Also test unavailable dependencies and long responses. The expected behavior may be a bounded explanation of uncertainty or a safe handoff instead of an apparently complete answer.
Use a scorecard with separate failure categories
Track factual support, task completion, citation quality, access correctness and appropriate abstention as distinct dimensions. Review important slices separately so a strong average does not conceal weak performance in a consequential business area. Record reviewer guidance and inspect agreement between reviewers before using their labels as a stable basis for comparison.
Automated evaluators can help prioritise review, but they also need validation against expert judgments. Preserve examples of disagreements and inspect them after major system changes. Report uncertainty and test set limitations when presenting results. A score is evidence about specified cases under specified conditions; it is not a universal guarantee of competence or a substitute for operational controls.
Connect evaluation to release and monitoring
Run the relevant evaluation set when changing retrieval, models, prompts, source transformations or permission logic. Compare both improvements and regressions with the previous version. Define release thresholds with the business owner and require targeted review for consequential failures. Keep the evaluated configuration identifiable so a production incident can be connected to the system version that produced it.
After release, collect user corrections and operational outcomes through an appropriate review process. Turn recurring failures into new evaluation cases while keeping the holdout set independent. Assign an owner to each corrective action and preserve a way to disable or narrow an unreliable workflow.