The forgotten step between designing and deploying AI features

September 24, 2026

The forgotten step between designing and deploying an AI feature is evaluation: a deliberate, repeatable check that the system works safely and reliably for its intended users and conditions. Evaluation connects design assumptions to real-world behavior through representative tests, risk reviews, monitoring plans, and release criteria. It matters because AI can fail in ways that ordinary software tests miss: Stanford’s 2024 AI Index reported that documented AI incidents rose to 123 in 2023, a 32.3% increase over 2022. A practical evaluation process can help teams find defects before launch, make deployment decisions on evidence, and catch changes after release.

Evaluation connects AI feature design to deployment

In this article, evaluation means the structured assessment of an AI system against its intended purpose, users, operating conditions, and risks. The National Institute of Standards and Technology (NIST) treats testing, evaluation, verification, and validation—often grouped as TEVV—as activities that help organizations understand whether an AI system meets its requirements and behaves as expected. Evaluation is not a final accuracy score or a one-time sign-off; it is evidence gathered throughout development and operation.

That distinction matters because a model can perform well on a benchmark yet fail in a product workflow. A support assistant might give plausible but unsupported answers; a classification model might work overall but perform poorly for a particular user group; a generative feature might expose sensitive information or change behavior after a model update. Evaluation therefore covers several connected questions: Does it work? For whom? Under what conditions? What could go wrong? How will the team know if performance changes?

Evaluation is broader than accuracy testing

Accuracy testing compares outputs with expected answers, but product evaluation also examines usefulness, robustness, fairness, privacy, security, and operational fit. NIST’s AI Risk Management Framework organizes trustworthy AI work around four functions: Govern, Map, Measure, and Manage. Its structure reinforces that measurement is only one part of the job: teams also need to define context, assign responsibility, and respond to identified risks.

Evaluation belongs between design and release—and continues afterward

The “forgotten step” is often treated as a single prelaunch test, when it is better understood as a lifecycle practice. The UK National Institute for Health and Care Excellence’s evidence standards for digital health technologies, for example, distinguish evidence needs according to a technology’s function and potential impact. The same principle applies beyond health: the higher the consequences of an AI-assisted decision, the stronger the evidence and oversight should be before users rely on it.

Pre-deployment evaluation translates AI design into release evidence

Pre-deployment evaluation is the set of checks conducted before an AI feature is made available to its intended audience. It should test the system in the context where it will be used, not merely in the environment where it was built. A useful evaluation plan specifies the intended use, the users and affected parties, the quality and risk measures, the test data, the acceptance thresholds, and the person empowered to make a release decision.

This process includes several related forms of assessment: offline model testing, end-to-end product testing, human review, safety and security testing, and limited rollout. These are complementary rather than interchangeable. A strong benchmark result cannot establish that an interface communicates uncertainty clearly, while a successful pilot cannot by itself prove that rare but serious failures are adequately controlled.

Offline tests measure performance against representative examples

Offline testing uses a fixed dataset or test suite to compare system outputs with expected outcomes or defined quality criteria. Teams should build tests from real task patterns, include difficult and unusual cases, separate development data from evaluation data, and document how examples were selected. For generative AI, evaluation can combine expert ratings, rule-based checks, and task-specific measures such as factual support, completeness, or correct tool use.

A test set is only as useful as its coverage. If a customer-service assistant is evaluated solely on common, well-phrased questions, its score may hide problems with misspellings, ambiguous requests, multilingual input, or requests outside its authority. The Organisation for Economic Co-operation and Development (OECD) has emphasized that AI systems should be assessed in relation to their context and foreseeable effects, not in isolation from deployment conditions.

Human review validates quality that metrics cannot capture

Human review is the structured inspection of system behavior by people with relevant domain or user expertise. It is especially useful when correctness depends on context, tone, completeness, or the consequences of an error. Reviewers need clear rubrics, examples of acceptable and unacceptable outputs, and a way to record disagreements; otherwise, ratings can reflect individual preference more than consistent evidence.

For high-impact uses, human oversight should be designed into the workflow rather than added as a vague promise. The European Union’s AI Act establishes requirements for certain high-risk AI systems, including risk management, documentation, human oversight, and monitoring. The law’s risk-based approach illustrates why evaluation criteria should reflect the system’s role and potential harm, not just the model’s technical novelty.

Red-team and safety tests probe failure modes deliberately

Red-team testing attempts to expose weaknesses by testing adversarial, misuse, and edge-case inputs. For a generative assistant, this may include prompt injection, requests for disallowed content, attempts to retrieve private information, or instructions that conflict with system rules. Security testing should also examine the surrounding application, including data access, tool permissions, logging, and safeguards against unauthorized actions.

The goal is not to prove that a system can never fail; it is to discover meaningful failure patterns and decide whether they can be mitigated to an acceptable level. NIST’s Generative AI Profile recommends identifying and managing risks specific to generative systems, including confabulation, information integrity, privacy, and security. Teams should turn each significant finding into an owner, mitigation, retest, and documented release decision.

Deployment evaluation monitors AI behavior in real use

Deployment evaluation is the ongoing assessment of an AI feature after release. It checks whether real-world use matches the assumptions made during design and testing. Users, inputs, external data, model versions, and business processes can change; consequently, prelaunch results are evidence about a defined test setting, not a guarantee of permanent performance.

Production evaluation commonly combines automated monitoring, user feedback, incident analysis, periodic human audits, and controlled experiments. These methods help teams detect quality regressions, new misuse patterns, and mismatches between product expectations and system behavior. The NIST AI Risk Management Framework’s “Manage” function emphasizes responding to risks and maintaining appropriate controls as conditions evolve.

Monitoring detects drift, incidents, and changing user needs

Monitoring is the continuous or scheduled collection of signals that indicate how a deployed feature is behaving. Depending on the system, useful signals can include task completion, user corrections, escalation rates, refusal rates, latency, output quality, and reports of harmful or unsupported responses. A change in input patterns or performance may indicate data drift, concept drift, a model update, or a change in the way users interact with the feature.

Monitoring should be paired with thresholds and response procedures. A dashboard without a defined owner or action plan can show a problem without preventing harm. The International Organization for Standardization’s ISO/IEC 42001 standard provides a management-system framework for organizations developing or using AI, including processes for continual improvement and addressing AI-related risks.

User feedback and incident reporting reveal real-world gaps

User feedback is evidence about whether the feature is understandable, useful, and safe in the situations people actually encounter. Product teams can collect it through clear reporting controls, support tickets, interviews, and structured review of escalations. Feedback mechanisms should avoid asking users to disclose unnecessary sensitive information, and teams should explain how reports are reviewed.

Incident reporting turns failures into operational learning. An AI incident may involve an incorrect recommendation, discriminatory outcome, privacy exposure, unsafe generated content, or an unexpected action taken through a connected tool. Stanford’s AI Index documented 123 AI-related incidents in 2023, up from 93 in 2022. Although incident counts depend on reporting and classification methods, the increase underlines why organizations need a process to log, investigate, and learn from failures rather than relying on informal awareness.

Controlled rollouts reduce the cost of learning

A controlled rollout releases a feature to a limited population or use case before expanding access. Common approaches include staged deployment, shadow testing, and feature flags that allow a team to disable the AI component quickly. These methods help compare expected and observed behavior while limiting exposure if the system underperforms.

Rollouts are not substitutes for pre-release safety work. They are a way to gather additional evidence under controlled conditions, with clear stop criteria and a fallback path. For example, a writing assistant might first be offered to internal staff, then to a small group of customers, while the team reviews sampled outputs and tracks corrections before widening access.

An AI evaluation plan makes release decisions actionable

An evaluation plan turns broad principles into a repeatable workflow. It aligns product, engineering, data science, security, legal, and domain experts around what the system is meant to do and what evidence is needed to ship it. The plan should be proportionate: a low-risk autocomplete feature does not require the same controls as a system influencing access to healthcare, employment, credit, or public services.

Define intended use, boundaries, and affected groups

Start by documenting the feature’s purpose, intended users, foreseeable misuse, excluded uses, and the decisions it may influence. Identify people who may be affected even if they are not direct users. This context determines which errors matter, which groups should be represented in testing, and what safeguards are necessary.

Choose measures and release thresholds before testing

Define success measures before reviewing results to reduce the temptation to select metrics that make performance look favorable. Measures may include task success, error severity, response latency, subgroup performance, escalation frequency, privacy exposure, and security findings. Set thresholds that reflect user impact, and specify what happens when a threshold is missed: mitigation, additional testing, a narrower release, or a decision not to launch.

Record evidence, ownership, and rollback conditions

A release record should capture the system version, evaluation data and methods, known limitations, unresolved risks, approvals, monitoring signals, and rollback conditions. This documentation supports accountability and makes results interpretable after a model or product change. It also helps teams avoid repeating tests without understanding what changed or why a previous decision was made.

Re-evaluate after material changes

A model replacement, prompt change, new data source, altered user population, or expanded use case can change system behavior. Teams should decide in advance which changes require targeted retesting and which require a full evaluation. Reusing old results without checking whether their assumptions still hold can create a gap between documented safety and actual operation.

Suggested visual: A lifecycle chart can show the evaluation loop as “define intended use → test representative cases → review risks → decide release → monitor production → investigate and improve.” A companion table could list each evaluation stage, its evidence, its accountable owner, and its launch or stop criteria.

Conclusion: Evaluation is the bridge from AI design to responsible deployment

Evaluation is the often-missed bridge between designing an AI feature and deploying it responsibly. Pre-deployment testing checks representative performance, human review, and safety; deployment evaluation monitors drift, incidents, and user experience; and a documented evaluation plan turns findings into clear release and rollback decisions. The rise in reported AI incidents reinforces that evaluation should be a continuing practice, not a box checked once before launch.

Before releasing an AI feature, teams should write down its intended use, define measurable acceptance criteria, test likely failure modes, assign monitoring owners, and specify how the system can be limited or disabled. For further guidance, consult NIST’s AI Risk Management Framework and Generative AI Profile, the OECD AI Principles, and applicable regulatory and sector-specific standards.

Sources: Stanford Institute for Human-Centered Artificial Intelligence, 2024 AI Index Report, https://aiindex.stanford.edu/report/; National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0), https://www.nist.gov/itl/ai-risk-management-framework; National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence; Organisation for Economic Co-operation and Development, OECD AI Principles, https://oecd.ai/en/ai-principles; European Union, Regulation (EU) 2024/1689 (Artificial Intelligence Act), https://eur-lex.europa.eu/eli/reg/2024/1689/oj; International Organization for Standardization, ISO/IEC 42001:2023 Artificial intelligence management system, https://www.iso.org/standard/81230.html; National Institute for Health and Care Excellence, Evidence standards framework for digital health technologies, https://www.nice.org.uk/corporate/ecd7