7 min read
Last updated:
The decision to ship an AI feature to your customers is no longer a product decision. As of 2026, it is a regulatory, legal, brand, and security decision rolled into a single launch. The EU AI Act risk-tier obligations are in active enforcement, the New York City local law on automated employment decisions has been litigated twice, California has expanded its AI transparency requirements, and your brand will absorb every confident wrong answer your model produces in front of a paying user. Engineering organizations that ship AI features without a structured pre-launch safety review are not moving faster than their competitors; they are accumulating undisclosed liability.
This is the framework we use with enterprise clients. It is not exhaustive. It is the minimum viable review that should gate any customer-facing AI deployment in 2026, regardless of model vendor or vertical.
Red-teaming methodology
The first failure mode of red-teaming in 2026 is treating it as a one-time event before launch. The model is not the only thing under test; the surrounding system, the prompt, the tool list, the retrieval corpus, and the rate-limiting policy are all attack surfaces, and any one of them can change weekly. Red-teaming must be a continuous program, with at least three distinct passes before a customer-facing launch.
The first pass is internal adversarial. Your own engineers, with no scoring criteria, attempt to break the system in any way they can imagine: extract the system prompt, get a refusal-bypass, generate output that violates your terms of service, escalate privileges through tool calls. Time-box this to forty hours of engineer time across three to five people. The goal is to surface the obvious vulnerabilities your design assumed away.
The second pass is structured red-teaming against a defined taxonomy. Use the categories from the NIST AI Risk Management Framework or Anthropic’s published red-team taxonomy: jailbreaks, prompt injection, harmful content, biased outputs, privacy violation, security exfiltration, denial of service. Run two hundred to five hundred prompts per category. Score with a rubric and track pass rates across model versions.
The third pass is external. For high-stakes deployments (financial advice, medical information, hiring, lending, content moderation at scale), pay a specialist firm. The market in 2026 is mature: Robust Intelligence, HiddenLayer, and several boutique firms run structured engagements that cost between forty and two hundred thousand dollars and produce a report your legal team can rely on if a regulator asks what diligence you performed.
Jailbreak surfaces
Treat every input field that reaches the model as untrusted. This is obvious for the user prompt; it is non-obvious for retrieval results, tool outputs, document uploads, and image inputs in multimodal flows. The 2024-2026 generation of indirect prompt injection attacks proved that an attacker who controls a document your agent reads can issue instructions the agent obeys. If your agent reads emails, web pages, PDFs, or any user-uploaded content, that content is an instruction surface.
Three concrete defenses, in order of importance:
- Strict separation between the system prompt, the user message, and any retrieved or tool-returned content. Use the role boundaries the model API provides; never concatenate untrusted text into the system prompt
- Output validation against an allow-list schema for any tool call that touches a sensitive resource (deletion, payment, escalation, external send)
- A second-pass classifier that scores the agent output for instruction-following from untrusted sources, flagging any response that appears to obey injected commands
None of these are silver bullets. The 2026 reality is that prompt injection cannot be fully prevented; it can only be contained by limiting what an injected instruction can actually accomplish. Design your tool list so that even a fully compromised agent cannot exfiltrate data, send unauthorized communications, or modify production state without a human-in-the-loop confirmation.
PII leakage
Three leakage paths matter, and they require different controls. The first is training-data memorization: the model emits PII from its pretraining corpus. With the major frontier vendors in 2026 this risk is low for properly licensed models but non-zero. The second is context bleed: PII from one user’s session appears in another user’s session. This is almost always a caching or logging bug rather than a model bug, and it is the most common cause of high-severity PII incidents. The third is over-disclosure: the model truthfully repeats PII the user supplied, but to the wrong audience or in the wrong context.
Your safety review needs an explicit test for each path. Generate synthetic PII (names, emails, social security numbers, credit card numbers in test ranges) and pass it through your pipeline. Verify that it does not appear in logs without redaction, in caches keyed by anything but session, in error messages returned to other users, or in agent responses sent to recipients who should not see it. Automate this test in your CI pipeline; do not rely on a one-time pre-launch check.
Brand-voice deviation
An AI feature that sounds like a different company is a feature that erodes trust faster than it builds it. Brand-voice review is not a marketing concern; it is a customer-retention concern that engineering owns when the system goes live. The mechanism is straightforward: define a voice spec (formality level, prohibited phrases, required disclaimers, tone in conflict situations), turn it into an evaluation set of two hundred conversations, and run the eval on every model upgrade and every prompt change. Track pass rate over time. A regression in voice pass rate is a launch blocker, not a documentation update.
Regulatory triggers
The EU AI Act risk categories that apply to most enterprise deployments in 2026 are limited risk and high risk. Limited risk imposes transparency obligations: users must know they are interacting with an AI system. High risk applies to employment, education, credit, insurance, law enforcement, critical infrastructure, and several other categories; it imposes documentation, conformity assessment, post-market monitoring, and human oversight obligations that take six to twelve months of work to implement and cannot be bolted on after launch.
The New York City local law on automated employment decisions requires a bias audit by an independent auditor before deployment, plus annual repetition. Several other US jurisdictions have followed: Colorado’s AI Act, Illinois’s Video Interview Act expansion, California SB 1047 implementation. If your AI feature touches hiring, lending, or housing decisions in any way, your legal counsel needs to be in the safety review from day one, not the week before launch.
Build a regulatory checklist for your specific feature, jurisdiction, and customer base. Maintain it as a living document. Treat regulatory non-compliance as a P0 incident on par with a security breach.
Kill-switch architecture
Every customer-facing AI feature must have a kill switch that an on-call engineer can flip in under sixty seconds. The switch must do three things: stop new inferences, drain in-flight requests with a graceful fallback (cached response, deterministic baseline, human handoff queue), and surface a status to users that does not say “AI feature is broken.”
The fallback path is where most kill switches fail in production. If the fallback is a static error message, your users have a worse experience than no AI feature at all. If the fallback is a different model, you have not killed the risk, you have moved it. The right answer for most enterprise products is a degraded-but-functional path: a search results page instead of an AI summary, a templated reply instead of a generated one, a human queue instead of an autonomous resolution. Design this path before launch and test it with the same rigor as a database failover.
Recommendation
Stand up a four-person AI safety review board: a senior engineer, a security engineer, a legal partner, and a product owner. Charter it to gate every customer-facing AI launch. Build a checklist that covers the six surfaces above. Make the review a sixty- to ninety-minute meeting with a written outcome: ship, ship with conditions, do not ship. The output of the meeting is the document you produce when a regulator, an enterprise customer’s procurement team, or your own board asks what diligence was performed. That document protects the company and forces the design conversations that prevent the worst incidents.
When this applies and when it does not
This framework applies to any AI feature where end users see model outputs, where the model influences decisions about users (recommendations, prioritization, eligibility), or where the model has authority to take action on a user’s behalf. It applies whether the feature is the headline product or a sidebar enhancement.
It does not apply with the same intensity to internal productivity tools (a code-completion plugin used by your engineers, an internal search assistant, a summarization tool for a specific employee role). Those still need a lighter review covering security and data handling, but the regulatory and brand surfaces are smaller. Match the depth of the review to the blast radius of the deployment, but never skip the review entirely just because the feature feels low-stakes during the demo.