Knowledge Base / Cheat-sheets
AI Guardrails in Testing
Risk framing and harm taxonomy, input and output controls, evaluation and red teaming, observability and governance, and a dedicated guardrail tooling roundup - five sections you can filter and toggle as you verify them. Progress is saved on this device.
What this checklist covers
Risk Framing & Scope
A guardrail you cannot name the failure for is decoration. Before any control is built, the harms in scope, the ways the model can be pushed into them, and the threshold that counts as 'safe enough' all have to be written down.
- The harms in scope are enumerated specifically (toxic output, PII leakage, unsafe clinical advice, prompt-injected data exfiltration) rather than gestured at as 'bad answers'.
- Each harm has a worst-case consequence attached, so guardrail effort is spent in proportion to damage rather than spread evenly.
- Every untrusted text path into the prompt is mapped - user input, retrieved documents, tool output, file contents - since any of them can carry an injection.
- Tool and function-calling permissions are modelled as capabilities the attacker inherits if injection succeeds, not as implementation detail.
- 'Safe enough' is expressed as a measurable rate on a named eval set, not as a subjective review verdict.
- False-positive cost is set alongside false-negative cost, so an over-blocking guardrail is treated as a real failure rather than a safe default.
Input & Output Controls
The runtime controls that sit either side of the model. These are the guardrails proper: what reaches the model, what is allowed back out, and how much of the answer has to be traceable to a source.
- Untrusted content is delimited and labelled as data in the prompt, so instructions embedded inside it are less likely to be followed as commands.
- Tests include direct injection ('ignore previous instructions') and indirect injection, where the payload arrives inside a retrieved document or web page.
- Output is screened before it reaches the user or a downstream system, not only logged after the fact.
- Refusals are tested for usefulness as well as correctness - a refusal that gives no path forward is a usability defect even when the safety call was right.
- For retrieval-backed answers, claims are checked against the retrieved context rather than against the tester's own knowledge.
- Behaviour when retrieval returns nothing relevant is tested - the correct answer is 'I don't have that', and a confident fabrication there is a top-severity defect.
Evaluation & Red Teaming
Guardrails are probabilistic, so they are verified statistically rather than by a single pass. That means a maintained eval set, adversaries who are actively trying to get through, and a way to notice when yesterday's numbers stop holding.
- The eval set contains both attack cases and benign cases, so the pass rate cannot be gamed by blocking everything.
- Every production incident is converted into an eval case, which is what stops the same failure shipping twice.
- Red teaming is run by people who did not build the guardrail, because the author's blind spots are exactly what needs finding.
- Jailbreak families are covered systematically - roleplay framing, hypothetical framing, incremental escalation, obfuscated payloads - not sampled at random.
- The eval suite runs on every prompt change, model version bump and retrieval-index rebuild - all three can move behaviour.
- Model versions are pinned in production, so a provider-side update cannot silently change behaviour between evaluations.
Observability, Governance & Release
What happens once the feature is live: whether a failure can be reconstructed, who catches what the automation misses, and what has to be written down for auditors and regulators.
- Prompt, retrieved context, model version and output are captured together, so an incident can be reproduced rather than guessed at.
- Logging respects the same privacy rules as the rest of the product - a trace store full of unredacted PII is its own breach.
- Actions that are irreversible or high-consequence require human confirmation before execution, not review afterwards.
- The reviewer is given enough context to disagree - a confirmation prompt with no evidence produces rubber-stamping, not oversight.
- The feature is classified against the applicable regime (EU AI Act risk tier, FDA SaMD, internal AI policy) before release, not during audit.
- Intended use and known limitations are documented in user-facing terms, so the boundary of safe use is visible to the person relying on it.
Guardrail Tooling
The implementation layer. Nothing here removes the need for the thinking above - a framework enforces the policy you wrote, it does not decide what the policy should be.
- NVIDIA NeMo Guardrails expresses conversational rails in a dedicated modelling language, suiting flows where the allowed topic path matters as much as the content.
- Guardrails AI validates output against declared schemas and validators, which fits structured-extraction pipelines better than open chat.
- promptfoo runs declarative prompt/model matrices from a config file and fits CI naturally, which makes it a good default for regression evals.
- DeepEval brings LLM evaluation into a pytest-shaped workflow, so evals live beside the rest of the Python test suite.
- Garak probes a model with a library of known attack families and reports which ones landed - a fast first pass before manual red teaming.
- Microsoft PyRIT scripts multi-turn automated attacks, which reaches escalation paths single-shot probes miss.
- LangSmith, Langfuse or Arize Phoenix capture full traces of prompt, retrieval and response, which is what makes incident reconstruction possible.
- Langfuse and Phoenix can be self-hosted, which matters when traces contain regulated data that cannot go to a vendor.