✓ the flagship method
Testing AI features that never give the same answer twice.
Right now you change a prompt, eyeball a few outputs, and ship. That’s not a test, that’s flying blind. You can’t improve what you don’t measure.
Here’s the actual method for putting non-deterministic AI features under test: golden datasets, evals as gates, prompt regression, and injection safety, the same system I install in a Foundation Sprint.
✓ why it's different
Determinism was the thing your old tests relied on.
A traditional test asserts an exact output. An LLM gives you a different output every run, and sometimes it’s quietly, confidently wrong. The fix isn’t “test harder,” it’s a different assertion model.
Three properties make AI features hard to test, and each has a concrete answer:
- Non-determinism. Same input, different output. Score against a rubric and a reference instead of asserting a string; run each case several times and assert on the distribution.
- Open-ended quality. “Good” is fuzzy. Pin it down with a golden dataset of human-labeled cases, that’s where the judgment gets encoded.
- Silent regression. A prompt tweak fixes one case and breaks three others you didn’t check. Wire evals into CI so a regression can’t merge.
✓ the backbone
Golden datasets: the unglamorous core.
A golden dataset is a curated set of inputs paired with a reference of what a good output looks like. It’s where error analysis turns into something you can test against, and it’s the asset you keep.
Start small and real: pull 40–60 cases from production logs, including the failures that embarrassed you. Label them. That set becomes the ground truth your evals score against, deterministic checks for the hard rules, rubric scores for the fuzzy quality.
# evals/test_support_tone.py
import pytest
from evals.harness import score, load_golden
GOLDEN = load_golden("support_replies.jsonl") # 60 curated cases
@pytest.mark.parametrize("case", GOLDEN, ids=lambda c: c["id"])
def test_reply_quality(case):
reply = run_feature(case["input"])
# Deterministic assertions first - cheap and exact.
assert case["must_not_contain"] not in reply.lower()
# Then a scored, rubric-based eval for the fuzzy parts.
result = score(
rubric="support_tone_v3",
input=case["input"],
output=reply,
reference=case["reference"],
)
assert result.score >= 0.8, result.explanation✓ unit tests for prompts
Evals only matter if they gate the release.
An eval suite that runs on someone’s laptop once a month is theater. Wire it into CI as a release gate, exactly like your unit tests, a regression blocks the merge.
# .github/workflows/evals.yml
name: evals
on: [pull_request]
jobs:
release-eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install -r requirements.txt
- name: Run release eval suite (blocks merge on regression)
run: pytest evals/ --maxfail=1 --eval-threshold=0.8
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}Now a prompt change is held to the same bar as a code change. No green build, no merge.
✓ did you fix it, or get lucky?
Know whether a prompt change helped or hurt.
Every prompt change is measured against the current baseline. If it improves one case but regresses four (or opens an injection hole), the suite says so before your users do.
# A prompt change must beat the current baseline, not just "look good."
$ vireo eval run --suite release --baseline main
prompt v36 (main) ........ 0.86 avg | injection 0 leaks
prompt v37 (this PR) ..... 0.83 avg | injection 1 leak
✗ FAIL quality regressed 0.86 -> 0.83 on 4 cases
✗ FAIL injection.safety new leak in 'ignore previous instructions'
did you fix it, or did you just get lucky? - blocked.Injection and output safety run in the same gate: prompt injection attempts, jailbreaks, and unsafe-output checks. The scanner is a commodity now; the value is interpreting the results and deciding what’s shippable.
✓ the bill nobody tests
API cost monitoring is a quality gate too.
A prompt change can quietly triple your token spend or latency. Track cost per eval run and per request, and alert on regressions. A 3x cost jump should fail the build, not surprise you at month-end.
Cost, latency, and quality are one system. The release gate that scores output quality should also fail on a cost or latency regression. That’s what “release-confident” actually means for an AI feature.
✓ questions
The questions teams ask first.
How do you test something that never returns the same answer twice?
You stop asserting exact strings and start scoring against a rubric and a reference set. Deterministic checks (must-not-contain, schema, refusal behavior) catch the hard failures; rubric-based evals score the fuzzy quality. Run each case a few times and assert on the distribution, not a single output.
Isn’t “LLM-as-judge” just guesswork with extra steps?
Only if you skip the error analysis. The judgment lives in the rubric and the golden set, what “good” means for your product, written down and validated against human-labeled cases. The model is just the scorer; the taste is yours, version-controlled.
We already use Promptfoo / Braintrust / LangSmith. Do we need you?
Those are good tools, I’ll happily configure them. But a tool doesn’t decide what to test, design your rubric, do error analysis, or own the release call. That’s the work that doesn’t commoditize, and it’s what the Foundation Sprint installs.
Where does this fit with normal QA?
Evals sit alongside your Playwright E2E and API tests, all gated in the same CI. Deterministic systems get deterministic tests; non-deterministic ones get scored evals. Same pipeline, same release gate, different assertion style.
Ship AI features with evidence behind them.
Run the free readiness grader to see where your AI testing actually stands, or book a call and we’ll talk through your specific features.