How I test LLM features that never return the same answer twice

The first time someone asks you to “write a test” for an LLM feature, your instinct fights you. A test asserts a known output. An LLM gives you a different output every run. So most teams quietly give up and fall back to the ritual: change a prompt, eyeball three or four outputs, and if it looked good, ship it.

That’s not testing. That’s vibe checks with extra steps. Here’s what I do instead.

Stop asserting strings. Start scoring.

The mental shift is this: you’re not checking equality, you’re checking quality. Two layers do the job.

Layer one, deterministic assertions. Even a non-deterministic output usually has hard rules that must always hold. It must not leak the system prompt. It must return valid JSON when you asked for JSON. It must refuse when it should refuse. These are exact, cheap, and fast, write them as normal assertions.

def test_hard_rules(case):
    out = run_feature(case["input"])
    assert "system prompt" not in out.lower()      # no leakage
    assert is_valid_json(out) is case["expects_json"]
    if case["should_refuse"]:
        assert is_refusal(out)

Layer two, scored evals. For the fuzzy part (“is this a good support reply?”), score the output against a reference using a rubric, and assert a threshold rather than equality.

def test_reply_quality(case):
    out = run_feature(case["input"])
    result = score(
        rubric="support_tone_v3",      # your written definition of "good"
        output=out,
        reference=case["reference"],
    )
    assert result.score >= 0.8, result.explanation

The judgment lives in support_tone_v3 and in the reference, not in the model doing the scoring. That distinction is the whole game, and it’s covered in depth on the LLM testing page.

Handle non-determinism head-on

A single run tells you almost nothing. Run each case a few times and assert on the distribution, not one lucky sample.

@pytest.mark.parametrize("case", GOLDEN)
def test_stable_quality(case):
    scores = [score("support_tone_v3", run_feature(case["input"]),
                     reference=case["reference"]).score
              for _ in range(5)]
    assert min(scores) >= 0.7        # no run falls off a cliff
    assert mean(scores) >= 0.8       # and the average is solid

Now a feature that’s “usually fine but occasionally terrible” fails, which is exactly the behavior that burns you at 2am.

The trap: “did you fix it, or did you just get lucky?”

You tweak a prompt to fix one bad case. It works. You ship. Three other cases you didn’t look at silently regressed. This is the single most common way AI features rot.

The fix is to measure every prompt change against the current baseline:

$ vireo eval run --suite release --baseline main

  prompt v36 (main) ...... 0.86 avg
  prompt v37 (this PR) ... 0.83 avg

✗ FAIL  quality regressed 0.86 -> 0.83 on 4 cases

“It looked better to me” is now a falsifiable claim. Either the numbers moved or they didn’t.

None of this matters until it’s a gate

An eval suite that runs on your laptop once a month is theater. The value shows up when a regression blocks the merge, the same way a failing unit test does.

# .github/workflows/evals.yml
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pytest evals/ --eval-threshold=0.8

That’s the difference between knowing your AI feature works and hoping it does.

Where to start

You don’t need a platform. You need 40 real cases, a written rubric, two layers of assertions, and a CI job. That’s a week of work and it ends the prompt-and-pray era for your team.

If you’d rather have it installed and handed over, golden sets, rubrics, regression suite, and CI gates wired to your stack, that’s exactly what a QA Foundation Sprint is. Either way: stop shipping on vibes.