VIREO Quality
Assurance

The AI Startup QA Checklist

A 25-point self-assessment for shipping AI features without breaking prod. Check the boxes, then read the score.

1. Foundations

Your top 3 user journeys have automated smoke tests
Tests run automatically in CI on every pull request
A failing test blocks a merge - it doesn't just warn
Core (non-AI) logic has meaningful unit tests
Test data is isolated and reset between runs
Secrets live in a vault, never in the repo or chat

2. Eval coverage

You have a golden dataset of labeled examples
AI outputs are scored against a rubric - not eyeballed
The golden set is version-controlled and grows from incidents
Most critical AI paths have some eval coverage

3. Prompt regression

A prompt change is compared to a baseline before shipping
Prompts and model configs are version-controlled and reviewed
You run each eval case multiple times and check the bad tail

4. Safety & robustness

You test for prompt injection and jailbreaks
AI outputs are validated before reaching users or systems
Graceful degradation when the model misbehaves or is down

5. Cost & guardrails

AI API cost and latency are monitored
A 3× token-cost regression would be caught before month-end
You have budget or rate guardrails on AI usage

6. CI gates & release

Evals run in CI, not just on someone's laptop
A failing eval can block a release
Releases are verified by an automated gate, not "push and watch"

7. Ownership

Someone owns "is this safe to ship," with criteria
There's a written definition of done / sign-off for AI features
After an incident, you add a regression case so it can't recur

Your score

0 / 25 checked

0–8 - High risk. No systematic safety net. Start with smoke tests + a CI gate.

9–15 - Partial coverage. Pieces exist, unevenly. Add evals + regression.

16–21 - Release-confident. Real coverage. Tighten ownership and depth.

22–25 - Green builds. Release-confident AI delivery. Keep it green.

Want this scored for you, with a prioritized 90-day roadmap? Take the free AI Release Readiness Grader - vireoqa.com/readiness

Vireo: a green bird that eats bugs for a living. On brand. · vireoqa.com · hello@vireoqa.com