The fix ships. Two releases later, the bug is back.
Regressions eat the sprint. A fix ships, the bug returns two releases later, and without regression coverage there’s no telling a real fix from a lucky one.
Fractional QA for AI-native startups
Ship AI features without breaking prod.
A working demo isn’t a reliable feature. AI outputs shift every run, regressions slip back in, and there’s rarely clean proof a change helped.
I install the complete QA function for AI-native startups: evals, prompt regression, CI gates, and senior release judgment. That’s what turns “ready to ship” into a call you can back with evidence.
$ vireo eval run --suite release
▸ golden dataset ......... ok
▸ prompt regression ...... ok
✗ FAIL✓ PASS injection.safety 0 leaks
✓ PASS eval.coverage complete
✓ PASS cost.guardrails within budget
● release.gate green build10+ years across regulated gaming and AI-native SaaS
the 2am problem
“Works in the demo” is not a release strategy.
Three things break AI-native teams. Most ship with at least one unsolved.
An LLM feature that behaves differently every deploy.
AI outputs change from one deploy to the next. Spot-checking a few by eye can’t catch a model that’s quietly, confidently wrong, or prove a prompt change actually helped.
Quality is whoever happens to try it before release.
With no gates, evals, or clear owner, quality falls to whoever clicks through before release, a hard place to improve from, because nothing’s being measured yet.
the fix
three tiers, one ladder
An installed QA function, not hours of manual testing.
Each tier feeds the next. Start small, prove value, expand. No open-ended hourly, ever.
QA Readiness Audit
$3,000 USD fixed
1–2 weeks · risk map + 90-day roadmap
A clear-eyed read of your repo, CI, release process, and AI-feature risk. You get the roadmap whether or not we work together, and the fee credits toward a Sprint if you move forward.
See what's includedQA Foundation Sprint
$15,000 USD base scope; add-ons quoted up front
30–45 days · the AI-quality system installed
Eval harness, prompt regression suites, injection-safety tests, CI gates, and a QA operating model. Playwright and CI are the included foundation; the AI-testing layer is the value.
See what's includedFractional QA Lead
from $5,000/mo USD AI Quality Lead tier from $7k
3-month min, then month-to-month
Senior QA ownership on retainer: I maintain the quality function, review PRs, gate releases, and send a weekly written quality status. Teams whose core risk is the model can step up to the AI Quality Lead tier, adding eval suites, regression gates, drift monitoring, and AI release gates.
See what's includedhow it works
Audit → Foundation → Fractional.
- 01
Audit
Two weeks. I review your repo, CI, release process, and AI features, then hand you a risk map and a prioritized 90-day roadmap. You keep it whether or not we work together again.
- 02
Foundation Sprint
30–45 days. I install the AI-quality system: eval harness, prompt regression suites, injection-safety tests, CI gates, and a QA operating model your engineers will actually follow.
- 03
Fractional QA Lead
Ongoing. I stay on as your QA lead (maintaining the framework, reviewing PRs, signing off releases, watching evals) for a fraction of a full-time hire.
who's behind it
the operator
One senior engineer. Proven playbook.
I’m Matt, an SDET / QA architect with 10+ years owning quality end-to-end across consumer SaaS, enterprise web, and high-stakes regulated platforms. I currently build the entire quality function for an AI-native, multi-tenant SaaS platform: TypeScript + Playwright across three apps, a tRPC API, and a full CI matrix on GitHub Actions.
Before that, five years on a multi-jurisdictional gaming platform serving 300+ casino locations across 7+ jurisdictions, zero-tolerance for defects, 100% compliance record. I’ve built a team’s first-ever QA system from nothing, and I bring that playbook to yours.
QA accelerates, never blocks. Build in, don’t bolt on. Automate the repeatable. Catch it where it’s cheap.
Read the full storyobjections, answered
Questions a CTO asks.
Why not just hire a full-time QA engineer?
You should, eventually. But a senior SDET is ~$160k plus four months of ramp building a framework from scratch on your dime. I install a working framework and process in 30–45 days at a fraction of the cost, and when you do hire, they inherit something that works instead of a blank repo. I’ll even help you write the job spec.
Can’t AI just do QA now?
AI is why I’m fast, it’s a force multiplier on top of judgment, not a replacement for it. Someone still has to decide what to test, what “good” means for your LLM outputs, and what’s safe to ship. That judgment layer is the product. The tools are commodities; the decisions are not.
How does testing an LLM feature even work?
You build golden datasets, write evals that score outputs against them, run prompt regression suites so a prompt change can’t silently break behavior, add injection-safety and cost guardrails, and wire all of it into CI as release gates. The full method is on the LLM testing page.
Why fixed-price instead of hourly?
Hourly pays me to be slow. Fixed price pays me to be good. Every package has a defined scope, deliverables, and price. No open-ended meters. If something falls outside scope, it becomes its own small fixed-price engagement.
What stacks do you cover?
I’m stack-agnostic; the patterns travel, so I fit into whatever you already run. I’m deepest in Playwright (TypeScript and Python/pytest) across web, API (REST/SOAP), database, mobile (Appium), accessibility (axe), visual, performance, and CI (GitHub Actions or your provider). For AI features: evals, prompt regression, output validation, injection safety, and API cost monitoring.
We ship in a regulated or compliance-heavy space. Still a fit?
Yes, often more so. The mechanics that matter transfer straight across from one regulated domain to the next: gating what is risky, proving a release is safe, and owning who signs off. Before AI-native SaaS I spent years in regulated gaming across 7+ jurisdictions with a 100% compliance record, so release discipline under audit pressure is familiar ground. The AI-testing layer sits on top of that same judgment.
Remote? Timezones?
Fully remote, based in Calgary, Canada. I work North American and European hours, so your test runs are triaged before standup. You get my hours and response times in writing at kickoff.
Not sure where your AI QA stands?
Run the free readiness grader. Twelve quick questions across the six pillars of shipping AI safely, and you'll get an instant read on where you're solid and where a bad release could slip through.
Score your readinessThe AI Startup QA Checklist
A 25-point self-assessment for shipping AI features without breaking prod. Enter your email and I’ll send it over, plus the occasional Green Builds email (one a month, no fluff).
One email a month. Unsubscribe anytime. Or read it right now.
One bad regression away from a lost week.
Book a 20-minute intro call. I’ll tell you honestly whether I can help and what the right next step is: audit, sprint, or nothing yet.