The QA setup every seed-stage AI startup actually needs (and what to skip)

Most QA advice is written for companies with 200 engineers and a compliance department. If you’re a 10-person AI startup, almost none of it applies. You don’t need a test pyramid poster. You need the smallest setup that stops you from shipping embarrassing breakage, and the discipline to skip everything else.

Here’s the honest version.

Build now

1. Smoke tests on your three critical paths. Not “full coverage.” The three flows that, if broken, mean you’re losing money or trust right now: sign-up/auth, the core AI interaction, and billing if you have it. Five to ten end-to-end tests, run on every PR.

test('core chat flow returns a grounded answer', async ({ page }) => {
  await page.goto('/app');
  await page.getByRole('textbox').fill('What is in my last invoice?');
  await page.getByRole('button', { name: 'Send' }).click();
  await expect(page.getByTestId('answer')).toContainText(/invoice/i);
});

2. A CI gate that blocks merge. A test suite nobody is forced to pass is decoration. Smoke on every PR, blocking. This one decision changes behavior more than any framework choice.

3. A tiny eval set for your AI feature. Twenty to forty real inputs with a reference of what “good” looks like, scored in CI. This is the part generic QA people skip and the part that actually protects an AI product. (How that works: testing LLM features.)

4. One written page: how a release gets approved. Who clicks deploy, what has to be green first, what happens when it breaks. One page. It prevents the “wait, who owns this?” incident.

Skip (for now)

• 100% coverage. A coverage number is a vanity metric at your stage. Cover what’s risky; ignore the rest.
• Visual regression, load testing, cross-browser matrices. Real eventually, premature now. You have ~12 users and three of them are your investors.
• A QA hire. At 10 engineers a full-time QA is usually the wrong spend, you don’t yet have enough surface to keep them busy on the right things. (When to actually hire one.)
• A heavyweight test framework with 14 plugins. Playwright, a CI file, and a folder. That’s it.

Deliberately defer

Some things you should plan for without building yet: prompt-injection red-teaming, API cost guardrails, and a full QA operating model. Write a one-line note for each so future-you knows it was a choice, not an oversight. Deferring on purpose is senior; forgetting is not.

The 80/20

If you do exactly four things, smoke the three critical paths, gate them in CI, score a small eval set, and write one release-approval page, you’ve eliminated the majority of the bugs that make a startup look amateur, in about a week of work.

That week is what a QA Foundation Sprint compresses and hands over, tuned to your stack, with the AI-testing layer built in. But you can start it yourself today. AI QA in a Box packages these exact four things as a runnable starter if you’d rather not build from a blank page. The worst setup is the one you keep meaning to build.