Playwright + GitHub Actions: a CI test pipeline that doesn't flake

A flaky test suite is worse than no suite. People learn to re-run red, then to ignore it, and within a month “green” means nothing. The goal of a CI pipeline isn’t just to run tests, it’s to stay trusted. Here’s the Playwright + GitHub Actions setup I install, and the habits that keep it from rotting.

Stage the pipeline so the fast feedback is fast

Don’t run everything on every push. Stage it by how much confidence each step buys versus how long it takes:

• Smoke on PR, your handful of critical-path tests, under ~5 minutes, blocking the merge.
• Regression on merge to main, the broader suite.
• Full + nightly, the long tail, plus anything slow (visual, cross-browser), on a schedule.

The smoke gate is the one that changes behavior. If a broken critical path can’t merge, quality stops being optional.

# .github/workflows/smoke.yml
name: smoke
on: [pull_request]
jobs:
  smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test --grep @smoke

Tag your critical paths with @smoke so this job stays small on purpose.

Shard for speed without the cost blowup

When the regression suite grows, don’t wait 20 minutes, shard it across parallel runners. Playwright has this built in:

jobs:
  regression:
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test --shard=${{ matrix.shard }}/4

Four shards turn a 20-minute run into ~5. You pay for four runners for five minutes instead of one for twenty, usually a wash, and developers get their answer four times faster.

Retries: where they help, where they hide bugs

Retries are a painkiller, not a cure. Set a small retry count in CI only so a one-off network blip doesn’t fail a whole run, but never locally, where you want to feel the flake:

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 1 : 0,
  reporter: process.env.CI ? [['blob'], ['github']] : 'list',
});

The trap: retries that quietly pass on the second attempt are hiding a real bug. Treat a test that needed a retry as a defect to investigate, not a success. If a test is genuinely unstable, quarantine it (skip with a ticket), don’t let it erode trust in the rest.

Kill the top five flake sources

Almost all flake comes from the same places:

1. Fixed waits. Delete every waitForTimeout. Use web-first assertions that retry until true: await expect(page.getByRole('button')).toBeVisible().
2. Shared state. Tests that depend on data another test created will fail when sharded. Create and tear down each test’s own data; run with full isolation.
3. Time. Tests that assume “today” break at midnight and across timezones. Freeze the clock.
4. Network. Real third-party calls flake. Mock them at the network layer for deterministic runs.
5. Order. If your suite only passes in one order, it’s not a suite, it’s a house of cards. Randomize order in CI and fix what falls over.

// Bad: sleeps and shared state
await page.waitForTimeout(3000);
await page.click('#submit'); // assumes a user seeded earlier

// Good: web-first wait, isolated data
const user = await createUser();
await page.goto(`/login?as=${user.id}`);
await expect(page.getByText('Dashboard')).toBeVisible();

Reporting the team actually reads

Merge sharded results into one HTML report and surface failures where people look. The github reporter annotates the PR directly; the blob reporter lets you combine shards:

- run: npx playwright merge-reports --reporter=html ./blob-reports
- uses: actions/upload-artifact@v4
  with: { name: playwright-report, path: playwright-report }

A red check that links straight to the failing trace is the difference between “someone will look at it eventually” and “fixed before lunch.”

---

None of this is exotic, it’s discipline encoded into config. A staged pipeline, sane retries, isolated tests, and a report people trust will outperform a thousand tests that nobody believes. If you’d rather have it built and handed over, tuned to your stack, that’s the foundation layer of a QA Foundation Sprint. And if your AI features need the same rigor, here’s how LLM testing fits into CI.