Setting up Allure reporting that people actually read

A test report has one job: turn a red build into a fix, fast. Plain CI logs don’t do that, people scroll, guess, and re-run. Allure does, but only if you set it up for signal instead of noise. Here’s the setup I use, and the parts most teams skip.

Install and generate

For pytest, it’s a plugin plus the Allure CLI (Java-based) to render results into HTML:

pip install allure-pytest
pytest --alluredir=allure-results
allure generate allure-results -o allure-report --clean

Playwright is the same idea with allure-playwright as the reporter. The allure-results folder is raw JSON; the CLI turns it into the report you actually browse.

Annotate for triage, not decoration

The default report already groups pass/fail. The value comes from annotations that let someone answer “what broke and where” without opening the code:

import allure

@allure.feature("Checkout")
@allure.story("Apply discount code")
@allure.severity(allure.severity_level.CRITICAL)
def test_expired_code_is_rejected(client):
    with allure.step("Submit an expired code"):
        resp = client.apply_code("EXPIRED2025")
    assert resp.status_code == 422

feature/story give you a behavior map; severity lets you read the report worst-first. Steps turn a failure into a timeline instead of one opaque stack trace.

Attach evidence on failure

A failing UI test with a screenshot, DOM, and the request/response attached is triaged in seconds. Wire it once in a fixture so every test benefits:

@pytest.fixture(autouse=True)
def _attach_on_fail(request, page):
    yield
    if request.node.rep_call.failed:
        allure.attach(
            page.screenshot(), name="screenshot",
            attachment_type=allure.attachment_type.PNG,
        )
        allure.attach(page.content(), name="dom",
                      attachment_type=allure.attachment_type.HTML)

For API tests, attach the request body and response. The rule: attach whatever you’d otherwise ask the author for in Slack.

Trends are the point, wire history in CI

A single run tells you today’s state. Trends tell you whether the suite is decaying. Allure builds flakiness and history graphs only if you carry the previous report’s history folder into the next run:

# copy prior history into results before generating
- run: cp -r ./allure-report/history ./allure-results/history || true
- run: allure generate allure-results -o allure-report --clean
- uses: actions/upload-artifact@v4
  with: { name: allure-report, path: allure-report }

Without that copy, every report looks like day one and you lose the flaky-test detection that makes Allure worth the setup.

When not to bother

For a tiny smoke suite, Allure is overkill, the built-in Playwright HTML report or pytest output is enough. Reach for Allure once you have enough tests and enough people that “who owns this failure” becomes a real question.

---

Reporting is the layer that keeps a suite trusted long after it’s written, and it’s part of what I set up in a QA Foundation Sprint, history and all. If your AI features need the same visibility, the same discipline applies to eval results in CI.