Tests

Find the flaky tests in your CI

Upload the JUnit XML of each test run. See which tests flake, which failures are new, and which tests are slow.

What you get

  • Flaky tests, ranked

    Every test that both passed and failed on a commit, with its flake rate and when it last flaked.

  • New failures first

    For each commit, the tests that failed and were never flaky, apart from the known flakes. Review the first list.

  • A comment for the pull request

    The upload answers with ready Markdown. One line in your workflow posts it.

    Read more in the docs
  • The slowest tests

    The tests with the highest mean duration over the last 20 commits.

  • History of each test

    A bar for each of the last 30 commits: passed, failed, flaky or skipped.

  • Any runner

    Vitest, Playwright, Jest, pytest, Maven and others write JUnit XML. Tests reads it in your upload, with no install.

    Read more in the docs

Upload in 3 steps

  1. Sign in, add your repo (an owner and a name, such as acme/widgets) and make an upload token. The token starts with tst_ and is shown once. Save it as a secret of your CI named TESTS_TOKEN.
  2. Make the report with your test runner: JUnit XML.
  3. Send it after your tests, even when they fail, with one curl.
Upload with curl
gzip -c junit.xml | curl --fail-with-body \
  -X POST "https://tests.utilarium.app/api/v1/upload?commit=$(git rev-parse HEAD)&branch=main&flag=unit" \
  -H "Authorization: Bearer $TESTS_TOKEN" \
  -H "Content-Type: application/xml" \
  -H "Content-Encoding: gzip" \
  --data-binary @-
GitHub Actions with Vitest
# .github/workflows/tests.yml
name: tests
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx vitest run --reporter=default --reporter=junit --outputFile.junit=junit.xml
        continue-on-error: true
      - name: Upload test results
        run: |
          gzip -c junit.xml | curl --fail-with-body -X POST \
            "https://tests.utilarium.app/api/v1/upload?commit=${{ github.event.pull_request.head.sha || github.sha }}&branch=${{ github.head_ref || github.ref_name }}&run=${{ github.run_id }}-${{ github.run_attempt }}&flag=vitest" \
            -H "Authorization: Bearer ${{ secrets.TESTS_TOKEN }}" \
            -H "Content-Encoding: gzip" \
            --data-binary @- | tee upload.json
        env:
          TESTS_TOKEN: ${{ secrets.TESTS_TOKEN }}

A second upload for the same commit, flag and run replaces the first, so a retried job is safe. The docs show Playwright and the pull request comment.

See your first flaky test in minutes

Add a repo, make a token and paste one curl into your CI. A test that passes after a retry shows up in the Flaky list.