Tests docs
The upload API, the answer, limits and what a flaky test is. Everything a CI job needs to send its test results.
On this page
The words
- Repo:
owner/name, lower case. It has one owner, the person who added it, and a default branch (main). - Run: one upload. One CI run, or one attempt of it, for one commit, one branch and one flag.
- Flag: the name of the CI job, such as
vitestorplaywright. It is empty by default. - Test: a
testcaseof the XML. Its key is itsclassname(orfile), two colons and itsname.
A test is passed, failed (it has a failure or an error) or skipped. If a test is in the file twice, a failure wins over a pass, and a pass over a skip.
Flaky tests
- A test is flaky at a commit when it passed and failed there, in the runs of that commit. A retry that passes after a failure makes a test flaky.
- A test is a known flake when it was flaky on an earlier commit of the repo.
- A new failure is a test that failed in every run of the commit and is not a known flake.
- The slowest tests are the highest mean duration over the last 20 commits of the default branch.
"Earlier" is by the time of the upload. Tests keeps the last 200 commits of each branch, so a flake older than that is forgotten.
POST /api/v1/upload
Send the JUnit XML as the body, with the token in a header. The body may be gzip (Content-Encoding: gzip).
commit: the 40 hex digits of the commit. Required.branch: the branch name. Required.run: an opaque id of the run, such as the CI run id and attempt. Optional: a random one is made. The same repo, commit, flag and run sent again replaces the first upload.flag: the job name. Optional.Authorization: Bearer tst_...: an upload token of the repo, made on its page.
<testsuites>
<testsuite name="src/orders.test.ts" tests="2" failures="1">
<testcase classname="src/orders.test.ts" name="creates an order" time="0.012">
<failure message="expected 2 to be 3"/>
</testcase>
<testcase classname="src/orders.test.ts" name="lists orders" time="0.004"/>
</testsuite>
</testsuites>The answer is 200 with the counts, the new failures and known flakes of the commit, the address of its page, and markdown: a pull request comment your CI can post. Errors are JSON { "ok": false, "error": "..." }: 400 for a bad query or XML, 401 for a missing or wrong token, 413 for a body that is too big.
{
"runId": "6f1c0a52-3b43-4d2f-8a58-0f7f5f0d9d11",
"commit": "0123456789abcdef0123456789abcdef01234567",
"tests": 120,
"passed": 117,
"failed": 2,
"skipped": 1,
"newFailures": [
{ "testId": 14, "name": "creates an order", "classname": "src/orders.test.ts" }
],
"knownFlakes": [
{ "testId": 9, "name": "reconnects", "classname": "src/socket.test.ts" }
],
"url": "https://tests.utilarium.app/dashboard/r/acme/widgets/commit/0123456789abcdef0123456789abcdef01234567",
"markdown": "### Tests by Utilarium\n\n1 new failure. 1 known flake. ..."
}One curl
gzip -c junit.xml | curl --fail-with-body \
-X POST "https://tests.utilarium.app/api/v1/upload?commit=$(git rev-parse HEAD)&branch=main&flag=unit" \
-H "Authorization: Bearer $TESTS_TOKEN" \
-H "Content-Type: application/xml" \
-H "Content-Encoding: gzip" \
--data-binary @-Put the token in a secret of your CI named TESTS_TOKEN. It is shown once, when you make it.
GitHub Actions
Vitest writes JUnit with --reporter=junit. The test step must not stop the job, so the results are sent when a test fails: that is when you want them.
# .github/workflows/tests.yml
name: tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
- run: npm ci
- run: npx vitest run --reporter=default --reporter=junit --outputFile.junit=junit.xml
continue-on-error: true
- name: Upload test results
run: |
gzip -c junit.xml | curl --fail-with-body -X POST \
"https://tests.utilarium.app/api/v1/upload?commit=${{ github.event.pull_request.head.sha || github.sha }}&branch=${{ github.head_ref || github.ref_name }}&run=${{ github.run_id }}-${{ github.run_attempt }}&flag=vitest" \
-H "Authorization: Bearer ${{ secrets.TESTS_TOKEN }}" \
-H "Content-Encoding: gzip" \
--data-binary @- | tee upload.json
env:
TESTS_TOKEN: ${{ secrets.TESTS_TOKEN }}Playwright writes JUnit with --reporter=junit. Send it with another flag.
- run: npx playwright test --reporter=junit
continue-on-error: true
env:
PLAYWRIGHT_JUNIT_OUTPUT_NAME: junit.xml
# then the same upload step, with flag=playwrightA pull request comment
The answer has markdown. Post it with the GitHub CLI after the upload step. It names the new failures apart from the known flakes, so a reviewer knows what to look at.
- name: Comment on the pull request
if: github.event_name == 'pull_request'
run: jq -r .markdown upload.json | gh pr comment ${{ github.event.pull_request.number }} --body-file -
env:
GH_TOKEN: ${{ github.token }}Limits
- 8 MB for the body sent, and 8 MB after gzip is opened.
- 50,000 test cases for each upload.
- 400 characters for a name and for a classname: more is cut.
- 20 repos for each person, 20 upload tokens for each repo.
- 10 repos or tokens made each minute.
- The last 200 commits of each branch. When a new commit lands, older ones with their results are deleted.