Linking Automation Batches to Test Runs
Short answer
CI pipelines produce automation execution batches—pass/fail/skip counts per Playwright run. Release management connects those batches to test runs on a release so scenario coverage from SmartTests rolls up alongside manual session captures. Teams see one complete picture of what was tested in the candidate version, across automation and manual work.
What is an automation execution batch?
When SmartTests run locally or in CI with the @testchimp/playwright reporter, TestChimp receives an automation batch:
| Property | Description |
|---|---|
| Batch invocation id | Unique id for the run |
| Per-test results | Pass, fail, skip per SmartTest file |
| Scenario links | Scenarios attached via // @Scenario: comments in test code |
| Metadata | Environment, release tag, branch, execution source (CI vs local), timestamps |
Batches appear under Executions → Automation Batches. For release work, the Candidate Automation Execution Batches panel on release analytics lists batches between the prior release commit and the current release commit (or recent candidates when no prior release exists).
Why link batches to test runs?
Linking answers: "Does this CI run count toward our release sign-off campaign?"
| Without link | With link to release test run |
|---|---|
| Batch visible only in execution history | Scenario rows update in test run viewer |
| May be excluded from release Overview | Included in pass/fail aggregates |
| Release tag might not match version | Linking updates release tag alignment |
| Manual and auto progress disconnected | Single progress bar per test run and release |
A batch can link to multiple test runs; a test run can include many batches. This many-to-many model matches how teams re-run CI across environments or reuse a green staging batch for multiple scoped runs.
Two ways batches count on a release
TestChimp includes batch executions in release aggregates when either:
- Release tag matches — the batch was reported with
release: v2.4.0(or your version string) via reporter config or CI env, or - Linked to a test run on this release — you linked the batch from the UI; linking also helps align the release tag
The Coverage column on the candidate batches table shows:
| Status | Meaning |
|---|---|
| Linked to test run | Batch linked to at least one test run on this release |
| Included via release tag | Release tag matches; counts without explicit link |
| Excluded from aggregates | Neither linked nor tagged—hover for guidance |
Prefer explicit links when the same release version string is reused across parallel campaigns or when CI did not set the release tag correctly.
Link from the release analytics panel
- Open a release → View release analytics
- Expand Candidate Automation Execution Batches
- Select the target environment (top bar)—batches filter to executions in that environment
- Find the batch row (date, CI/local source, release tag, pass/fail progress)
- Click the link icon in the row actions
- In the popover, select one or more active test runs on this release (filter by branch/environment/release)
- Save
The batch's SmartTest results flow into each linked test run's scenario table. Refresh stats on the test run and Refresh release data on the release if aggregates look stale.
Click a batch row to open the batch execution viewer with per-test detail.
Link from Executions → Automation Batches
The same Link to test runs action is available:
- From the batch list Actions column
- From the batch viewer header
Use this when you are triaging CI from the executions hub rather than starting from the release page.
How scenario coverage attaches
SmartTests link to scenarios with in-code comments:
// @Scenario: plans/scenarios/checkout/complete-purchase.md
test('checkout completes with valid card', async ({ page }) => {
// ...
});
When a batch links to a test run:
- Each test result in the batch resolves to linked scenario ids
- Scenarios in scope of the test run update their status from the batch result
- Last execution wins per scenario when multiple batches or manual sessions link to the same run
- Requirement coverage on the release reflects the combined evidence
Tests without @Scenario links still appear in the batch viewer but do not move scenario rows—fix links in Git and re-run CI for traceability.
CI setup: tag batches with release and environment
Configure the Playwright reporter so batches arrive with correct metadata:
// playwright.config.ts (example)
reporter: [
['@testchimp/playwright', {
environment: process.env.TESTCHIMP_ENV ?? 'staging',
release: process.env.RELEASE_VERSION, // e.g. v2.4.0 from CI
}],
],
| Variable | Purpose |
|---|---|
TESTCHIMP_ENV | Filters batches per environment on the release |
RELEASE_VERSION | Auto-includes batch in release aggregates when tag matches |
Even with correct tags, linking to a named test run makes intent explicit for sign-off ("this staging CI run = Sprint 42 regression").
See Run SmartTests in CI (Playwright) for full reporter setup.
End-to-end release picture
| Channel | How it attaches | What it contributes |
|---|---|---|
| CI automation | Link batch → test run (or release tag) | Scenario status from SmartTests |
| Manual QA | Extension → test run at capture | Scenario status + step evidence |
| Release intelligence | Aggregates all linked runs | Overview, coverage, delta, ExploreChimp, TrueCoverage |
Troubleshooting
| Symptom | Check |
|---|---|
| Batch not in candidate list | Release commit SHA set? Prior release defined? Batch in commit/time window? |
| Excluded from aggregates | Link to test run or fix release tag on batch |
| Scenarios stay not attempted | Test run scope includes scenario folder? @Scenario link in test? |
| Wrong environment stats | Environment selector on release analytics matches CI TESTCHIMP_ENV |
| Stale progress | Refresh stats on test run; Refresh release data on release |
Related documentation
- Release management overview
- Release intelligence
- Manual sessions to test runs
- Test runs
- Linking scenarios in SmartTests
- Run SmartTests in CI
Frequently asked questions
How do I attach CI test results to a release?
Open the release analytics page, expand Candidate Automation Execution Batches, and link the batch to an active test run on that release. Alternatively use Link to test runs from Executions → Automation Batches. Set release and environment in the @testchimp/playwright reporter so batches can auto-include via release tag.
What is the difference between a release tag and linking to a test run?
A matching release tag includes the batch in release overview aggregates automatically. Linking to a test run also attaches scenario-level results to that named campaign and can align the release tag. Use links when you need explicit sign-off to a specific regression run.
Do automation batches update scenario coverage on a test run?
Yes, when SmartTests include // @Scenario: links to plan scenarios and those scenarios are in the test run scope. The latest linked batch or manual session determines each scenario pass/fail status.
Can one CI batch link to multiple test runs?
Yes. The mapping modal supports many-to-many relationships. A single green staging batch might count toward both a smoke test run and a full regression run if both scopes apply.
Why is my batch excluded from release aggregates?
The batch release tag does not match this version and it is not linked to a test run on this release. Link it from the candidate batches panel or fix the release value in your CI reporter configuration.
Unify CI and manual evidence on every release
Link Playwright batches to test runs on your release so scenario coverage reflects automation and manual testing together.