Performance Regression Testing: Compare Load Runs Across Releases
Short answer
Performance regression testing asks whether this version is slower or more error-prone than a comparable prior run—not whether a one-off k6 chart “looked fine.” You overlay the same journey on the same environment, profile, and dataset, then read p95, fail rate, and 4xx / 5xx. TestChimp stamps every wrapper run with the git HEAD SHA, auto-tags untagged runs onto the release whose git cut contains that SHA, and lets you compare 1.3 to 1.2 on the release Performance Tests panel—so load and volume data sit in the same QA story as SmartTests, manual runs, and release checks.
Part of Performance testing guides.
Who this is for
Teams that already run load or volume tests and still ship surprises: “p95 crept 2× over three releases and nobody compared the charts.”
Thresholds are not enough
A threshold says “p95 under 800 ms on this profile.” That can stay green while:
- Last release was 410 ms and this one is 790 ms (soft degrade)
- Fail rate doubled from 0.2% to 0.8% still under a 1% gate
- 5xx appeared where last time you only had 4xx (different incident class)
Regression testing is the second question: are we worse than last comparable run? Both belong on a release. Product UI: compare against a prior run.
What “comparable” means
Overlay is only a ship signal when comparison keys match:
- Same journey / composite identity (
testchimp.id) - Same environment class (do not overlay laptop Docker on staging)
- Same profile (smoke ≠ load ≠ soak ≠ spike)
- Same dataset version (1k rows ≠ 100k rows)
- Same LLM / mock latency profile (0 ms payments stub ≠ 200 ms)
TestChimp lists Recommended candidates on the detail page and treats Other as directional only. CLI compare-perf-to-baseline exits nonzero when comparison.regressed is true. A missing or mismatched baseline is incomparable, never a pass. See comparison keys.
Auto-tagging runs to a release (git SHA)
Two mechanisms keep performance on the release record without a spreadsheet:
1. Explicit stamp on a release-scoped run
/testchimp run performance tests for release 1.3.0
The agent uses prior release SHA → cut SHA as the git range (not “whatever is on my laptop”), runs related journeys, and sets TESTCHIMP_RELEASE plus TESTCHIMP_GIT_COMMIT_SHA on wrapper ingest. Workflow: run-perf-tests — release-scoped.
2. Automatic contribution from git cut
You do not have to remember a label on every local run. Wrapper ingest stores the git commit SHA. When a release exists with a git cut, TestChimp stamps untagged perf runs whose SHA sits in that release’s commit set (prior SHA → current SHA) onto that version—and unstamps runs that fall out of the cut.
That is how a k6 job on the release branch shows up on the release Performance Tests panel next to functional batches: HEAD SHA in, release git SHAs as the matcher.
You can still edit the release label on the run detail page when you need a correction.
Consolidated QA story for the release
A ship decision is incomplete if “Playwright is green” while checkout p95 doubled.
| Evidence on the release | Role |
|---|---|
| SmartTests / automation batches | Functional correctness |
| Manual sessions | Auditable human paths |
| Release checks | UX / SAST / DAST / secrets / deps |
| Performance Tests panel | Load + volume: threshold, p95, 4xx/5xx, overlays vs prior version |
| Release intelligence | What changed in the git range |
Related k6 selection for a cut uses the same git range as other release workflows: scenarios, operations, and paths touched between SHAs. If coverage is insufficient, run-perf-tests asks to nest create-perf-tests rather than silently skipping or inventing load.
How to read a degrade
| Signal | Read it as |
|---|---|
| Threshold Failed vs prior Passed | Hard gate miss — start here |
| p95 up, threshold still green | Soft degrade vs last comparable run |
| Fail rate up + 5xx spike | Server/dependency under load or volume |
| Fail rate up + 4xx only | Journey, auth, stock, or seed — do not “scale k8s” first |
| VU chart never ramped | Invalid experiment; fix env/script, do not blame the product |
Full chart guide: viewing results.
Baseline promote (CI)
Named baselines (promote-perf-baseline / list-perf-baselines) are the agent/CI form of “last good release on staging / load profile / dataset X.” Compare only on matching keys. Never retarget the baseline or drop stub latency to 0 ms to force green.
Related
- Metrics · Load testing · k6 vs JMeter vs Locust
- Viewing results · Run-perf-tests · Release management
Frequently asked questions
What is performance regression testing?
It compares a new build’s load or volume results to a comparable prior run (same journey, environment, profile, dataset) to see whether latency or error rates got worse—even when absolute SLOs still pass.
How do I compare k6 results across releases?
Run the same journey with the same profile and dataset on each cut, stamp git SHA (and optionally TESTCHIMP_RELEASE), then overlay the prior release in Executions or open the release Performance Tests panel. TestChimp auto-tags untagged runs whose commit sits in the release git cut.
Why not just look at the latest k6 HTML report?
A single report has no prior series, no comparison keys, and no attachment to the release record. You cannot tell 1.3 vs 1.2, or whether you compared smoke to load. Ingest plus overlay is the regression workflow.
How does TestChimp attach performance tests to a release?
Wrappers record TESTCHIMP_GIT_COMMIT_SHA. TestChimp stamps untagged perf runs onto the release whose prior→current git SHAs include that commit. Release-scoped /testchimp run-perf-tests also sets TESTCHIMP_RELEASE. Both surfaces feed the release Performance Tests panel.
What if the overlay run used a different dataset?
Then it is incomparable. TestChimp lists it under Other if you insist on a directional overlay, but recommended comparisons and compare-perf-to-baseline require matching environment, profile, dataset, and mock/latency keys.
Ship with checkout p95, not only a green PR
Run `/testchimp run performance tests for release 1.2.0` so related k6 journeys stamp the cut SHA, land on the release, and overlay the previous version’s p95 and 5xx.