18 posts tagged with "Testing"

TL;DR: Agents built the product. Agents authored the tests that verify the implementation. The missing piece is accountability—has anyone actually looked at the test to ensure it truly verifies the scenario? Verified Tests puts that sanity check in the platform: inspect screen captures, steps, and code, then stamp a Verified Badge on the covering scenario.

Full product guide: Verified Tests.

The missing piece is accountability​

Agents are very good at getting a path green.

The SmartTest runs. The scenario is linked. Requirement coverage looks healthier than last sprint.

That is not the same as: this test actually verifies the behaviour the scenario describes.

Has anyone actually looked at the test?

A scenario annotation is cheap to add—and agents add them at speed. They can link the right scenario to a script that asserts a toast, skips the important check, or walks a cousin flow that happens to share a button. The dashboard still counts it as coverage.

We used to paper over that with implicit knowledge: the person who wrote the test was the person who knew what it proved. Agents broke that assumption. Volume went up. Inspection did not.

Green is an execution fact. Linked is a mapping. Verified is the accountability layer: a person has sanity-checked that the covering test truly verifies the scenario.

That is the problem Verified Tests solves.

Why this is necessary in the agentic era​

Before agents, test authoring was slow enough that “who wrote this?” was a reasonable proxy for “who checked this?”

That proxy is dead.

Agents implement. Agents author SmartTests. Agents attach scenario ids because the plan files told them to. You still need a durable record that a person opened the execution, looked at what actually happened, read the script, and agreed: yes, this test verifies that scenario.

Otherwise requirement traceability is a spreadsheet of claims with nicer UI. Coverage insights tell you what ran. They do not tell you whether the test is the right test.

Introducing Verified Tests​

TestChimp now lets you Verify tests in the platform—without bouncing out to a repo or a recording elsewhere.

Open a SmartTest execution and you already have what you need to make an informed decision:

• Screen captures from the run
• Steps the test actually took
• The code, via View Test, so you can read the script next to the evidence

Then use the check badge next to the SmartTest name. Same visual language as a verify badge elsewhere on the internet, because the job is the same: this was inspected.

Three states on the test:

• Hollow — none of the linked scenarios are verified
• Grey — some, not all
• Blue — every linked scenario for that test is verified

For one scenario, you confirm. For several, you verify per scenario—or Mark all as verified. Hover a filled badge and you see who verified it.

Un-verify is deliberate: it records manually unverified, not “never happened.” The audit trail stays.

Status lives in TestChimp, not in source. Agents can keep authoring Playwright. Humans stamp the claim when they have actually looked.

A Verified Badge on scenario coverage​

Plans, requirement coverage, and test runs already show recent execution results per scenario.

You now also see a Verified Badge on each scenario—an extra layer of assurance that the product is being tested properly, not only that something ran green.

• Filled — a user has sanity-checked a covering test for that scenario
• Hollow — the scenario has tests, but nobody has inspected them yet
• Omitted — no linked tests, so there is nothing to verify

Clicking the coverage-side badge does not flip the bit. It tells you how: open an execution, inspect the run, then use the check badge next to the test name. Verification is an inspection action, not a bulk paint on a dashboard.

That is the point. If it were too easy, we would have rebuilt the original lie at a larger scale.

How this fits the rest of TestChimp​

If you’ve been following along:

• Test Planning as Code puts stories and scenarios in Git so agents have a real task
• Scenario annotations declare which SmartTest claims which scenario
• Requirement traceability rolls execution results up the plan
• Verified Tests records that a human checked the claim—not just that CI was green

Planned reality → linked automation → inspected coverage → release confidence. The middle of that chain was the hole once agents started writing both sides.

Frequently asked questions​

What is a verified test?​

A verified test in TestChimp is a SmartTest a project member has inspected and confirmed actually covers a linked scenario. It is more than a green run or a scenario annotation: someone looked at the screen captures, steps, and code, and stamped the claim.

How is verified different from linking a test to a scenario?​

Linking (a Playwright annotation with type: 'scenario' and #TS-<n>) is a coverage claim. Verification is a human sanity check of that claim. You can have linked, green, and still unverified.

Where do I verify a SmartTest?​

Open a SmartTest execution (click a coverage square). Inspect the screen captures, steps, and test code, then use the check badge after the test name. Coverage-train badges are read-only hints.

What does the Verified Badge on a scenario mean?​

A filled badge means at least one covering SmartTest has been manually verified for that scenario. Hollow means tests exist but have not been sanity-checked yet. No linked tests → no badge.

Does renaming a scenario reset verification?​

No, as long as the scenario ordinal (#TS-n) stays the same. Linking a different scenario is a new claim and starts unverified.

Can I un-verify?​

Yes. Confirming un-verify sets the link to manually unverified (distinct from never verified). Who/when hover applies to the verified state.

What if the test has no linked scenarios?​

It cannot be marked verified. There is no scenario claim to inspect. Link the scenario first.

When agents run your SDLC, trust—but verify.

Docs: Verified Tests · Link tests to scenarios · Requirement traceability.

Agent coding is very good at getting the feature to work.

A user can check out. The API returns 200. The SmartTest is green. You merge.

Then ten concurrent checkouts queue behind one chatty query. Or the reports page, which was snappy with three invoices in the seed DB, falls over on a tenant that actually uses the product.

That gap is not a mystery. Agents optimize for the path in front of them—one user, empty-ish data, mocked collaborators that return in 0 ms. Concurrency and data volume are different questions. Functional tests do not answer them.

So we shipped performance testing as a first-class TestChimp surface: k6 in your repo, agentic workflows that author the right journeys (and keep composites honest), related runs after a PR or a release, and an Executions view that compares this run to a prior one.

Full product guide: Performance Testing.

The questions that actually matter​

Not “did k6 print a chart.” The questions product and eng already have:

We keep load and volume as separate axes. “Make it heavier” by turning both knobs hides which one broke.

Agents author journeys. You still own capacity.​

/testchimp create-perf-tests does not invent a k6 file from vibes.

It ranks real scenarios (priority, semantic coverage, get-requirement-coverage --include-perf). It uses redacted REAL E2E interaction shapes—method, path template, schema, status class, timing distribution—not cookies, tokens, or production bodies. When TrueCoverage is mature, relative demand helps order the queue and suggest composite weights.

What it will not do: copy a production RPS into load.js. TestChimp telemetry tells you what is worth testing. You (or run-perf-tests.policy.md) still set VUs, duration, and dataset size. Smoke is the default while authoring. Load/volume wait on an explicit capacity decision.

Composites are a weighted mix of journeys—“typical overall load,” not isolated degrade detection. Adding a journey to a composite is always a prompted approval. Silent membership is how you accidentally change the mix and then argue about the chart.

Outbound deps (payments, email, LLMs, partner APIs) get harness mocks with realistic latency. Hitting Stripe in a load test is expensive and flaky. Stubbing it at 0 ms is worse: you never see pool exhaustion. Either outcome is false confidence.

Folder layout, metadata, and wrappers: How performance tests are organized.

Related runs after a PR. Release range when you cut.​

Perf load/volume does not execute inside /testchimp test. Functional smoke should stay fast. k6 execution is /testchimp run-perf-tests or CI k6/scripts/run.sh. Run QA does write plans/smart-smoke/<branch>/related-perf-tests.json when k6/journeys exists, so the next CI --impacted run picks up the affected journeys.

On a PR, we select related journeys from the change set—scenarios, operations, path templates—then run them through k6/scripts/run.sh --impacted. Bare k6 run is not a TestChimp run: no ingest, no Executions charts.

On a release, the prompt from the release page is:

/testchimp run performance tests for release 1.2.0

The git range is prior SHA → cut SHA, not whatever happens to be checked out. Ingest stamps TESTCHIMP_RELEASE so the release panel and Executions list light up. If existing journeys do not cover that range, the agent asks whether to author (nested create-perf-tests on the same range) instead of silently inventing a suite or pretending coverage exists.

/testchimp upkeep-perf is the long loop: stale scenario links, drifted contracts, composite membership, thresholds that should not be quietly weakened to make compare green.

/testchimp init-perf scaffolds k6/ once. /testchimp import-perf-tests brings Locust/JMeter/Gatling/Artillery/k6-from-elsewhere into that tree. Source VU counts stay out of load profiles until you approve capacity.

Workflow walkthrough: Performance testing workflows.

Results you can overlay, not a one-off HTML report​

Wrapper runs ingest a summary and attach downsampled timeseries (p95, fail rate, VUs, and the rest of the k6 dump). Executions → Performance Tests lists them—optionally grouped by batch. Open a run for threshold, p95, fail rate, duration, metric-over-time, and VUs.

Then click Add Comparison Run. Recommended candidates share the comparison keys (environment, profile, dataset, LLM mode, mock/latency profile). The overlay re-bases both series on elapsed time so you are not comparing Tuesday 4pm wall clock to Wednesday 9am.

That is the degrade question: same kind of run, previous comparable result, is this worse? Green against a loose threshold can still be a regression versus last Tuesday. CLI compare-perf-to-baseline is the same contract for agents and CI—it exits nonzero when comparison.regressed is true. A mismatched baseline is incomparable, not a pass.

Viewing results.

How this fits the rest of TestChimp​

If you’ve been following along:

• SmartTests (now grouped as Functional Testing in the docs) prove the path works for a user
• Smart Smoke keeps that suite runnable in a CI budget
• TrueCoverage says which journeys real people actually take
• Performance testing asks whether those journeys still hold when many people take them—or when the tenant already has history

Planned reality → functionally tested reality → load/volume-tested reality → production reality. The middle was the hole for teams whose agents ship faster than anyone can stare at an EXPLAIN plan.

Frequently asked questions​

What is TestChimp performance testing?​

Grafana k6 scripts in your mapped tests folder (k6/journeys and k6/composites), authored and selected by agentic workflows, ingested into Executions so you can chart a run and overlay a comparable prior run.

Do I need this if my SmartTests are green?​

Yes if you care about concurrency or tenant data volume. Agents (and humans) routinely ship designs that are correct for one user on a small seed database and fail under the conditions production actually creates.

Does /testchimp test run k6?​

No. Keep functional PR smoke fast. Use /testchimp run-perf-tests (or a dedicated CI job that calls the same wrappers).

Will TestChimp pick my VU count from production traffic?​

No. TrueCoverage and interaction timings are relative signals—what to test, how to weight a mix. Absolute VUs, RPS, duration, and dataset size come from policy or an explicit approval.

How do I see if we got slower?​

Open the run → Add Comparison Run → pick a recommended candidate. Or compare-perf-to-baseline in CI. Compare only matching environment / profile / dataset / mock profile.

Docs: Intro · Organization · Workflows · Results.

Concept: What is load testing? · Volume testing · k6 vs JMeter vs Locust · Compare across releases.

Your agents have been writing tests for a few months, and now you have 500+ E2E tests taking an eternity to run. Sounds familiar?

Say hello to Smart Smoke.

With agents authoring tests en masse, the problem has changed. Coverage gaps close fast—but:

• Your CI bill keeps growing as you run the full suite on every PR
• Worse, you wait hours before knowing if anything broke

Today, smoke suites are often manually managed. Tag lists. Folder filters. A @smoke set someone curated last quarter. That leaves a lot of useful signals on the table—especially as test suites grow at an unprecedented pace.

What really interested me here—as a lover of algorithms—was that this is essentially a classic optimization problem:

Given N minutes, how do you maximize semantic coverage + PR impact?

First: find what the PR can break​

An agent identifies the tests relevant to the change—impacted scenarios from your plans, linked SmartTests via scenario annotations, plus anything newly authored on the branch. That related set lands in plans/smart-smoke/<branch>/related-tests.json.

That’s the seed. Then comes the fun part: covering as much ground as possible.

Paint the canvas within a fixed budget​

Imagine your tests laid out as nodes on a semantic canvas. The problem becomes:

Paint the canvas by selecting test nodes to maximize the covered spread—within a fixed time budget.

We don’t just grab the nearest cluster. We iteratively pick the next test that adds the most new ground—so selection spreads across the plane instead of camping in one corner.

Then weigh the terrain​

Pure geometry isn’t enough. We weigh the terrain using signals like scenario priority, recency, test stability, execution time, historical failures—so packing prefers tests that are both informative and practical.

Seeds always include related tests, tagged smoke (e.g. @smoke), and newly authored branch tests. Packing fills whatever budget remains—until the bin is full.

The result: the best subset of tests to run as smoke for a given PR—giving you the most confidence within a fixed time budget.

Less CI cost. Less waiting. More confidence.

Same Playwright command​

Smart Smoke isn’t a new runner. You keep your SmartTests and the same npx playwright test—opt in per run:

export TESTCHIMP_SMART_SMOKE_ENABLED=true
# optional: time budget, suite %, tags, related-tests-only
npx playwright test

Non-selected tests skip with reason smart-smoke (distinct from an explicit test.skip).

It plugs into /testchimp test as Phase 5, or standalone via /testchimp run smart smoke.

Full reference: Smart Smoke Runs · How it works · Configuration.

Plugin: @testchimp/playwright (≥ 0.2.20).

Smoke responsibly.

TL;DR: We’ve shipped API contract governance in TestChimp. Point us at your OpenAPI roots, opt in to API payload capture on SmartTest runs, and the APIs surface shows which operations—and which request/response fields and status codes—your automation actually exercised. Uncovered holes become Create issue, Create test, or Ignore gap—accountable work, not a spreadsheet of “we should cover that someday.”

The contract nobody audits​

Your OpenAPI (or Swagger) file is not documentation fluff. It is the business contract of the product: methods, paths, request shapes, response codes, and fields clients depend on.

UI E2E already touches that contract. Checkout hits POST /orders. Declined payment hits POST /payments. A rare enum or a 422 body? Often never.

Without a contract lens, teams hit the same failure modes:

API contract governance is not a separate API-testing product. It is the audit and triage layer that turns traffic your SmartTests already generate into accountable coverage of the contract.

Full product guide: API Contract Governance.

How it works​

Three inputs. One surface.

1. Discover the contract — Configure OpenAPI root path(s) from your connected repo (APIs → Configure). TestChimp parses them into services and operations.
2. Observe traffic — Run SmartTests with TESTCHIMP_ENABLE_API_CAPTURE=1 (and @testchimp/playwright ≥ 0.2.17). Matching HTTP interactions upload with the batch; coverage denorm runs after—not on the hot ingest path.
3. Govern gaps — List view for the portfolio; detail view for schema fields and response codes. From any hole: file an issue, copy an agent prompt to create/update tests, or ignore the gap when it is intentional.

Capture is off by default so ordinary runs stay light. Opt in when you want Operations coverage—same pattern as other optional runtime features. Setup detail: Set up API contract governance and the runtime plugin.

From endpoint scores to schema fields​

The APIs list is the portfolio view: service picker, coverage-band summary, method/path, spec-defined vs observed-only, coverage score, latency, covering-test chips.

Click a row and you get the interesting part—field-level truth:

• Request schema and query parameters with covering tests per field
• Response schemas nested by status code
• undocumented tags when traffic (or denorm) sees a field or code the OpenAPI root never declared
• Empty covering tests when the contract exists but automation never exercised it

That is the difference between “we hit /orders” and “we never sent delivery.type = same-day, and nobody owns that hole.”

List and detail walkthroughs: Operations list · Operation detail.

Governance that sticks: issue, test, or ignore​

Coverage without triage is another dashboard nobody opens.

Wherever coverage is missing—operation, field, or response code—TestChimp offers the same decision set:

Agents already understand operation-id / field / response-code scopes. The UI just stops the gap from living only in someone’s head.

How this fits the rest of TestChimp​

If you’ve followed us:

• Requirement quality governance hardens the plan contract before agents spend tokens
• Release governance hardens the ship decision for a version
• API contract governance hardens what your automation proved against the HTTP contract—the surface clients and integrations actually depend on

SmartTests remain the traffic source. Issues remain the backlog. Create-tests remains the authoring path. The APIs page is where contract debt becomes visible and governable—next to TrueCoverage for behaviour-aligned RUM, not instead of it.

Planned reality → tested reality → production reality. The API surface was the missing middle for teams whose “coverage” stopped at green UI journeys.

Frequently asked questions​

What is API contract governance in TestChimp?​

API contract governance maps OpenAPI operations to real and mocked SmartTest traffic so you can see endpoint and schema-field coverage, then triage gaps with create issue, create test, or ignore gap.

Do I need dedicated API tests?​

No. Coverage is inferred from HTTP traffic your SmartTests already generate when capture is enabled. You can still author dedicated API specs; the governance surface tracks both.

Why is my coverage empty after configuring OpenAPI?​

Parsing the spec discovers operations. Scores and covering tests need runs with TESTCHIMP_ENABLE_API_CAPTURE=1. Capture is off by default on purpose.

What does Ignore gap do?​

It marks an uncovered operation, field, or response code as intentionally ungoverned for now—without creating an issue or a test—so teams can acknowledge debt without false urgency.

Spec defined No vs undocumented—what’s the difference?​

Spec defined No on an operation means the whole method/path is missing from the OpenAPI root (observed in traffic only). An undocumented tag on a field or response code means that leaf was seen but not declared—classic contract drift.

Try it​

1. Connect git and open APIs → Configure — select your OpenAPI root file(s)
2. Wait for parse/sync; pick the service in the list
3. Run SmartTests with TESTCHIMP_ENABLE_API_CAPTURE=1
4. Refresh APIs — open a low-score or uncovered operation
5. Create issue, Create test, or Ignore gap on the holes that matter

Start here:

• API Contract Governance
• Setup
• List view
• Detail view
• Playwright runtime plugin
• TestChimp skill

Know the contract. Cover the edge cases—before unaudited branches ship.

Further reading​

TestChimp

• API Contract Governance
• Set up API contract governance
• SmartTests runtime plugin
• Create tests
• Issue management

Related posts

• Release Governance: Ship When the Evidence Says So
• Requirement quality governance (DeFOSPAM)
• TrueCoverage
• Boiling the QA lake

TL;DR: We’ve shipped Automations in TestChimp—event-driven runs of the same catalog workflows you already invoke with /testchimp. When an issue is created, a story moves to ready, a CI batch fails, or a release lifecycle status changes, TestChimp can hand the work to a cloud agent via OpenHands or a labelled GitHub Issue. Aggregation collapses noisy bursts; optional human gates keep high-blast-radius runs under review. Local agents stay for interactive work. Automations cover the complementary case: the platform watches, then starts the agent for you.

The prompt-pasting tax​

If you’ve been living in the agentic QA loop, you already know the playbooks:

/testchimp fix issue BUG-1042
/testchimp implement US-181
/testchimp fix test execution …
/testchimp run QA

That works. It also creates a new bottleneck: someone has to notice the event and paste the prompt.

Agents compressed execution. They did not invent routing. Automations are that routing layer—on the same workflows, policies, and workflow-execution-id traceability you already use.

Full product docs: Automations.

What is a TestChimp automation?​

An automation is three parts:

Trigger + Action + Config

Policies stay *.policy.md in Git. Automations optionally attach them—the same files local /testchimp runs use. Workflows still define how the agent works. Automations define when that work starts and how the cloud agent is reached.

Project event  →  matching automation  →  aggregation window
                                              ↓
                                    workflow execution queued
                                              ↓
                         (optional) human approve invoke / plan
                                              ↓
                    OpenHands conversation  or  labelled GitHub Issue
                                              ↓
                         /testchimp <workflow> … --workflow-execution-id …

Two ways to reach a cloud agent​

Configure once under Project Settings → Automations, then pick a strategy per automation.

Via GitHub Issue (great default when Git is connected)​

TestChimp opens an issue on your mapped repo. Body = the full /testchimp … prompt (plus the workflow execution id). Labels always include testchimp, plus any extras you configure.

Point Copilot coding agent, an OpenHands issue resolver, Codex, or your own runner at those labels. The agent checks out the repo, runs the prompt with the TestChimp skill + MCP, and reports back—so Executions → Workflow Executions stays current.

Via OpenHands​

TestChimp starts an OpenHands conversation with the rendered prompt (and branch when known). Same project OpenHands config you may already use for one-click bug fixes / OpenHands integration.

Deep dive: Cloud agents for automations.

Aggregation and human gates (because autonomy without brakes is chaos)​

Matching events should not always mean “spawn an agent immediately.”

Default rate limit: 5 cloud agent invocations per hour per project (shared). Over-cap runs stay Queued—not dropped.

Two independent human gates (off by default):

1. Approve before invoke — review the rendered task before the cloud agent starts
2. Approve before plan execute — agent plans first; you approve before execute

Pending work surfaces in Executions → Workflow Executions. You can also copy the prompt and run it locally—the workflow-execution-id keeps the timeline accurate either way.

Mechanics: How automations work.

Recipes teams actually want​

Task templates use {{issue.id}}, {{story.title}}, {{batch.failed_count}}, {{project.repo_url}}, and friends—with autocomplete in the wizard.

More starters: Typical automation setups.

Why this matters in the agentic era​

We already argued that skills are SaaS distribution—the playbook travels with the agent. We argued that requirements must be governed before agents spend tokens. We argued that release governance is the ship decision, not a spreadsheet.

Automations close another gap: intent without attendance.

Without them, QA-on-autopilot still needs a human dispatcher. With them, the same /testchimp catalog becomes a control plane: events in → policy-backed workflow executions out → auditable status in the product. Local Claude / Cursor for interactive loops. Cloud agents for the overnight and the “while you were in standup” cases.

That is how you boil the lake without standing next to the kettle (boiling the QA lake).

Frequently asked questions​

What are TestChimp Automations?​

Automations are event-driven rules that run TestChimp catalog workflows via a cloud agent when something changes in your project—issues, stories, scenarios, test executions, CI batches, or releases—optionally aggregated and gated by human approval.

How do Automations differ from Workflows?​

Workflows define how an agent should work (skill playbook + optional policy). Automations define when that work starts and which cloud invocation path to use. Both share the same execution timeline and workflow-execution-id reporting.

OpenHands vs GitHub Issue—which should I pick?​

Use GitHub Issue if you want any label-watching coding agent (Copilot, OpenHands resolver, Codex, custom bots) and already have the TestChimp GitHub App. Use OpenHands if TestChimp should start the conversation directly and you standardize on OpenHands cloud or self-hosted.

Will Automations spam my agents on every CI flake?​

Not if you configure them well. Prefer Buffered aggregation for batches, exclude ExploreChimp reporters on auto-fix rules, use conditions (status = BATCH_INVOCATION_FAILED, severity = HIGH), and keep the project hourly invocation cap. High-risk workflows should start with human gates on.

Can I still run workflows locally?​

Yes. Automations don’t replace /testchimp in your IDE. From a pending execution you can also copy the prompt—including the workflow execution id—and run it with a local agent so status still updates in TestChimp.

Where do I create and monitor Automations?​

Create under Workflows → Automations (or contextual Automations CTAs on Issues, Plans, batches, releases, and workflow pages). Configure strategies and rate limits in Project Settings → Automations. Monitor and approve under Executions → Workflow Executions.

Try it​

1. Connect GitHub (and optionally OpenHands) under Project Settings → Automations
2. Open Workflows → Automations → Create Automation
3. Pick a trigger (start from a popular condition template)
4. Choose a workflow, task template, policy, and invocation strategy
5. Leave Buffered aggregation on; add invoke or plan-execute approval for risky jobs
6. Trigger a matching event—then watch Workflow Executions

Start here:

• Automations overview
• Create an automation
• Cloud agents
• Typical setups
• Workflows
• TestChimp skill

When something changes, the agent should already be working—not waiting for you to paste a prompt.

Further reading​

TestChimp

• Automations
• How automations work
• Cloud agents
• OpenHands integration
• GitHub integration
• Fix issue
• Implement Requirement

Related posts

• From Bug Report to Pull Request (OpenHands)
• SKILLs as SaaS distribution
• Requirement quality governance
• Release governance
• Boiling the QA lake
• Test planning as code

Concepts

• OpenHands
• GitHub Copilot coding agent
• Agent Skills

TL;DR: We’ve shipped release governance in TestChimp—the workflow that answers one question before you deploy: has this version been tested enough, in the right places, with evidence we can audit? A release rolls up test runs, manual session captures, CI automation batches, release checks (UX + security), and release intelligence—so product, QA, and engineering share one readiness picture. CI and agents can gate on the same data via API.

The release confidence problem​

Shipping fast only works when you can trust what was validated.

Most teams already run plenty of QA. What they lack is a version-level contract:

Release governance is not bureaucracy. It is the shortest path to deploy confidence when stakeholders need a shared, auditable answer—not a status emoji in #releases.

What is release governance?​

In software delivery, release governance is how a team decides a version is ready to ship: what was in scope, what was tested, what risk remains, and who can prove it.

Industry practice usually spans test planning, requirement traceability, regression campaigns (test runs), security review, and a release gate—a pass/fail policy before production. Traditional tools often split that across a TMS, CI dashboards, a security scanner tab, and a spreadsheet.

Release governance in TestChimp keeps that decision on one surface: the release—a versioned milestone (for example v2.4.0) with git commit context, focus areas from your plan, test runs, checks, and analytics.

Full walkthrough: Release Management.

The TestChimp release model​

If you’ve followed us, the pieces already exist as Test Planning as Code, Test Runs, and requirement traceability. Release governance is the campaign layer that binds them to a ship candidate.

Open Releases in the sidebar → create a version → create test runs from the release viewer. Progress, checks, and analytics stay anchored to that label and commit range.

Evidence that rolls up: manual + automation​

Auditable manual testing​

Traditional TMS tools record an outcome. Release sign-off usually needs evidence.

With TestChimp, testers:

1. Capture a session in the Chrome extension (or add a record in the test run viewer)
2. Pick the active test run on the release
3. Get steps, screenshots, notes, bugs, and tester identity rolled into release overview and requirement coverage

“QA said pass” becomes a reproducible session tied to the scenario and version. Details: Manual sessions to test runs.

CI that counts toward the release​

Automation batches from the Playwright reporter show up under Executions. For release work, link them to a test run—especially from the Candidate Automation Execution Batches panel (batches between prior and current release commits).

Linking answers: Does this CI run count toward our sign-off campaign? Scenario status updates in the same progress view as manual work. Guide: Linking automation batches.

One release. One progress bar. Both execution types.

Release Checks: UX and security on the ship candidate​

Test runs answer scenario pass/fail. Release Checks answer the adjacent ship questions—from the same release page:

You queue a check from Run Release Check…, paste the /testchimp run… prompt into a TestChimp-upskilled agent, and triage Report / View Bugs beside test runs—not in a separate security silo.

Overview: Release Checks.

Release intelligence: beyond pass/fail counts​

Release intelligence connects what changed since the prior release to what was tested and what exploratory work found.

Use Overview in standups. Drill into analytics when you need requirement-level proof or scope risk (“scenarios added in delta, still not attempted”). Refresh after new commits, linked batches, or manual sessions.

Full detail: Release intelligence.

Programmatic release gating​

Governance that only lives in a UI is incomplete for agentic and CI-first teams.

get_release_details returns gate-oriented JSON for a version label: in-scope test aggregations per environment, open-issue stats, release-check summaries, and per-scenario detail. Your pipeline applies your policy—we don’t hard-code “block if any P0 failed.”

testchimp get-release-details --version '1.2.0'

Same data the release page shows—consumable by GitHub Actions, agents, or a custom quality gate. Docs: Programmatic release gating.

How this differs from a classic TMS release​

We are not asking you to maintain a parallel TestRail library forever—import scenarios if you need to migrate, then keep Git as source of truth. Honest comparison: TestChimp vs TestRail.

Why this matters in the agentic era​

Agents compress authoring and execution. They do not invent a shared definition of “ready to ship.”

Without release governance, you get faster green CI and louder Slack threads—still no auditable answer for this version. With it, the same agents that run /testchimp test and /testchimp run security scan feed a release surface humans and pipelines can gate on.

Requirements quality (DeFOSPAM governance) hardens the contract upstream. Release governance hardens the ship decision downstream. Together they close the loop we care about: planned reality → tested reality → production reality (boiling the QA lake).

Frequently asked questions​

What is release governance in TestChimp?​

Release governance is TestChimp’s workflow for validating a specific application version before deploy: create a release (version + git commit + optional plan scope), attach test runs, link manual sessions and CI batches, run release checks (UX and security), review release intelligence, and optionally gate CI with get_release_details.

What is the difference between a release and a test run?​

A release is the version milestone (v2.4.0) with metadata, focus areas, checks, and rolled-up intelligence. A test run is a scoped validation campaign inside that release—selected scenario folders, collaborators, due date, and linked executions. One release typically has multiple test runs.

What are release checks?​

Release Checks are QA activities launched from the release detail page: ExploreChimp UX Checks plus DAST (ZAP), SAST (Semgrep), secrets (Gitleaks), and dependency scans (Trivy). Reports and bugs stay scoped to the ship candidate next to test runs.

How do I know when a release is ready to ship?​

Review overview metrics for the target environment, requirement coverage for focus areas, delta analytics for untested plan changes, release-check reports and bugs, and ExploreChimp / TrueCoverage warnings. When in-scope scenarios meet your bar with evidence attached, complete the test runs and deploy—or enforce the same bar in CI via the gating API.

Does TestChimp enforce a fixed release gate policy?​

No. The UI and get_release_details expose the data. Your team (or pipeline) decides thresholds—for example zero failed P0 scenarios, no open critical security issues, or required completed scans.

Do manual results need to be linked to a test run?​

For results to count in release overview and requirement coverage, link manual sessions to an active test run on that release—easiest at capture time in the Chrome extension. Unlinked sessions stay in execution history but do not roll up to release progress.

Try it​

1. Open Releases → New Release (set version, commit SHA, prior release, focus areas)
2. Create New Test Run from the release viewer
3. Capture a manual session or link a Playwright batch
4. Queue a Release Check and run it with your agent
5. Open View release analytics—then gate CI with get-release-details when you’re ready

Start here:

• Release Management
• Release Checks
• Release intelligence
• Programmatic release gating
• TestChimp skill

Ship when the evidence says so—not when the spreadsheet says “LGTM.”

Further reading​

TestChimp

• Release Management
• Release Checks
• Release intelligence
• Manual sessions to test runs
• Linking automation batches
• Test Runs
• Release Management API Guide

Related posts

• Test Runs: Turn Testing Into Release Confidence
• Requirement quality governance (DeFOSPAM)
• Requirement traceability
• Boiling the QA lake
• Test planning as code

Concepts & standards

• ISTQB: test run
• Requirements traceability
• ISO/IEC/IEEE 29148 — requirements engineering
• OWASP ZAP · Semgrep · Gitleaks · Trivy

We’ve spent a lot of energy making agents better at writing code and tests.

We’ve spent far less making the inputs to those agents less terrible.

Most modern teams don’t have an “AI coding” problem. They have a requirements problem—and agents make it louder. Give Claude a vague story with “fast,” “easy,” and a missing error path, and it will happily invent behaviour. Give Playwright-authoring agents an untestable scenario, and they will automate the guess. Garbage in doesn’t just produce garbage out anymore. It produces confident garbage—at merge velocity.

That isn’t a new insight in software engineering. It’s just newly expensive.

We’ve known what “good” looks like for decades​

Quality requirements aren’t a 2026 invention.

Agile teams have INVEST (Independent, Negotiable, Valuable, Estimable, Small, Testable)—Bill Wake’s checklist for backlog items that don’t sabotage the sprint. The “T” is the one that keeps biting agentic workflows: if you can’t write a test for it in principle, you don’t have a requirement—you have a vibe.

Formal requirements engineering went further. ISO/IEC/IEEE 29148 spells out characteristics like unambiguous, complete, singular, and verifiable—measurable properties of a requirement, not a gut feel in a grooming meeting. Industry write-ups of the standard make the same point in plain language: “user-friendly” isn’t a requirement; a verifiable threshold is (overview of ISO 29148 quality criteria).

And for scenario thinking, Specification by Example and Given-When-Then have been the antidote to “as a user I want stuff so that value” for years.

The gap was never knowing. The gap was doing it continuously, on every story, without a two-day workshop and a whiteboard full of sticky notes.

Enter DeFOSPAM​

DeFOSPAM is a seven-lens mnemonic from Paul Gerrard’s Business Story Method (with Jonathon Wright / OpenTest.AI)—popularized recently by OpenRequirements.AI as an agentic requirements-validation approach. The goal is blunt:

A perfect requirement lets the reader predict the behaviour of every feature in all circumstances.

Where that prediction fails, DeFOSPAM tells you why—systematically:

Paul has talked about this as structured appraisal of requirements for years—including how AI can help walk the checklist without replacing human judgment (Analyzing and improving requirements — Richard Seidl podcast). OpenRequirements frames the same idea as specialist analyst agents over living documentation (OpenRequirements.AI; methodology notes on GitHub).

We didn’t invent DeFOSPAM. We operationalized it where our plans already live.

Requirement quality governance in TestChimp​

If you’ve been following us, you know the thesis: Test Planning as Code—stories and scenarios as markdown in Git, workflows layered on for humans, context for agents. Traceability without the spreadsheet circus (requirement traceability).

The next piece is governance: not just having plans, but knowing whether those plans are good enough for agents to build and test against.

In TestChimp you can run agentic DeFOSPAM-style checks on a story, a scenario, or a plans folder. What you get back isn’t a vague “needs more detail” comment:

1. Scores across clarity, completeness, testability, consistency, ambiguity risk, and scenario coverage—plus an overall score
2. Findings with severity (critical / major / minor)
3. Suggested fixes you can apply or ignore
4. Tracked state in Plans → Insights → Requirement quality, so quality is a backlog you can govern—not a chat transcript that evaporates

Applied and ignored findings stay out of the penalty box on re-score. The board shows remaining work.

Full walkthrough: Requirement Quality Governance. Agent playbook: /testchimp analyze requirement quality.

Cloud vs local agent: pick for context, not vibes​

There are two ways to run the checks.

TestChimp Cloud — one click from the story/scenario editor. Fast. No IDE setup. Costs an AI credit. Great when you want a quality pass without leaving the platform.

Local coding agent (recommended when you can) — install the TestChimp skill and run:

/testchimp analyze requirement quality of US-181
/testchimp analyze requirement quality of plans/stories/billing

Why we recommend local when it fits: the agent can ground findings in your code—fixtures, seed routes, existing scenarios, how “subscription” is actually modelled. Cloud is the simpler path; local is the richer one. Same governance surface either way—results land in Insights.

Authoring fits the same loop. Upskill the agent, then:

/testchimp author story for <objective>

Playbook: Author Plans. Docs: Authoring test scenarios.

Why this matters now​

In a human-only world, ambiguous requirements wasted meetings and produced “works as designed” arguments.

In an agentic world, they waste tokens, CI minutes, and PR cycles—and they poison the feedback loop we care about: planned reality → tested reality → production reality (boiling the QA lake).

Agents are extraordinary executors. They are mediocre mind-readers.

If you want high-quality dev output and high-quality test output, start earlier than /testchimp test. Harden the story. Score it. Fix the critical findings. Then let agents implement and automate.

Requirements stop being static tickets. They become governed assets—the same way we already treat code.

Wrapping up​

We believe the next leverage in agentic QA isn’t another “generate tests” button. It’s making the contract agents work from clear, detailed, and unambiguous—and keeping that quality visible over time.

DeFOSPAM gives the lenses. INVEST and ISO 29148 gave the vocabulary. TestChimp puts governance on the plans you already sync to Git.

If that resonates, start here:

• Requirement Quality Governance
• Run Requirement Quality Checks
• Test Planning as Code
• TestChimp skill

References and further reading​

• OpenRequirements.AI — DeFOSPAM / living requirements
• OpenRequirementsAI on GitHub — DeFOSPAM methodology notes
• Paul Gerrard / Richard Seidl — Analyzing and improving requirements (DeFOSPAM)
• Gerrard Consulting
• Agile Alliance — INVEST
• Bill Wake — INVEST in Good Stories, and SMART Tasks (XP123)
• ISO/IEC/IEEE 29148 — Systems and software requirements engineering
• ISO 29148 quality characteristics (overview)
• Martin Fowler — GivenWhenThen
• Gojko Adzic — Specification by Example
• ISTQB — Test planning

TL;DR: When coding agents can write dozens of Playwright tests in a single session, the bottleneck shifts from authoring to governance: are the new tests distinct and useful, or just near-duplicates of what you already have? Semantic Graph is a free, open-source CLI that scans your suite, embeds each test semantically, clusters related tests, and renders an interactive graph so you—and your agent—can spot redundancy before it compounds.

The new problem: agents author tests en masse​

For most of the last decade, the hard part of E2E testing was throughput: humans could not write and maintain enough tests to keep up with product velocity.

That constraint is collapsing. With Claude Code, Cursor, and agent skills like the TestChimp skill, a single prompt can produce a folder of well-formed Playwright specs in minutes. Coverage gaps that used to take a sprint to close can shrink to an afternoon.

The bottleneck has moved.

When an agent is rewarded for adding tests—closing coverage gaps, responding to PR feedback, or filling in scenarios from a test plan—it has no innate sense of "this already exists, slightly reworded." Left unchecked, suites balloon with:

• Duplicate tests that assert the same behaviour under different titles
• Near-duplicates that differ only in fixture data or selector phrasing
• Clustered redundancy where five tests all exercise the same checkout edge case
• Invisible overlap across folders, because no human (and no agent) holds the entire suite in working memory

This is the QA equivalent of boiling the lake in the wrong direction: lots of heat, little new coverage. Worse, duplicate tests inflate CI time, confuse failure triage, and give a false sense of depth—your line count grows while your behavioural breadth stalls.

The question is no longer "Can we write more tests?" It is:

"Are we writing useful, distinct tests—or just duplicative ones?"

That question needs a semantic answer, not a filename diff.

What is Semantic Graph?​

Semantic Graph is an open-source tool from TestChimp that maps your Playwright test suite by meaning, not syntax.

It is published as @testchimp/semantic-graph on npm and lives in the TestChimp/semantic-graph repository. Run one command against your tests directory; the CLI:

1. Scans *.spec.ts, *.test.ts, and related Playwright files
2. Parses each test's suite path, title, intent comments, scenario annotations, and body
3. Embeds the canonical test text with an embedding model (OpenAI or Voyage AI)
4. Clusters tests by semantic similarity using DBSCAN
5. Lays out a 2D graph with UMAP so similar tests appear close together
6. Names clusters with a lightweight LLM pass (e.g. "auth", "checkout", "api-contracts")
7. Serves a local interactive UI at http://localhost:3859

No database. No TestChimp account required. Embeddings are computed in memory each run—ideal for local audits, pre-merge reviews, or giving an agent a structural view of the suite before it authors more tests.

How it works (the pipeline)​

Understanding the pipeline helps you interpret the graph—and tune how agents use it.

1. Parse tests into embedding-ready text​

The core library (@testchimp/semantic-graph-core) includes a vendored Playwright-aware parser. For each test it builds canonical text:

Suite: checkout > guest flow
Test: rejects expired coupon at payment step
Body:
Scenario: Guest checkout with invalid coupon
// intent: verify error copy and no charge created
await page.goto('/checkout');
...

Parsing captures intent comments and scenario annotations—the same metadata agents should be authoring anyway when following requirement traceability conventions. Two tests with different selectors but the same intent will land close together in embedding space.

2. Embed with cosine similarity​

Each test's text is sent to an embedding API in batches (default model: text-embedding-3-small for OpenAI, voyage-4 for Voyage). The tool computes cosine similarity between vectors and applies configurable thresholds:

These thresholds mirror how humans judge redundancy: not byte-identical, but "would a failure in one make the other pointless?"

3. Cluster with DBSCAN​

Similar embeddings are grouped with DBSCAN density clustering—no need to pick k clusters upfront. Each cluster gets an LLM-generated label (e.g. "settings-page", "admin-tasks") so the legend is readable at a glance.

4. Visualize with UMAP + D3​

A seeded UMAP projection maps high-dimensional embeddings to 2D coordinates. The bundled UI (built with D3.js) renders:

• Graph view — nodes as tests, edges as similarity links; click a node to see nearest neighbours and duplicate flags
• Clusters view — grouped list with colour-coded legend
• Folder tree — scope the graph to a directory or single file

Zoom into tests/checkout/ before a refactor. Scan the whole suite before a release. Hand the URL to an agent and ask it to propose merges.

Why this matters for agentic QA workflows​

Semantic Graph is not a replacement for TrueCoverage—production-informed prioritization—or requirement traceability. It solves a orthogonal problem: intra-suite redundancy.

Here is where it fits in a modern agent loop:

Before the agent writes​

Run Semantic Graph and attach the cluster summary to the agent's context. Instructions become concrete:

"We already have four tests in the checkout cluster covering coupon validation. Do not add another unless you are testing a different failure mode."

This is cheaper and more reliable than asking the agent to grep test titles.

After the agent writes​

Re-run the graph on the PR branch. New nodes that snap onto existing clusters—or spike duplicate scores above 0.92—are review flags. Pair with CI the same way you gate on lint or coverage deltas.

During suite health reviews​

Quarterly "suite diet" sessions used to mean spreadsheets and gut feel. Now: filter to clusters with high internal similarity, merge or delete, and measure CI time recovered.

Complement to production signals​

TrueCoverage tells you what behaviours users need tested. Semantic Graph tells you whether your existing tests are saying the same thing twice. Both are necessary for a suite that is broad and lean.

What you see in the UI​

The demo above shows the full workflow:

1. Left panel — folder tree mirroring your repo layout; click a folder or file to scope the view
2. Graph mode — force-directed layout; proximate nodes are semantically alike
3. Clusters mode — tests bucketed with named themes
4. Popover — click any test to see top similar neighbours, similarity scores, and potential duplicate badges

The UI ships inside the npm package—no separate install. It is the same "freebie" static app published as @testchimp/semantic-graph-viz in the monorepo for anyone who wants to embed or fork it.

Try it yourself​

Prerequisites​

• Node.js 18+
• An API key for embeddings (and cluster naming):
  • OpenAI — one key covers embeddings + LLM, or
  • Anthropic + Voyage — Claude for cluster labels, Voyage for embeddings (Anthropic does not ship an embedding API)

Quick start (OpenAI)​

export PROVIDER=openai
export API_KEY=sk-...

npx @testchimp/semantic-graph visualize --tests-dir ./tests

Open the printed URL (default port 3859). Add --verbose for embedding progress and diagnostics.

Claude + Voyage​

export PROVIDER=anthropic
export API_KEY=sk-ant-...
export VOYAGE_API_KEY=pa-...

npx @testchimp/semantic-graph visualize --tests-dir ./tests

All options​

See the README for environment variables, monorepo build instructions, and npm publish details.

Continuous governance with TestChimp​

Semantic Graph is deliberately local and standalone—a flashlight you can shine on any Playwright repo, TestChimp customer or not.

For continuous duplicate detection, requirement traceability, release confidence, and keeping suites healthy as agents keep authoring, see TestChimp—the git-native QA governance platform built for agentic teams. Install the TestChimp Agent Skill and run /testchimp test after each PR to orchestrate coverage, exploration, and plan alignment in one loop.

FAQ​

What test file types are supported?​

The scanner picks up *.spec.ts, *.spec.js, *.test.ts, *.test.js, and .mjs / .cjs variants under your chosen root—standard Playwright test layouts.

Does it require a TestChimp account?​

No. Semantic Graph runs entirely locally. You only need embedding (and optionally LLM) API keys.

How is this different from code coverage?​

Code coverage measures which lines executed. Semantic Graph measures whether test intentions overlap. A suite can have high line coverage and still be full of redundant scenarios.

How is this different from duplicate detection by test name?​

Titles lie. Agents especially love paraphrasing: "should reject invalid coupon" vs "guest user sees error for expired promo code." Embeddings capture the full body and intent, not the string on line one.

Can I use it in CI?​

Today the primary interface is the local visualize command and JSON APIs (/api/graph, /api/similar). For CI gates, parse the API responses or run before review and archive the graph output. Continuous server-side governance is on the TestChimp platform roadmap.

What embedding models are supported?​

Defaults: text-embedding-3-small (OpenAI) and voyage-4 (Voyage). Override with EMBEDDING_MODEL. LLM cluster naming defaults to gpt-5-nano or claude-3-5-haiku-latest.

Is the source code open?​

Yes. MIT-licensed monorepo: github.com/TestChimp/semantic-graph. Packages: @testchimp/semantic-graph-core, @testchimp/semantic-graph, @testchimp/semantic-graph-viz.

Summary​

Agentic QA solved test authoring at scale. The next discipline is test distinctness at scale—ensuring every new spec adds behavioural breadth, not noise.

Semantic Graph gives you a semantic map of your Playwright suite: embeddings for meaning, DBSCAN for clusters, UMAP for intuition, and a local UI for humans and agents alike. Run it before you merge agent-authored tests. Run it when CI gets slow. Run it when you suspect the lake is boiling but not reducing risk.

Get started: github.com/TestChimp/semantic-graph · npx @testchimp/semantic-graph visualize

References and further reading​

• TestChimp Semantic Graph repository — source, README, and issue tracker
• @testchimp/semantic-graph on npm — CLI package
• Playwright Test documentation — supported project layouts
• OpenAI Embeddings guide — text-embedding-3-small and related models
• Voyage AI documentation — embeddings when using Claude as the LLM provider
• UMAP: Uniform Manifold Approximation and Projection — dimensionality reduction for the 2D layout
• DBSCAN clustering — density-based cluster assignment
• Fixtures in agentic test automation — complementary TestChimp blog on Arrange-layer quality
• TrueCoverage for agentic QA — production-informed test prioritization
• TestChimp Agent Skills — orchestrate QA workflows in Claude and Cursor

TL;DR: TestChimp now has Test Runs—named validation campaigns that roll up scenario progress across manual sessions and automation batches. If you have used test runs in TestRail, Qase, or PractiTest, the concept will feel familiar. What is different is that scope, progress, and drill-down inherit the folder structure of your test plan—not a flat, manually curated case list copied into yet another container.

What is a test run?​

In software testing, a test run is the execution of a defined set of tests against a specific version or build of the system under test. The ISTQB glossary defines it as “the execution of a test suite on a specific version of the test object.” Test execution—the process of running those tests and recording outcomes—is a core part of the fundamental test process described in ISO/IEC/IEEE 29119.

In practice, teams use test runs to answer a release question: Given the scenarios we committed to validate for this sprint or version, how far along are we—and what is still failing?

Traditional test management systems such as TestRail and Qase model a run as a container: selected test cases, assignees, pass/fail/blocked status, and often milestone or environment context. TestRail’s guidance notes that runs are typically created per sprint or release so managers can track progress in real time.

Test Runs in TestChimp preserve that coordination purpose while changing what sits underneath—the plan, the executions, and how progress rolls up.

The gap test runs are meant to close​

Most teams already know the shape of a release cycle:

• a defined set of scenarios to validate
• manual testers working through critical paths
• automation running in CI on every build
• a lead asking, “Are we done yet?”

Traditional tools answer the last question with a run—but the artifacts rarely stay connected.

User stories often live in Jira or similar issue trackers. Scenarios live in a TMS. Manual evidence sits in screenshots and Slack. Automation results sit in GitHub Actions, Jenkins, or a Playwright CI report. Requirement traceability—linking requirements to verifying tests, as described in ISO/IEC/IEEE 29148—is often maintained in spreadsheets or a test traceability matrix that goes stale.

The run becomes another manually curated list, disconnected from how the product is organized and how work actually happens.

We built Test Runs in TestChimp to close that loop without duplicating your plan in a flat case catalog.

Same concept, different foundation​

A Test Run in TestChimp is still a time-bound validation campaign: a title, optional environment and release context, collaborators, a due date, and a scope of scenarios to validate.

What changes is everything underneath.

You are not maintaining a parallel catalog. You are pointing a run at the test plan you already have.

One run, both execution types​

The most common fracture in enterprise QA is two parallel tracks:

• manual validation tracked in a test management tool
• automated validation tracked in CI or a vendor dashboard

Qase’s own documentation describes the tension: auto-generated CI run names pile up quickly, and teams need runs that “tell a story at a glance” when reviewing overnight failures before a release.

A Test Run in TestChimp is deliberately execution-type agnostic. Link a manual session from exploratory regression. Link tonight’s Playwright batch. Link both to the same run. Scenario status reflects the latest relevant execution—whether a human marked a session passed or CI reported a SmartTest failure.

That is the same unified coverage story we told with manual testing and traceability—now packaged for release-scale questions instead of only folder-level requirement traceability insights.

Folder-based progress, not flat lists​

Because TestChimp organizes stories and scenarios as markdown files in folders (Test Planning as Code), a test run inherits something traditional tools struggle to offer: scoped views at any granularity.

Select the root of the run and see overall progress for the whole release. Select checkout/ and see only checkout scenarios. Select a single story file and see exactly what is left on that requirement.

No re-tagging. No re-grouping cases into ad hoc suites every sprint. The folder structure you already use for planning becomes the structure you use for reporting—the same principle as coverage at any folder level in Test Planning.

That matters when:

• feature teams own folders, not individual case IDs
• a release spans several modules but not the entire backlog
• you need a standup answer for one area without re-filtering a 2,000-row grid

Trend charts in the run viewer show how passing, failing, and not-attempted counts move over time—useful for daily readouts without exporting to a spreadsheet.

Why this fits the agentic era​

Test runs are not a throwback to heavyweight process. They are a lightweight coordination layer on top of artifacts agents can already read.

Your scenarios are files. Your tests link with scenario annotations (requirement traceability in code). Executions feed the same traceability graph whether they are manual or automated. A test run simply names the campaign—“Sprint 42 regression”, “v2.1 sign-off”—and gives humans a place to see progress while agents keep authoring against the same plan.

We are not replacing CI dashboards or extension manual capture. We are giving product and QA leads a single pane for this validation cycle, grounded in requirements rather than orphaned case records.

See it in action​

For step-by-step setup—creating a run, defining scope, linking batches and sessions, reading the viewer—see Test Runs in the docs.

Frequently asked questions​

What is a test run in software testing?​

A test run is a structured execution of a selected set of tests against a specific build, release, or milestone. The ISTQB glossary defines it as running a test suite on a particular version of the system under test. Teams use runs to track who tested what, record pass/fail outcomes, and report release readiness.

How is a TestChimp Test Run different from a TestRail or Qase test run?​

The coordination goal is the same: scope a set of tests, track progress, report status. The foundation is different. Traditional tools copy flat test cases into a run container (TestRail runs, Qase test runs). TestChimp scopes runs from your existing plans folder tree and aggregates results from linked manual sessions and automation batches—without maintaining a duplicate case list.

Can one test run include both manual testing and test automation?​

Yes. TestChimp Test Runs are execution-type agnostic. Link manual sessions captured via the Chrome extension and automation batches from Playwright CI to the same run. Each scenario’s status reflects the latest linked execution, whether the outcome came from a human or from CI.

Do I need to duplicate test cases to create a test run?​

No. You select scope from folders and files already in Test Planning. Scenarios remain the same markdown artifacts your team authors and version-controls; the run is a pointer and progress lens, not a second catalog.

What is folder-based test run progress?​

Because stories and scenarios live in a nested folder structure (Test Planning as Code), the test run viewer lets you drill into any folder or file and see passing, failing, and not-attempted counts for just that subtree. Root shows the full run; authentication/ shows auth only—without re-tagging cases or rebuilding suites each sprint.

How do Test Runs relate to requirement traceability?​

Requirement traceability links requirements and scenarios to executions over time—supporting the verification relationships described in standards such as ISO/IEC/IEEE 29148. Test Runs add a named campaign layer: a due date, collaborators, explicit scope, and release-oriented progress for one validation cycle. Traceability is ongoing product health; test runs are this regression or this release sign-off.

When should we use test runs vs Test Planning insights alone?​

Use Test Planning insights when you want continuous coverage visibility for a folder, environment, and time range. Use Test Runs when you need a time-bound campaign with assigned collaborators, a due date, and a dedicated dashboard fed by executions you link during that cycle—similar to how teams use TestRail test runs per sprint, but unified across manual and automated work.

Can we link executions to a test run after they complete?​

Yes. Automation batches can be linked from Executions → Automation Batches (list or batch viewer). Manual sessions can be linked at capture time in the extension or afterward from the manual session viewer. Many-to-many linking is supported—a batch or session can belong to multiple active runs.

Try it​

Open Test Runs from the TestChimp sidebar, create a run scoped to the folder your team owns, and link the next manual session or automation batch you execute.

If you are comparing approaches, our requirement traceability post explains the foundation; this feature adds the campaign layer on top when you need to track a specific release or regression cycle end to end.

We are iterating on collaborator workflows, PDF reporting, and deeper agent integration. Feedback welcome—especially from teams migrating off TestRail-style run models.

Further reading​

TestChimp

• Test Runs (documentation)
• Test Planning as Code
• Requirement traceability
• Manual test session capture
• Run SmartTests in CI (Playwright)

Test management & QA concepts

• ISTQB: test run
• ISTQB: test execution
• TestRail: test runs and results (best practices)
• TestRail: importing automated test results
• Qase: test runs concepts
• Requirements traceability (overview)
• ISO/IEC/IEEE 29148 — requirements engineering
• ISO/IEC/IEEE 29119 — software testing standard

Automation & CI

• Playwright: continuous integration
• GitHub Actions documentation

Related posts

• Manual testing with traceability
• Test planning as code
• Requirement traceability, without the spreadsheet circus

TL;DR: If your product ships both a web app and native mobile apps, you are probably maintaining two automation codebases that repeat the same Arrange logic—users, listings, payments, feature flags—before any UI step runs. TestChimp Multi-Platform Projects put Playwright (web), Mobilewright (iOS/Android), and API tests in one Git-connected scaffold, with shared business logic for world-state setup and platform-specific UI tests, coverage, and UX analytics. UI interactions stay platform-specific; test infrastructure does not have to—and neither does your requirements, TrueCoverage, or Atlas view of quality.

The hidden cost of “Appium for mobile, Playwright for web”​

Cross-platform products rarely differ at the data layer. A booking marketplace needs the same primitives whether the customer taps Book in Safari or in your iOS app:

• A test user with a known identity
• Inventory (for example, a few property listings)
• A valid payment method linked to that user
• Whatever else your domain requires before the flow under test is meaningful

None of that is inherently web or mobile. It is application state—the Arrange phase in the classic Arrange → Act → Assert model (Martin Fowler on Given-When-Then).

Yet the dominant split for years has been:

Teams end up with parallel helper libraries, duplicate seed scripts, and drift—web tests create users one way, mobile tests another, and failures become “which stack is wrong?” instead of “did we break the product?”

The Act and Assert steps should differ by surface: selectors, gestures, and viewport behaviour are platform-specific. The Arrange layer often should not.

Why Mobilewright changes the consolidation story​

Mobilewright brings native iOS and Android automation closer to the Playwright mental model: async tests, auto-waiting, project matrices in config, and fixtures that feel familiar if you already run npx playwright test.

That alignment matters for multi-platform engineering, not only for “mobile testing” as an isolated workstream:

• Same language and patterns (commonly TypeScript/JavaScript in one repo)
• Same CI habits (config projects, parallel workers, artifact uploads)
• Same opportunity to share code for factories, API clients, and database seeding

TestChimp already extended the plan → repo → agent → CI loop to native mobile (native mobile testing announcement). Multi-Platform Projects are the next step: one TestChimp project type and one tests tree for teams that ship web and mobile together.

What TestChimp Multi-Platform Projects provide​

When you create a TestChimp project with type Multi-Platform, the platform scaffolds a single tests/ directory that includes:

• web/ — browser SmartTests via Playwright (playwright.config.js, web/e2e/, web/pages/, web/fixtures/)
• mobile/ — native UI tests via Mobilewright (mobilewright.config.ts, mobile/e2e/common|ios|android/, mobile/pages/, mobile/fixtures/)
• api/ — platform-agnostic HTTP specs (often the fastest way to Arrange and to assert backend state)
• shared/ — cross-suite helpers and fixture factories (seed users, auth builders)—excluded from test discovery, intended for reuse
• setup/ — global setup run once before suites in both configs

Platform-specific UI code lives in platform-specific folders. Business logic that creates entities and prepares situations can live in shared/, api/fixtures/, or factories imported by both web and mobile specs.

tests/
  setup/
  shared/              ← shared Arrange logic (users, listings, payments, flags)
  api/
    fixtures/
  mobile/
    fixtures/
    pages/
    e2e/
      common/
      ios/
      android/
  web/
    fixtures/
    pages/
    e2e/
  playwright.config.js
  mobilewright.config.ts

Result for QA and platform teams:

• Less duplicated infrastructure — one place to update “premium user with saved card”
• Less maintenance — fix seeding once; web and mobile suites consume the same factories
• More consistency — the same world-state definitions drive cross-platform regression

Smart Steps (ai.act, ai.verify) remain web-only today; native mobile continues to use standard Mobilewright APIs for UI Act steps. For platform capabilities and CI notes, see Mobile testing.

One project, platform-specific coverage and UX intelligence​

Consolidating tests in one repo does not mean blending web and mobile into one misleading coverage number. Multi-Platform Projects keep one TestChimp project and one plans/tests Git mapping, while treating Web, iOS, and Android as first-class execution platforms everywhere insights matter.

Think of it as: shared requirements and shared Arrange code, sliced execution and analytics per surface.

Requirement traceability (Test Planning)​

Requirement traceability links scenarios in Git to SmartTest runs. On a Multi-Platform project, the Insights tab and scenario execution history respect an execution scope that includes platform alongside environment, release, branch, and time range.

• Choose Web, iOS, or Android to see which scenarios passed or failed on that surface.
• Drill into a user story to view execution history filtered to the platform you care about—useful when mobile lags web or when a shared scenario is covered by both web/e2e/ and mobile/e2e/ specs.
• Folder roll-ups in Test Planning still work; the platform dimension answers questions like “Is checkout covered on iOS in QA this week?” without spinning up a second project.

Agents and CI should report runs with the correct platform identity (via @testchimp/playwright / Mobilewright reporter wiring) so linked scenario-annotated tests attribute to the right slice. Your plans can describe behaviour once; coverage status reflects where that behaviour is actually exercised.

TrueCoverage (production-informed gaps)​

TrueCoverage compares real user journeys (RUM) with automation coverage (test-tagged events). Each surface has its own instrumentation path—@testchimp/rum-js on web, testchimp-rum-ios and testchimp-rum-android on native—with TESTCHIMP_PROJECT_TYPE set to web, ios, or android as described in Instrumenting your app.

On Multi-Platform projects, the TrueCoverage execution scope offers the same Web / iOS / Android selector. That keeps comparisons honest:

• Production events from the iOS app are not mixed with web test runs when you evaluate gaps.
• Agents prioritizing fixtures and tests can target the platform where users actually hit the gap—for example high drop-off on Android checkout vs healthy web funnel.

Instrument every surface you ship; scope analytics one platform at a time when deciding what to automate next.

Atlas (UX bugs on the right surface)​

Atlas is TestChimp’s app-structure map: screens and states, with UX and non-functional bugs tagged where ExploreChimp or SmartTests observed them. For multi-platform products, the SiteMap is not a single blurred tree—you browse and triage per platform.

• A platform selector (Web, iOS, Android) loads the screen-state tree for that execution platform.
• Bugs discovered during exploration or annotated runs are associated with screen-state context on that platform, so a layout regression on mobile does not drown in unrelated web noise.
• markScreenState checkpoints in web Playwright tests and mobile Mobilewright tests feed the vocabulary ExploreChimp and Atlas use; platform-specific folders keep Act steps separate while structure stays comparable across surfaces.

That matters for engineering leads reviewing quality: you open Atlas, pick iOS, and see UX issues on the iOS SiteMap—assign owners per screen, run targeted ExploreChimp from a node, and track fix status without conflating desktop-only flows.

Arrange vs Act: what to share (and what not to)​

This is the same insight as fixtures and Object Mother patterns in xUnit-style testing (xUnit Test Patterns — test fixture, Object Mother): push incidental complexity of setup out of the test body and into reusable, composable building blocks. Agents authoring tests benefit even more when Arrange is API-backed rather than repeated through slow UI clicks (fixtures in agentic automation).

How to get started​

1. Sign in to TestChimp and open Add project.
2. Choose project type Multi-Platform (web + native mobile in one codebase).
3. Connect Git and map your plans/ and tests/ folders (same workflow as web-only projects).
4. Run your usual agent workflow—for example /testchimp test after a PR—using the TestChimp skill on Claude or Cursor.

Docs to read next:

• Mobile testing (iOS and Android) — configs, CI, and web vs mobile capability table
• Creating SmartTests — Playwright additions on web; Mobilewright on native
• Instrumenting your app for TrueCoverage — RUM on web and native when you want production-informed coverage

If your team already runs separate web and mobile automation repos, migrating Arrange into shared/ and api/ first—before moving UI specs—is usually the lowest-risk path. You keep platform runners; you stop duplicating the world behind them.

Frequently asked questions​

Is Multi-Platform the same as creating separate web and mobile TestChimp projects?​

No. Multi-Platform is one project and one scaffold where both Playwright and Mobilewright configs and folder layouts coexist. Separate Web and Mobile project types still exist when you only need one surface.

Do I have to abandon Appium to use this?​

TestChimp’s native path is Mobilewright, not Appium. Teams often adopt it when they want Playwright-like authoring and shared TypeScript with web suites. If you are standardized on Appium, compare effort to maintain duplicate Arrange code versus migrating Act layers over time while centralizing setup in API tests first.

Can API tests really replace UI for Arrange?​

For many domains, yes—and Playwright’s request context (and direct HTTP clients in api/*.spec.js) are the fastest, least flaky way to reach a given situation. UI Act remains necessary to validate what users see and tap; UI Arrange is usually optional once APIs or admin seeds exist (QA in production).

What’s the biggest win if we already have Playwright on web?​

The win is often consolidation of test infrastructure, not “another mobile runner.” Mobilewright lets mobile join the same repo conventions as web so agents and engineers maintain one mental model for fixtures, plans, and CI.

If plans and tests are in one repo, is coverage merged across web and mobile?​

No—not by default. Requirement coverage, TrueCoverage comparisons, and Atlas navigation use an explicit platform dimension on Multi-Platform projects (Web, iOS, Android). Shared scenarios in plans/ can be linked from both web/ and mobile/ tests; the platform scope shows where those links actually ran and passed.

Further reading​

TestChimp

• Mobile testing (iOS and Android)
• Requirement traceability
• Atlas SiteMap
• Screen-state annotations
• SmartTests introduction
• TrueCoverage introduction
• Instrumenting your app for TrueCoverage
• Workflows

Playwright & Mobilewright

• Playwright documentation
• Playwright test fixtures
• Mobilewright
• Mobile Use — cloud devices for Mobilewright

Patterns & quality engineering

• Arrange-Act-Assert (xUnitPatterns.com)
• Given-When-Then (Martin Fowler)
• Test fixture (xUnitPatterns.com)
• Object Mother (Martin Fowler)
• Page Object Model — Selenium wiki
• Appium documentation

Try it

• TestChimp — create a Multi-Platform project and connect your repository. Feedback welcome via your usual support channel or community touchpoints linked from the product.

Shipping both web and mobile? The duplication you feel in test automation is often in the Arrange layer—not in the product. Multi-Platform Projects let you maintain that layer once, run Playwright and Mobilewright where users actually interact, and still read requirements, TrueCoverage, and Atlas with clear per-platform signal.