Verified Tests

In brief: Agents can write the product and the tests. Verified Tests is the accountability layer: inspect screen captures, steps, and code in the execution viewer, then mark that a covering SmartTest truly verifies the scenario. Coverage views show a Verified Badge when someone has sanity-checked that test.

Why verify

Linking a SmartTest to a scenario is a claim—this test covers that behaviour. A green run is only an execution fact. Neither answers: has anyone actually looked at the test to ensure it verifies the scenario?

Verification is that sanity check. Open an execution, review what the test did (screen captures and steps) and what it is (the code), then stamp the claim. Status is stored in TestChimp, not in Playwright source—so agents can keep authoring tests, and humans record that they inspected them.

Badge states

The check badge sits after the test name in the execution viewer (not shown for manual sessions):

• Hollow — none of the linked scenarios are verified (default, or after un-verify)
• Grey (partial) — some but not all linked scenarios are verified
• Blue (verified) — every linked scenario for that test is verified
• Blue with ! (stale) — verified, but an agent later patched the test (or git-sync stale is on). The badge stays filled; a red ! overlaps the top-right. Coverage trains use the same overlay when a covering mapping is stale and no sibling is still freshly verified.

On coverage displays, the badge is either hollow or filled. There is no grey state there because the unit is a scenario: filled when any covering SmartTest is verified. The badge is hidden when the scenario has no linked tests, and hidden everywhere when Enable verified badges is off in project settings.

How to verify or un-verify

1. Open a SmartTest execution (click a coverage square).
2. Review the screen captures, steps, and test code (View Test) so you can decide whether the test actually verifies the scenario.
3. Use the check badge after the test name.
4. No linked scenarios — a message explains the test cannot be verified.
5. One scenario — confirm to verify or un-verify. When verified, hover shows who verified it.
6. Several scenarios — open the popover to verify per scenario, or Mark all as verified (already-verified rows keep their original verifier). There is no “un-verify all.”
7. Stale (!) — click the badge. The popover still shows who originally verified, plus Marked as stale on a human-readable date (the date links to the workflow execution when one was recorded). In the execution viewer only, use Re-Verify to stamp VERIFIED again (same fan-out as verify). Coverage popovers are read-only — they do not offer Re-Verify.

Un-verify records manually unverified (not “never verified”). Hover of who/when is shown only for verified links. Do not un-verify a stale badge through the verify/un-verify confirm — use Re-Verify, or un-verify after it is fresh again.

Renaming a scenario without changing its ordinal (#TS-n) keeps verification. Linking a different scenario is a new claim and starts unverified.

Project settings

Project Settings → Verified Tests (defaults):

• Enable verified badges — on. Off hides badges everywhere and treats nested stale options as off.
• Mark as stale when an agent reports a test fix — on. Agents call mark-tests-for-review after fixing existing tests; the platform always stores an audit row, then applies VERIFIED_STALE when this is on.
• Use LLM judgement before marking stale — on. When on, stale applies only if reported confidence is below the threshold (default 80; equal to the threshold stays verified). When off, every reported fix is marked stale.
• Mark as stale when test code changes on git sync — off. When on, inbound parse/sync UPDATED test bodies also go stale.

Learn more in product — this page.

Coverage trains

Coverage squares still show recent execution results. The Verified Badge sits before the train — after the priority icon on plans/requirement coverage, and in a narrow unnamed column on named test run tables. Filled means a user has sanity-checked a covering test; hollow means tests exist but have not been inspected yet. It is omitted when the scenario has no linked tests. Clicking it does not change verification — it explains to open an execution and use the viewer badge.

View Test

From the execution viewer, View Test opens a read-only file popover so you can inspect the script next to the run. View Full Test still opens the SmartTests editor. Back still returns to the previous list or batch view.