Skip to main content

Human Review of Agent-Written Tests

In brief: Review agent-written tests by checking they can fail, assert the right outcomes, avoid skip/retry crutches, link to a real scenario, and leave evidence. TestChimp Verified Tests records that a person did that inspection.

How do you review tests written by AI coding agents?​

A practical, vendor-neutral review looks at the test the same way you would review a colleague's PR, with extra attention because the same agent may have written the product code. For each important test, a reviewer should check:

  1. It can fail. Change the product behaviour (or imagine the mutation). Does the test fail for the right reason, or only when the whole page goes blank?
  2. Assertions are scoped. The test asserts outcomes from the scenario (state, API effect, user-visible result), not only that "something rendered" or that a toast appeared.
  3. No skip/retry crutches added to hide flakes. Watch for new test.skip, broadened timeouts, or retries that paper over a real bug instead of fixing the product or the selector.
  4. It is linked to a requirement scenario. There is a written story or scenario the test is meant to cover, ideally written independently of the implementation.
  5. Evidence exists. Screenshots, traces or videos from a real run support what the script claims (follow your runner's config).

Green alone is not that review. A scenario link alone is not that review. Someone has to open the run and the code and decide.

Longer checklist and failure modes: How to trust AI-written tests. Narrative: Who checks the tests your AI agent wrote?.

How TestChimp supports the review (Verified Tests)​

TestChimp does not replace the judgment above. It records that a person performed it.

  • Open a SmartTest execution. Inspect screen captures, steps and test code (View Test).
  • Use the check badge after the test name to stamp a Verified claim per linked scenario (or mark all linked scenarios).
  • Badge states: hollow (none verified), grey (partial), blue (all linked scenarios verified), blue with ! (stale: verified earlier, then an agent patched the test or git-sync stale is on).
  • By default, when an agent reports a fix to an existing test, the platform can mark verification stale until someone re-verifies (LLM judgement on by default: stale only if reported confidence is below 80). Teams can mark every reported fix stale, or turn the setting off.
  • Status lives in TestChimp, not in Playwright source. It is the only sole-platform asset called out for exit-safe ownership.

Full behaviour: Verified Tests. Announcement: Verified Tests: trust but verify.

Where this fits in the git-native loop​

  1. Scenarios live as Markdown in your repo (Test planning as code).
  2. Agents author or patch Playwright/Mobilewright SmartTests linked with #TS-n annotations.
  3. CI or /testchimp run QA produces runs and evidence (Playwright config controls screenshots/traces; default is last screenshot on pass and full trace on fail).
  4. A person reviews and stamps Verified Tests.
  5. Release views show which scenarios have a person-checked covering test.

For teams replacing a hosted TMS while using coding agents, see the git-native alternatives: TestRail, Qase, Azure DevOps Test Plans, Xray.

Frequently asked questions​

How do you review tests written by AI coding agents?​

Check that the test can fail for the right reason, asserts scenario outcomes, did not gain skip/retry crutches, is linked to a written scenario, and has run evidence. Then record who reviewed it. TestChimp Verified Tests is one way to record that stamp.

Does a green CI run mean the agent-written test is good?​

No. Green means the code and the test currently agree. If the same agent wrote both, that agreement can hide a wrong requirement.

What makes a Verified badge go stale?​

By default, when an agent reports a fix to an existing test and (with LLM judgement on) reported confidence is below 80, verification becomes stale until someone re-verifies. Optional: mark stale on every reported fix, or when test code changes on git sync. Details in Verified Tests.

Do I need TestChimp to review agent-written tests?​

No. The checklist works with any stack. TestChimp adds a durable, coverage-visible record of who verified which scenario link.

Does Verified Tests prove correctness?​

No. It is a recorded human sanity check, not a formal proof. A reviewer can still approve a weak test.

How to trust AI-written tests, Verified Tests, Who checks the tests your AI agent wrote?, and Verified Tests announcement.

FAQ

How do you review tests written by AI coding agents?

Check that the test can fail for the right reason, asserts scenario outcomes, did not gain skip/retry crutches, is linked to a written scenario, and has run evidence. Then record who reviewed it.

Does a green CI run mean the agent-written test is good?

No. Green means the code and the test currently agree. If the same agent wrote both, that agreement can hide a wrong requirement.

What makes a Verified badge go stale?

By default, when an agent reports a fix to an existing test and (with LLM judgement on) reported confidence is below 80, verification becomes stale until someone re-verifies.

Do I need TestChimp to review agent-written tests?

No. The checklist works with any stack. TestChimp adds a durable, coverage-visible record of who verified which scenario link.

Related documentation