Claude Writes Your Tests. Who Decides Which Ones Matter?
Last updated: 2026-10-20
TL;DR: If Claude or Cursor already writes your tests, keep it that way. The hard part was never typing the test. It is knowing which tests matter, whether anyone checked them, and whether the release is ready. TestChimp measures your QA posture (requirement coverage, API contract coverage and real user behaviour coverage) so your coding agent knows exactly where the gaps are and can close them. Claude writes tests. TestChimp tells it which ones matter.
"Claude can write tests too, so why bother?"
It's a fair question, and we hear it often. The answer starts with agreeing: yes, Claude can write tests, and it should. A coding agent with Playwright and a decent CLAUDE.md will produce a working end-to-end test in minutes.
But look at what happens a few months after a team starts doing this at scale. The suite has grown fast. CI is mostly green. And yet nobody on the team can answer a few basic questions:
- Are our requirements actually tested?
- How much of our API is exercised?
- Are the flows our users actually use covered?
- Which of these tests has a person ever looked at?
- What should we test next?
- Are we ready to ship this release?
None of these are test-writing problems. Writing more tests doesn't answer them. They are measurement problems, and what you can't measure, you can't improve.
What the agent can't see on its own
A coding agent works from its context window: the code it's touching, the prompt, maybe a rules file. That is plenty for writing a test. It isn't enough to decide which tests are worth writing.
| The agent knows | The agent doesn't know |
|---|---|
| The code it just changed | What was decided in last week's planning call |
| The test it was asked to write | Which requirements have no tests at all |
| The endpoints in the files it opened | Which API operations and response codes nothing exercises |
| How the feature should work, per the prompt | Which flows real users take most, and where they drop off |
| That its test passed | Whether anyone has checked the test asserts the right thing |
So the agent does the reasonable thing: it writes tests near whatever it built most recently. Coverage clusters. Important but unglamorous flows stay untested. And because the agent wrote both the code and the test, a misread requirement shows up in both, and the suite stays green.
Measure first: three questions a dev lead should be able to answer
TestChimp gives your agent, and your team, three measures of what is actually covered.
1. Requirement coverage: are your requirements tested?
Stories and scenarios live as markdown files in your repo, in a plans/ folder that syncs with TestChimp. Tests declare the scenario they cover with a Playwright annotation:
test('guest can cancel a booking', {
annotation: [{ type: 'scenario', description: '#TS-214' }],
}, async ({ page }) => {
// ...
});
Coverage rolls up per folder, release and environment, so you can see which scenarios have no tests, no recent runs or a failing latest run.
Requirements also go stale. Decisions now happen in meetings and in coding-agent chats, and rarely make it back into a spec. MeetBots capture what was agreed on calls, and AgentWatch spots product decisions made in Claude Code or Cursor sessions. Both propose story and scenario updates that you approve, so the plans your agent tests against stay current.
2. API contract coverage: how much of your product surface is tested?
Connect your OpenAPI specs and run your tests with API capture on. TestChimp shows per-operation coverage, plus request field, response field and response code coverage. Traffic that hits undocumented paths or fields is flagged too. Each gap can go straight to your agent as a create-tests task.
3. Real user behaviour coverage: are the flows your users actually use tested?
TrueCoverage captures the same user events in production and during test runs, on web, iOS and Android. Lay them over each other and you see which real user pathways have no test coverage, ranked by demand, duration and drop-off. That turns "what should we test next?" from a guess into a ranked list.
Then improve: let the agent close the gaps
Measurement is only useful if it changes what the agent does next. That is where the TestChimp skill comes in. Install it in Claude Code, Cursor or another agent host, and you get workflows that turn the measures into work:
/testchimp run QAon a PR: works out what the change affects, updates plans, writes or fixes tests for the scenarios in scope, and runs the related tests./testchimp upkeepon a cadence: pulls the highest-value requirement and TrueCoverage gaps, fixes recently failing tests, and keeps instrumentation current.- Smart-Smoke: impact analysis picks the tests related to a change and an optimal subset of a large suite, so every PR doesn't have to run everything.
Your agent still writes the tests. It just starts from a prioritised list of gaps instead of a blank page. The result is usually fewer, better-aimed tests rather than a bigger suite.
You can run these yourself in your agent, or let them run automatically on events like a failing CI batch or a release moving to ready.
Keep humans on the parts that need judgment
Agent-written tests need a second pair of eyes, and that person shouldn't have to re-read the whole suite. In TestChimp, a person inspects the run (steps, screen captures and code) and marks a test verified. When an agent later patches a verified test, the test is flagged for human re-verification by default, so a quiet "fix" that loosens an assertion doesn't slip through.
Run evidence such as screenshots, traces and video follows your Playwright config, so you decide how much is captured.
Know whether the release is ready
All of this comes together per release. Each release gets one clear picture of how ready it is to ship: test runs, requirement coverage, and verification status. Verified tests show a verified badge; unverified tests are visible. The gate can be checked programmatically, so CI can ask "is this release ready?" and get an answer based on evidence, not on a green tick.
What doesn't change
- Your tests stay standard. Playwright on web, Mobilewright for native iOS and Android, in your own repo.
npx playwright teststill runs them. - Your plans stay readable. Markdown with frontmatter, reviewed by pull request.
- Your agent stays your agent. TestChimp supplies the signals and the playbooks. Claude or Cursor does the work.
Who this is for
If your team only just found out that Claude can write tests, enjoy it. You'll feel the problems above soon enough. If you are already shipping a lot of agent-written code and your suite has grown faster than your confidence in it, that is the point where measurement pays off.
Claude writes tests. TestChimp tells it which ones matter.
Frequently asked questions
If Claude already writes our tests, what does TestChimp add?
Measurement and direction. TestChimp shows requirement coverage, API contract coverage and real user behaviour coverage, then feeds the gaps to your agent through workflows like /testchimp run QA and /testchimp upkeep. Your agent writes the tests; TestChimp decides which ones are worth writing.
Do I have to change test frameworks?
No. Tests are standard Playwright specs on web and Mobilewright specs on native mobile. Optional natural-language steps (ai.act, ai.verify, ai.extract) are available on web.
How do I know an agent-written test checks the right thing?
Have a person inspect it and mark it verified in TestChimp. If an agent later edits a verified test, it is flagged for re-verification by default.
Does TestChimp support mobile apps?
Yes. Native iOS and Android are supported through Mobilewright, and TrueCoverage works on web, iOS and Android when instrumented.
How do I get started?
Connect your Git repo, map a tests folder and a plans folder, install the TestChimp skill in your agent, run /testchimp init, then run /testchimp run QA on your next PR.