Skip to main content

QA on Autopilot with TestChimp + Claude

TestChimp is built for humans and AI agents to share the same QA workflow. When you pair the TestChimp platform with Claude (or any agent that can load skills and MCP tools), you get a repeatable loop: plans and coverage live in TestChimp, git stays the source of truth for automation, and the agent implements tests, instrumentation, and environment wiring using the same conventions your team uses.

This section describes the TestChimp skill for coding agents—three command flows that turn that idea into a day-to-day practice.

What you get​

  • Single workflow for planning and execution — Product and QA author and manage test plans in TestChimp (user stories, scenarios, stable ids). Changes sync to your repository as markdown under your mapped plans tree. The agent reads those files to know what to cover and links automation back to scenarios—no guessing requirements only from code diffs.
  • Coverage you can reason about — TestChimp tracks requirement-aligned coverage (tests linked to scenarios) and, with TrueCoverage, real user behaviour from production-like traffic. After TrueCoverage is set up, the agent can use analytics in audit flows to close gaps that matter in practice, not only on paper.
  • Tests that stay maintainable — SmartTests are Playwright with optional execution-time intelligence via ai.act / ai.verify / ai.extract (ai-wright), so the agent writes stable scripts and uses AI where it helps, not everywhere.
  • A complete harness, not one-off scripts — One-time init establishes the test folder scaffold (setup → e2e → api projects), seed and teardown strategy (dedicated endpoints where possible), TrueCoverage instrumentation when you opt in, world-state scripts for reusable data posture, and ephemeral environment options (for example via EaaS) when isolated full-stack runs are required.

Together, that is QA on autopilot: the platform owns what should be covered and how well you are covered; the agent owns implementing and evolving the automation and instrumentation; git ties it to your releases.

The TestChimp skill (three commands)​

The skill is published as testchimp-skills (github.com/testchimphq/testchimp-skills). Install it so your agent loads SKILL.md (for example clone into your skills directory as testchimp so updates stay easy). The skill routes work like this:

CommandPurpose
/testchimp initBootstrap the repo: dependencies, MCP, markers, Playwright + reporter, CI hints, seed strategy, environment strategy, optional TrueCoverage.
/testchimp testAfter a PR’s implementation is in place, plan → setup → execute → cleanup: align plans, add or update SmartTests and API tests, run and fix, report.
/testchimp auditOn a schedule, or when triggered (for example after deploy), analyze requirement coverage and execution history, and when TrueCoverage is enabled compare real usage to tests, then plan and execute improvements.

Details for each flow:

  • Init — Skill installation, two-folder mapping (plans + tests), markers, API key, MCP, scaffold and harness.
  • Test — The phased test workflow for a PR: planning against synced plans, environment and world-state, authoring UI and API tests linked to scenarios.
  • Audit — Ongoing coverage and behaviour review, gap closure, and the closed feedback loop with TestChimp insights.

Integration in one picture​

  1. In TestChimp, connect your Git repository and map two folders: one to plans (markdown test plans) and one to tests (SmartTests / Playwright). TestChimp adds empty marker files (.testchimp-plans, .testchimp-tests) at those roots so tools and agents know which directories are canonical.
  2. Teams edit plans in the product; sync creates or updates the markdown in git. The agent reads plans/ for scenarios and ids and writes tests under the mapped tests root, using comments such as // @Scenario: #TS-… for traceability.
  3. Configure testchimp-mcp-client with TESTCHIMP_API_KEY (project-scoped) so the agent can query coverage, execution history, branch endpoints, and TrueCoverage analytics where applicable.

Ongoing use​

  • When a feature or fix is ready for QA in a PR, run /testchimp test. The outcome is new or updated automation (API and/or UI depending on the scenario), linked to the right scenarios, and missing plan items surfaced so authors can add stories or scenarios in TestChimp and sync again.
  • Periodically or on a trigger (for example post-deploy or nightly), run /testchimp audit. The outcome is a deliberate improvement pass: requirement gaps, failing history, and—with TrueCoverage—gaps between real user behaviour and what tests exercise, turned into changes in tests and instrumentation over time.

That is the closed loop: coverage monitored (requirements + behaviour where TrueCoverage is on), insights from TestChimp, and automation and instrumentation updated by the agent so quality strategically improves instead of drifting.

See also​