Skip to main content

/testchimp audit — Close coverage and behaviour gaps

Use /testchimp audit on a schedule (for example nightly or weekly) or when triggered by events you care about—after a deployment, after merge to main, or whenever you want portfolio-level QA improvement without tying everything to a single PR.

While /testchimp test optimizes for one change set, audit optimizes for the whole product over time: requirement coverage, execution history, and—when TrueCoverage is enabled—gaps between real user behaviour and what your tests exercise (drop-offs, dwell, funnels, high-traffic paths).

What audit does​

The agent collaborates with the TestChimp platform—the same insights you see in the app—and turns them into an ordered plan that optimizes for business impact, then implements tests and instrumentation.

  1. Requirement coverage — Call get_requirement_coverage with optional scope (tests/... or plans/... platform paths, file paths, branch, environment). Turn on flags for non-covered stories and scenarios when you need an explicit gap list.
  2. Execution history — Call get_execution_history with a matching scope to see recent pass/fail, flakes, and errors—so fixes target what is actually failing in CI or scheduled runs.
  3. TrueCoverage (when instrumentation is on) — Use RUM analytics tools: list environments, summarize events and funnels, drill into details, child trees, transitions, time series, and metadata keys. Compare real usage (for example production-like baseExecutionScope) to test-tagged traffic on a staging or automation scope (comparisonExecutionScope with automationEmitsOnly: true where appropriate) so manual noise does not inflate “covered.”
  4. Plan — Order work by impact: combine missing requirement coverage, failing history, and high-traffic / high-risk TrueCoverage gaps (for example dropoff and dwell insights).
  5. Execute — Implement SmartTests, API tests, and instrumentation updates; re-run locally or in CI.

Overall outcome​

  • Requirement-aligned coverage improves because tests stay linked to scenarios and gaps are visible and actionable.
  • User behaviour–aligned coverage improves when TrueCoverage is enabled: the agent prioritizes journeys that matter in production, not only what is easy to script.
  • Insights stay in the loop—TestChimp aggregates traceability, runs, and RUM so the team gets a single place to see whether quality is converging, not a one-off spreadsheet per release.

That is the closed feedback loop: measure (plans + runs + TrueCoverage) → decide (audit plan) → change (tests and app emits) → measure again.

When to run audit vs test​

SituationPrefer
A specific PR is ready for QA automation/testchimp test
You want portfolio-level gap closure, post-release hygiene, or periodic quality improvement/testchimp audit

You can run both: test after each meaningful PR; audit on a cadence or after major releases so strategic coverage catches up with product and real usage.

See also​