Skip to main content

Playwright TestChimp runtime (@testchimp/playwright)

In brief: The @testchimp/playwright plugin reports runs to TestChimp, correlates user events during tests for TrueCoverage, and enables resilient ai.act/ai.verify execution in CI.

The @testchimp/playwright package is the Playwright-side bridge between your SmartTests (normal npx playwright test runs) and TestChimp. Install it during /testchimp init and keep it enabled in CI so every run feeds the same feedback loop as Run QA and Upkeep.

Runtime: reporter, execution telemetry, RUM augmentation, and AI steps

What it does​

1. Report test execution to TestChimp​

The reporter collects pass/fail, steps, errors, timing, and (when configured) screenshots for failing steps. That data powers execution history, QA intelligence dashboards, and traceability between runs, files, and scenarios—without replacing the Playwright CLI or your CI runner.

See the published package: @testchimp/playwright on npm.

2. Augment user events with test identity (TrueCoverage)​

If your app emits RUM events (web: @testchimp/rum-js; iOS / Android: testchimp-rum-ios, testchimp-rum-android), the runtime ensures events produced during automated runs carry test / run context—page injection on web and automation deep links on native when TESTCHIMP_PROJECT_TYPE is ios or android—so TestChimp can correlate them with production traffic on the same event taxonomy. Setup: Instrumenting your app.

That alignment is what makes TrueCoverage useful: you compare what users do to what tests exercise, not just “did this test file pass.”

3. Execute natural-language steps in the runner​

SmartTests can include AI-native steps (ai.act, ai.verify, ai.extract, …). The reporter coordinates those steps with TestChimp backends so they execute wherever you run Playwright (local, CI, preview)—execution-time intelligence in the script, not only at authoring time.

For authoring patterns, see Smart Tests and Run QA.

4. ExploreChimp (optional UX analytics)​

When EXPLORECHIMP_ENABLED is set and the batch id / API key are present, the runtime sends checkpointed UX telemetry (DOM, screenshots, console, optional network and metrics) keyed off the markScreenState Playwright fixture—so analysis runs along the same pathways as your UI SmartTests. Pure API tests (no browser journey) are out of scope.

Use Run ExploreChimp for a dedicated exploration pass, or enable ExploreChimp inside Run QA / Upkeep when the policy or plan opts in. See Screen-State Annotations.

Configuration (minimal)​

In playwright.config.ts (or .js), register the reporter and import the runtime module so TrueCoverage tagging and step plumbing apply:

import { defineConfig } from '@playwright/test';

// Enables runtime behaviour (e.g. test info for RUM / TrueCoverage)
import '@testchimp/playwright/runtime';

export default defineConfig({
  reporter: [
    ['list'],
    ['@testchimp/playwright/reporter', { verbose: true, reportOnlyFinalAttempt: true, captureScreenshots: true }],
  ],
});

Set TESTCHIMP_API_KEY (and any project-specific env your repo documents) in CI and local shells so the reporter can authenticate. See your repo’s @testchimp/playwright README or the npm package page for the full option list.

How this feeds TrueCoverage and upkeep​

  • Production vs test — TrueCoverage compares real-user event streams to automation-tagged streams (often scoped so manual traffic does not inflate “covered”). The runtime’s role is to make test-origin events comparable to prod: same events, metadata, and scopes your team defined in plans/events/ and instrumentation.
  • Gap signals — When production stops emitting an event (or it drops sharply) but tests still drive that event, that can indicate deprecated product surface still covered by automation—a candidate for suite cleanup (always confirm with product intent before deleting tests).
  • Prioritization — TrueCoverage surfaces event-level metrics (for example Demand, Depth, Duration, Drop-off) so you can prioritize where users spend time, where they abandon, and which journeys deserve new tests or exploration. You do not need to duplicate those definitions here: see How TrueCoverage metrics work and the dashboard overview.

Where this sits in the agent workflow​

  1. /testchimp init — Adds @testchimp/playwright, wires reporter + runtime import, and aligns CI env with TESTCHIMP_API_KEY.
  2. Run QA — Authors and runs SmartTests with the reporter so results and behaviour land in TestChimp; optional ExploreChimp when the policy/plan opts in.
  3. Run ExploreChimp — Runs ExploreChimp as the primary task on selected UI specs.
  4. This page — What the runner contributes continuously (including outside agent sessions).
  5. Upkeep — Consumes accumulated run + RUM + plan signals to improve the portfolio; may include targeted ExploreChimp.

See also​

FAQ

Is this a custom test runner?

No—you use the standard Playwright runner; the plugin is a reporter and runtime hook added during /testchimp init.

Why tag events during test runs?

Same event taxonomy as production RUM lets TrueCoverage show which real user pathways your CI executions actually cover.

When do ai.act steps run agentically?

At execution time in CI or locally when a SmartTest calls ai.act, ai.verify, or ai.extract instead of brittle selectors.

Related documentation