Skip to main content

/testchimp test — Test the PR

Use /testchimp test when implementation for a pull request is done and you want the agent to run the full PR QA workflow before merge: align with test plans, close coverage gaps the platform surfaces, and leave SmartTests and API tests linked to scenarios.

The skill follows a fixed phase order: Plan → Setup → Execute → Cleanup.

What this command is meant to do​

After development on a branch, the agent:

  • Finds missing stories or scenarios for the work shipped and surfaces them so owners can add them in TestChimp and sync.
  • Chooses which scenarios to test for this PR and builds a matrix (UI vs API vs manual follow-ups).
  • Instruments TrueCoverage (when enabled) for new journeys or interactions—updates app emits and plans/events/ as needed.
  • Updates seed endpoints or world-state scripts when data posture must change for the new behaviour.
  • Uses TestChimp insights (MCP: requirement coverage, execution history, TrueCoverage analytics where configured) to prioritize what to automate for this PR.
  • Provisions a test environment when the plan calls for isolation (for example PR-scoped full stack); destroys it in Cleanup.
  • Brings environments to world-states via ensureWorldState and shared *.world.js scripts so author-time and run-time state match.
  • Uses Playwright for UI; for API tests, follows your repo’s patterns (including intercepting or driving HTTP where you author API-level cases).
  • Authors SmartTests with natural-language steps (ai.act, ai.verify, …) where they reduce flake or encode intent that selectors alone cannot.
  • Links tests to scenarios with in-code comments (for example // @Scenario: #TS-…) for requirement coverage visibility.

How this fits the broader workflow​

  • Plans are authored and managed in TestChimp and synced to git under your mapped plans tree. The agent reads those markdown files (stories, scenarios, ids such as #TS-…) to decide what must be covered for this PR.
  • Tests live under the mapped tests tree. The agent authors or updates API tests and UI SmartTests depending on risk (UI-facing vs contract/backend vs both). Each automated case is tied to the scenario under test via traceability comments and file placement so TestChimp coverage stays meaningful.
  • TrueCoverage (when enabled) keeps behaviour-aligned coverage: the agent can add or adjust semantic events, metadata, and plans/events/ docs when the PR changes journeys users care about—so later audit runs can compare real usage to tests.

Phase 1: Plan​

Goals:

  • Find existing stories and scenarios affected by the PR; use MCP (get_requirement_coverage, get_execution_history) for gaps and recent failures.
  • List missing stories or scenarios for new or changed behaviour—the agent should not only automate; it should surface missing plan items so owners can add them in TestChimp and sync.
  • Build a test matrix: for each case, UI, API, or manual; link to scenario ids; for API-heavy cases, follow the repo’s API patterns alongside creating Smart Tests.
  • Choose environment strategy using plans/knowledge/ai-test-instructions.md from init: persistent + preview when enough, ephemeral full stack when backend or data isolation matters. For ephemeral targets, use MCP provisioning tools when EaaS is configured.
  • Call out infra gaps (missing seed APIs, missing world-state scripts).

TrueCoverage gate: Respect the skill’s .truecoverage_setup flag so you do not half-enable instrumentation.

Plan artifact: A markdown summary (often under a scratch or plans area) capturing scenarios, gaps, matrix, prerequisites, and environment choice—this drives the next phases.

Phase 2: Setup​

  • Provision an ephemeral environment when the plan requires it; poll until ready or escalate; destroy in Cleanup.
  • Implement missing seed wiring, world-state scripts, and plan updates (including MCP create/update for scenarios when the product is the source of truth for new ids).
  • If TrueCoverage work was planned, update app emits and plans/events/ before relying on those signals in tests.

Phase 3: Execute​

  • Author UI tests using SmartTest patterns (write-smarttests in the skill): stable selectors where possible, ai.act / ai.verify / ai.extract when execution-time intelligence reduces flake.
  • Author API tests using captured flows or contracts where your process defines them.

World-states and execution order​

For each UI case that depends on seeded data, ensureWorldState (or your equivalent) runs before browser automation—the browser pass assumes the backend already matches the target state. World-states are few, reusable states (backed by tests/setup/world-states/*.world.js in the scaffold), not a new state per test. The Plan phase must name the target world-state per case and any missing scripts or seed APIs; Setup lands those scripts before you rely on them in Execute.

Phase 4: Cleanup and report​

  • Tear down ephemeral environments and temporary storage state files.
  • Report pass/fail, API vs UI counts, new world-states, TrueCoverage changes, and anything that still needs a human.

CI: what playwright-testchimp does on runs​

When tests run in CI (or locally with the reporter), the playwright-testchimp plugin:

  • Calls back to TestChimp to execute natural-language steps expressed as ai.act / similar in the test.
  • Records user-event emits during the run so TrueCoverage can compare test behaviour to production behaviour.
  • Sends execution metadata so requirement coverage and history stay accurate in the app.
  • Coordinates world-states with the patterns you defined so authoring and execution stay in sync—supporting reliable, repeatable runs.

Expected deliverables​

  • A plan artifact for the PR (working notes; location is up to your team).
  • New or updated scenarios in TestChimp when gaps were filled, then synced.
  • New or updated SmartTests and API tests with scenario links.
  • A short report of what ran and what is left open.

Next: for ongoing improvement across the whole project or release, run /testchimp audit on a cadence or trigger.