Skip to main content

/testchimp test — Test the PR

PR QA workflow: analyze, plan, execute, validate, optional ExploreChimp, cleanup

Use /testchimp test when implementation for a pull request is done and you want the agent to run the full PR QA workflow before merge: align with test plans, close coverage gaps the platform surfaces, and leave SmartTests and API tests linked to scenarios—with optional ExploreChimp UX analytics when the team opts in.

The skill follows a fixed phase order:

Analyze → Plan → Execute → Validate → Phase 5 (ExploreChimp, optional) → Phase 6 (Cleanup and report)

Each phase has completion gates (checklists); the agent does not skip ahead silently. Details and exact checklist wording live in the TestChimp skill (references/testing-process.md).

What this command is meant to do​

After development on a branch, the agent:

  • Analyzes the change: diff vs base, existing stories/scenarios, coverage and execution history via MCP where available, and fixture/seed posture implied by the work.
  • Plans in a branch plan file under your mapped plans tree: inventory of tests, Arrange / Act / Assert per test, infra summaries, user approval, and—when UI is in scope—whether to run ExploreChimp after validation (yes vs N/A, target spec list, cost acknowledgment).
  • Executes the approved plan in dependency order: seed/probe endpoints, environment (re)start if backend changed, fixtures, SmartTests/API tests, TrueCoverage doc updates when planned.
  • Validates cross-cutting quality: scenario link comments (// @Scenario: …), markScreenState placement for meaningful UI transitions, and Atlas vocabulary (list-screen-states / upsert-screen-states) before treating UI work as done.
  • Optionally runs ExploreChimp (Phase 5) when Phase 2, section 6 — ExploreChimp (optional) in the branch plan is yes and the user approved that scope: same Playwright specs, with EXPLORECHIMP_ENABLED and batch correlation, after functional tests and Validate are in good shape—see /testchimp explore for what that analytics pass does.
  • Cleans up (Phase 6): stop local stacks, destroy ephemeral environments, and produce a short final report (including ExploreChimp N/A vs executed).

Other bullets (unchanged in intent):

  • Instruments TrueCoverage (when enabled) for new journeys or interactions—updates app emits and plans/events/ as needed.
  • Provisions isolated environments when the plan requires it; destroys them in Cleanup.
  • Authors SmartTests with natural-language steps (ai.act, ai.verify, …) where they reduce flake or encode intent selectors alone cannot.
  • Links tests to scenarios with in-code comments for requirement coverage in the app.

How this fits the broader workflow​

  • Plans are authored and managed in TestChimp and synced to git under your mapped plans tree. The agent reads those markdown files (stories, scenarios, ids such as #TS-…) to decide what must be covered for this PR.
  • Tests live under the mapped tests tree. The agent authors or updates API tests and UI SmartTests depending on risk. Each automated case is tied to the scenario under test via traceability comments.
  • TrueCoverage (when enabled) keeps behaviour-aligned coverage so later /testchimp evolve runs can compare real usage to tests.
  • ExploreChimp reuses UI SmartTests as exploration vehicles when enabled—never a substitute for functional assertions; it adds UX-oriented signals tied to markScreenState checkpoints.

Phase overview (PR workflow)​

Phase 1: Analyze​

Discover gaps, branch-plan file location, candidate tests, environment strategy from plans/knowledge/ai-test-instructions.md, and (for UI-heavy changes) candidate specs for a possible ExploreChimp pass—the yes/N/A decision is recorded only in Plan.

Phase 2: Plan​

Produce a written, user-approved branch plan: tests to add/change, infra, section 6 — ExploreChimp (optional) (yes + target files or N/A with rationale), approval and structure guard, and workflow checklists for later phases.

Phase 3: Execute​

Implement seeds, probes, fixtures, app instrumentation, and test code; run Playwright/API tests until the planned automation is green.

Fixtures and execution order​

For each UI case that depends on seeded data, fixture setup (or equivalent shared hooks) runs before browser automation—the browser pass assumes the backend already matches the target state. Prefer few, reusable fixtures under tests/fixtures/ (merged with mergeTests and installTestChimp on the merged test), not bespoke setup per test. The Plan phase must name the target data posture per case and any missing fixture or seed API work; Execute lands that wiring before you rely on it in the spec.

Phase 4: Validate​

Audit scenario links, markScreenState on meaningful transitions, and Atlas consistency for touched UI specs.

Phase 5: ExploreChimp (optional)​

Run only when the approved plan’s section 6 is yes. Uses the same SmartTest files with ExploreChimp env described on /testchimp explore.

Phase 6: Cleanup and report​

Tear down ephemeral environments and temporary state; report pass/fail, infra changes, TrueCoverage updates, ExploreChimp outcome, and open follow-ups.

CI: what @testchimp/playwright does on runs​

When tests run in CI (or locally with the reporter), the @testchimp/playwright plugin:

  • Calls back to TestChimp to execute natural-language steps expressed as ai.act / similar in the test.
  • Records user-event emits during the run so TrueCoverage can compare test behaviour to production behaviour.
  • Sends execution metadata so requirement coverage and history stay accurate in the app.
  • Supports ExploreChimp when env flags are on—checkpointed UX telemetry for analysis; see Runtime plugin.

Expected deliverables​

  • A branch plan for the PR (under plans/knowledge/branch_test_plans/ per team mapping).
  • New or updated scenarios in TestChimp when gaps were filled, then synced.
  • New or updated SmartTests and API tests with scenario links and, for UI flows, markScreenState where required.
  • A short report including optional ExploreChimp results or explicit N/A.

Next: for ongoing improvement across the whole project or release, run /testchimp evolve on a cadence or trigger. For a standalone exploration batch on selected UI specs, use /testchimp explore.

See also​