Skip to main content

Introduction to Smart Smoke Runs

In brief: Smart Smoke Runs pick the optimal SmartTest subset for a change within a time, count, or suite-% budget—maximizing semantic coverage without running the full suite.

The problem​

Agent-authored SmartTests close coverage gaps fast. Suites that used to take weeks to grow now add dozens of specs per PR. That is good for quality—and hard on CI.

Running everything on every change stops scaling: PR merges stretch to hours, feedback loops die, and teams either abandon smoke gates or invent brittle grep filters that miss the riskiest paths.

What Smart Smoke is​

Smart Smoke is TestChimp’s budget-aware test selection for Playwright SmartTests. You set a sizing constraint (for example “about 20 minutes” or “20% of the suite”). Smart Smoke returns the highest-ROI subset for the current change—tests related to impacted scenarios, tagged smoke tests, newly authored branch tests, and (in budgeted mode) additional tests packed for maximum semantic coverage.

Semantic packing within a time budget

It is not a new test runner. You keep the same npx playwright test command. Selection is opt-in per run via environment variables and the @testchimp/playwright plugin (≥ 0.2.20). Tests outside the selected set are skipped with skip-reason smart-smoke (distinct from an explicit test.skip).

When to use Smart Smoke​

SituationUse Smart Smoke?
PR / feature-branch confidence after authoring or validating SmartTestsYes — default Phase 5 inside /testchimp test
Standalone “smoke this change” after connect-to-test-envYes — /testchimp run smart smoke
Nightly / release full regressionUsually no — run the full suite (or a dedicated regression tag)
Greenfield repo with no existing scenarios / linked testsN/A — document rationale on the plan
Docs-only or non-product changeN/A — skip with rationale

Prefer related-tests-only when you want tight PR confidence without expanding the suite. Use budgeted smoke when you want broader ROI within a time/count/% cap.

How it fits the TestChimp stack​

  1. Plans — stories and scenarios in plans/ give semantic meaning to what a change might break.
  2. Scenario annotations — SmartTests link to #TS-… ids so impact analysis resolves to concrete tests.
  3. Related-tests file — agents write plans/smart-smoke/<branch>/related-tests.json for the change scope.
  4. Plugin + selection API — with TESTCHIMP_SMART_SMOKE_ENABLED=true, @testchimp/playwright loads related tests and (unless related-only) asks TestChimp to pack the budgeted set.
  5. Same Playwright run — selected tests execute; others skip as smart-smoke.

Modes at a glance​

ModeEnvEffect
Related-tests-only (safe default)TESTCHIMP_SMART_SMOKE_RELATED_TESTS_ONLY=trueRuns only locators from related-tests.json (no selection API packing).
Budgeted smokeEnabled without related-onlySelects related ∪ tagged ∪ newly authored ∪ semantically packed tests within the budget.

Budget knobs (env overrides playwright.config use.testchimpSmartSmoke):

  • TESTCHIMP_SMART_SMOKE_MAX_TIME_BUDGET_MINS
  • TESTCHIMP_SMART_SMOKE_MAX_TESTS
  • TESTCHIMP_SMART_SMOKE_SUITE_PERCENTAGE (default 20 if no size constraint is set)
  • TESTCHIMP_SMART_SMOKE_INCLUDE_TAGS (e.g. smoke)

Full reference: Configuration. Algorithm detail: How it works.

Prerequisites​

  • Playwright SmartTests with @testchimp/playwright ≥ 0.2.20
  • TESTCHIMP_API_KEY on the runner process (and backend/ingress URLs when not on SaaS prod)
  • Scenario-linked tests (Playwright annotation with type: 'scenario') for meaningful related sets
  • Optional but recommended: suite tags such as @smoke from global.policy.md

Next steps​