How Smart Smoke Works
In brief: Smart Smoke identifies change-impacted tests, weights the suite with prioritization signals, then iteratively packs the next best semantic gain until the budget is full.
Smart Smoke selection is meaning-first, not path-first. File folders and naming conventions help humans navigate a suite; Smart Smoke optimizes for semantic coverage—how much of the impacted scenario / story space a budgeted subset covers.
Pipeline overview
Change scope (PR / plans / area)
↓
1) Impact analysis → related TestLocators
↓
2) Seed set = related ∪ tagged (@smoke …) ∪ newly authored on branch
↓
3) (Budgeted mode) Weight candidates + pack for max semantic gain
↓
4) Stop at budget → return Smart Smoke set
↓
Playwright runs selected tests; others skip as smart-smoke
In related-tests-only mode, steps 3–4 are skipped: the plugin runs exactly the related locators from disk.
1) Identify impacted areas
For a feature branch (or explicit plans scope), the agent / workflow:
- Reads the PR/diff and relevant
plans/stories/+plans/scenarios/ - Selects scenarios likely affected (same feature area, journey, APIs, or sibling scenarios under an overlapping story)
- Resolves linked SmartTests via Playwright scenario annotations (
type: 'scenario',description: '#TS-…') - Always includes newly authored or materially changed tests from the run
- Writes the result to:
plans/smart-smoke/<branch>/related-tests.json
On the semantic plane, impacted scenarios (triangles) link to related tests (squares). Those tests become the mint seed set; the rest of the suite stays gray until packing.
related-tests.json is blocking for smart-smoke Execute (and required at the end of create-tests). See Configuration → related-tests.json.
2) Apply prioritization signals
Semantic distance alone is not enough. Budgeted mode weights candidates so packing prefers tests that are both informative and practical to run.
Typical weighting signals (combined into a score with semantic distance):
| Signal | What it captures |
|---|---|
| Scenario priority | Business / risk importance of linked scenarios |
| Stability | Recent pass rate / flake risk |
| Recency | How recently the test ran (stale tests may deserve a slot) |
| Execution time | Prefer packing denser coverage per minute |
On the illustration, the terrain brightens or dims as these signals apply—selection is not pure geometric clustering.
Server-side selection uses a bounded packer over embeddings for tests, scenarios, and stories (with hard caps so pathological mappings cannot blow up runtime). Approximate score mix:
| Component | Role |
|---|---|
| Semantic coverage gain | Dominant — maximize new ground covered |
| Priority | Prefer higher-risk scenarios |
| Stability | Prefer trustworthy signal |
| Recency | Prefer under-exercised tests in-window |
| Time | Prefer efficient packing into the budget |
Exact weights evolve; treat the table as the intent of the packer, not a frozen formula.
3) Select for max semantic gain
With seeds in place and the terrain weighted, Smart Smoke iteratively picks the next test that adds the most additional semantic coverage—spreading across the plane rather than clustering in one neighborhood—until the budget says stop.
What “budget” means
You may constrain size in one of three ways (env wins over use.testchimpSmartSmoke):
| Constraint | Meaning |
|---|---|
| Max time (mins) | Pack until estimated runtime fills the bin |
| Max tests | Pack until N tests are selected |
| Suite percentage | Pack until ~N% of the suite is selected |
If no size constraint is set in budgeted mode, @testchimp/playwright defaults to suitePercentage = 20 and logs a warning.
When multiple constraints are present, selection honours the most restrictive effective cap.
Must-include seeds
Before packing extras, the selected set always includes (deduped):
- Related tests from
related-tests.json - Tagged tests matching
includeTags(e.g.@smoke) - Newly authored on the branch (no default-branch twin)—so PR-authored specs are never dropped by the packer
Then packing adds tests that expand coverage within remaining budget capacity.
4) Stop at budget, return the set
When the bin is full, selection stops and returns the Smart Smoke set. The plugin writes a selection sidecar; during the suite, tests not in the list are skipped with reason smart-smoke.
You still get normal Playwright reporting and TestChimp execution ingest for the tests that ran. Skipped-as-smart-smoke is intentional—not a failure.
Enablement (same runner)
export TESTCHIMP_SMART_SMOKE_ENABLED=true
export TESTCHIMP_BRANCH_NAME="$(git branch --show-current)" # local / agent runs
# optional: RELATED_TESTS_ONLY, MAX_TIME_BUDGET_MINS, etc.
npx playwright test
There are no Playwright CLI flags for Smart Smoke. See Configuration.
Related vs full suite
| Full suite | Smart Smoke | |
|---|---|---|
| Goal | Exhaustive confidence | Max meaning per minute |
| Typical trigger | Nightly / release | PR / /testchimp test Phase 5 |
| Selection | All (or manual tags) | Related ∪ tagged ∪ packed |
| Skip reason | — | smart-smoke for non-selected |
Use Smart Smoke to keep the merge gate honest and fast. Keep full regression where breadth matters more than latency.
See also
- Introduction — when to use
- Configuration — env, config, CI
- Run Smart Smoke — agent workflow checklist