How Smart Smoke Works

In brief: Smart Smoke identifies change-impacted tests, weights the suite with prioritization signals, then iteratively packs the next best semantic gain until the budget is full.

Smart Smoke selection is meaning-first, not path-first. File folders and naming conventions help humans navigate a suite; Smart Smoke optimizes for semantic coverage—how much of the impacted scenario / story space a budgeted subset covers.

Pipeline overview

Change scope (PR / plans / area)
        ↓
1) Impact analysis → related TestLocators
        ↓
2) Seed set = related ∪ tagged (@smoke …) ∪ newly authored on branch
        ↓
3) (Budgeted mode) Weight candidates + pack for max semantic gain
        ↓
4) Stop at budget → return Smart Smoke set
        ↓
Playwright runs selected tests; others skip as smart-smoke

In related-tests-only mode, steps 3–4 are skipped: the plugin runs exactly the related locators from disk.

---

1) Identify impacted areas

For a feature branch (or explicit plans scope), the agent / workflow:

1. Reads the PR/diff and relevant plans/stories/ + plans/scenarios/
2. Selects scenarios likely affected (same feature area, journey, APIs, or sibling scenarios under an overlapping story)
3. Resolves linked SmartTests via Playwright scenario annotations (type: 'scenario', description: '#TS-…')
4. Always includes newly authored or materially changed tests from the run
5. Writes the result to:

plans/smart-smoke/<branch>/related-tests.json

On the semantic plane, impacted scenarios (triangles) link to related tests (squares). Those tests become the mint seed set; the rest of the suite stays gray until packing.

related-tests.json is blocking for smart-smoke Execute (and required at the end of create-tests). See Configuration → related-tests.json.

---

2) Apply prioritization signals

Semantic distance alone is not enough. Budgeted mode weights candidates so packing prefers tests that are both informative and practical to run.

Typical weighting signals (combined into a score with semantic distance):

Signal	What it captures
Scenario priority	Business / risk importance of linked scenarios
Stability	Recent pass rate / flake risk
Recency	How recently the test ran (stale tests may deserve a slot)
Execution time	Prefer packing denser coverage per minute

On the illustration, the terrain brightens or dims as these signals apply—selection is not pure geometric clustering.

Server-side selection uses a bounded packer over embeddings for tests, scenarios, and stories (with hard caps so pathological mappings cannot blow up runtime). Approximate score mix:

Component	Role
Semantic coverage gain	Dominant — maximize new ground covered
Priority	Prefer higher-risk scenarios
Stability	Prefer trustworthy signal
Recency	Prefer under-exercised tests in-window
Time	Prefer efficient packing into the budget

Exact weights evolve; treat the table as the intent of the packer, not a frozen formula.

---

3) Select for max semantic gain

With seeds in place and the terrain weighted, Smart Smoke iteratively picks the next test that adds the most additional semantic coverage—spreading across the plane rather than clustering in one neighborhood—until the budget says stop.

What “budget” means

You may constrain size in one of three ways (env wins over use.testchimpSmartSmoke):

Constraint	Meaning
Max time (mins)	Pack until estimated runtime fills the bin
Max tests	Pack until N tests are selected
Suite percentage	Pack until ~N% of the suite is selected

If no size constraint is set in budgeted mode, @testchimp/playwright defaults to suitePercentage = 20 and logs a warning.

When multiple constraints are present, selection honours the most restrictive effective cap. An env time budget alone does not inherit a template suitePercentage from use.testchimpSmartSmoke.

Packing runs against the platform SmartTest inventory (git-synced; default∪feature CoW merge with de-dupe). Clients do not send the local Playwright suite as candidates.

Must-include seeds

Before packing extras, the selected set always includes (deduped):

• Related tests from related-tests.json
• Tagged tests matching includeTags (e.g. @smoke)
• Newly authored on the branch (no default-branch twin)—so PR-authored specs are never dropped by the packer

Then packing adds tests that expand coverage within remaining budget capacity.

---

4) Stop at budget, return the set

When the bin is full, selection stops and returns the Smart Smoke set. The plugin writes a selection sidecar; during the suite, tests not in the list are skipped with reason smart-smoke.

You still get normal Playwright reporting and TestChimp execution ingest for the tests that ran. Skipped-as-smart-smoke is intentional—not a failure.

---

Review what the batch covered

After a Smart Smoke run finishes, open the batch execution in TestChimp (Test Executions → the invocation for that run). In the Overview pane, use View in Semantic Canvas.

That opens Semantic Canvas with Tests loaded and borders set to Execution status from batch… for that invocation. Pass / fail / skipped paint on the map so you can see which automation neighborhoods the smoke batch exercised—and which nearby tests stayed out of budget (neutral border when a test was not in the batch).

You can also pick Execution status from batch… on the Tests type card in Semantic Canvas and choose any recent batch from the picker—the same overlay used by the batch-viewer deeplink.

---

Enablement (same runner)

export TESTCHIMP_SMART_SMOKE_ENABLED=true
export TESTCHIMP_BRANCH_NAME="$(git branch --show-current)"   # local / agent runs
# optional: RELATED_TESTS_ONLY, MAX_TIME_BUDGET_MINS, etc.
npx playwright test

There are no Playwright CLI flags for Smart Smoke. See Configuration.

---

Related vs full suite

	Full suite	Smart Smoke
Goal	Exhaustive confidence	Max meaning per minute
Typical trigger	Nightly / release	PR / /testchimp test Phase 5
Selection	All (or manual tags)	Related ∪ tagged ∪ packed
Skip reason	—	smart-smoke for non-selected

Use Smart Smoke to keep the merge gate honest and fast. Keep full regression where breadth matters more than latency.

See also

• Introduction — when to use
• Configuration — env, config, CI
• Semantic Canvas — review batch coverage on the similarity map
• Run Smart Smoke — agent workflow checklist