Probe Assert vs UI Assertions — When Each Wins
Short answer
Probe Assert wins when business truth lives in API, DB, or async jobs—orders, balances, entitlements, webhooks. UI assertions win for layout, copy, accessibility, and navigation the user actually sees. Authoritative E2E uses probes first; UI checks are optional polish. The expired-coupon pattern is the canonical example.
The assertion split
E2E tests must answer: did the product do the right thing? UI and backend often disagree:
- Clients show optimistic success before server confirms
- Toasts use generic copy that passes while partial writes fail
- Webhooks and jobs finish after the UI already moved on
- A/B tests change banners without changing business rules
Treating visible text as proof is the #1 gotcha in startup E2E.
Probe Assert — authoritative state
A probe is a test-only endpoint (or direct query) that returns structured business state:
// Probe answers: "What is the cart discount in cents?"
const res = await request.get(`/api/test/probe-cart/${cartId}`);
const { discountCents, orderStatus } = await res.json();
Build probes alongside seed routes—Arrange creates the world; Assert reads it back.
When probe Assert wins
| Situation | Why UI fails | Probe checks |
|---|---|---|
| Payments & billing | Redirect pages lie; 3DS async | Invoice status, charge amount |
| Coupons & pricing | Toast before recalc | Line items, tax, discount cents |
| Auth & tenancy | Cookie set but claims wrong | Session probe, role flags |
| Webhooks & jobs | UI idle while queue pending | Job status poll via probe |
| Inventory & ledger | "Added to cart" optimistic | Stock count, reservation row |
| Multi-step checkout | Step indicator ahead of API | Order draft state |
Expired-coupon pattern (canonical)
The expired-coupon pattern demonstrates the full stack:
- Arrange — seed route creates cart with known SKU and coupon slot
- Act — Playwright applies an expired code at checkout
- Assert (probe) — discount remains zero; no order row with discount applied
- Assert (UI, optional) — error message visible—never the only check
UI might show "Invalid coupon" while the backend incorrectly applied a discount. Only the probe catches that.
Example scenario
Situation: Shopper applies an expired coupon at checkout.
Expected outcome: No discount; no order created.
Why UI-only automation breaks: Toast says 'Invalid coupon' but order row exists with discount applied.
- Arrange: Seed cart with SKU and expired coupon code via test API.
- Act: Playwright enters coupon and clicks Apply.
- Assert: Probe: discountCents === 0 and orderStatus === null.
TestChimp workflow: TrueCoverage showed rising checkout errors for coupon edge cases before manual QA scheduled.
Same Arrange/Act/Assert pattern as expired-coupon checkout.
UI assertions — what the user experiences
UI checks remain valuable—they just are not authoritative for business outcomes.
When UI assertion wins
| Situation | What to assert | Notes |
|---|---|---|
| Navigation & routing | URL, active tab, breadcrumb | User cannot proceed if lost |
| Form validation (client) | Inline error, disabled submit | Before server round-trip |
| Layout & responsive | Element visible, not clipped | ExploreChimp complements this |
| Copy & compliance | Legal text, pricing disclaimer | Regulated surfaces |
| Accessibility | Roles, labels, focus order | getByRole over brittle CSS |
| Empty & error states | Skeleton, retry button | UX contract, not DB row |
When UI is supplementary
Use UI asserts after probe confirms backend truth—or as non-blocking polish:
// Authoritative — must pass
await expect.poll(async () => {
const res = await request.get(`/api/test/probe-order/${orderId}`);
return (await res.json()).status;
}, { timeout: 15_000 }).toBe('paid');
// Optional UX — nice to have
await expect(page.getByText(/thank you for your order/i)).toBeVisible();
If the probe fails, the test fails—regardless of toast text.
Decision matrix
| Question | If yes → | If no → |
|---|---|---|
| Does correctness depend on DB/API/job state? | Probe Assert | UI may suffice |
| Can UI show success before async work completes? | Probe Assert | UI assert OK |
| Is the risk copy/layout/accessibility only? | UI assert | Add probe if money/identity involved |
| Does prod bug report say "UI looked fine"? | Probe Assert | Revisit UI scope |
| Parallel CI shares staging entities? | Fix Arrange (seed) + probe | — |
Combining both (recommended pattern)
Arrange → seed route (per runId)
Act → Playwright (short path)
Assert → probe poll (required)
Assert → UI locator (optional)
| Layer | Failure means |
|---|---|
| Probe | Ship blocker — business rule broken |
| UI | UX regression — fix or demote to warning |
Link specs to markdown scenarios: // @Scenario: checkout/expired-coupon (traceability).
Common mistakes
| Mistake | Symptom | Fix |
|---|---|---|
| Toast-only Assert | Prod incidents, green CI | Probe-first guide |
| Probe without Arrange | Flaky polls | Per-run seed with same runId |
waitForTimeout then UI check | Random flakes | expect.poll on probe (flaky waits) |
| Asserting API in UI test ad hoc | Duplicated query logic | Centralize probe routes |
| Deleting probe to unblock CI | Coverage hole | Fix harness; check TrueCoverage |
Full symptom write-up: UI-only assertions miss backend bugs.
How TestChimp enforces the split
/testchimp init— scaffolds seed and probe routes so Assert has an authoritative hook- SmartTests — Playwright Act with probe Assert in the same spec
/testchimp test— agents repair probes and UI locators together in PR context- TrueCoverage — prioritizes probe gaps on high-traffic prod paths
See TestChimp approach to test automation and E2E foundations.
Frequently asked questions
Should we remove all UI assertions from E2E?
No—keep UI asserts for navigation, layout, accessibility, and copy. Remove UI as the *only* proof of business outcomes. Probes assert money, identity, and async state; UI asserts what the user sees.
What is a probe in E2E testing?
A test-only API or query endpoint that returns structured business state—cart totals, order status, subscription row—so Assert does not rely on optimistic UI. Build probes next to seed routes in `/testchimp init`.
Why does the expired-coupon example use probes?
Coupons touch pricing engines and order rows. UI can show an error while a discount still applies. The expired-coupon pattern seeds a cart, applies the code in Playwright, and polls a probe for discount cents and order existence.
Our tests pass UI but prod fails—what changed?
Add probe Assert on the failing business dimension first. Read the UI-only gotcha guide, then scaffold probes via seed routes. Agents can add probes in the same PR via `/testchimp test`.
Make probes authoritative in your E2E suite
Scaffold seed routes and probe Assert with /testchimp init—stop trusting toasts alone.