UI-Only Assertions Miss Backend Bugs
Short answer
Green toasts and “Success!” banners are not proof of business outcomes. Probe Assert on API or DB state is authoritative; UI checks are optional polish. This is the core lesson of the expired-coupon pattern.
Part of Common E2E testing gotchas.
Symptom
- E2E passes; production shows unpaid orders, missing refunds, or wrong tenant data
- Webhook handler stubbed in staging but UI still shows success
- Toast text updated; test still passes while API returns 500
Root cause
UI is eventually consistent and optimistic. Clients render success before server confirms—or show generic errors while partial writes occurred.
Fix: probe-first Assert
await page.getByRole('button', { name: 'Apply coupon' }).click();
// Optional UI — do not rely on alone
await expect(page.getByText(/discount applied/i)).toBeVisible({ timeout: 5_000 }).catch(() => {});
// Authoritative
await expect.poll(async () => {
const res = await request.get(`/api/test/probe-cart/${cartId}`);
return (await res.json()).discountCents;
}, { timeout: 15_000 }).toBe(500);
Build probes in seed routes guide.
Example scenario
Situation: Shopper applies an expired coupon at checkout.
Expected outcome: No discount; no order created.
Why UI-only automation breaks: Toast says 'Invalid coupon' but order row exists with discount applied.
- Arrange: Seed cart with expired coupon via /api/test/seed-cart for this runId.
- Act: Submit checkout in UI.
- Assert: Probe shows zero paid orders and discountCents=0; UI error optional.
TestChimp workflow: Link spec to markdown scenario with // @Scenario:; /testchimp test keeps probe Assert when UI copy changes.
Same Arrange/Act/Assert pattern as expired-coupon checkout.
When UI asserts are OK
| OK | Not OK alone |
|---|---|
| Error message visible after probe confirms failure | Toast text without probe |
| Button disabled during submit | Button enabled = success |
| Accessibility role present | Snapshot of entire page |
ExploreChimp and non-functional gaps
Functional probes catch wrong state; they miss confusing errors and slow flows. Run ExploreChimp on SmartTest paths after probe coverage exists (non-functional guide).
Anti-patterns
| Anti-pattern | Why it fails | Better approach |
|---|---|---|
| Assert redirect URL only | Handler never ran | Poll probe for paid status |
| Mock API in UI test only | Integration gap | Real route + test env stub at network boundary |
| Snapshot checkout page | Copy change | Probe + structure |
TestChimp workflow
Document business rules in markdown test plans—agents authoring via /testchimp test add probes when scenarios say “no order created.” Requirement traceability shows which specs enforce revenue rules.
Related
Frequently asked questions
Is asserting API responses in Playwright an anti-pattern?
No—probe endpoints in test env are the recommended Assert layer. Avoid brittle coupling to internal schema in UI tests; use stable probe fields.
We only have UI access—no test APIs yet?
Add minimal /api/test/probe-* routes gated to non-prod. /testchimp init scaffolds this—cheaper than production incidents.
Should we delete all UI assertions?
Keep lightweight UI checks for regressions users see—after probes pass. Probes gate release; UI optional.
How do record-replay tools handle this?
They assert what was recorded—usually visible UI. Without probes, backend regressions slip through.
Can ExploreChimp replace probes?
No—ExploreChimp finds UX and non-functional issues on paths you already cover functionally. Probes stay authoritative for business state.
How do scenarios help?
Markdown plans state invariants (no order on expired coupon). // @Scenario: links make missing probes obvious in coverage reports.
Apply these patterns in your repo
Run `/testchimp init` to connect TestChimp to your repo, then `/testchimp test` on PRs to turn these patterns into maintained SmartTests. Use `/testchimp evolve` when you want to expand coverage as your app grows.