UI-Only Assertions Miss Backend Bugs
Short answer
Green toasts and “Success!” banners are not proof of business outcomes. Probe Assert on API or DB state is authoritative; UI checks are optional polish. This is the core lesson of the expired-coupon pattern.
Part of Common E2E testing gotchas.
Symptom
- E2E passes; production shows unpaid orders, missing refunds, or wrong tenant data
- Webhook handler stubbed in staging but UI still shows success
- Toast text updated; test still passes while API returns 500
Root cause
UI is eventually consistent and optimistic. Clients render success before server confirms—or show generic errors while partial writes occurred.
Fix: probe-first Assert
await page.getByRole('button', { name: 'Apply coupon' }).click();
// Optional UI — do not rely on alone
await expect(page.getByText(/discount applied/i)).toBeVisible({ timeout: 5_000 }).catch(() => {});
// Authoritative
await expect.poll(async () => {
const res = await request.get(`/api/test/probe-cart/${cartId}`);
return (await res.json()).discountCents;
}, { timeout: 15_000 }).toBe(500);
Build probes in seed routes guide.
Example scenario
Situation: Shopper applies an expired coupon at checkout.
Expected outcome: No discount; no order created.
Why UI-only automation breaks: Toast says 'Invalid coupon' but order row exists with discount applied.
- Arrange: Seed cart with expired coupon via /api/test/seed-cart for this runId.
- Act: Submit checkout in UI.
- Assert: Probe shows zero paid orders and discountCents=0; UI error optional.
TestChimp workflow: Link spec to markdown scenario with scenario annotations; /testchimp test keeps probe Assert when UI copy changes.
Same Arrange/Act/Assert pattern as expired-coupon checkout.
When UI asserts are OK
| OK | Not OK alone |
|---|---|
| Error message visible after probe confirms failure | Toast text without probe |
| Button disabled during submit | Button enabled = success |
| Accessibility role present | Snapshot of entire page |
ExploreChimp and non-functional gaps
Functional probes catch wrong state; they miss confusing errors and slow flows. Run ExploreChimp on SmartTest paths after probe coverage exists (non-functional guide).
Anti-patterns
| Anti-pattern | Why it fails | Better approach |
|---|---|---|
| Assert redirect URL only | Handler never ran | Poll probe for paid status |
| Mock API in UI test only | Integration gap | Real route + test env stub at network boundary |
| Snapshot checkout page | Copy change | Probe + structure |
TestChimp workflow
Document business rules in markdown test plans—agents authoring via /testchimp test add probes when scenarios say “no order created.” Requirement traceability shows which specs enforce revenue rules.
Related
Frequently asked questions
Is asserting API responses in Playwright an anti-pattern?
No—probe endpoints in test env are the recommended Assert layer. Avoid brittle coupling to internal schema in UI tests; use stable probe fields.
We only have UI access—no test APIs yet?
Add minimal /api/test/probe-* routes gated to non-prod. /testchimp init scaffolds this—cheaper than production incidents.
Should we delete all UI assertions?
Keep lightweight UI checks for regressions users see—after probes pass. Probes gate release; UI optional.
How do record-replay tools handle this?
They assert what was recorded—usually visible UI. Without probes, backend regressions slip through.
Can ExploreChimp replace probes?
No—ExploreChimp finds UX and non-functional issues on paths you already cover functionally. Probes stay authoritative for business state.
How do scenarios help?
Markdown plans state invariants (no order on expired coupon). scenario annotation links make missing probes obvious in coverage reports.
Apply these patterns in your repo
Run `/testchimp init` to connect TestChimp to your repo, then `/testchimp test` on PRs to turn these patterns into maintained SmartTests. Use `/testchimp evolve` when you want to expand coverage as your app grows.