Over-Mocking E2E Misses Integration Bugs
Short answer
page.route stubbing every JSON response makes specs fast and green while auth, validation, and DB writes never run. E2E should hit real app routes in a test environment and probe authoritative state—mock only non-deterministic boundaries (LLM, payment webhooks, third-party quotas) with AIMock for models, not your own REST layer.
Part of Common E2E testing gotchas.
Symptom
- E2E passes; production 500s on checkout or chat send
- Refactor breaks API contract; UI tests still green
- Mock returns
{ ok: true }while handler never persisted - "Integration tests" are Playwright with MSW intercepting your own backend
- Chat specs assert mocked text; tool calls and RAG retrieval never exercised
Root cause
E2E collapses into UI unit tests by mocking the wrong layer:
page.route('**/api/**', route => route.fulfill({ body: '{}' }))at file top- MSW handlers duplicate production shapes that drift silently
- LLM mocked—but so are cart, auth, and thread APIs
- No probe Assert; mocks define expected behavior, not the server
Unit tests should mock collaborators; E2E should prove wiring across UI, API, and persistence—with narrow exceptions.
Fix: real routes, narrow mocks, probe Assert
1. Default: no mock for your own API
test('apply coupon updates cart', async ({ page, request, runId }) => {
await request.post('/api/test/seed-cart', {
data: { runId, cartId: `cart-${runId}`, couponCode: 'SAVE10' },
});
await page.goto(`/cart?cartId=cart-${runId}`);
await page.getByRole('button', { name: 'Apply' }).click();
await expect.poll(async () => {
const res = await request.get(`/api/test/probe-cart?runId=${runId}`);
return (await res.json()).discountCents;
}).toBe(1000);
});
Real /api/cart/apply runs—validation, auth middleware, and DB update included. See seed routes and probe Assert.
2. Mock only external or non-deterministic boundaries
| OK to stub | Keep real |
|---|---|
| LLM completion (AIMock) | Your chat API route + tool router |
| Stripe webhook replay fixture | Create-checkout session route |
| SendGrid send (capture payload) | User signup + DB row |
| Third-party geocoder quota | Address validation handler |
For conversational features: stub upstream model, not thread persistence—conversational UI guide.
3. AIMock for LLM only (not your REST layer)
// SmartTests / runner — AIMock at model boundary
await aiMock.setResponse({
threadId: runId,
messages: [{ role: 'assistant', content: 'Your refund is processed.' }],
});
await page.goto(`/support?threadId=${runId}`);
await page.getByRole('textbox', { name: 'Message' }).fill('Refund my order');
await page.getByRole('button', { name: 'Send' }).click();
await expect.poll(async () => {
const res = await request.get(`/api/test/probe-refund?runId=${runId}`);
return (await res.json()).status;
}).toBe('completed');
AIMock stabilizes wording; probes prove tool dispatch and side effects. Depth: AI web apps vertical and LLM output validation.
4. When page.route is acceptable
- Fault injection: 503 once to test error UI (then real route on retry)
- Third-party SDK you do not control in test env
- Not replacing your monolith API for convenience
Document injected faults in markdown scenarios; keep happy path on real routes.
Anti-patterns
| Anti-pattern | Why it fails | Better approach |
|---|---|---|
Mock all /api/* in Playwright | Zero integration coverage | Test env + seed/probe |
| Duplicate DTOs in MSW handlers | Drift from OpenAPI | Real handler or contract tests |
| AIMock on cart and chat APIs | Misses router bugs | AIMock on LLM only |
| Assert mocked JSON in UI test | Proves mock, not app | Probe DB state |
| No E2E because "we have unit tests" | Wiring bugs ship | Narrow E2E on critical paths |
TestChimp workflow
/testchimp init scaffolds real seed/probe routes—not blanket page.route stubs—so SmartTests exercise true integration paths. /testchimp test on PRs removes over-broad mocks when markdown scenarios require backend invariants; agents add probes and reserve AIMock for LLM steps per when to use ai.act.
Related
- UI-only assertions
- Conversational UI testing
- AI web apps vertical
- LLM output validation
- When to use ai.act / ai.verify
- Seed routes and probe Assert
- Playwright network
Frequently asked questions
Should Playwright tests mock API responses?
Sparingly—for third-party quotas, fault injection, or non-deterministic LLM output via AIMock. Your own REST routes should run for real in a test environment with seed and probe Assert, or you are testing mocks not integration.
What is AIMock vs mocking the whole app?
AIMock stubs upstream LLM responses while your API routes, tool router, auth, and UI run for real. Mocking every /api call turns E2E into UI unit tests that miss wiring bugs.
MSW in Playwright—good or bad?
MSW is fine for third-party or browser-only boundaries. Using MSW to replace your entire backend in E2E duplicates contracts and drifts silently—prefer test env servers and probes.
How do I test AI chat without mocking everything?
Seed thread with runId, AIMock model text for deterministic CI, hit real chat API routes, probe refund or tool side effects—see conversational UI guide.
Our E2E is fast because we mock APIs—is that OK?
Fast green builds that never hit your handlers do not catch auth, validation, or persistence bugs. Keep a small set of integration specs on real routes; mock only true external boundaries.
Unit tests vs E2E mocking boundaries?
Unit tests mock collaborators by design. E2E proves cross-layer wiring—UI to API to DB. Overlap is fine; E2E should not re-mock what unit tests already isolated unless injecting a fault.
Can TestChimp help remove over-mocking?
Yes—/testchimp init scaffolds seed and probe routes on real handlers. /testchimp test refactors specs that stub internal APIs when markdown scenarios state backend invariants agents can enforce with probes.
When is page.route still correct?
One-shot 503 fault injection, blocking analytics beacons, or stubbing a third-party SDK. Happy-path checkout, auth, and chat should not use blanket page.route on your own API.
Apply these patterns in your repo
Run `/testchimp init` to connect TestChimp to your repo, then `/testchimp test` on PRs to turn these patterns into maintained SmartTests. Use `/testchimp evolve` when you want to expand coverage as your app grows.