Hardcoded Test Data Flakes in CI
Short answer
test@example.com, PROMO2024, and "tomorrow's date" in specs work until someone uses them in manual QA, a prior run exhausts the coupon, or CI timezone differs. Mint data in seed routes per runId—never assume staging rows stay pristine.
Part of Common E2E testing gotchas.
Symptom
- Spec failed Monday, passed Tuesday—no code change
- "User already exists" or "Coupon limit reached" on fresh branches
- Date-bound tests fail on UTC vs local midnight
- Manual QA reports "I used that test account"
Root cause
Specs assume immutable staging fixtures that are actually shared and mutable:
- Fixed emails, phones, SSNs in
fill() - Promo codes checked into git
new Date()for "expires tomorrow" assertions- SKU with inventory 1 consumed by parallel workers
Hardcoded data is the Arrange half of parallel CI collisions and world-state mutation.
Fix: mint data in Arrange
Identifiers — never hard-code
const runId = `data-${test.info().workerIndex}-${Date.now()}`;
const email = `e2e-${runId}@test.local`;
await request.post('/api/test/seed-user', { data: { runId, email } });
await page.getByLabel('Email').fill(email);
Coupons and entitlements — create per run
const { code } = await request.post('/api/test/seed-coupon', {
data: { runId, discountPercent: 15, maxRedemptions: 1 },
}).then(r => r.json());
Dates — freeze clock or seed absolute instants
await page.clock.install({ time: new Date('2025-06-15T12:00:00Z') });
// or seed route sets subscription trialEnd to known ISO string
See dates and timezones.
Assert outcomes via probe—not leftover UI text
await expect.poll(async () => {
const probe = await request.get(`/api/test/probe-user?runId=${runId}`).then(r => r.json());
return probe.plan;
}).toBe('pro');
Full harness: seed routes and probe Assert.
Data hygiene checklist
| Hard-coded | Replace with |
|---|---|
test@company.com | e2e-${runId}@test.local from seed |
SAVE20 in spec | seed-coupon response code |
product-id-42 | Seed catalog row keyed to runId |
2024-12-31 in assert | Clock install or seeded trialEnd |
| "First row in table" | Probe count for runId |
Anti-patterns
| Anti-pattern | Why it fails | Better approach |
|---|---|---|
| "Everyone knows" staging user | Manual QA collides | Mint per runId |
Delete row in afterEach only | Races under parallel | Isolated seed at Arrange |
| Copy prod snapshot to staging | PII + drift | Synthetic seed API |
| Skip test when coupon used | Coverage rots | New coupon per run |
faker in spec without DB key | UI unique, DB orphan | Seed route owns insert |
TestChimp workflow
/testchimp init replaces hard-coded Arrange blocks with seed route calls in generated fixtures. /testchimp test on PRs rewrites specs that still embed test@ emails when agents read scenario markdown—keeps SmartTests aligned with isolation rules.
Related
Frequently asked questions
Is faker enough for unique test data?
Faker helps UI strings but does not insert DB rows. Pair unique strings with seed routes that persist entities keyed to runId—or probes will not find them.
Our staging DB is reset nightly—why do we still flake?
PRs run many times per day; parallel workers collide within hours. Nightly reset does not fix intra-day or parallel contention.
Can we use a CSV of test users?
Static CSVs exhaust like hard-coded emails. Generate users programmatically or via seed API per run.
How do we handle one-time promo codes?
Seed route creates a fresh code with maxRedemptions: 1 scoped to runId. Never check a shared code into git.
Date tests fail only in CI—timezone?
CI runs UTC; laptops may be local. Use page.clock.install or seed absolute instants—see date/timezone pattern guide.
Should teardown delete test users?
Isolation at Arrange is primary. Teardown is backup; under parallel CI, another worker may read before delete completes.
Does /testchimp init remove hard-coded data?
It scaffolds seed routes and fixture patterns agents use to replace fixed emails and coupons in SmartTests—run it once per repo, then /testchimp test maintains specs on PRs.
What about third-party sandboxes (Stripe test mode)?
Still mint unique customer ids and payment methods per runId where the API allows—shared Stripe test clocks and customers collide like DB rows.
Apply these patterns in your repo
Run `/testchimp init` to connect TestChimp to your repo, then `/testchimp test` on PRs to turn these patterns into maintained SmartTests. Use `/testchimp evolve` when you want to expand coverage as your app grows.