How to check that AI-written Playwright tests can actually fail

A green suite only tells you something if the tests could have gone red.

That sounds obvious, but it is the problem I keep hearing from teams that let Claude Code, Cursor or Codex write their end-to-end tests. A recent r/ClaudeCode thread put it well. The author gave the agent every advantage: data-testids in the markup, seed credentials in CLAUDE.md, an architecture doc and Playwright MCP. The suite still had:

• data-testids that exist nowhere in the app
• a query parameter the app never reads
• a test navigating to a route that does not exist
• test.skip() used every time a test needed data that wasn't already there
• after a feature was deleted entirely, one test stayed green, because its assertion checked for text anywhere on the page

The last one is the dangerous kind. Broken tests get noticed. Tests that pass while the feature is broken don't.

This post covers the checks we use to catch these, in rough order of cost. The Playwright snippets were run against @playwright/test 1.64.1.

Why agent-written tests pass for the wrong reasons

These patterns show up whichever model writes the tests:

Pattern	What it looks like	Why it passes
Assertion too broad	expect(page.getByText('Hotel Aurora')).toBeVisible()	The text also appears in a header, sidebar or recent-items list
Assertion on the action, not the outcome	await expect(saveButton).toBeEnabled() after clicking save	Proves the button exists, not that anything was saved
Skipped instead of seeded	test.skip(!booking, 'no booking found')	Skipped tests are not failures in most CI views
Soft or missing assertions	A test that only clicks and navigates	No expect at all, so nothing can fail except a timeout
Mocked away	page.route('**/api/**', ...) returning a fixture	The real backend is never exercised

Agents tend to do these things because they are rewarded for getting the suite to go green. A comment in the r/Playwright thread on Test Agents describes the same thing as a "fix-run-fail-fix loop". If the only signal is green or red, green wins.

Check 1: Make the agent prove the test goes red first

The cheapest check is the old TDD rule. A new test should fail before the change and pass after it. For a bug fix, ask the agent to run the test against the unfixed code and paste the failure output into the PR description.

You can enforce part of this in CI by running the new or changed spec files against the base branch:

# In a PR job (checkout with fetch-depth: 0): which spec files changed?
git fetch origin main
CHANGED=$(git diff --name-only origin/main...HEAD -- 'tests/**/*.spec.ts')
[ -z "$CHANGED" ] && exit 0  # no new or changed specs, nothing to check

# Check out the app code from main, keep the new tests
git checkout origin/main -- src/
npx playwright test $CHANGED --reporter=line && {
  echo "New tests pass on main. Do they test the change?"
  exit 1
}

This only fits tests written for a behaviour change. For tests that cover existing behaviour, you need the next check.

Check 2: Break the app on purpose with page.route

You don't need a full mutation testing tool to see whether a test is sensitive to the thing it claims to cover. Playwright's network routing lets you corrupt one response and confirm the test fails.

Say the agent wrote this:

// tests/search.spec.ts
import { test, expect } from '@playwright/test';

test('search filters results by destination', async ({ page }) => {
  await page.goto('/search?destination=lisbon');
  await expect(page.getByText('Hotel Aurora')).toBeVisible();
});

Now write a "should fail" twin that removes the expected hotel from the API response:

// tests/sabotage/search.sabotage.spec.ts
import { test, expect } from '@playwright/test';

test('search test notices when results are wrong', async ({ page }) => {
  test.fail(true, 'sabotaged response: the assertion must not pass');

  await page.route('**/api/search*', async (route) => {
    const response = await route.fetch();
    const json = await response.json();
    json.results = json.results.filter((h: { name: string }) => h.name !== 'Hotel Aurora');
    await route.fulfill({ response, json });
  });

  const searched = page.waitForResponse('**/api/search*');
  await page.goto('/search?destination=lisbon');
  await searched; // make sure the sabotaged response actually arrived
  // Same assertion as the real test. We EXPECT this to fail.
  await expect(page.getByText('Hotel Aurora')).toBeVisible({ timeout: 3000 });
});

Playwright's test.fail() marks the test as expected to fail. It is reported as passed when it fails, and as an error when it unexpectedly passes.

If the sabotage test passes, your real test is too broad. In this example it passes when "Hotel Aurora" also appears in a "recently viewed" sidebar. That is exactly the failure from the Reddit thread.

Two traps we hit while testing this:

• Wait for the sabotaged response. Without await searched, the assertion can pass on the sidebar before the search results load. The test then ends while the route handler is still running, Playwright reports a "Test ended" error from route.fetch, and test.fail() counts that error as the expected failure. The sabotage twin looks fine while proving nothing.
• The real test has the same race. getByText('Hotel Aurora') can match the sidebar before results render, so the original test passes even when search is slow or broken. Scoping the assertion (next check) fixes both.

Don't write these for every test. Write them for the flows that make you money (signup, checkout, billing) and for any test that has never once failed in CI.

Check 3: Scope assertions to the part of the page that matters

The fix for broad assertions is usually scoping. Playwright 1.64 added Locator.within(), which reads naturally:

const results = page.getByRole('region', { name: 'Search results' });
const hotel = page.getByRole('heading', { name: 'Hotel Aurora' });

await expect(hotel.within(results)).toBeVisible();

Chaining (results.getByRole(...)) works the same way on older versions. Either way, the sidebar can no longer make the test pass.

Better still, assert on the outcome instead of on the presence of text:

const cards = page.getByRole('article').within(results);
await expect(cards).toHaveCount(3);
await expect(cards.first()).toContainText('Lisbon');

Check 4: Catch the lazy patterns with lint, not review

Some patterns should never reach a PR. eslint-plugin-playwright already has rules for most of them:

// eslint.config.mjs
import playwright from 'eslint-plugin-playwright';

export default [
  {
    ...playwright.configs['flat/recommended'],
    files: ['tests/**/*.ts'],
    rules: {
      ...playwright.configs['flat/recommended'].rules,
      'playwright/no-skipped-test': 'error',
      'playwright/expect-expect': 'error',
      'playwright/no-wait-for-timeout': 'error',
      'playwright/no-networkidle': 'error',
      'playwright/no-conditional-in-test': 'error',
      'playwright/no-force-option': 'error',
    },
  },
];

expect-expect catches tests with no assertions. no-skipped-test catches the "skip instead of seed" habit. no-networkidle matters because networkidle is itself a common source of flakiness. The Zenn team found this when their own agent skill recommended it (write-up).

Put the same rules in your CLAUDE.md, AGENTS.md or .cursor/rules so the agent avoids them in the first place. Keep lint as the backstop, because agents don't always follow instructions.

Check 5: Remove random failures so red means something

Flaky tests teach a team to ignore red, and then the checks above don't matter. Three recent Playwright features help here:

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  retryStrategy: 'isolated', // 1.62: retries run at the end, one at a time
  use: { trace: 'retain-on-failure' },
});

# 1.64: random order surfaces tests that depend on each other
npx playwright test --shuffle
# reproduce with the printed seed
npx playwright test --shuffle 271828182

// 1.63: tests sharing a resource never run at the same time
test('update account settings', { lock: 'user-settings' }, async ({ page }) => {
  // ...
});

Sources: Playwright release notes for 1.62, 1.63 and 1.64. Treat a test that passes only on retry as a bug to fix.

Check 6: Seed data so there is nothing to skip

Agents use test.skip() when the state they need isn't there. Give them a fixture that creates it through your API, so skipping is never the easy path:

// tests/fixtures.ts
import { test as base } from '@playwright/test';

type Booking = { id: string; propertyName: string };

export const test = base.extend<{ booking: Booking }>({
  booking: async ({ request }, use) => {
    const res = await request.post('/api/test/bookings', {
      data: { property: 'hotel-aurora', nights: 2 },
    });
    const booking = (await res.json()) as Booking;
    await use(booking);
    await request.delete(`/api/test/bookings/${booking.id}`);
  },
});
export { expect } from '@playwright/test';

Tell the agent in your rules file: "If a test needs data, use or add a fixture in tests/fixtures.ts. Never skip."

A checklist you can paste into your PR template

### E2E tests in this PR
- [ ] Each new test was seen failing (output pasted, or the CI base-branch check ran)
- [ ] Assertions are scoped to a region/component, not the whole page
- [ ] Assertions check an outcome (data, count, state), not just that an element exists
- [ ] No test.skip, waitForTimeout, networkidle or force: true (lint passes)
- [ ] Test data comes from fixtures, not from whatever is in the DB
- [ ] Critical flows have a sabotage twin marked test.fail()
- [ ] Ran with --shuffle at least once

Where this leaves you

None of this is new testing theory. What changed is the author: very fast, very eager to see green, and unaware of which parts of your page matter. The checks that scale best are the ones the agent can't skip: lint rules, the base-branch check and sabotage twins on your money paths.

We hit the same problem building TestChimp. That is why, in TestChimp, a test only gets a "verified" badge after a person on the team has inspected it, and an agent's later fix can flag that badge as stale for another look. You don't need a platform to get most of the value, though. The checklist above is a good start.

FAQ

How do I know if an AI-generated test is useful?

Break the behaviour it claims to cover and see if it fails. Corrupt the API response with page.route, remove the feature behind a flag, or run the test against the base branch. If it stays green, it is not testing what its name says.

Should I let an agent auto-fix failing Playwright tests?

Only with guardrails. Allow fixes to locators and waits. Block any change that removes or loosens assertions, adds retries or adds test.skip, and require a human to review the diff. In the r/Playwright thread above, people found the healer agent useful for selector drift and weak on timing problems.

Is mutation testing practical for end-to-end tests?

Full mutation testing is usually too slow for E2E suites. A cheaper middle ground is targeted fault injection: a few "sabotage" tests on critical flows that corrupt network responses and are marked test.fail().

Which lint rules catch the most common agent mistakes?

From eslint-plugin-playwright: expect-expect, no-skipped-test, no-wait-for-timeout, no-networkidle, no-force-option and no-conditional-in-test.

What Playwright version do I need for Locator.within() and --shuffle?

Playwright 1.64. Test locks need 1.63 and retryStrategy: 'isolated' needs 1.62. On older versions, chain locators and use --repeat-each to look for flakiness.