10 posts tagged with "Smart Tests"

TL;DR: Agents built the product. Agents authored the tests that verify the implementation. The missing piece is accountability—has anyone actually looked at the test to ensure it truly verifies the scenario? Verified Tests puts that sanity check in the platform: inspect screen captures, steps, and code, then stamp a Verified Badge on the covering scenario.

Full product guide: Verified Tests.

The missing piece is accountability​

Agents are very good at getting a path green.

The SmartTest runs. The scenario is linked. Requirement coverage looks healthier than last sprint.

That is not the same as: this test actually verifies the behaviour the scenario describes.

Has anyone actually looked at the test?

A scenario annotation is cheap to add—and agents add them at speed. They can link the right scenario to a script that asserts a toast, skips the important check, or walks a cousin flow that happens to share a button. The dashboard still counts it as coverage.

We used to paper over that with implicit knowledge: the person who wrote the test was the person who knew what it proved. Agents broke that assumption. Volume went up. Inspection did not.

Green is an execution fact. Linked is a mapping. Verified is the accountability layer: a person has sanity-checked that the covering test truly verifies the scenario.

That is the problem Verified Tests solves.

Why this is necessary in the agentic era​

Before agents, test authoring was slow enough that “who wrote this?” was a reasonable proxy for “who checked this?”

That proxy is dead.

Agents implement. Agents author SmartTests. Agents attach scenario ids because the plan files told them to. You still need a durable record that a person opened the execution, looked at what actually happened, read the script, and agreed: yes, this test verifies that scenario.

Otherwise requirement traceability is a spreadsheet of claims with nicer UI. Coverage insights tell you what ran. They do not tell you whether the test is the right test.

Introducing Verified Tests​

TestChimp now lets you Verify tests in the platform—without bouncing out to a repo or a recording elsewhere.

Open a SmartTest execution and you already have what you need to make an informed decision:

• Screen captures from the run
• Steps the test actually took
• The code, via View Test, so you can read the script next to the evidence

Then use the check badge next to the SmartTest name. Same visual language as a verify badge elsewhere on the internet, because the job is the same: this was inspected.

Three states on the test:

• Hollow — none of the linked scenarios are verified
• Grey — some, not all
• Blue — every linked scenario for that test is verified

For one scenario, you confirm. For several, you verify per scenario—or Mark all as verified. Hover a filled badge and you see who verified it.

Un-verify is deliberate: it records manually unverified, not “never happened.” The audit trail stays.

Status lives in TestChimp, not in source. Agents can keep authoring Playwright. Humans stamp the claim when they have actually looked.

A Verified Badge on scenario coverage​

Plans, requirement coverage, and test runs already show recent execution results per scenario.

You now also see a Verified Badge on each scenario—an extra layer of assurance that the product is being tested properly, not only that something ran green.

• Filled — a user has sanity-checked a covering test for that scenario
• Hollow — the scenario has tests, but nobody has inspected them yet
• Omitted — no linked tests, so there is nothing to verify

Clicking the coverage-side badge does not flip the bit. It tells you how: open an execution, inspect the run, then use the check badge next to the test name. Verification is an inspection action, not a bulk paint on a dashboard.

That is the point. If it were too easy, we would have rebuilt the original lie at a larger scale.

How this fits the rest of TestChimp​

If you’ve been following along:

• Test Planning as Code puts stories and scenarios in Git so agents have a real task
• Scenario annotations declare which SmartTest claims which scenario
• Requirement traceability rolls execution results up the plan
• Verified Tests records that a human checked the claim—not just that CI was green

Planned reality → linked automation → inspected coverage → release confidence. The middle of that chain was the hole once agents started writing both sides.

Frequently asked questions​

What is a verified test?​

A verified test in TestChimp is a SmartTest a project member has inspected and confirmed actually covers a linked scenario. It is more than a green run or a scenario annotation: someone looked at the screen captures, steps, and code, and stamped the claim.

How is verified different from linking a test to a scenario?​

Linking (a Playwright annotation with type: 'scenario' and #TS-<n>) is a coverage claim. Verification is a human sanity check of that claim. You can have linked, green, and still unverified.

Where do I verify a SmartTest?​

Open a SmartTest execution (click a coverage square). Inspect the screen captures, steps, and test code, then use the check badge after the test name. Coverage-train badges are read-only hints.

What does the Verified Badge on a scenario mean?​

A filled badge means at least one covering SmartTest has been manually verified for that scenario. Hollow means tests exist but have not been sanity-checked yet. No linked tests → no badge.

Does renaming a scenario reset verification?​

No, as long as the scenario ordinal (#TS-n) stays the same. Linking a different scenario is a new claim and starts unverified.

Can I un-verify?​

Yes. Confirming un-verify sets the link to manually unverified (distinct from never verified). Who/when hover applies to the verified state.

What if the test has no linked scenarios?​

It cannot be marked verified. There is no scenario claim to inspect. Link the scenario first.

When agents run your SDLC, trust—but verify.

Docs: Verified Tests · Link tests to scenarios · Requirement traceability.

Your agents have been writing tests for a few months, and now you have 500+ E2E tests taking an eternity to run. Sounds familiar?

Say hello to Smart Smoke.

With agents authoring tests en masse, the problem has changed. Coverage gaps close fast—but:

• Your CI bill keeps growing as you run the full suite on every PR
• Worse, you wait hours before knowing if anything broke

Today, smoke suites are often manually managed. Tag lists. Folder filters. A @smoke set someone curated last quarter. That leaves a lot of useful signals on the table—especially as test suites grow at an unprecedented pace.

What really interested me here—as a lover of algorithms—was that this is essentially a classic optimization problem:

Given N minutes, how do you maximize semantic coverage + PR impact?

First: find what the PR can break​

An agent identifies the tests relevant to the change—impacted scenarios from your plans, linked SmartTests via scenario annotations, plus anything newly authored on the branch. That related set lands in plans/smart-smoke/<branch>/related-tests.json.

That’s the seed. Then comes the fun part: covering as much ground as possible.

Paint the canvas within a fixed budget​

Imagine your tests laid out as nodes on a semantic canvas. The problem becomes:

Paint the canvas by selecting test nodes to maximize the covered spread—within a fixed time budget.

We don’t just grab the nearest cluster. We iteratively pick the next test that adds the most new ground—so selection spreads across the plane instead of camping in one corner.

Then weigh the terrain​

Pure geometry isn’t enough. We weigh the terrain using signals like scenario priority, recency, test stability, execution time, historical failures—so packing prefers tests that are both informative and practical.

Seeds always include related tests, tagged smoke (e.g. @smoke), and newly authored branch tests. Packing fills whatever budget remains—until the bin is full.

The result: the best subset of tests to run as smoke for a given PR—giving you the most confidence within a fixed time budget.

Less CI cost. Less waiting. More confidence.

Same Playwright command​

Smart Smoke isn’t a new runner. You keep your SmartTests and the same npx playwright test—opt in per run:

export TESTCHIMP_SMART_SMOKE_ENABLED=true
# optional: time budget, suite %, tags, related-tests-only
npx playwright test

Non-selected tests skip with reason smart-smoke (distinct from an explicit test.skip).

It plugs into /testchimp test as Phase 5, or standalone via /testchimp run smart smoke.

Full reference: Smart Smoke Runs · How it works · Configuration.

Plugin: @testchimp/playwright (≥ 0.2.20).

Smoke responsibly.

TL;DR: We’ve shipped API contract governance in TestChimp. Point us at your OpenAPI roots, opt in to API payload capture on SmartTest runs, and the APIs surface shows which operations—and which request/response fields and status codes—your automation actually exercised. Uncovered holes become Create issue, Create test, or Ignore gap—accountable work, not a spreadsheet of “we should cover that someday.”

The contract nobody audits​

Your OpenAPI (or Swagger) file is not documentation fluff. It is the business contract of the product: methods, paths, request shapes, response codes, and fields clients depend on.

UI E2E already touches that contract. Checkout hits POST /orders. Declined payment hits POST /payments. A rare enum or a 422 body? Often never.

Without a contract lens, teams hit the same failure modes:

API contract governance is not a separate API-testing product. It is the audit and triage layer that turns traffic your SmartTests already generate into accountable coverage of the contract.

Full product guide: API Contract Governance.

How it works​

Three inputs. One surface.

1. Discover the contract — Configure OpenAPI root path(s) from your connected repo (APIs → Configure). TestChimp parses them into services and operations.
2. Observe traffic — Run SmartTests with TESTCHIMP_ENABLE_API_CAPTURE=1 (and @testchimp/playwright ≥ 0.2.17). Matching HTTP interactions upload with the batch; coverage denorm runs after—not on the hot ingest path.
3. Govern gaps — List view for the portfolio; detail view for schema fields and response codes. From any hole: file an issue, copy an agent prompt to create/update tests, or ignore the gap when it is intentional.

Capture is off by default so ordinary runs stay light. Opt in when you want Operations coverage—same pattern as other optional runtime features. Setup detail: Set up API contract governance and the runtime plugin.

From endpoint scores to schema fields​

The APIs list is the portfolio view: service picker, coverage-band summary, method/path, spec-defined vs observed-only, coverage score, latency, covering-test chips.

Click a row and you get the interesting part—field-level truth:

• Request schema and query parameters with covering tests per field
• Response schemas nested by status code
• undocumented tags when traffic (or denorm) sees a field or code the OpenAPI root never declared
• Empty covering tests when the contract exists but automation never exercised it

That is the difference between “we hit /orders” and “we never sent delivery.type = same-day, and nobody owns that hole.”

List and detail walkthroughs: Operations list · Operation detail.

Governance that sticks: issue, test, or ignore​

Coverage without triage is another dashboard nobody opens.

Wherever coverage is missing—operation, field, or response code—TestChimp offers the same decision set:

Agents already understand operation-id / field / response-code scopes. The UI just stops the gap from living only in someone’s head.

How this fits the rest of TestChimp​

If you’ve followed us:

• Requirement quality governance hardens the plan contract before agents spend tokens
• Release governance hardens the ship decision for a version
• API contract governance hardens what your automation proved against the HTTP contract—the surface clients and integrations actually depend on

SmartTests remain the traffic source. Issues remain the backlog. Create-tests remains the authoring path. The APIs page is where contract debt becomes visible and governable—next to TrueCoverage for behaviour-aligned RUM, not instead of it.

Planned reality → tested reality → production reality. The API surface was the missing middle for teams whose “coverage” stopped at green UI journeys.

Frequently asked questions​

What is API contract governance in TestChimp?​

API contract governance maps OpenAPI operations to real and mocked SmartTest traffic so you can see endpoint and schema-field coverage, then triage gaps with create issue, create test, or ignore gap.

Do I need dedicated API tests?​

No. Coverage is inferred from HTTP traffic your SmartTests already generate when capture is enabled. You can still author dedicated API specs; the governance surface tracks both.

Why is my coverage empty after configuring OpenAPI?​

Parsing the spec discovers operations. Scores and covering tests need runs with TESTCHIMP_ENABLE_API_CAPTURE=1. Capture is off by default on purpose.

What does Ignore gap do?​

It marks an uncovered operation, field, or response code as intentionally ungoverned for now—without creating an issue or a test—so teams can acknowledge debt without false urgency.

Spec defined No vs undocumented—what’s the difference?​

Spec defined No on an operation means the whole method/path is missing from the OpenAPI root (observed in traffic only). An undocumented tag on a field or response code means that leaf was seen but not declared—classic contract drift.

Try it​

1. Connect git and open APIs → Configure — select your OpenAPI root file(s)
2. Wait for parse/sync; pick the service in the list
3. Run SmartTests with TESTCHIMP_ENABLE_API_CAPTURE=1
4. Refresh APIs — open a low-score or uncovered operation
5. Create issue, Create test, or Ignore gap on the holes that matter

Start here:

• API Contract Governance
• Setup
• List view
• Detail view
• Playwright runtime plugin
• TestChimp skill

Know the contract. Cover the edge cases—before unaudited branches ship.

Further reading​

TestChimp

• API Contract Governance
• Set up API contract governance
• SmartTests runtime plugin
• Create tests
• Issue management

Related posts

• Release Governance: Ship When the Evidence Says So
• Requirement quality governance (DeFOSPAM)
• TrueCoverage
• Boiling the QA lake

Manual testing still finds what automation misses—but too often, the path from a good manual run to a reliable automated test is broken.

Teams try Playwright codegen or record-replay tools, get a script quickly, and then spend weeks fighting flakes: shared data, missing assertions, no link back to the scenario, and no fit with POMs or fixtures already in the repo.

Today we’re announcing a workflow we recommend for turning manual sessions into SmartTests: capture with traceability, then let a coding agent upskilled with TestChimp author automation that actually belongs in your codebase.

The problem with “just record it”​

Record-replay—including Playwright codegen—optimizes for mirroring UI clicks. That is not the same as authoring a repeatable test.

Real automation needs:

• Arrange: seed data, fixtures, run-scoped entities
• Act: the journey that matters (often shorter than what a human clicked through)
• Assert: UI checks and backend state where outcomes live

Recorders capture the act layer well. They usually skip arrange and assert, and they never know which business scenario you were proving.

The result is familiar: tests that pass once on a developer machine, then fail in CI because the world-state was never set up—or because the script asserts the wrong thing (or nothing at all).

What we do instead​

TestChimp connects manual execution, test planning, and agent-authored Playwright in one loop.

1) Capture the manual session—with scenario context​

Use the TestChimp Chrome extension Manual tab to record a session while exercising your app. Start from Test Planning so the scenario is pre-linked (recommended), or link a scenario as part of the workflow.

What gets stored:

• Step-by-step actions and screenshots
• Linked scenarios (business context)
• Environment and release metadata
• Pass/fail outcome and optional bugs/notes

The session is auditable manual evidence and the reference for automation—not a throwaway recording.

2) Generate prompt → coding agent​

Open the session in TestChimp (Executions → Manual Sessions) and click Copy test generate prompt. Paste it into your agent host (Cursor, Claude Code, etc.) with the TestChimp skill installed.

The agent pulls rich context via get-manual-session-details (CLI or MCP):

• Recorded steps
• Linked scenarios and scenario steps
• Screenshots for visual grounding
• Project layout and existing POMs, fixtures, seed/probe endpoints

It uses the manual walkthrough as reference, navigates the app to validate selectors, and writes a SmartTest that reuses your harness—not a blind replay file.

3) Continuous improvement—not one-shot codegen​

Authoring does not stop at the first green run. TestChimp’s feedback loop surfaces coverage gaps (planned scenarios and TrueCoverage behaviour signals). Your agent runs /testchimp test on PRs and /testchimp evolve on a schedule or after deploys to close gaps, extend fixtures, and keep tests aligned with how users actually behave (Workflows).

The Web IDE is where you view tests, run them, and see insights aligned with your test folder structure—not where we expect most authoring to happen anymore.

How this differs from record-replay vendors​

Tools like mabl, Katalon, and Testim (and codegen at the framework level) center on capture → replay. They can speed up first script creation, but they typically:

• omit fixture-backed world-state
• lack in-repo scenario traceability at authoring time
• rarely generate backend probe assertions
• produce tests that do not compose with your existing Playwright patterns

TestChimp’s manual-to-auto path is informed agent authoring: session + scenario + screenshots + your repo conventions → repeatable Playwright in Git. See the full comparison: Why record-replay falls short in creating repeatable tests.

When to use which path​

Get started​

1. Install the Chrome extension and add the TestChimp skill to your coding agent.
2. Capture a manual session from a linked scenario (manual test capture guide).
3. Copy test generate prompt and let the agent author the SmartTest (Creating SmartTests).
4. Wire /testchimp test into your PR flow and schedule /testchimp evolve for portfolio upkeep.

Manual testing stays human. Automation becomes engineering-grade—because the agent authors like an engineer who read the scenario, not like a recorder that only heard the clicks.

TL;DR: If your product ships both a web app and native mobile apps, you are probably maintaining two automation codebases that repeat the same Arrange logic—users, listings, payments, feature flags—before any UI step runs. TestChimp Multi-Platform Projects put Playwright (web), Mobilewright (iOS/Android), and API tests in one Git-connected scaffold, with shared business logic for world-state setup and platform-specific UI tests, coverage, and UX analytics. UI interactions stay platform-specific; test infrastructure does not have to—and neither does your requirements, TrueCoverage, or Atlas view of quality.

The hidden cost of “Appium for mobile, Playwright for web”​

Cross-platform products rarely differ at the data layer. A booking marketplace needs the same primitives whether the customer taps Book in Safari or in your iOS app:

• A test user with a known identity
• Inventory (for example, a few property listings)
• A valid payment method linked to that user
• Whatever else your domain requires before the flow under test is meaningful

None of that is inherently web or mobile. It is application state—the Arrange phase in the classic Arrange → Act → Assert model (Martin Fowler on Given-When-Then).

Yet the dominant split for years has been:

Teams end up with parallel helper libraries, duplicate seed scripts, and drift—web tests create users one way, mobile tests another, and failures become “which stack is wrong?” instead of “did we break the product?”

The Act and Assert steps should differ by surface: selectors, gestures, and viewport behaviour are platform-specific. The Arrange layer often should not.

Why Mobilewright changes the consolidation story​

Mobilewright brings native iOS and Android automation closer to the Playwright mental model: async tests, auto-waiting, project matrices in config, and fixtures that feel familiar if you already run npx playwright test.

That alignment matters for multi-platform engineering, not only for “mobile testing” as an isolated workstream:

• Same language and patterns (commonly TypeScript/JavaScript in one repo)
• Same CI habits (config projects, parallel workers, artifact uploads)
• Same opportunity to share code for factories, API clients, and database seeding

TestChimp already extended the plan → repo → agent → CI loop to native mobile (native mobile testing announcement). Multi-Platform Projects are the next step: one TestChimp project type and one tests tree for teams that ship web and mobile together.

What TestChimp Multi-Platform Projects provide​

When you create a TestChimp project with type Multi-Platform, the platform scaffolds a single tests/ directory that includes:

• web/ — browser SmartTests via Playwright (playwright.config.js, web/e2e/, web/pages/, web/fixtures/)
• mobile/ — native UI tests via Mobilewright (mobilewright.config.ts, mobile/e2e/common|ios|android/, mobile/pages/, mobile/fixtures/)
• api/ — platform-agnostic HTTP specs (often the fastest way to Arrange and to assert backend state)
• shared/ — cross-suite helpers and fixture factories (seed users, auth builders)—excluded from test discovery, intended for reuse
• setup/ — global setup run once before suites in both configs

Platform-specific UI code lives in platform-specific folders. Business logic that creates entities and prepares situations can live in shared/, api/fixtures/, or factories imported by both web and mobile specs.

tests/
  setup/
  shared/              ← shared Arrange logic (users, listings, payments, flags)
  api/
    fixtures/
  mobile/
    fixtures/
    pages/
    e2e/
      common/
      ios/
      android/
  web/
    fixtures/
    pages/
    e2e/
  playwright.config.js
  mobilewright.config.ts

Result for QA and platform teams:

• Less duplicated infrastructure — one place to update “premium user with saved card”
• Less maintenance — fix seeding once; web and mobile suites consume the same factories
• More consistency — the same world-state definitions drive cross-platform regression

Smart Steps (ai.act, ai.verify) remain web-only today; native mobile continues to use standard Mobilewright APIs for UI Act steps. For platform capabilities and CI notes, see Mobile testing.

One project, platform-specific coverage and UX intelligence​

Consolidating tests in one repo does not mean blending web and mobile into one misleading coverage number. Multi-Platform Projects keep one TestChimp project and one plans/tests Git mapping, while treating Web, iOS, and Android as first-class execution platforms everywhere insights matter.

Think of it as: shared requirements and shared Arrange code, sliced execution and analytics per surface.

Requirement traceability (Test Planning)​

Requirement traceability links scenarios in Git to SmartTest runs. On a Multi-Platform project, the Insights tab and scenario execution history respect an execution scope that includes platform alongside environment, release, branch, and time range.

• Choose Web, iOS, or Android to see which scenarios passed or failed on that surface.
• Drill into a user story to view execution history filtered to the platform you care about—useful when mobile lags web or when a shared scenario is covered by both web/e2e/ and mobile/e2e/ specs.
• Folder roll-ups in Test Planning still work; the platform dimension answers questions like “Is checkout covered on iOS in QA this week?” without spinning up a second project.

Agents and CI should report runs with the correct platform identity (via @testchimp/playwright / Mobilewright reporter wiring) so linked scenario-annotated tests attribute to the right slice. Your plans can describe behaviour once; coverage status reflects where that behaviour is actually exercised.

TrueCoverage (production-informed gaps)​

TrueCoverage compares real user journeys (RUM) with automation coverage (test-tagged events). Each surface has its own instrumentation path—@testchimp/rum-js on web, testchimp-rum-ios and testchimp-rum-android on native—with TESTCHIMP_PROJECT_TYPE set to web, ios, or android as described in Instrumenting your app.

On Multi-Platform projects, the TrueCoverage execution scope offers the same Web / iOS / Android selector. That keeps comparisons honest:

• Production events from the iOS app are not mixed with web test runs when you evaluate gaps.
• Agents prioritizing fixtures and tests can target the platform where users actually hit the gap—for example high drop-off on Android checkout vs healthy web funnel.

Instrument every surface you ship; scope analytics one platform at a time when deciding what to automate next.

Atlas (UX bugs on the right surface)​

Atlas is TestChimp’s app-structure map: screens and states, with UX and non-functional bugs tagged where ExploreChimp or SmartTests observed them. For multi-platform products, the SiteMap is not a single blurred tree—you browse and triage per platform.

• A platform selector (Web, iOS, Android) loads the screen-state tree for that execution platform.
• Bugs discovered during exploration or annotated runs are associated with screen-state context on that platform, so a layout regression on mobile does not drown in unrelated web noise.
• markScreenState checkpoints in web Playwright tests and mobile Mobilewright tests feed the vocabulary ExploreChimp and Atlas use; platform-specific folders keep Act steps separate while structure stays comparable across surfaces.

That matters for engineering leads reviewing quality: you open Atlas, pick iOS, and see UX issues on the iOS SiteMap—assign owners per screen, run targeted ExploreChimp from a node, and track fix status without conflating desktop-only flows.

Arrange vs Act: what to share (and what not to)​

This is the same insight as fixtures and Object Mother patterns in xUnit-style testing (xUnit Test Patterns — test fixture, Object Mother): push incidental complexity of setup out of the test body and into reusable, composable building blocks. Agents authoring tests benefit even more when Arrange is API-backed rather than repeated through slow UI clicks (fixtures in agentic automation).

How to get started​

1. Sign in to TestChimp and open Add project.
2. Choose project type Multi-Platform (web + native mobile in one codebase).
3. Connect Git and map your plans/ and tests/ folders (same workflow as web-only projects).
4. Run your usual agent workflow—for example /testchimp test after a PR—using the TestChimp skill on Claude or Cursor.

Docs to read next:

• Mobile testing (iOS and Android) — configs, CI, and web vs mobile capability table
• Creating SmartTests — Playwright additions on web; Mobilewright on native
• Instrumenting your app for TrueCoverage — RUM on web and native when you want production-informed coverage

If your team already runs separate web and mobile automation repos, migrating Arrange into shared/ and api/ first—before moving UI specs—is usually the lowest-risk path. You keep platform runners; you stop duplicating the world behind them.

Frequently asked questions​

Is Multi-Platform the same as creating separate web and mobile TestChimp projects?​

No. Multi-Platform is one project and one scaffold where both Playwright and Mobilewright configs and folder layouts coexist. Separate Web and Mobile project types still exist when you only need one surface.

Do I have to abandon Appium to use this?​

TestChimp’s native path is Mobilewright, not Appium. Teams often adopt it when they want Playwright-like authoring and shared TypeScript with web suites. If you are standardized on Appium, compare effort to maintain duplicate Arrange code versus migrating Act layers over time while centralizing setup in API tests first.

Can API tests really replace UI for Arrange?​

For many domains, yes—and Playwright’s request context (and direct HTTP clients in api/*.spec.js) are the fastest, least flaky way to reach a given situation. UI Act remains necessary to validate what users see and tap; UI Arrange is usually optional once APIs or admin seeds exist (QA in production).

What’s the biggest win if we already have Playwright on web?​

The win is often consolidation of test infrastructure, not “another mobile runner.” Mobilewright lets mobile join the same repo conventions as web so agents and engineers maintain one mental model for fixtures, plans, and CI.

If plans and tests are in one repo, is coverage merged across web and mobile?​

No—not by default. Requirement coverage, TrueCoverage comparisons, and Atlas navigation use an explicit platform dimension on Multi-Platform projects (Web, iOS, Android). Shared scenarios in plans/ can be linked from both web/ and mobile/ tests; the platform scope shows where those links actually ran and passed.

Further reading​

TestChimp

• Mobile testing (iOS and Android)
• Requirement traceability
• Atlas SiteMap
• Screen-state annotations
• SmartTests introduction
• TrueCoverage introduction
• Instrumenting your app for TrueCoverage
• Workflows

Playwright & Mobilewright

• Playwright documentation
• Playwright test fixtures
• Mobilewright
• Mobile Use — cloud devices for Mobilewright

Patterns & quality engineering

• Arrange-Act-Assert (xUnitPatterns.com)
• Given-When-Then (Martin Fowler)
• Test fixture (xUnitPatterns.com)
• Object Mother (Martin Fowler)
• Page Object Model — Selenium wiki
• Appium documentation

Try it

• TestChimp — create a Multi-Platform project and connect your repository. Feedback welcome via your usual support channel or community touchpoints linked from the product.

Shipping both web and mobile? The duplication you feel in test automation is often in the Arrange layer—not in the product. Multi-Platform Projects let you maintain that layer once, run Playwright and Mobilewright where users actually interact, and still read requirements, TrueCoverage, and Atlas with clear per-platform signal.

TL;DR: TestChimp now supports native mobile app testing on both iOS and Android. This brings the same seamless workflow we unlock for your web testing - just say "/testchimp test".

What shipped​

Mobile is not a separate product bolted on the side. It is the same plan → repo → agent → CI loop you use for web SmartTests, extended to native apps via Mobilewright—a Playwright-style API and toolchain for iOS and Android.

Create a TestChimp project with project type iOS or Android, connect Git for your plans and tests folders, install the TestChimp skill on Claude or Cursor, and after each PR say /testchimp test. The platform keeps doing what you expect: wiring RUM, reading scenarios, closing coverage gaps, and surfacing analytics—now on screens that live inside your app, not only in the browser.

For setup details and parity tables, see Mobile testing (iOS and Android).

Five value props for Claude-based test authoring—four are live on mobile​

TestChimp’s agentic QA model rests on five pillars. On native mobile, four are fully supported today:

Smart Steps remain web-only for now. Native mobile tests use standard Mobilewright APIs for UI interaction—the same deterministic, async execution model you know from Playwright, without the intent-comment layer on top.

Everything else—the closed loops between requirements, production behaviour, fixtures, and tests—carries over.

The same seamless workflow as web​

You do not need a new playbook. The habit stays the same:

1. Install the TestChimp skill on Claude or Cursor.
2. After each PR, run /testchimp test (or your team’s equivalent in the agent host).

TestChimp then orchestrates the work you would otherwise stitch together manually:

• RUM libraries — Wire up testchimp-rum-ios and testchimp-rum-android so production and test runs speak the same event vocabulary.
• Instrumentation — Understand real user behaviour: segments, interaction flows, and scenarios—not just “the app launched.”
• Plans and stories — Read markdown scenarios, pull requirement traceability insights, and see what is still untested.
• Test authoring — Author Mobilewright tests to cover gaps, with traceability annotations where your plan expects them.
• Spot analytics — Run ExploreChimp-style analysis on new screens: visuals, logs, network.

You still get continuous transparency of QA posture in one platform—requirements, coverage, failures, and exploration—whether the surface is a browser tab or a native view controller.

Familiar tests, less flakiness​

Mobile tests are authored in a Playwright-familiar style via Mobilewright: auto-waits, async execution, and fixtures that behave like the ecosystem you already trust on web. That consistency matters when agents (and humans) move between repos that ship both web and mobile.

Fair credit where it is due: the reliability characteristics of that execution model come from Mobilewright—and we are grateful they exist. Mobilewright moved our timeline for serious native support forward by at least a year. If you need cloud-hosted real devices in CI, Mobile Use integrates with the same stack.

What to do next​

• Docs: Mobile testing (iOS and Android) — feature parity, CI notes, and links to RUM instrumentation.
• TrueCoverage on device: Instrumenting your app for TrueCoverage.
• Agent commands: Workflows — init, test, explore, evolve.

If you are already on TestChimp for web, create an iOS or Android project, point Git at your plans and tests folders, and run /testchimp test on your next mobile PR. Smart Steps will follow; the feedback loops you care about for shipping quality are already there.

End-to-end tests are a necessary evil: they are the last line of defense that something actually works in a real browser—but they break often enough that the suite becomes a burden instead of a trustworthy signal.

There are three main sources of variance that make E2E tests unreliable. Understanding them is the first step toward a suite you can actually rely on.

1. World-state variance​

This is what happens when your tests run in a different world-state than the one they were written against. A common cause is a shared environment where manual testing and automated runs both happen. The world changes between runs; the next run fails for reasons that have nothing to do with the code under test.

This kind of variance does more than flake tests. It also slows feedback: if those environments only get updates after PRs merge to main, you get weaker root-cause isolation and more expensive triage when something breaks.

2. System variance​

These are variances built into the stack itself: network latency, transient failures, UI paint timing, and so on. Mature frameworks like Playwright address a lot of this with built-in waiting, auto-waiting locators, and expect polling—so a big slice of “flakiness” is really tooling and patterns, not fate.

3. Product variance​

Even in a steady state (with no product change) — modern web apps are not as simple as calculators. Behavior can be inherently non-deterministic, and that is only more true now that AI often sits in the user journey (for example, a splash or offer that appears only sometimes). Much of that variance may be irrelevant to what a given test is trying to prove.

When tests are authored with a fragile, UI-selector-heavy approach, those product-level variances show up as broken steps. The test is coupled to incidental UI, not to intent.

Solve for these three kinds of variance, and you get a suite that is finally trustworthy.

World-state variance → controlled environments​

Use ephemeral environments loaded into known, predefined world-states, so each run matches the state the test was authored against as closely as possible.

System variance → solid automation primitives​

This is largely where mature frameworks shine. With Playwright, you get strong primitives for timing and stability—so you are not reinventing waits on every test.

Product variance → intent where the UI is messy​

This is where agentic steps in tests can help: natural-language instructions executed by an AI, instead of brittle coupling to selectors—only on the messy, flaky parts of the flow.

There is no free lunch: natural-language steps tend to be slower, costlier, and harder to debug than plain script. The goal is to use them surgically, not everywhere.

SmartTests: intent-based Playwright scripts​

That tradeoff is what TestChimp SmartTests are designed around: intent-based Playwright scripts.

They are still scripts for the most part, with an extra capability when you need flexibility—the parts of the app that fight selector-based automation.

Instead of:

await page.locator('.anticon.anticon-plus-square.ant-tree-switcher-line-icon > svg').nth(1).click();

Now you can write:

await ai.act('Expand the tree displayed in the left pane');

Only where you need it—the brittle, shifting UI—not for every line of the test.

Because the tests remain Playwright-based, system variance is handled by the same patterns and tooling you already trust. Run them in ephemeral environments with controlled world-states, and you have a test suite with all 3 variances accounted for - a suite that you can trust.

And yes, we are cooking up something on the ephemeral-environment side too. Stay tuned...

References​

The themes above—non-deterministic tests, UI-level flakiness, and mitigations—are well documented in both industry practice and research. These sources are a good starting point if you want to go deeper.

1. Martin Fowler, Eradicating Non-Determinism in Tests — A widely cited overview of why tests become non-deterministic (including async and shared-state issues) and how to structure tests to get repeatable results.
   https://martinfowler.com/articles/nonDeterminism.html

2. Google Testing Blog (George Pirocanac), Test Flakiness - One of the main challenges of automated testing (Dec 2020) — Describes categories of flakiness and why inconsistent automated tests slow development; follow-up posts in the same series expand on causes and responses.
   https://testing.googleblog.com/2020/12/test-flakiness-one-of-main-challenges.html

3. Google Testing Blog (John Micco), Flaky Tests at Google and How We Mitigate Them (May 2016) — Early, concrete account of flaky tests at scale, including mitigation strategies and discussion of where UI tests skew flaky.
   https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html

4. Wing Lam, Stefan Winter, Anjiang Wei, Tao Xie, Darko Marinov, Jonathan Bell, A Large-Scale Longitudinal Study of Flaky Tests, Proc. ACM Program. Lang. 4, OOPSLA, Article 202 (2020). Peer-reviewed study of when tests become flaky and how changes in code, tests, and dependencies contribute.
   https://doi.org/10.1145/3428270
   Conference entry: https://2020.splashcon.org/details/splash-2020-oopsla/78/A-Large-Scale-Longitudinal-Study-of-Flaky-Tests

5. Microsoft Playwright, Auto-waiting (Actionability) — Official documentation for pre-action checks (visible, stable, receiving events, enabled, etc.) that reduce timing-driven failures.
   https://playwright.dev/docs/actionability

6. Microsoft Playwright, Assertions — Describes auto-retrying assertions (expect) that wait until conditions hold, complementary to actionability for stable checks.
   https://playwright.dev/docs/test-assertions

7. Heroku / 12factor, Dev/prod parity (The Twelve-Factor App) — Classic framing for keeping development, staging, and production sufficiently aligned so “works in my environment” mismatches show up earlier; relevant when reasoning about world-state and shared environments.
   https://12factor.net/dev-prod-parity

8. Google Research (Diego Cavalcanti), De-Flake Your Tests: Automatically Locating Root Causes of Flaky Tests in Code at Google, ICSME 2020 — Empirical work on locating flaky-test root causes in code at Google scale; reports high accuracy for the proposed technique in their evaluation.
   https://research.google/pubs/de-flake-your-tests-automatically-locating-root-causes-of-flaky-tests-in-code-at-google/

TestChimp tests have always been plain Playwright under the hood — with extra capabilities like plain-English steps and lightweight scenario linking via code comments. That gives you fixtures, hooks, page objects, and the test organization you expect from a serious engineering setup.

Most QA teams are a mix of technical and non-technical teammates. Code-only authoring keeps contribution narrow. A separate no-code tool often means a second suite that drifts from the “real” tests and never gets the same CI treatment.

We added Simplified View in the web IDE so you do not have to choose.

What Simplified View Is​

Simplified View is a no-code surface for creating and editing SmartTests that still compiles to fully functional Playwright scripts. Everyone works on the same test; people just choose how they interact with the tests.

Your teammates can:

• Add plain English steps - that are run agentically.
• Use structured building blocks for common actions — less boilerplate and fewer syntax slips.
• Drop in free-form code when you need it — custom waits, tricky selectors, helpers: full Playwright, no lock-in.

You pick the level of code per step and per person, not one rule for the whole team.

Why this matters​

Non-technical members can contribute directly to test automation — not only by filing tickets for engineers to translate later. They build and edit steps in Simplified View; the result is still Playwright your automation folks can refine, reuse, and run in the same pipelines as everything else.

That lifts throughput for the whole team: more people can ship checks in parallel, fewer scenarios sit in a queue waiting for a coder, and engineers spend time on structure and hard cases instead of retyping flows from docs. Underneath, it stays real Playwright — deterministic runs, familiar debugging, ExploreChimp, CI, and Git workflows you already rely on.

Getting Started​

Open a SmartTest in the web IDE and switch to Simplified View to author or edit steps. When you need the full script, switch to code view; both views stay aligned with the same underlying test.

For more on creating and editing SmartTests, see Creating Smart Tests.

Further Reading​

If you’re interested in how no-code and low-code approaches impact QA team velocity and collaboration in general, these resources provide useful perspectives:

• No-Code vs Low-Code Test Automation (DevAssure) – A clear breakdown of how no-code enables faster onboarding and broader team contribution, while low-code supports scaling and collaboration across QA and engineering: https://www.devassure.io/blog/no-code-vs-low-code-test-automation/ 

• Benefits of No-Code Testing for Modern QA Teams – Explains how removing dependency on automation engineers allows teams to move faster, increase productivity, and ship more tests in parallel: https://sedstart.com/blog/benefits-of-no-code-testing-for-modern-qa-teams/ 

• How Low-Code/No-Code Tools Improve QA Speed (ImpactQA) – Shows how visual test creation reduces test cycle time, accelerates feedback loops, and shortens release cycles: https://www.impactqa.com/infographics/how-low-code-no-code-automation-tools-are-transforming-qa-with-faster-test-cycles/ 

• Low-Code/No-Code Testing Advantages (Jignect) – Highlights how these tools enable non-developers (testers, analysts) to actively contribute to automation, reducing bottlenecks in QA workflows: https://jignect.tech/qa-test-automation-with-low-code-no-code-tools-advantages-and-limitations/
  

• Low-Code vs No-Code Testing Benefits (Aegis Softtech) – Covers how reusable components and visual workflows reduce onboarding time and standardize test creation across teams: https://www.aegissofttech.com/insights/low-code-vs-no-code-test-automation/ 

We’re excited to announce that SmartTests now fully support the core Playwright testing patterns and constructs you know and love. This means you can write maintainable, well-structured test suites that leverage Playwright’s powerful features while still getting all the AI-powered adaptability that makes TestChimp SmartTests special.

What Are SmartTests?​

For those new to SmartTests, you can think of a SmartTest as a Playwright scripts with couple of twists:

Intent Comments:​

SmartTest Steps include intent comments that describe what you’re trying to accomplish. When a test runs, it executes as a standard Playwright script for speed and determinism. But when a step fails, our AI agent steps in to fix the issue on the fly and raises a PR with the changes – giving you the best of both worlds: fast script execution and intelligent adaptability.

Screen-state annotations:​

Markers that specify the screen and state the UI is at a given step in the script. These annotations are authored and used by ExploreChimp to tag the bugs to the correct screen-state in the SiteMap.

What's New: Full Playwright Compatibility​

SmartTests now support all the essential Playwright patterns that help you build professional, maintainable test suites:

1. Hooks for Setup and Teardown​

SmartTests now support all four Playwright hooks at both file and suite levels:

– beforeAll– Run once before all tests in a suite – afterAll – Run once after all tests in a suite – beforeEach – Run before each test – afterEach – Run after each test

This means you can set up test data, initialize page objects, authenticate users, and clean up resources exactly as you would in standard Playwright tests.

2. Page Object Models (POMs)​

SmartTests fully support the Page Object Model pattern, allowing you to encapsulate page interactions in reusable classes. This keeps your tests clean, maintainable, and aligned with best practices.

Example:

import { Page } from '@playwright/test';

class SignInPage {
constructor(private page: Page) {}

async navigate() {
await this.page.goto('/signin');
}

async login(email: string, password: string) {
await this.page.fill('#email', email);
await this.page.fill('#password', password);
await this.page.click('#sign-in-button');
}
}

test('user can sign in', async ({ page }) => {
const signInPage = new SignInPage(page);
await signInPage.navigate();
await signInPage.login('user@example.com', 'password123');
});

3. Fixtures for File Uploads​

SmartTests support Playwright fixtures, making it easy to handle file uploads and other test artifacts. Upload your fixture files (like test data, images, or documents) under the fixtures folder in the SmartTests tab, and they will be available during test execution.

4. Playwright Configuration​

SmartTests folder contains a playwright.config.js file in your project to configure the Playwright execution environment. This is essential for:

• Browser Authentication: Set up HTTP basic auth for staging environments
• Custom Headers: Add authorization tokens, API keys, or custom headers
• Base URLs: Configure default URLs for your test environment
• Viewport Settings: Set default browser viewport sizes And more: All standard Playwright configuration options

Example playwright.config.js:

const { defineConfig } = require(‘@playwright/test’);

module.exports = defineConfig({
use: {
baseURL: ‘https://staging.example.com’,
httpCredentials: {
username: ‘staging-user’,
password: ‘staging-password’
},
extraHTTPHeaders: {
‘Authorization’: ‘Bearer your-token’,
‘X-Environment’: ‘staging’
}
}
});
“

5. Test Suites with Multiple Tests​

SmartTests support organizing multiple tests in a single file using Playwright’s test.describe() blocks. You can create nested suites, group related tests together, and apply suite-level hooks – just like in standard Playwright.

Why This Matters​

These additions mean SmartTests are now fully compatible with Playwright’s ecosystem. You can:

✅ Write maintainable tests using industry-standard patterns like POMs and hooks

✅ Organize your test suite with proper grouping and structure

✅ Handle complex setups with configuration files and fixtures

✅ Reuse existing Playwright knowledge without learning new patterns

✅ Still get AI-powered fixes when tests fail – the best of both worlds!

Getting Started​

If you’re already using SmartTests, you can start using these features immediately. Just structure your tests using standard Playwright patterns, and SmartTests will handle the rest.

For new users, SmartTests work just like Playwright tests – with the added benefit of AI-powered failure recovery & stepwise execution enabling guided exploration.

What's Next?​

SmartTests continue to evolve, and we’re committed to maintaining full compatibility with Playwright’s ecosystem while adding intelligent features that make testing easier and more reliable. Stay tuned for more updates!

Got questions or feedback? We’d love to hear from you! Drop us a line at contact@testchimp.io.

Ok, first a quick recap on SmartTests:

SmartTests are plain playwright scripts, with intent comments before steps, that enables hybrid execution (fallback to agent mode execution when needed).

SmartTests are used by ExploreChimp to guide its explorations in pre-defined pathways, along which it identifies UX issues of the webapp such as performance, visual glitches, usability, content and more.

The Challenge: Context for Bugs​

When ExploreChimp finds bugs, it tags them with the “Screen” and “State” where they were captured. This context helps with troubleshooting and understanding when issues occur.

• A Screen is a conceptual view of your application: Dashboard, Homepage, Shopping Cart, etc.
• A State represents a specific situation within that screen: Empty Cart vs Cart with Items, Logged In vs Logged Out, etc.

ExploreChimp autonomously determines current screen and state based on the steps taken and the current screenshot. While this makes getting started easier, it may not always align with your mental model / the granularity you want things tracked at.

The Solution: Screen-State Annotations​

Now you can add explicit screen-state markers directly in your SmartTest scripts. These annotations tell ExploreChimp exactly which screen and state the app is at at a given point in the test, ensuring bugs are tagged with the context you care about.

How It Works​

After ExploreChimp runs, if the script didn’t contain screen-state markers, it updates the script with screen-state annotations it determined during the walk.

If you don’t want agent to update the script, you can turn it off by unchecking “Update script with screen-state annotations” under Advanced Settings (in the Exploration config wizard).

You can edit these annotations to match your conceptual model. For example, you may want to track UX bugs for “Cart with out-of-stock items” vs “Cart with in-stock items.” instead of the agent suggested states.

On the next run, ExploreChimp uses your annotations instead of guessing, so bugs are tagged consistently with your terminology.

Here is an example of a SmartTest with screen-state annotations:

test('Shopping Cart Flow', async ({ page }) => {
  // Navigate to homepage
  await page.goto('https://example.com');
  // @Screen: Homepage @State: Default

  // Search for a product
  await page.getByPlaceholder('Search products').fill('laptop');
  await page.getByRole('button', { name: 'Search' }).click();
  // @Screen: Search Results @State: With Results

  // Add item to cart
  await page.getByRole('link', { name: /laptop/i }).first().click();
  await page.getByRole('button', { name: 'Add to Cart' }).click();
  // @Screen: Shopping Cart @State: Cart with Items

  // Proceed to checkout
  await page.getByRole('button', { name: 'Proceed to Checkout' }).click();
  // @Screen: Checkout @State: Payment Step
});

Benefits​

• Consistent bug tagging: Bugs are tagged consistantly using your terminology, not AI-generated labels.

• Better organization: View bugs by screen-state in Atlas → SiteMap with your own categories.

• Easy refinement: Edit annotations to match your mental model easily – no need to retrain or reconfigure.

Getting Started​

• Run ExploreChimp on your SmartTest (annotations are added automatically).

• Review and edit the annotations in your script to match your terminology.

• The next time ExploreChimp is run on that test, it will use your annotations for consistent bug tagging.

The annotations are simple comments, so they don’t affect test execution – they’re purely for ExploreChimp’s context understanding.