Introduction to SmartTests

Platforms: On web, a SmartTest is a Playwright script with TestChimp additions below. On native mobile (iOS / Android), tests are Mobilewright scripts with the same planning, traceability, and ExploreChimp story—see Mobile testing for setup, CI, and web vs mobile capability parity (Smart Steps and some web-only integrations differ).

What is a SmartTest?

A SmartTest is a Playwright script (web) optionally enhanced with:

• AI-native steps (web today): ai.act and ai.verify that enable natural language-driven actions and verifications
• Scenario annotations: Link to your test scenarios with a comment - no need of manually maintained excel sheets to get coverage insights
• Screen–state checkpoints: The markScreenState Playwright fixture (via installTestChimp in tests/fixtures/index.js) records which screen and state the app is in at stable points—used by ExploreChimp and Atlas for bug tagging (Atlas, Screen states)

It combines the speed and determinism of traditional test scripts with the adaptability of AI while providing accurate guidance to Exploratory agents.

Guiding ExploreChimp and exploratory UX testing

SmartTests are the GPS for ExploreChimp: the Playwright script is the deterministic path, and await markScreenState('Screen', 'State') marks checkpoints where DOM, screenshots, console, network (optional), and performance telemetry are attributed for analysis. That keeps UX analytics aligned to real user journeys your suite already covers—see /testchimp explore for the command-level workflow and Screen-State Annotations for fixture usage.

Script-shaped exploration is more repeatable than an unconstrained crawl: you choose which specs to run (PR scope, feature folder, or TrueCoverage-prioritized paths in /testchimp evolve), and ExploreChimp analyzes along those steps.

The TestChimp product also offers exploratory agents that can extend beyond a single recorded script; SmartTests and markScreenState still anchor Atlas and bug routing when both models are used together.

Your Functional Tests - on Agentic steroids

On web, SmartTests are pure Playwright scripts with optional AI-native steps (ai.act, ai.verify etc.) allowing for selective inclusion of Agentic assistance during test execution (not yet on native mobile—see Mobile testing). Web SmartTests have full support for:

• Hooks: beforeEach, afterEach, beforeAll, afterAll
• Page Object Models (POMs): Organize page interactions in reusable classes
• Test suites: Group related tests together
• Config files: playwright.config.js or playwright.config.ts
• Artifacts: File uploads and other artifacts
• All other Playwright features: Everything you can do in Playwright

You get all the benefits of standard Playwright testing, plus the added advantage of hybrid execution:

• Execute as script by default: SmartTests run as fast, deterministic Playwright scripts
• Agentic intervention on tricky steps: For the tricky / flaky portions of your test, you can invoke ai.act steps, which triggers the agent to step in and do that step and return the normal script execution.
• Requirement traceability built-in: Unlike traditional approaches that require wrangling multiple systems, manually maintained excel sheets and cumbersome processes, You can simply annontate your tests with // @Scenario: comments, which links the test to the scenarios, enabling contextual insights.

SmartTests vs. Purely Agentic Functional Tests

Compared to purely agentic functional tests, SmartTests offer significant advantages:

Aspect	SmartTests	Purely Agentic Tests
Speed	Fast (script execution by default)	Slow (every step requires AI processing)
Cost	Low (AI only used on failures)	High (AI processing for every step)
Determinism	Deterministic (standard script execution)	Non-deterministic (AI decisions vary)
Scalability	Highly scalable (runs like standard tests)	Limited scalability (cost and time constraints)
Transparency	Full transparency (script changes via PRs)	Black box (AI decisions not always clear)
Vendor Lock-in	No vendor lock-in (standard Playwright scripts, run anywhere)	Vendor lock-in (requires proprietary test runners)

SmartTests provide the best of both worlds: the speed, determinism, and scalability of traditional scripts, with the adaptability of AI agents when needed.

SmartTests vs. Pure Playwright Scripts

Compared to pure Playwright scripts, SmartTests offer additional capabilities while maintaining full compatibility:

Aspect	SmartTests	Pure Playwright Scripts
Execution	Hybrid execution (script by default, agentic on failures)	Script execution only
AI-Native Steps	ai.act / ai.verify on web; not yet on native mobile (details)	Standard Playwright actions only
Self-Repair	Automatic repair on failures with transparent PRs in CI	Manual debugging and fixes required
Resilience	Adapts to UI changes automatically when steps fail	Breaks when selectors or UI change

SmartTests are standard Playwright scripts at their core, so you can run them with your existing Playwright setup, CI/CD pipelines, and tooling. The AI enhancements are optional and only activate when needed, giving you the reliability of scripts with the adaptability of AI.

Anatomy of a SmartTest

A SmartTest consists of several key components:

Structure and Components

• Playwright script elements: Standard Playwright test code
• AI-native steps: Agents can be invoked with ai.act and ai.verify commands as needed.
• Scenario comments: Annotation comments that link the test to scenarios it covers.
• Screen–state checkpoints (recommended): markScreenState fixture calls after stable UI boundaries—used by ExploreChimp and Atlas (details)
• Standard test structure: Setup, test steps, assertions, and teardown

Intent Comments (Optional)

Intent comments are optional annotations that provide context about what each test step is trying to accomplish. They help the agent understand the purpose of each step, enabling it to adapt when things change. For example:

// Navigate to the login page
await page.goto('https://example.com/login');

// Fill in the username field
await page.fill('#username', 'testuser');

These comments help the agent understand the purpose of each step, enabling it to adapt when things change.

Screen–state markers (markScreenState)

Use the markScreenState fixture on your merged test (see Creating Smart Tests and Screen-State Annotations):

import { test, expect } from '../fixtures';

test('login', async ({ page, markScreenState }) => {
  await page.goto('https://example.com/login');
  await markScreenState('Login', 'Empty form');
});

Legacy // @screen: / // @state: line comments may still exist on older specs; new work should use markScreenState only.

Playwright Script Elements

SmartTests are full Playwright scripts, so they include all standard Playwright features:

• Page objects
• Locators
• Assertions
• Actions (click, fill, etc.)
• Navigation
• And all other Playwright capabilities

Full Playwright Feature Set Support

SmartTests support the complete Playwright feature set, including:

• Hooks: beforeEach, afterEach, beforeAll, afterAll
• Config files: playwright.config.js or playwright.config.ts
• Artifacts: File uploads and other artifacts
• Project fixtures: tests/fixtures/index.js with mergeTests and installTestChimp for TrueCoverage (web page metadata; native automation URLs when TESTCHIMP_PROJECT_TYPE is ios / android), markScreenState, and ExploreChimp—see Creating Smart Tests and Instrumenting your app
• Page Object Models (POMs): Organize page interactions in reusable classes
• Test suites: Group related tests together
• And all other Playwright features: Everything you can do in Playwright, you can do in SmartTests

This means you can use your existing Playwright knowledge and patterns when creating SmartTests.

AI-Native Steps (web projects)

Smart Steps (ai.act, ai.verify, …) are supported on web SmartTests today. They are not yet available on native mobile Mobilewright projects—see Mobile testing.

SmartTests support AI-native steps that allow you to write tests using natural language for actions and verifications. These steps leverage vision intelligence to understand and interact with your UI.

ai.act - Natural Language Actions

Use ai.act to perform actions described in plain English. The AI analyzes the current page state and executes the necessary steps to achieve your objective:

import { ai } from 'ai-wright';

// Perform a login action using natural language
await ai.act('Log in as alice@example.com with password TestPass123', {
  page,
  test,
});

The AI will:

1. Analyze the current page state using screenshots and DOM
2. Identify the necessary UI elements (username field, password field, login button)
3. Execute the actions to complete the objective
4. Handle edge cases like modals or blockers automatically

ai.verify - Vision-Driven Assertions

Use ai.verify for natural language assertions that check visual and contextual requirements:

// Verify a requirement using natural language
await ai.verify('The toast should say "Message sent"', {
  page,
  test,
}, {
  confidence_threshold: 85, // Optional: set confidence threshold (default 70%)
});

ai.verify uses vision intelligence to:

• Analyze the current page visually
• Check if the requirement is met
• Provide confidence scores
• Fail the test if the requirement isn't met or confidence is too low

Benefits of AI-Native Steps

• Resilient to UI changes: AI steps adapt when selectors or layouts change
• Natural language: Write tests in plain English, making them more readable
• Vision intelligence: Uses screenshots and visual analysis for better accuracy
• Handles complexity: Automatically handles modals, dynamic content, and edge cases
• Combines with traditional steps: Mix AI-native steps with standard Playwright code

When to Use AI-Native Steps

• Complex interactions: When actions require multiple steps or decision-making
• Visual verifications: When you need to verify visual elements or layouts
• Dynamic content: When UI elements change frequently or are hard to locate
• Natural language clarity: When plain English makes the test intent clearer

You can combine AI-native steps with traditional Playwright code in the same SmartTest, choosing the best approach for each part of your test.

How SmartTests Run

SmartTests are run as plain scripts by default. When a step fails, the agent chimes in to execute the failing portion agentically (using optional intent comments if present), and continues with the script execution.

• Fast, deterministic execution of scripts
• Adaptability of agents when things go wrong

How SmartTests Are Used to Learn About Your App

SmartTests serve as a "GPS" for TestChimp's exploration agent:

1. Run ExploreChimp on a SmartTest or test folder: Right-click on any test or folder and select "Run ExploreChimp"
2. Mapping screens and states: ExploreChimp walks through your app guided by the tests, mapping out different screens/states it observes and how to navigate through them
3. Learning unexplored paths: The agent learns unexplored paths that can be used for autonomous expansion of tests

This knowledge is captured in the Atlas SiteMap (structure) and Atlas Behaviour Map (navigation pathways), which TestChimp uses to provide intelligent assistance across your QA workflow.