Introduction to SmartTests

What is a SmartTest?

A SmartTest is a Playwright script optionally enhanced with:

• AI-native steps: ai.act and ai.verify that enable natural language-driven actions and verifications
• Intent comments: Human-readable descriptions accompanying code blocks that are used for agentic fallback execution
• Screen-state annotations: Tags that identify which screen and state the app is at at a given step in the test

It combines the speed and determinism of traditional test scripts with the adaptability of AI while providing accurate guidance to Exploratory agents.

Purpose of SmartTest

Primary Purpose: Guiding Exploratory Agent for UX Testing

The primary purpose of SmartTests is to guide TestChimp's exploratory agent through your webapp for UX testing. SmartTests serve as a "GPS" that directs ExploreChimp to navigate your application, identify screens and states, and discover UX issues that traditional automated tests miss.

Test script-based execution makes exploratory testing more deterministic and enables systemic coverage - as opposed to a blank state agent just wandering around without direction. By following structured test pathways, ExploreChimp can:

• Systematically explore: Users can specify to traverse a specific path and catch bugs along the way
• Intelligently pick routes: Since the full map of journeys is described by the tests, the agent can intelligently select which routes to explore
• Expand methodically: Use the tests as the base, and venture from there to discover related areas and edge cases

When you run ExploreChimp on a SmartTest or test folder, the agent:

• Follows the test pathways to explore your application
• Maps out screens and states in the Atlas SiteMap
• Analyzes multiple data sources (DOM, screenshots, network, console, metrics) to identify UX bugs
• Tags bugs at the specific screen-state where they occur

This makes SmartTests essential for comprehensive UX testing that scales beyond manual testing while maintaining the structure and predictability of script-based execution.

Secondary Benefit: Functional Testing Suite

As a secondary benefit, SmartTests can serve as your functional testing suite. Since SmartTests are essentially standard Playwright scripts with full support for:

• Hooks: beforeEach, afterEach, beforeAll, afterAll
• Page Object Models (POMs): Organize page interactions in reusable classes
• Test suites: Group related tests together
• Config files: playwright.config.js or playwright.config.ts
• Artifacts: File uploads and other artifacts
• All other Playwright features: Everything you can do in Playwright

You get all the benefits of standard Playwright testing, plus the added advantage of hybrid execution:

• Execute as script by default: SmartTests run as fast, deterministic Playwright scripts
• Agentic intervention on failures: When a step fails, the agent automatically intervenes and executes the failing portion agentically (based on optional intent comments if present)
• Transparent changes: The agent makes changes to the script and raises a PR with the changes, providing complete transparency

SmartTests vs. Purely Agentic Functional Tests

Compared to purely agentic functional tests, SmartTests offer significant advantages:

Aspect	SmartTests	Purely Agentic Tests
Speed	Fast (script execution by default)	Slow (every step requires AI processing)
Cost	Low (AI only used on failures)	High (AI processing for every step)
Determinism	Deterministic (standard script execution)	Non-deterministic (AI decisions vary)
Scalability	Highly scalable (runs like standard tests)	Limited scalability (cost and time constraints)
Transparency	Full transparency (script changes via PRs)	Black box (AI decisions not always clear)
Vendor Lock-in	No vendor lock-in (standard Playwright scripts, run anywhere)	Vendor lock-in (requires proprietary test runners)

SmartTests provide the best of both worlds: the speed, determinism, and scalability of traditional scripts, with the adaptability of AI agents when needed.

SmartTests vs. Pure Playwright Scripts

Compared to pure Playwright scripts, SmartTests offer additional capabilities while maintaining full compatibility:

Aspect	SmartTests	Pure Playwright Scripts
Execution	Hybrid execution (script by default, agentic on failures)	Script execution only
AI-Native Steps	Support for ai.act and ai.verify natural language steps	Standard Playwright actions only
Self-Repair	Automatic repair on failures with transparent PRs in CI	Manual debugging and fixes required
Resilience	Adapts to UI changes automatically when steps fail	Breaks when selectors or UI change

SmartTests are standard Playwright scripts at their core, so you can run them with your existing Playwright setup, CI/CD pipelines, and tooling. The AI enhancements are optional and only activate when needed, giving you the reliability of scripts with the adaptability of AI.

Anatomy of a SmartTest

A SmartTest consists of several key components:

Structure and Components

• Playwright script elements: Standard Playwright test code
• Intent comments (optional): Human-readable descriptions of what each step is trying to accomplish
• Screen-state annotation comments (optional): Tags that identify which screen and state the test is currently in
• Standard test structure: Setup, test steps, assertions, and teardown

Intent Comments (Optional)

Intent comments are optional annotations that provide context about what each test step is trying to accomplish. They help the agent understand the purpose of each step, enabling it to adapt when things change. For example:

// Navigate to the login page
await page.goto('https://example.com/login');

// Fill in the username field
await page.fill('#username', 'testuser');

These comments help the agent understand the purpose of each step, enabling it to adapt when things change.

Screen-State Annotation Comments

Screen-state annotations identify which screen and state the test is currently in. The annotation comment comes after the step that navigates to or establishes that screen-state:

await page.goto('https://example.com/login');
// @screen: Login @state: Empty form

These annotations are used during explorations to tag bugs at the specific screen-state where they occur. If annotations are not present, the agent will deduce screen states during exploration and update the script with annotation comments after the corresponding steps.

Playwright Script Elements

SmartTests are full Playwright scripts, so they include all standard Playwright features:

• Page objects
• Locators
• Assertions
• Actions (click, fill, etc.)
• Navigation
• And all other Playwright capabilities

Full Playwright Feature Set Support

SmartTests support the complete Playwright feature set, including:

• Hooks: beforeEach, afterEach, beforeAll, afterAll
• Config files: playwright.config.js or playwright.config.ts
• Artifacts: File uploads and other artifacts (test fixtures are not currently supported)
• Page Object Models (POMs): Organize page interactions in reusable classes
• Test suites: Group related tests together
• And all other Playwright features: Everything you can do in Playwright, you can do in SmartTests

This means you can use your existing Playwright knowledge and patterns when creating SmartTests.

AI-Native Steps

SmartTests support AI-native steps that allow you to write tests using natural language for actions and verifications. These steps leverage vision intelligence to understand and interact with your UI.

ai.act - Natural Language Actions

Use ai.act to perform actions described in plain English. The AI analyzes the current page state and executes the necessary steps to achieve your objective:

import { ai } from 'ai-wright';

// Perform a login action using natural language
await ai.act('Log in as alice@example.com with password TestPass123', {
  page,
  test,
});

The AI will:

1. Analyze the current page state using screenshots and DOM
2. Identify the necessary UI elements (username field, password field, login button)
3. Execute the actions to complete the objective
4. Handle edge cases like modals or blockers automatically

ai.verify - Vision-Driven Assertions

Use ai.verify for natural language assertions that check visual and contextual requirements:

// Verify a requirement using natural language
await ai.verify('The toast should say "Message sent"', {
  page,
  test,
}, {
  confidence_threshold: 85, // Optional: set confidence threshold (default 70%)
});

ai.verify uses vision intelligence to:

• Analyze the current page visually
• Check if the requirement is met
• Provide confidence scores
• Fail the test if the requirement isn't met or confidence is too low

Benefits of AI-Native Steps

• Resilient to UI changes: AI steps adapt when selectors or layouts change
• Natural language: Write tests in plain English, making them more readable
• Vision intelligence: Uses screenshots and visual analysis for better accuracy
• Handles complexity: Automatically handles modals, dynamic content, and edge cases
• Combines with traditional steps: Mix AI-native steps with standard Playwright code

When to Use AI-Native Steps

• Complex interactions: When actions require multiple steps or decision-making
• Visual verifications: When you need to verify visual elements or layouts
• Dynamic content: When UI elements change frequently or are hard to locate
• Natural language clarity: When plain English makes the test intent clearer

You can combine AI-native steps with traditional Playwright code in the same SmartTest, choosing the best approach for each part of your test.

How SmartTests Run

SmartTests are run as plain scripts by default. When a step fails, the agent chimes in to execute the failing portion agentically (using optional intent comments if present), and continues with the script execution.

• Fast, deterministic execution of scripts
• Adaptability of agents when things go wrong

How SmartTests Are Used to Learn About Your App

SmartTests serve as a "GPS" for TestChimp's exploration agent:

1. Run ExploreChimp on a SmartTest or test folder: Right-click on any test or folder and select "Run ExploreChimp"
2. Mapping screens and states: ExploreChimp walks through your app guided by the tests, mapping out different screens/states it observes and how to navigate through them
3. Learning unexplored paths: The agent learns unexplored paths that can be used for autonomous expansion of tests

This knowledge is captured in the Atlas SiteMap (structure) and Atlas Behaviour Map (navigation pathways), which TestChimp uses to provide intelligent assistance across your QA workflow.