SmartTests in your code base

In brief: SmartTests live in your repository as ordinary Playwright files so developers use familiar PR workflows while TestChimp adds planning links, CI reporting, and agent orchestration.

Overview

SmartTests are saved as standard Playwright test files (.spec.js or .spec.ts). They are ordinary specs—no proprietary format or special filename prefix—so they work with your existing Playwright tooling, PR review, and CI.

What makes them SmartTests is how you use them with TestChimp: scenario links (Playwright annotation with type: 'scenario'), optional AI steps, ExploreChimp markers, and sync into a mapped tests folder—not a distinct file extension.

File Naming Convention

Use Playwright’s normal naming:

• JavaScript: *.spec.js
• TypeScript: *.spec.ts

This design enables:

• Familiar tooling: Same editor, CLI, and CI commands as any Playwright suite
• Flexible layout: Separate SmartTests from other suites by folder, not by filename
• GitHub sync: .spec.js / .spec.ts files in the mapped folder are synchronized between TestChimp and your repository

Default Playwright Behavior

By default, Playwright runs all *.spec.js / *.spec.ts files under your configured testDir. This means:

• SmartTests run like any other specs when they live under that directory
• No special runner needed: npx playwright test works out of the box

Separating SmartTests from Other Suites

Because SmartTests use the same .spec extension as regular Playwright tests, use folder layout (and Playwright testDir / project config) when you want different run behavior.

Point Playwright at a non-SmartTest folder

If SmartTests live in a dedicated folder and your functional suite lives elsewhere, set testDir (or a Playwright project) to the functional-tests path only:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests/e2e',  // regular suite only
  // SmartTests in another folder (e.g. tests/smart-tests/) are not discovered
});

Command-line path filter

Run only a subtree:

npx playwright test tests/e2e
# or
npx playwright test tests/smart-tests

Modes of Usage

Mode 1: SmartTests as Guides to Explore Agents

Recommended approach: Use SmartTests primarily as guides for TestChimp's exploratory agents.

Configuration:

• Keep SmartTests in a dedicated folder (or map that folder for GitHub Sync)
• Point your day-to-day Playwright testDir / CI job at your functional suite only
• SmartTests remain in the repository and serve as pathways for ExploreChimp

Benefits:

• Clear separation of concerns: SmartTests guide exploration, functional tests verify behavior
• No interference with your standard test suite
• SmartTests focus on documenting user journeys and pathways

Example Configuration:

// playwright.config.js
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests/e2e',
  // SmartTests under tests/smart-tests/ (or similar) are excluded from this config
});

Mode 2: SmartTests as Your E2E Test Suite

You can use SmartTests as your primary end-to-end test suite by pointing Playwright at the SmartTests folder (or keeping everything in one suite) and running normally.

Important Limitations: While SmartTests support most of the Playwright ecosystem, the following features are not currently supported:

• Test fixtures (extending test behavior): Custom test fixtures that modify test behavior are not supported
• Custom test extensions: Extending the test object with custom properties or methods
• Advanced fixture composition: Complex fixture dependencies and composition patterns
• Custom test reporters: While standard Playwright reporters work, custom reporter implementations may have limitations
• Worker-specific configurations: Advanced worker isolation and configuration options

Supported Features:

• ✅ Test suites (test.describe)
• ✅ Hooks (test.beforeEach, test.afterEach, test.beforeAll, test.afterAll)
• ✅ Fixture assets (file uploads via setInputFiles)
• ✅ Page Object Model (POM) file references
• ✅ playwright.config.js configuration
• ✅ Standard Playwright assertions and matchers
• ✅ Standard Playwright locators and actions

Recommendation: We recommend keeping SmartTests focused on being guides to explore agents rather than your primary E2E test suite. This approach provides:

• Clear objectives: SmartTests document pathways, functional tests verify behavior
• Better maintainability: Separation of concerns between exploration guides and verification tests
• Flexibility: Use SmartTests for exploration while maintaining traditional tests for critical paths

Test Folder Organization

You have flexibility in how you organize SmartTests in your project:

Option 1: SmartTests in Standard Test Folder

Place SmartTests alongside your other Playwright tests in the same directory:

tests/
  ├── login.spec.js              # Playwright / SmartTest
  ├── dashboard.spec.js          # Playwright / SmartTest
  ├── user-journey.spec.js       # SmartTest
  └── checkout-flow.spec.js      # SmartTest

Benefits:

• All tests in one location
• Can reuse your existing POM files and fixture assets
• Easy to see the full suite together
• Simple folder structure

Note: When this folder is mapped for GitHub Sync, TestChimp synchronizes .spec.js / .spec.ts files (plus pages/ and fixtures/ as described below).

Option 2: SmartTests in Separate Folder

Create a dedicated folder for SmartTests to provide better separation:

tests/
  ├── e2e/
  │   ├── login.spec.js
  │   └── dashboard.spec.js
  └── smart-tests/
      ├── user-journey.spec.js
      └── checkout-flow.spec.js

Benefits:

• Clear separation between SmartTests and functional tests
• Easier to apply different Playwright testDir / project configs or CI jobs
• Better organization for teams using SmartTests primarily as exploration guides

Configuration: When using a separate folder, point Playwright at the suite you want to run:

// playwright.config.js
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests/e2e',  // Only run regular tests from e2e folder
  // Map tests/smart-tests/ (or the whole tests/ tree) in TestChimp for sync / agents
});

GitHub Sync Behavior

When using GitHub Sync, TestChimp synchronizes Playwright test files from the configured directory:

• Included: All .spec.js and .spec.ts files in the synced folder (and subfolders)
• Excluded: Configuration files and other non-test assets outside the synced support folders
• Recursive: Subdirectories are scanned for .spec.js / .spec.ts files
• Pages, Fixtures: The sync also syncs pages/ and fixtures/ folders. *.page.js / *.page.ts files in pages/ and all files in fixtures/ are synced.

Map the folder that should be managed with TestChimp (often a dedicated SmartTests directory, or your whole tests/ tree if that is your SmartTest suite).

Best Practices

1. Use SmartTests as exploration guides: Keep SmartTests focused on documenting user journeys and pathways for exploratory agents
2. Maintain functional tests separately: Use a separate folder (or Playwright project) for critical verification and regression testing when you need clear run boundaries
3. Exclude SmartTests from CI/CD when they are guides only: Point CI at your functional testDir so exploration guides do not mix into gate jobs unintentionally
4. Organize by purpose: Prefer separate folders when you have many SmartTests alongside functional tests
5. Document your approach: Make it clear to your team whether SmartTests are guides or part of the gate suite

By following these practices, you can effectively integrate SmartTests into your Playwright-based testing workflow while maintaining clear separation of concerns.