Skip to main content

SmartTests in your code base

Overview​

SmartTests are saved as .smart.spec.js files, which allows for easy inclusion or exclusion in your normal Playwright test suite runs. This naming convention provides flexibility in how you integrate SmartTests with your existing test infrastructure.

File Naming Convention​

SmartTests use the .smart.spec.js file extension, which follows Playwright's standard .spec.js pattern with a .smart prefix. This design enables:

  • Easy identification: Quickly distinguish SmartTests from regular Playwright tests
  • Flexible execution: Include or exclude SmartTests from test runs using Playwright's built-in filtering
  • GitHub sync: Only .smart.spec.js files are synchronized between TestChimp and your repository

Default Playwright Behavior​

By default, Playwright runs all *.spec.js files in your test directory. This means:

  • SmartTests are included by default: Since .smart.spec.js files match the *.spec.js pattern, they will be executed when you run npx playwright test
  • No configuration needed: Your SmartTests work out of the box with standard Playwright commands

Excluding SmartTests from Test Runs​

If you want to exclude SmartTests from your normal Playwright test runs, you can use Playwright's test filter options:

Using Test Match Pattern​

In your playwright.config.js, you can configure the testMatch option to exclude .smart.spec.js files:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  testMatch: /^((?!.*\.smart\.spec\.js).)*\.spec\.js$/,
  // ... other config
});

Using Command Line Filter​

You can also exclude SmartTests when running tests from the command line:

npx playwright test --grep-invert "\.smart\.spec\.js"

Modes of Usage​

Mode 1: SmartTests as Guides to Explore Agents​

Recommended approach: Use SmartTests primarily as guides for TestChimp's exploratory agents.

Configuration:

  • Add *.smart.spec.js to your test ignore/filter pattern
  • SmartTests remain in your repository but are excluded from standard test runs
  • Tests serve as pathways for ExploreChimp to follow during exploratory testing

Benefits:

  • Clear separation of concerns: SmartTests guide exploration, functional tests verify behavior
  • No interference with your standard test suite
  • SmartTests focus on documenting user journeys and pathways

Example Configuration:

// playwright.config.js
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testMatch: /^((?!.*\.smart\.spec\.js).)*\.spec\.js$/,
  // Regular tests run normally
  // SmartTests are excluded but available for exploration
});

Mode 2: SmartTests as Your E2E Test Suite​

You can use SmartTests as your primary end-to-end test suite by running Playwright normally (no filtering needed).

Important Limitations: While SmartTests support most of the Playwright ecosystem, the following features are not currently supported:

  • Test fixtures (extending test behavior): Custom test fixtures that modify test behavior are not supported
  • Custom test extensions: Extending the test object with custom properties or methods
  • Advanced fixture composition: Complex fixture dependencies and composition patterns
  • Custom test reporters: While standard Playwright reporters work, custom reporter implementations may have limitations
  • Worker-specific configurations: Advanced worker isolation and configuration options

Supported Features:

  • ✅ Test suites (test.describe)
  • ✅ Hooks (test.beforeEach, test.afterEach, test.beforeAll, test.afterAll)
  • ✅ Fixture assets (file uploads via setInputFiles)
  • ✅ Page Object Model (POM) file references
  • ✅ playwright.config.js configuration
  • ✅ Standard Playwright assertions and matchers
  • ✅ Standard Playwright locators and actions

Recommendation: We recommend keeping SmartTests focused on being guides to explore agents rather than your primary E2E test suite. This approach provides:

  • Clear objectives: SmartTests document pathways, functional tests verify behavior
  • Better maintainability: Separation of concerns between exploration guides and verification tests
  • Flexibility: Use SmartTests for exploration while maintaining traditional tests for critical paths

Test Folder Organization​

You have flexibility in how you organize SmartTests in your project:

Option 1: SmartTests in Standard Test Folder​

Place SmartTests alongside your regular Playwright tests in the same directory:

tests/
  ├── login.spec.js           # Regular Playwright test
  ├── dashboard.spec.js        # Regular Playwright test
  ├── user-journey.smart.spec.js  # SmartTest
  └── checkout-flow.smart.spec.js # SmartTest

Benefits:

  • All tests in one location
  • Can reuse your existing POM files, fixture assets in SmartTests
  • Easy to see both test types together
  • Simple folder structure

Note: Only .smart.spec.js test files are synchronized with TestChimp via GitHub Sync.

Option 2: SmartTests in Separate Folder​

Create a dedicated folder for SmartTests to provide better separation:

tests/
  ├── e2e/
  │   ├── login.spec.js
  │   └── dashboard.spec.js
  └── smart-tests/
      ├── user-journey.smart.spec.js
      └── checkout-flow.smart.spec.js

Benefits:

  • Clear separation between SmartTests and functional tests
  • Easier to apply different configurations or filters
  • Better organization for teams using SmartTests primarily as exploration guides

Configuration: When using a separate folder, you can configure Playwright to run tests from specific directories:

// playwright.config.js
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests/e2e',  // Only run regular tests from e2e folder
  // SmartTests in smart-tests/ folder are excluded
});

GitHub Sync Behavior​

When using GitHub Sync, TestChimp only synchronizes .smart.spec.js test files from the configured directory:

  • Included: All .smart.spec.js files in the synced folder (and subfolders).

  • Excluded: Regular .spec.js files, configuration files, and other non-SmartTest files

  • Recursive: Subdirectories are scanned for .smart.spec.js files

  • Pages, Fixtures: The sync also syncs pages/ and fixtures/ folders. *.page.js files in pages/ folder and all files in fixtures/ folder are synced.

This ensures that only SmartTests are managed by TestChimp, while your regular test files remain under your control.

Best Practices​

  1. Use SmartTests as exploration guides: Keep SmartTests focused on documenting user journeys and pathways for exploratory agents
  2. Maintain functional tests separately: Use regular Playwright tests for critical verification and regression testing
  3. Exclude SmartTests from CI/CD: If using SmartTests as guides, exclude them from your standard test runs to avoid confusion.
  4. Organize by purpose: Consider using separate folders if you have many SmartTests alongside functional tests
  5. Document your approach: Make it clear to your team whether SmartTests are guides or part of the test suite

By following these practices, you can effectively integrate SmartTests into your Playwright-based testing workflow while maintaining clear separation of concerns.