Run SmartTests in CI with Playwright Runner

Overview

You can run SmartTests using the standard Playwright test runner in your CI pipeline, with the playwright-testchimp-reporter configured to automatically report test execution results to TestChimp (and handle AI step invocations). This enables continuous requirement traceability, and AI-native steps in scripts, without changing how you run your tests.

Why Use Playwright Runner in CI?

Running SmartTests with the standard Playwright runner provides:

Familiar workflow: Use the same Playwright commands and configuration you're already using
Standard CI integration: Works with any CI/CD platform that supports Node.js
Continuous traceability: Test executions are automatically tracked for requirement coverage
No vendor lock-in: Your tests remain standard Playwright scripts that run anywhere
Flexible execution: Run all tests or filter to specific tests, folders, or tags

Installation

Install the playwright-testchimp-reporter package:

npm install --save-dev playwright-testchimp-reporter

Or with yarn:

yarn add -D playwright-testchimp-reporter

The package is available on npm: playwright-testchimp-reporter

Configuration

Basic Setup

In TestChimp, we have setup a playwright.config.js file under the tests folder, which has the reporter configuration already done for you. So you don't need to do anything in addition - once you sync the TestChimp tests folder to your repo. Simply configure your CI action to run tests (remember to cd tests in to the tests folder first, since the playwright.config.js file is inside the tests folder).

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['list'], // Standard list reporter
    ['playwright-testchimp-reporter', {
      // Configuration options (see below)
    }]
  ],
  // ... rest of your config
});

Configuration Options

The reporter can be configured using environment variables.

Environment Variable	Description	Required
`TESTCHIMP_API_KEY`	API key for authentication	Yes
`TESTCHIMP_PROJECT_ID`	Project identifier	Yes
`TESTCHIMP_BACKEND_URL`	API URL (default: https://featureservice.testchimp.io)	No
`TESTCHIMP_RELEASE`	Release/version identifier	No
`TESTCHIMP_ENV`	Environment name (e.g., staging, prod)	No

CI/CD Setup

GitHub Actions

Complete sample action file: link

Example GitHub Actions workflow:

name: Run SmartTests

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Install Playwright browsers
        run: npx playwright install --with-deps
      
     - name: Run Playwright tests
        run: |
          cd tests
          npx playwright test
        env:
          TESTCHIMP_API_KEY: ${{ secrets.TESTCHIMP_API_KEY }}
          TESTCHIMP_PROJECT_ID: ${{ secrets.TESTCHIMP_PROJECT_ID }}
          TESTCHIMP_RELEASE: "default"
          TESTCHIMP_ENV: "QA"

GitLab CI

Example GitLab CI configuration:

test:
  image: node:18
  before_script:
    - npm ci
    - npx playwright install --with-deps
  script:
    - cd tests
    - npx playwright test
  variables:
    TESTCHIMP_API_KEY: $TESTCHIMP_API_KEY
    TESTCHIMP_PROJECT_ID: $TESTCHIMP_PROJECT_ID
    TESTCHIMP_ENV: staging
    TESTCHIMP_RELEASE: $CI_COMMIT_REF_NAME

Jenkins

Example Jenkins pipeline:

pipeline {
  agent any
  environment {
    TESTCHIMP_API_KEY = credentials('testchimp-api-key')
    TESTCHIMP_PROJECT_ID = credentials('testchimp-project-id')
    TESTCHIMP_ENV = 'staging'
    TESTCHIMP_RELEASE = env.BUILD_NUMBER
  }
  stages {
    stage('Test') {
      steps {
        sh 'npm ci'
        sh 'npx playwright install --with-deps'
        sh 'cd tests'
        sh 'npx playwright test'
      }
    }
  }
}

Enabling AI-Native Steps with ai-wright

playwright-testchimp-reporter automatically pulls in the ai-wright library, and wires up the testchimp credential variables, so you don't need to do anything special in addition.

How Customer Reporter Works

When you run tests with the reporter configured:

Test execution: Tests run normally using the standard Playwright runner
Automatic reporting: The reporter captures test execution details:
- Test name, file path, and suite structure
- Step-by-step execution results
- Pass/fail status
- Error messages (if any)
- Screenshots for failing steps (if enabled)
Backend ingestion: Execution data is sent to TestChimp backend
Coverage tracking: Test executions are automatically linked to scenarios based on // @Scenario: comments in your tests
Traceability updates: Coverage information is updated in real-time in the TestChimp platform

Linking Tests to Scenarios

To enable requirement traceability, make sure you have added // @Scenario comments in tests:

test('login test', async ({ page }) => {
// @Scenario: User can log in with valid credentials
  // ... test code
});

For more details, see Linking Scenarios from SmartTests.

Benefits

Running SmartTests in CI with the Playwright runner provides:

Continuous traceability: Every CI run contributes to requirement coverage tracking
No workflow changes: Use standard Playwright commands and configuration
Automatic tracking: No manual steps required - executions are tracked automatically
Historical data: Build up a history of test executions over time
Environment tracking: Track coverage separately for different environments (staging, production, etc.)
Release tracking: Associate test executions with specific releases or versions

Branch-specific runs

When running tests in CI for a branch (e.g. a pull request), you often want to run against that branch’s deployment preview URL (PR preview) instead of a fixed QA URL. You can do this by having CI pass the preview URL into the test run.

Set BASE_URL in CI: In your CI workflow, set the BASE_URL environment variable to the PR preview or deployment preview URL for the current branch (e.g. from your hosting provider’s API or a predefined variable like VERCEL_PREVIEW_URL).
Tests pick it up: The Playwright runner and your SmartTests use process.env.BASE_URL (or Playwright’s baseURL if configured from it). When CI sets BASE_URL to the preview URL, tests run against that deployment without changing test code.
Reporter and branch: Ensure your CI sets TESTCHIMP_RELEASE (and optionally branch or build id) so executions are attributed correctly in TestChimp. The reporter sends results to the platform so branch-specific runs appear in branch-scoped insights.

Example (GitHub Actions with a hypothetical preview URL):

- name: Run Playwright tests
  run: |
    cd tests
    npx playwright test
  env:
    TESTCHIMP_API_KEY: ${{ secrets.TESTCHIMP_API_KEY }}
    TESTCHIMP_PROJECT_ID: ${{ secrets.TESTCHIMP_PROJECT_ID }}
    TESTCHIMP_RELEASE: ${{ github.ref_name }}
    BASE_URL: ${{ env.PREVIEW_URL }}   # e.g. from a previous step that sets PR preview URL

For configuring branch-specific URLs in the TestChimp project (template and overrides), see Branch Specific Test Execution.

Best Practices

Use environment variables: Store API credentials as secrets in your CI/CD platform
Set environment and release: Use TESTCHIMP_ENV and TESTCHIMP_RELEASE to track executions by environment and version
Enable screenshots: Keep captureScreenshots: true for better debugging of failures
Link to scenarios: Add // @Scenario: comments to enable requirement traceability
Review coverage: Regularly check requirement coverage in the TestChimp platform to identify gaps
Branch previews: For PR/branch runs, set BASE_URL to the preview URL so tests run against the branch deployment; see Branch-specific runs above

Troubleshooting

Path resolution issues

Make sure you are running your playwright-runner from WITHIN the tests folder (and not the root folder of the repo). So if your tests are in repo_root/tests, then in the CI action, make sure you first cd tests as a step (refer above examples). This is since our playwright.config.js file and the environment files are managed directly under the tests folder.

Reporter Not Sending Data

Check credentials: Verify TESTCHIMP_API_KEY and TESTCHIMP_PROJECT_ID are set correctly
Enable verbose logging: Set verbose: true to see detailed logs
Check network: Ensure your CI environment can reach the TestChimp API

Tests Not Linked to Scenarios

Verify scenario comments: Ensure // @Scenario: comments are present in your tests
Check scenario titles: Scenario titles must match exactly (case-sensitive)
Sync your tests folder in TestChimp: Make sure you have synced your repo tests folder in TestChimp -> SmartTests page.

Missing Screenshots

Enable screenshot capture: Set captureScreenshots: true in config
Check Playwright config: Ensure Playwright is configured to capture screenshots on failure

Running SmartTests in CI with the Playwright runner enables continuous requirement traceability while maintaining the flexibility and familiarity of standard Playwright test execution.

Overview​

Why Use Playwright Runner in CI?​

Installation​

Configuration​

Basic Setup​

Configuration Options​

CI/CD Setup​

GitHub Actions​

GitLab CI​

Jenkins​

Enabling AI-Native Steps with ai-wright​

How Customer Reporter Works​

Linking Tests to Scenarios​

Benefits​

Branch-specific runs​

Best Practices​

Troubleshooting​

Path resolution issues​

Reporter Not Sending Data​

Tests Not Linked to Scenarios​

Missing Screenshots​