Run SmartTests in CI with Playwright Runner

Overview

You can run SmartTests using the standard Playwright test runner in your CI pipeline, with the playwright-testchimp-reporter configured to automatically report test execution results to TestChimp. This enables continuous requirement traceability without changing how you run your tests.

Why Use Playwright Runner in CI?

Running SmartTests with the standard Playwright runner provides:

• Familiar workflow: Use the same Playwright commands and configuration you're already using
• Standard CI integration: Works with any CI/CD platform that supports Node.js
• Continuous traceability: Test executions are automatically tracked for requirement coverage
• No vendor lock-in: Your tests remain standard Playwright scripts that run anywhere
• Flexible execution: Run all tests or filter to specific tests, folders, or tags

Installation

Install the playwright-testchimp-reporter package:

npm install --save-dev playwright-testchimp-reporter

Or with yarn:

yarn add -D playwright-testchimp-reporter

The package is available on npm: playwright-testchimp-reporter

Configuration

Basic Setup

Add the reporter to your playwright.config.ts:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['list'], // Standard list reporter
    ['playwright-testchimp-reporter', {
      // Configuration options (see below)
    }]
  ],
  // ... rest of your config
});

Configuration Options

The reporter can be configured using environment variables.

Environment Variable	Description	Required
TESTCHIMP_API_KEY	API key for authentication	Yes
TESTCHIMP_PROJECT_ID	Project identifier	Yes
TESTCHIMP_BACKEND_URL	API URL (default: https://featureservice.testchimp.io)	No
TESTCHIMP_TESTS_FOLDER	Base folder for relative path calculation	No
TESTCHIMP_RELEASE	Release/version identifier	No
TESTCHIMP_ENV	Environment name (e.g., staging, prod)	No

Complete Example

import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  reporter: [
    ['list'],
    ['playwright-testchimp-reporter', {
      // Credentials from environment variables
      // Optional: override defaults
      reportOnlyFinalAttempt: true,
      captureScreenshots: true,
      verbose: process.env.CI === 'true', // Verbose in CI
    }]
  ],
  use: {
    baseURL: 'https://example.com',
  },
});

CI/CD Setup

GitHub Actions

Example GitHub Actions workflow:

name: Run SmartTests

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Install Playwright browsers
        run: npx playwright install --with-deps
      
      - name: Run tests
        env:
          TESTCHIMP_API_KEY: ${{ secrets.TESTCHIMP_API_KEY }}
          TESTCHIMP_PROJECT_ID: ${{ secrets.TESTCHIMP_PROJECT_ID }}
          TESTCHIMP_ENV: staging
          TESTCHIMP_RELEASE: ${{ github.ref_name }}
        run: npx playwright test

GitLab CI

Example GitLab CI configuration:

test:
  image: node:18
  before_script:
    - npm ci
    - npx playwright install --with-deps
  script:
    - npx playwright test
  variables:
    TESTCHIMP_API_KEY: $TESTCHIMP_API_KEY
    TESTCHIMP_PROJECT_ID: $TESTCHIMP_PROJECT_ID
    TESTCHIMP_ENV: staging
    TESTCHIMP_RELEASE: $CI_COMMIT_REF_NAME

Jenkins

Example Jenkins pipeline:

pipeline {
  agent any
  environment {
    TESTCHIMP_API_KEY = credentials('testchimp-api-key')
    TESTCHIMP_PROJECT_ID = credentials('testchimp-project-id')
    TESTCHIMP_ENV = 'staging'
    TESTCHIMP_RELEASE = env.BUILD_NUMBER
  }
  stages {
    stage('Test') {
      steps {
        sh 'npm ci'
        sh 'npx playwright install --with-deps'
        sh 'npx playwright test'
      }
    }
  }
}

How It Works

When you run tests with the reporter configured:

1. Test execution: Tests run normally using the standard Playwright runner
2. Automatic reporting: The reporter captures test execution details:
   • Test name, file path, and suite structure
   • Step-by-step execution results
   • Pass/fail status
   • Error messages (if any)
   • Screenshots for failing steps (if enabled)
3. Backend ingestion: Execution data is sent to TestChimp backend
4. Coverage tracking: Test executions are automatically linked to scenarios based on // @Scenario: comments in your tests
5. Traceability updates: Coverage information is updated in real-time in the TestChimp platform

Linking Tests to Scenarios

To enable requirement traceability, add scenario comments to your tests:

// @Scenario: User can log in with valid credentials
test('login test', async ({ page }) => {
  // ... test code
});

The reporter automatically extracts these comments and links test executions to scenarios. For more details, see Linking Scenarios from SmartTests.

Benefits

Running SmartTests in CI with the Playwright runner provides:

• Continuous traceability: Every CI run contributes to requirement coverage tracking
• No workflow changes: Use standard Playwright commands and configuration
• Automatic tracking: No manual steps required - executions are tracked automatically
• Historical data: Build up a history of test executions over time
• Environment tracking: Track coverage separately for different environments (staging, production, etc.)
• Release tracking: Associate test executions with specific releases or versions

Best Practices

1. Use environment variables: Store API credentials as secrets in your CI/CD platform
2. Set environment and release: Use TESTCHIMP_ENV and TESTCHIMP_RELEASE to track executions by environment and version
3. Enable screenshots: Keep captureScreenshots: true for better debugging of failures
4. Link to scenarios: Add // @Scenario: comments to enable requirement traceability
5. Review coverage: Regularly check requirement coverage in the TestChimp platform to identify gaps

Open Source Library

The playwright-testchimp-reporter is an open-source library available on GitHub:

Repository: https://github.com/testchimphq/playwright-testchimp-reporter

You can:

• View the source code
• Report issues
• Contribute improvements
• See the latest updates and releases

Troubleshooting

Reporter Not Sending Data

• Check credentials: Verify TESTCHIMP_API_KEY and TESTCHIMP_PROJECT_ID are set correctly
• Enable verbose logging: Set verbose: true to see detailed logs
• Check network: Ensure your CI environment can reach the TestChimp API

Tests Not Linked to Scenarios

• Verify scenario comments: Ensure // @Scenario: comments are present in your tests
• Check scenario titles: Scenario titles must match exactly (case-sensitive)
• Review test file paths: Ensure test files are in the expected location

Missing Screenshots

• Enable screenshot capture: Set captureScreenshots: true in config
• Check Playwright config: Ensure Playwright is configured to capture screenshots on failure

Running SmartTests in CI with the Playwright runner enables continuous requirement traceability while maintaining the flexibility and familiarity of standard Playwright test execution.