Skip to main content

Running SmartTests in CI/CD

Overview​

You can run SmartTests as part of your CI/CD pipeline using the TestChimp GitHub Action. This enables automated testing on every commit, pull request, or release.

GitHub Actions Setup​

Basic Configuration​

Add the TestChimp GitHub Action to your workflow file (.github/workflows/test.yml):

name: Run SmartTests

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
      - name: Run TestChimp SmartTests
        uses: awarelabshq/testchimp-github-testrunner@v1
        with:
          testchimp-api-key: ${{ secrets.TESTCHIMP_API_KEY }}
          project-id: ${{ secrets.TESTCHIMP_PROJECT_ID }}

Configuration Options​

  • testchimp-api-key: Your TestChimp API key (store as GitHub secret)
  • project-id: Your TestChimp project ID (store as GitHub secret)
  • test-folder: Optional - specific folder to run tests from
  • parallel: Optional - run tests in parallel

How SmartTests Run in CI/CD​

Default Execution​

By default, SmartTests are executed as plain Playwright scripts:

  • Fast, deterministic execution
  • No agent intervention unless a step fails
  • Standard test output and reporting

Agent Intervention on Failures​

When a test step fails:

  1. The agent automatically intervenes
  2. It attempts to fix the failing portion agentically
  3. Changes are made to the script
  4. A Pull Request is created with the changes
  5. You have complete transparency into what was changed and why

This provides:

  • Fast execution when everything works
  • Automatic adaptation when things break
  • Transparency through PRs showing all changes

PR Creation for Script Changes​

When the agent makes changes to fix a failing test:

  1. A new branch is created with the fixes
  2. A Pull Request is opened with:
    • The original failing test
    • The changes made by the agent
    • Explanation of why changes were needed
  3. You can review, modify, or merge the PR as needed

This ensures you always know what changes were made and can review them before merging.

Best Practices​

  1. Store credentials securely: Use GitHub Secrets for API keys and project IDs
  2. Review agent changes: Always review PRs created by the agent before merging
  3. Test in parallel: Use parallel execution for faster test runs
  4. Monitor test results: Set up notifications for test failures
  5. Version your tests: Keep SmartTests in sync with your codebase using GitHub Sync

Running SmartTests in CI/CD ensures your application is continuously tested and issues are caught early in the development cycle.