Setting up cloud agents

In brief: Shared repo prerequisites (skill, mcp.json, TESTCHIMP_API_KEY secret) plus accurate, provider-specific setup for agents that pick up TestChimp's labelled GitHub issues—and OpenHands for direct API or issue-resolver paths.

When an automation uses Via GitHub Issue, TestChimp opens an issue on your mapped repository. The body is a full /testchimp … prompt (including --workflow-execution-id). Your coding agent must watch the labels, load the TestChimp skill, talk to the platform through @testchimp/cli MCP, and report progress so Workflow Executions updates.

This page covers that listener setup. For how TestChimp chooses OpenHands vs GitHub Issue, see Cloud agents.

Shared prerequisites (all GitHub Issue listeners)

Do these once per repository before configuring a specific agent product.

1. Connect GitHub in TestChimp

Install the TestChimp GitHub App and select the repository for the project.
Confirm labels under Project Settings → Automations (built-in testchimp, plus any extras from Integrations → GitHub).
Create an automation that uses Via GitHub Issue.

2. Store the project API key as a GitHub secret

In TestChimp → Project Settings → Key management, copy the project API key.
In the GitHub repo → Settings → Secrets and variables → Actions, add:
- TESTCHIMP_API_KEY — required
- Optionally TESTCHIMP_PROJECT_ID (TrueCoverage / RUM) and TESTCHIMP_BACKEND_URL (staging / enterprise / self-hosted only)

Never commit the raw key. Agents and Actions should inject it at runtime.

3. Commit a project MCP config that runs `@testchimp/cli@latest`

Add a committed MCP file the skill can find. Preferred paths (first match wins for the skill walk-up):

Path	Typical host
`.cursor/mcp.json`	Cursor
`.mcp.json`	Claude Code
`mcp.json`	Generic / multi-host

Use npx with @testchimp/cli@latest so cloud runs always pull the current CLI:

{
  "mcpServers": {
    "testchimp": {
      "command": "npx",
      "args": ["-y", "@testchimp/cli@latest", "mcp"],
      "env": {
        "TESTCHIMP_API_KEY": "${TESTCHIMP_API_KEY}",
        "TESTCHIMP_PROJECT_ID": "${TESTCHIMP_PROJECT_ID}"
      }
    }
  }
}

${TESTCHIMP_API_KEY} is a placeholder for runtime injection. How each host expands secrets differs—see the provider sections below. Do not paste a live key into the committed file.

For enterprise / non-prod, also set TESTCHIMP_BACKEND_URL in the same env block (and export it into any CLI / Playwright child process).

4. Install the TestChimp skill in the repo

Cloud agents only see skills that live in the repository (or that the host installs into the run). Commit the skill from testchimphq/testchimp-skills:

mkdir -p .agents/skills
git clone --depth 1 https://github.com/testchimphq/testchimp-skills.git .agents/skills/testchimp
# Optional: drop .git if your org prefers a vendored copy without nested remotes

Also accepted by common hosts (pick what your agent loads):

.cursor/skills/testchimp/
.claude/skills/testchimp/
.github/skills/testchimp/ (GitHub Copilot)

Keep the skill updated (git pull in that directory, or re-clone). The skill’s preamble expects this tree so /testchimp … workflows resolve.

5. What the agent must do when an issue appears

Read the issue body and run the /testchimp … command as written (preserve --workflow-execution-id and --mode=non-interactive when present).
Prefer MCP tools from the testchimp server; fall back to CLI only after exporting TESTCHIMP_API_KEY (and backend URL when configured) into the shell.
Report agent actions so the workflow execution completes in TestChimp.

Cursor Cloud Agent

Official docs: Cloud agents, Automations, MCP capabilities, Skills.

Trigger on TestChimp labels

Connect the repo to Cursor (GitHub App / Cloud Agents dashboard).
Create an automation at cursor.com/automations (or Agents Window → Automations).
Add a GitHub → Issue label changed trigger for your TestChimp labels (at least testchimp).
Set the automation repository to the same repo TestChimp maps.
Prompt the agent to execute the issue body as a TestChimp workflow (skill + MCP), for example:

When this issue is labelled for TestChimp, run the /testchimp command in the issue body exactly. Load the TestChimp skill from .agents/skills/testchimp (or .cursor/skills/testchimp). Use the TestChimp MCP tools. Do not invent a different workflow.

:::caution Issue label trigger UI Cursor’s docs list Issue label changed. Community reports (2026) that some Automations UIs only expose PR label triggers even though the backend documents issue labels. If you cannot select an issue-label trigger, use Issue comment / @cursor on the issue, or a thin GitHub Action that calls the Cursor Cloud Agents API when testchimp is applied—until the UI exposes issue labels for your account. :::

You can also mention @cursor on an issue to start a session manually.

Skill

Commit the skill under .cursor/skills/testchimp/ or .agents/skills/testchimp/ so cloud runs clone it with the repo.

MCP and `TESTCHIMP_API_KEY`

Cursor Cloud Agents load MCP from the Cloud Agents / team dashboard, not reliably from repo .cursor/mcp.json alone. Cursor staff guidance: configure servers at cursor.com/agents (personal) or Dashboard → Integrations & MCP (team).

Add a stdio server matching the committed template:

Command: npx
Args: -y, @testchimp/cli@latest, mcp
Env: set TESTCHIMP_API_KEY (and optional project id / backend URL) in the dashboard secret fields—values are encrypted and redacted after save

Still commit .cursor/mcp.json (or mcp.json) so the TestChimp skill can discover the same shape for CLI export and local agents. Dashboard env is what the cloud MCP process actually receives.

Ensure the cloud environment has Node.js so npx works (Cloud agent setup).

Claude Code (GitHub Action)

Official docs: Claude Code GitHub Actions, anthropics/claude-code-action, custom automations, configuration / MCP.

Setup

Install the Claude GitHub App on the repo (or run /install-github-app in Claude Code).
Add repository secrets: ANTHROPIC_API_KEY, TESTCHIMP_API_KEY (and optional TestChimp project / backend secrets).
Commit the skill under .claude/skills/testchimp/ (or .agents/skills/testchimp/) and a project .mcp.json as in Shared prerequisites.
Add a workflow that runs when TestChimp labels are applied. With a prompt input the action runs in automation mode (no @claude mention required).

Example (adapt label names to match TestChimp):

name: TestChimp via Claude Code
on:
  issues:
    types: [labeled]

permissions:
  contents: write
  pull-requests: write
  issues: write
  id-token: write

jobs:
  testchimp:
    if: github.event.label.name == 'testchimp'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Claude Code on TestChimp issue
        uses: anthropics/claude-code-action@v1
        env:
          TESTCHIMP_API_KEY: ${{ secrets.TESTCHIMP_API_KEY }}
          TESTCHIMP_PROJECT_ID: ${{ secrets.TESTCHIMP_PROJECT_ID }}
        with:
          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
          prompt: |
            Execute the TestChimp workflow described in issue #${{ github.event.issue.number }}.
            Use the issue body as the authoritative /testchimp command (including flags).
            Load the TestChimp skill from .claude/skills/testchimp or .agents/skills/testchimp.
            Prefer TestChimp MCP tools; export TESTCHIMP_API_KEY into any CLI/Playwright child process.
          claude_args: |
            --mcp-config '{"mcpServers":{"testchimp":{"command":"npx","args":["-y","@testchimp/cli@latest","mcp"],"env":{"TESTCHIMP_API_KEY":"${{ secrets.TESTCHIMP_API_KEY }}","TESTCHIMP_PROJECT_ID":"${{ secrets.TESTCHIMP_PROJECT_ID }}"}}}}'
            --allowedTools mcp__testchimp__*

:::tip MCP config for the Action Pass TestChimp MCP as inline JSON on --mcp-config in claude_args. File-path --mcp-config values can be dropped when the action also injects built-in GitHub MCP JSON (known merge behaviour). Explicitly allow TestChimp tools (mcp__testchimp__* or listed tools). :::

Interactive fallback: omit prompt and mention @claude on the issue so Claude responds in tag mode—still requires the skill + MCP wiring above.

OpenAI Codex

Official docs: Codex GitHub Action (openai/codex-action@v1), Codex GitHub code review, Customization / skills & MCP.

What is natively supported

Path	Issue-label listener?	Notes
Codex GitHub App / cloud	Not as a first-class issue-label product	Documented GitHub triggers centre on pull request review (`@codex review`) and non-review `@codex` comments on PRs, which start a cloud chat with the PR as context—not “watch label `testchimp` on issues.”
`openai/codex-action@v1`	Yes (you wire it)	Official Action runs `codex exec` in CI. Trigger on `issues: [labeled]` yourself and pass the issue body as the prompt.

Recommended path for TestChimp automations

Add OPENAI_API_KEY and TESTCHIMP_API_KEY (plus optional TestChimp secrets) to the repo.
Commit the TestChimp skill (e.g. .agents/skills/testchimp/) and mcp.json / .mcp.json.
Add a workflow:

name: TestChimp via Codex
on:
  issues:
    types: [labeled]

permissions:
  contents: write
  pull-requests: write
  issues: write

jobs:
  testchimp:
    if: github.event.label.name == 'testchimp'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5

      - name: Run Codex on TestChimp issue
        uses: openai/codex-action@v1
        env:
          TESTCHIMP_API_KEY: ${{ secrets.TESTCHIMP_API_KEY }}
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          sandbox: workspace-write
          prompt: |
            Execute this TestChimp workflow exactly (from GitHub issue #${{ github.event.issue.number }}):

            ${{ github.event.issue.body }}

            Load the TestChimp skill under .agents/skills/testchimp.
            Use @testchimp/cli@latest MCP (npx) with TESTCHIMP_API_KEY from the environment.

Point codex-args / codex-home at an MCP config that registers TestChimp if your Codex profile expects a config file; otherwise ensure the skill preamble can export the key from the committed mcp.json and the job env.

:::note Codex cloud vs Action Do not assume that enabling Codex code review on the repo will pick up TestChimp-labelled issues. For issue-driven TestChimp automations, use openai/codex-action (or another explicit CI bridge). PR @codex remains useful for review/fix-on-PR flows, which is a different entrypoint than TestChimp’s GitHub Issue strategy. :::

GitHub Copilot coding agent

Official docs: Kick off a task, Configure MCP servers, Add agent skills, Agent environment setup.

Trigger model (important)

Copilot cloud agent starts when you assign the issue to Copilot (Assignees → Copilot), not merely when a label is added. TestChimp’s GitHub Issue strategy creates and labels the issue; it does not assign Copilot.

Practical options:

Manual / semi-manual: After TestChimp opens the issue, assign Copilot (optional custom agent / prompt in the assign dialog).
Automation bridge: Add a small GitHub Action on issues: [labeled] that assigns Copilot via the GitHub API / GraphQL when the label is testchimp (requires a token with permission to assign Copilot; follow current GitHub Copilot API docs for your plan).
Agents panel: Start a task from github.com/copilot/agents with a prompt that points at the issue.

:::warning Labels alone are not enough Watching testchimp without assigning Copilot (or starting an agents-panel task) will not run the coding agent. Plan for assignment or an API bridge. :::

Skill

Install the TestChimp skill as a project skill:

mkdir -p .github/skills
# clone or copy testchimp skill → .github/skills/testchimp/SKILL.md

.claude/skills/ and .agents/skills/ are also discovered per GitHub’s skills docs. Prefer committing under .github/skills/testchimp/ for Copilot cloud agent.

MCP and secrets

Repo Settings → Copilot → MCP servers — paste an mcpServers JSON entry for TestChimp (type: local or stdio, command: npx, args: ["-y","@testchimp/cli@latest","mcp"], tools: ["*"] or an allowlist).
Store the key as an Agents secret named with the required prefix, e.g. COPILOT_MCP_TESTCHIMP_API_KEY.
Reference it in MCP env, for example "TESTCHIMP_API_KEY": "$COPILOT_MCP_TESTCHIMP_API_KEY".

Only secrets/variables prefixed with COPILOT_MCP_ are available to Copilot MCP config. Copilot cloud agent supports MCP tools only (not MCP resources/prompts), and does not support remote MCP that requires OAuth.

Optional: .github/workflows/copilot-setup-steps.yml with a single job named copilot-setup-steps to install Node / repo deps before the agent runs.

Also commit mcp.json / .mcp.json for the TestChimp skill’s discovery walk-up (local agents and preamble), even though Copilot’s runtime MCP is configured in repo Settings.

OpenHands

OpenHands appears in TestChimp in two ways. Both need the same GitHub repo, the TestChimp skill, and TestChimp MCP (@testchimp/cli@latest + API key).

Official docs: MCP settings, CLI MCP, Adding skills, GitHub Action / resolver, OpenHands integration (TestChimp).

A. TestChimp → OpenHands API (invocation strategy)

Used when the automation action selects OpenHands.

In OpenHands (cloud or self-hosted), connect the same GitHub repository TestChimp maps.
Create an OpenHands API key.
In TestChimp → Project Settings → Automations (or Integrations → OpenHands), paste the key and set cloud vs self-hosted URL. Teams tier required.
In OpenHands (Cloud UI Settings → MCP, or CLI ~/.openhands/mcp.json / project MCP), register TestChimp:

{
  "mcpServers": {
    "testchimp": {
      "command": "npx",
      "args": ["-y", "@testchimp/cli@latest", "mcp"],
      "env": {
        "TESTCHIMP_API_KEY": "<from OpenHands secret store / env — not committed>"
      }
    }
  }
}

Install the skill into the repo OpenHands checks out, e.g.:

mkdir -p .agents/skills
git clone --depth 1 https://github.com/testchimphq/testchimp-skills.git .agents/skills/testchimp

Or in a conversation: /add-skill pointing at your vendored skill path / GitHub tree once published under your org. OpenHands loads project skills from .agents/skills/.

Commit a discoverable mcp.json / .mcp.json in the repo (with ${TESTCHIMP_API_KEY} placeholders) so the skill preamble can resolve keys when the runtime injects them.

When the automation fires, TestChimp starts an OpenHands conversation whose initial_user_msg is the /testchimp … prompt. Plan-execute approval starts a new conversation (not an in-place resume).

B. OpenHands GitHub resolver (issue labels)

Use this when TestChimp uses Via GitHub Issue and OpenHands should pick up the labelled issue in GitHub Actions.

Follow the OpenHands Resolver README: copy .github/workflows/openhands-resolver.yml, set LLM_API_KEY (and optional PAT_TOKEN / PAT_USERNAME).
Default trigger label is fix-me. Either:
- Add fix-me as an extra issue label in TestChimp GitHub config so created issues carry both testchimp and fix-me, or
- Change the resolver workflow / OPENHANDS_MACRO / label filter to watch testchimp (see OpenHands custom configuration variables).
Ensure the resolver environment has Node, can run npx @testchimp/cli@latest, and receives TESTCHIMP_API_KEY (repository secret exported into the job).
Commit .agents/skills/testchimp and repo MCP config as above.

Macro alternative: comment @openhands-agent (or your OPENHANDS_MACRO) on the issue—scoped to that comment plus the issue description.

Capability summary

Agent	Native “label on issue” listen	Skill install	TestChimp MCP
Cursor Cloud	Documented Issue label changed automation (verify UI availability)	`.cursor/skills` / `.agents/skills` in repo	Dashboard / team MCP (stdio `npx @testchimp/cli@latest`); commit `mcp.json` for skill discovery
Claude Code Action	Yes — `issues: [labeled]` + `prompt`	`.claude/skills` / `.agents/skills`	Inline `--mcp-config` JSON + `TESTCHIMP_API_KEY` secret
OpenAI Codex	Not native for issues; use `openai/codex-action` on labeled issues	`.agents/skills` (+ `AGENTS.md` as needed)	Job `env` + skill/`codex` MCP profile
GitHub Copilot	No — requires assign Copilot (or API/agents panel)	`.github/skills/testchimp`	Repo Settings → Copilot → MCP + `COPILOT_MCP_*` secrets
OpenHands (API)	N/A (TestChimp starts conversation)	`.agents/skills/testchimp`	OpenHands MCP settings / `mcp.json`
OpenHands (resolver)	Yes — default `fix-me` (remap or dual-label)	Same	Secrets on the Actions job + MCP/skill in repo

Troubleshooting

Symptom	Check
Issue created, agent idle	Provider trigger (label vs assign vs `@mention`); wrong repo; label name mismatch
Agent runs but cannot call TestChimp	MCP not registered; key missing in host secret store; `npx`/Node missing in cloud VM
`401` from TestChimp	Export `TESTCHIMP_BACKEND_URL` when using staging/enterprise; key must match that backend
Skill not used	Skill not committed under a path the host loads; outdated clone
Workflow execution stuck in TestChimp	Agent must keep `--workflow-execution-id` and report agent actions

Shared prerequisites (all GitHub Issue listeners)​

1. Connect GitHub in TestChimp​

2. Store the project API key as a GitHub secret​

3. Commit a project MCP config that runs @testchimp/cli@latest​

4. Install the TestChimp skill in the repo​

5. What the agent must do when an issue appears​

Cursor Cloud Agent​

Trigger on TestChimp labels​

Skill​

MCP and TESTCHIMP_API_KEY​

Claude Code (GitHub Action)​

Setup​

OpenAI Codex​

What is natively supported​

Recommended path for TestChimp automations​

GitHub Copilot coding agent​

Trigger model (important)​

Skill​

MCP and secrets​

OpenHands​

A. TestChimp → OpenHands API (invocation strategy)​

B. OpenHands GitHub resolver (issue labels)​

Capability summary​

Troubleshooting​

Related​

Shared prerequisites (all GitHub Issue listeners)

1. Connect GitHub in TestChimp

2. Store the project API key as a GitHub secret

3. Commit a project MCP config that runs `@testchimp/cli@latest`

4. Install the TestChimp skill in the repo

5. What the agent must do when an issue appears

Cursor Cloud Agent

Trigger on TestChimp labels

Skill

MCP and `TESTCHIMP_API_KEY`

Claude Code (GitHub Action)

Setup

OpenAI Codex

What is natively supported

Recommended path for TestChimp automations

GitHub Copilot coding agent

Trigger model (important)

Skill

MCP and secrets

OpenHands

A. TestChimp → OpenHands API (invocation strategy)

B. OpenHands GitHub resolver (issue labels)

Capability summary

Troubleshooting

Related