QA bots for your team

In brief: Each team member connects a QA bot to a TestChimp project over OAuth, points TestChimp's webhook at it, and onboards it with their role. The bot then proposes requirement updates, E2E tests, issue fixes, and batch triage, and acts only after approval.

A QA bot is a team member's QA counterpart for one TestChimp project. It runs in a bot host (today: Grok Bot) with the TestChimp CLI, the testchimp agent skill, and a remote MCP connection to TestChimp. TestChimp sends it the events its owner cares about, such as their pushes, assignments, releases, and test batches. The bot suggests the next QA step and waits for approval before it acts.

Every team member sets up their own bot. Bots act only with their owner's permissions.

What the bot does

You choose this capability	The bot watches	It proposes (and runs after you approve)
Requirement updates	Your git pushes, meetings you add the bot to	The AgentWatch story/scenario update plan for your push, or updates from a meeting summary
E2E authoring	Your git pushes	New or updated E2E tests for the changed behaviour, then asks you to verify them
Issue fixes	Issues assigned to you	A fix plan for the issue
Manual test coordination	Manual scenarios assigned to you	A walkthrough of the steps, or automating the scenario
Test batch fixes	E2E and k6 batches on your branches	Fixes for failing tests, or a performance investigation for threshold breaches
QA posture	Releases created or changing status	A short go / no-go heads-up

It also runs a few routines: a daily self-update check, a weekday reminder of what's on your plate, and a weekly QA posture digest for QA leads and PMs.

Before you start

• Project init is done (one time per project, usually by a QA lead): /testchimp project init sets the plans/tests folders, test environment, and CI. The bot checks this during onboarding and tells you what's missing.
• A Grok Bot you can configure.
• Access to your computer for the bot if you want local flows: folder mapping, AgentWatch requirement updates and local test runs. These always run on your own computer, never on the bot's cloud computer, because AgentWatch reads the coding-agent chats and repo clone on your machine. You grant access in Grok; the bot asks before running anything that changes your machine.
• Node.js 20+ on your computer.

Set up your bot

1. Create the bot from the template

Create a Grok Bot from the TestChimp template. The grok-template.json manifest in the skill repo lists everything the bot needs:

• Connector: remote MCP at https://mcp.testchimp.io/mcp (OAuth).
• CLI: npm i -g @testchimp/cli@latest (0.1.85 or newer).
• Skill: testchimp, with system instructions from assets/bot/instructions.md.
• Routines: daily self-update, weekday reminder, weekly digest.

2. Connect to TestChimp (OAuth consent)

On its first run the bot adds the TestChimp connector itself (https://mcp.testchimp.io/mcp); you don't paste any URL. Click Add, then Authorize. Grok opens TestChimp's OAuth consent screen: sign in, pick the project, and approve. The bot gets a token for that project and your user. You don't need to paste an API key.

3. Point TestChimp's webhook at the bot

1. In Grok, copy the bot's webhook URL and webhook key.
2. In TestChimp, open User Settings → My Bots (/user-settings?tab=my-bots), pick the project on the left, then click the bot's card.
3. Paste the Webhook URL and Webhook key, then click Save webhook.
4. Click Check Connection. TestChimp sends a test event; the card shows Verified when the bot answered.

The same page shows your subscriptions, the bot's profile, and a Pause all switch. View recent deliveries lists the latest 50 events sent to the bot (queued, delivered, acked, expired).

4. Onboard the bot

Start a conversation with the bot ("set up my QA bot"). It walks you through:

1. The project's init status.
2. Your role (QA lead, PM, QA engineer, developer) and responsibilities in your own words.
3. Capabilities, pre-selected for your role. Confirm or adjust them.
4. Registering your profile and event subscriptions with TestChimp.
5. Webhook verification (if you skipped step 3).
6. Your local repo folder. On your computer, the bot runs testchimp workspace map to link your clone to the project. It checks that the folder is a git repo and that its remote matches the project's connected repository. The mapping is stored in ~/.testchimp/projects.json, which TestChimp Studio uses too.
7. A summary of what it will watch for and your reminder times.

Say "change my focus" anytime to redo the role and capability steps.

5. Connect AgentWatch (requirement updates)

For requirement updates from your pushes, the bot asks AgentWatch for the plan it drafted from your coding-agent chats. AgentWatch runs headless on the machine where you code. TestChimp Studio is not needed.

During onboarding the bot asks to run this on your computer:

testchimp bot connect --project-id <projectId>

Your browser opens the TestChimp consent page. Keep the project selected and click Allow. The page warns that your personal access key and the project's API key will be stored on this computer, in ~/.testchimp/agentwatch/credentials.json (readable only by you). Approving also turns AgentWatch on for that project. The bot then queries npx -y @testchimp/agentwatch query, shows you the plan, and runs it only after you approve.

To remove the stored keys, run testchimp bot disconnect --project-id <projectId>. If you already use TestChimp Studio, its sign-in and AgentWatch toggle work too (how).

Troubleshooting

Symptom	Fix
Check Connection fails	Re-copy the webhook URL and key from Grok, click Save webhook, and check again. The URL must be reachable over HTTPS.
No events arrive	Confirm the webhook shows Verified and Pause all is off. Check that your subscriptions in My Bots cover the event, for example git pushes are filtered to your own commits. Delivery can take 10–15 minutes.
Events keep coming back	The bot isn't acking. Ask it to check its CLI version (testchimp bot compat) and bot id.
Bot says it can't reach your computer	Grant the bot access to your computer in Grok. Folder mapping and AgentWatch only run there.
Bot says the folder is not mapped	Ask it to rerun the folder step, or run testchimp workspace map --project-id <id> --folder <path> yourself.
REPO_MISMATCH when mapping	The folder's git remote doesn't match the repository connected in Project Settings → Integrations. Pick the right clone, or fix the integration.
folder already mapped to project …	The folder belongs to another project. Map it with --reassign if you meant to move it.
AgentWatch: agentwatch_disabled or auth_required	No AgentWatch keys for this project on this machine. Run testchimp bot connect --project-id <id> (or sign in to TestChimp Studio and enable AgentWatch).
AgentWatch: below_baseline	AgentWatch is still learning this repo. Requirement updates start after the baseline is built.
AgentWatch: tick_in_progress	AgentWatch is analysing. The bot retries a few minutes later.
AgentWatch: agentsview_unavailable	The first run indexes your local coding-agent sessions, which can take a few minutes. The bot retries later.
Bot says its skill or CLI is outdated	Approve the update it proposes (npm i -g @testchimp/cli@latest, re-sync the skill).

Related

• AgentWatch — how plans are drafted from coding-agent chats
• TestChimp Studio
• Meeting Bots — meeting summaries the bot can turn into requirement updates