Put QA on Auto-Pilot with Workflow Automations
TL;DR: We’ve shipped Automations in TestChimp—event-driven runs of the same catalog workflows you already invoke with /testchimp. When an issue is created, a story moves to ready, a CI batch fails, or a release lifecycle status changes, TestChimp can hand the work to a cloud agent via OpenHands or a labelled GitHub Issue. Aggregation collapses noisy bursts; optional human gates keep high-blast-radius runs under review. Local agents stay for interactive work. Automations cover the complementary case: the platform watches, then starts the agent for you.
The prompt-pasting tax
If you’ve been living in the agentic QA loop, you already know the playbooks:
/testchimp fix issue BUG-1042
/testchimp implement US-181
/testchimp fix test execution …
/testchimp run QA
That works. It also creates a new bottleneck: someone has to notice the event and paste the prompt.
| Failure mode | What actually happens |
|---|---|
| Latency | High-severity bug sits until a human opens the IDE |
| Inconsistency | Different people phrase the same job differently |
| Missed signals | Failed CI batch, ready story, release → Ready—none of them auto-start work |
| No audit trail | Chat history ≠ a tracked workflow execution with status and policy version |
| Noisy bursts | Ten related failures → ten agent sessions unless someone batches by hand |
Agents compressed execution. They did not invent routing. Automations are that routing layer—on the same workflows, policies, and workflow-execution-id traceability you already use.
Full product docs: Automations.
What is a TestChimp automation?
An automation is three parts:
Trigger + Action + Config
| Part | Answers | Examples |
|---|---|---|
| Trigger | When? | Entity (issue, story, scenario, test execution, batch, release) + event + optional AND conditions |
| Action | What? | Execute a catalog workflow with a task template, optional policy, and invocation strategy |
| Config | How loudly / how carefully? | Aggregation (Immediate / Buffered / Windowed) + optional human approval before invoke and/or before plan execute |
Policies stay *.policy.md in Git. Automations optionally attach them—the same files local /testchimp runs use. Workflows still define how the agent works. Automations define when that work starts and how the cloud agent is reached.
Project event → matching automation → aggregation window
↓
workflow execution queued
↓
(optional) human approve invoke / plan
↓
OpenHands conversation or labelled GitHub Issue
↓
/testchimp <workflow> … --workflow-execution-id …
Two ways to reach a cloud agent
Configure once under Project Settings → Automations, then pick a strategy per automation.
Via GitHub Issue (great default when Git is connected)
TestChimp opens an issue on your mapped repo. Body = the full /testchimp … prompt (plus the workflow execution id). Labels always include testchimp, plus any extras you configure.
Point Copilot coding agent, an OpenHands issue resolver, Codex, or your own runner at those labels. The agent checks out the repo, runs the prompt with the TestChimp skill + MCP, and reports back—so Executions → Workflow Executions stays current.
Via OpenHands
TestChimp starts an OpenHands conversation with the rendered prompt (and branch when known). Same project OpenHands config you may already use for one-click bug fixes / OpenHands integration.
| OpenHands | Via GitHub Issue | |
|---|---|---|
| Who starts the agent | TestChimp (API) | Your label-watching agent |
| Needs | OpenHands API key + mapped GitHub repo | TestChimp GitHub App + repo |
| Continuation (plan → execute) | New conversation | Comment on the same issue |
| Fit | You standardize on OpenHands | You want any GitHub-native agent queue |
Deep dive: Cloud agents for automations.
Aggregation and human gates (because autonomy without brakes is chaos)
Matching events should not always mean “spawn an agent immediately.”
| Policy | Behaviour |
|---|---|
| Immediate | Every match fulfills (still subject to the project hourly cap) |
| Buffered (default) | Wait for a quiet period (≥ 5 minutes), bounded by max window / max event count |
| Windowed | Fulfill on fixed hour boundaries—digest style |
Default rate limit: 5 cloud agent invocations per hour per project (shared). Over-cap runs stay Queued—not dropped.
Two independent human gates (off by default):
- Approve before invoke — review the rendered task before the cloud agent starts
- Approve before plan execute — agent plans first; you approve before execute
Pending work surfaces in Executions → Workflow Executions. You can also copy the prompt and run it locally—the workflow-execution-id keeps the timeline accurate either way.
Mechanics: How automations work.
Recipes teams actually want
| Signal | Workflow | Notes |
|---|---|---|
| High-severity issue created (not ExploreChimp noise) | fix-issue | Immediate or short buffer; start with invoke approval |
| Story / scenario → ready | implement | Buffered; prefer plan-execute approval |
| Automation batch completed with failures | fix-test-execution / upkeep | Buffered so one CI push → one agent run |
| Release → Ready | run-qa / run-release-check | Immediate; gate the composite |
| Steady drip of ready work | upkeep | Windowed hourly/daily digest |
Task templates use {{issue.id}}, {{story.title}}, {{batch.failed_count}}, {{project.repo_url}}, and friends—with autocomplete in the wizard.
More starters: Typical automation setups.
Why this matters in the agentic era
We already argued that skills are SaaS distribution—the playbook travels with the agent. We argued that requirements must be governed before agents spend tokens. We argued that release governance is the ship decision, not a spreadsheet.
Automations close another gap: intent without attendance.
Without them, QA-on-autopilot still needs a human dispatcher. With them, the same /testchimp catalog becomes a control plane: events in → policy-backed workflow executions out → auditable status in the product. Local Claude / Cursor for interactive loops. Cloud agents for the overnight and the “while you were in standup” cases.
That is how you boil the lake without standing next to the kettle (boiling the QA lake).
Frequently asked questions
What are TestChimp Automations?
Automations are event-driven rules that run TestChimp catalog workflows via a cloud agent when something changes in your project—issues, stories, scenarios, test executions, CI batches, or releases—optionally aggregated and gated by human approval.
How do Automations differ from Workflows?
Workflows define how an agent should work (skill playbook + optional policy). Automations define when that work starts and which cloud invocation path to use. Both share the same execution timeline and workflow-execution-id reporting.
OpenHands vs GitHub Issue—which should I pick?
Use GitHub Issue if you want any label-watching coding agent (Copilot, OpenHands resolver, Codex, custom bots) and already have the TestChimp GitHub App. Use OpenHands if TestChimp should start the conversation directly and you standardize on OpenHands cloud or self-hosted.
Will Automations spam my agents on every CI flake?
Not if you configure them well. Prefer Buffered aggregation for batches, exclude ExploreChimp reporters on auto-fix rules, use conditions (failed_count > 0, severity = HIGH), and keep the project hourly invocation cap. High-risk workflows should start with human gates on.
Can I still run workflows locally?
Yes. Automations don’t replace /testchimp in your IDE. From a pending execution you can also copy the prompt—including the workflow execution id—and run it with a local agent so status still updates in TestChimp.
Where do I create and monitor Automations?
Create under Workflows → Automations (or contextual Automations CTAs on Issues, Plans, batches, releases, and workflow pages). Configure strategies and rate limits in Project Settings → Automations. Monitor and approve under Executions → Workflow Executions.
Try it
- Connect GitHub (and optionally OpenHands) under Project Settings → Automations
- Open Workflows → Automations → Create Automation
- Pick a trigger (start from a popular condition template)
- Choose a workflow, task template, policy, and invocation strategy
- Leave Buffered aggregation on; add invoke or plan-execute approval for risky jobs
- Trigger a matching event—then watch Workflow Executions
Start here:
When something changes, the agent should already be working—not waiting for you to paste a prompt.
Further reading
TestChimp
- Automations
- How automations work
- Cloud agents
- OpenHands integration
- GitHub integration
- Fix issue
- Implement Requirement
Related posts
- From Bug Report to Pull Request (OpenHands)
- SKILLs as SaaS distribution
- Requirement quality governance
- Release governance
- Boiling the QA lake
- Test planning as code
Concepts