Skip to main content

Exit-Safe Test Management: Keep Your Tests and Plans in Your Own Git Repo

· 20 min read
Nuwan Samarasekera
Nuwan Samarasekera
Founder & CEO, TestChimp

Last updated: 2026-09-30

TL;DR: TestChimp has two layers. The platform layer runs the QA loop and governance: it orchestrates agents working through product risks and coverage gaps, and holds verification status, execution history and coverage signals. The repo layer is yours: tests are standard Playwright specs and plans are markdown stories and scenarios, in your own git repo and synced both ways with TestChimp. If you leave, the repo and its full git history stay with you. Verified Tests status is the only thing that lives solely in TestChimp; nothing else is lost when you leave, because everything else is either in git or covered by TestChimp's full download of data not stored in git.

What does vendor lock-in look like in AI testing tools?​

Lock-in in testing tools rarely looks like a contract clause. It looks like one of these:

  • Tests stored as records in a hosted tool. You can read them in the vendor's UI, but leaving means an export you did not test until you needed it.
  • A proprietary test format or runner. The tests only execute inside the vendor's platform.
  • Requirements and test cases in a database. Plans are reachable only through the vendor's UI or API, so your coding agents cannot read them and migration means a data export project.
  • History that does not travel. Teams that have moved between test management tools report that imports keep case titles and steps but flatten the history (a practitioner comment, not a TestChimp claim).

AI test generation adds a new version of the same question: when an agent writes hundreds of tests for you, where do those tests live, and in what format? If the answer is "in the vendor's system", your investment sits on their side of the door.

TestChimp is built the other way around. Tests are standard Playwright (or Mobilewright for native mobile) files in your repo. Plans are markdown files with frontmatter in your repo. The platform adds workflow, verification and coverage intelligence on top of those files. Per the docs, it orchestrates a feedback loop so agents keep seeing product risk and coverage gaps and act on them, while your files stay the source of truth.

What lives in your repo?​

AssetWhere it livesFormat
Automated testsYour repo, in the folder you map as the tests rootStandard Playwright .spec.ts / .spec.js (Mobilewright for native mobile), no proprietary file extension
Requirements: user storiesYour repo, in the plans folder (for example plans/stories/)Markdown with YAML frontmatter
Requirements: test scenariosYour repo, in the plans folder (for example plans/scenarios/)Markdown with YAML frontmatter
Knowledge files and optional event specsYour repo, in the plans folderMarkdown
Traceability linksInside your test codeA Playwright annotation, { type: 'scenario', description: '#TS-<n>' }, so reviewers see links in the PR
Performance testsYour repo, in a reserved k6/ tree under the tests rootk6 scripts
CI and fixturesYour repoYour existing pipeline config, plus the reporter plugin

You choose the folder names. The tests and plans folders are mapped in the project's repository settings, and TestChimp writes small marker files (.testchimp-tests, .testchimp-plans) so agents can find the roots. Sync is two-way (bi-directional). Changes made in TestChimp reach your repo as pull requests or merge requests on branches prefixed testchimp-, so they go through your normal review, and commits to the mapped plans and tests folders come back into TestChimp through the provider webhooks. Plans are kept aligned with your default branch, while SmartTests sync branch by branch.

What lives in the platform instead:

  • Verification status. Verified Tests badges and who verified what are stored in TestChimp, not in source. This is the only thing that exists solely in TestChimp.
  • Data not stored in git. Execution history and coverage roll-ups (test-run results reported from CI, requirement coverage views, and release intelligence), coverage signals (TrueCoverage, which compares production user events with test runs; API contract coverage, which is API schema coverage built from captured API traffic; and semantic coverage, where stories, scenarios, tests and events are mapped by meaning in the Semantic Canvas), and workflow objects (issues, releases, test runs, and cloud meeting transcripts) are platform data rather than files in your repo. TestChimp supports a full download of all data that is not stored in git, so this data is not lost when you leave.
  • Stable IDs. Story and scenario IDs are tracked in the platform and referenced in the markdown frontmatter, which is how status and traceability stay consistent.

This split is deliberate. What a test does and what the product should do are files you own. Who inspected which test is platform state (Verified Tests status), and the remaining platform-side data can be downloaded in full. Meet-Bots and AgentWatch feed the plans side: they propose story and scenario updates from meetings and coding-agent chats, and you approve them into the plans folder, so the files stay current.

Details: SmartTests in your codebase, Code repository overview, GitHub sync for SmartTests, Requirement planning as code, Export plans to Git.

What does leaving TestChimp look like?​

Concretely, in the order a team would do it:

  1. Merge everything pending. Merge or close any open testchimp- pull requests so your default branch has the latest plans and tests. From here the repo is complete on its own.
  2. Keep your files. The plans folder, the tests folder, the k6/ tree, your Playwright config and your CI pipeline are already yours, with full git history. Nothing needs to be exported for these.
  3. Replace optional AI steps. If tests use ai.act, ai.verify or ai.extract, those steps run through the @testchimp/playwright plugin. Replace them with plain Playwright locators and assertions before you remove the plugin. Tests that never used AI steps are already plain Playwright.
  4. Remove the integration wiring. Take the @testchimp/playwright reporter out of playwright.config, delete the TESTCHIMP_API_KEY secret from CI, and remove agent wiring you added (MCP config, the ChimpHands workflow file if you used it, and the TrueCoverage RUM SDK if you instrumented an app).
  5. Disconnect the repository. Remove the GitHub App or the GitLab project access token and webhook. Stop allowing the testchimp- branch prefix if you had added a rule for it.
  6. Download your platform-side data. TestChimp supports a full download of all data that is not stored in git. Verified Tests status is the only thing that lives solely in TestChimp; nothing else is lost when you leave.
  7. Keep or clean up the markers. The marker files and the #TS-<n> annotations are harmless metadata. Plain Playwright ignores the annotations, and you can keep them as a readable link from each test to its scenario, or strip them.

Your tests keep running with npx playwright test, and your plans stay readable in any editor, Git host or agent. What you lose is the platform layer: the Verified Tests status and badges, and the hosted workflows and views. Apart from that status, your data is either in your repo or available through the full download.

Migrating in from TestRail or Jira​

Both imports are documented at Import from External Sources (Linear is supported too). Both start in Test Planning, Import / Export menu, then Import from... and the source you want.

Jira (user stories):

  • What you provide: Jira URL, email, an Atlassian API token, the project key, and optionally a JQL query to narrow what is imported. Credentials are used for the import and are not stored.
  • What you choose: which Jira issue type is treated as a user story (required) and, optionally, which is treated as a test scenario. Leave the scenario type blank if your test cases live in TestRail or another tool, and only stories are imported.
  • What you get: stories become TestChimp user stories, and scenarios (if you chose a type) become TestChimp scenarios linked to their parent story. You choose file naming (external ID, TestChimp ID, or AI deduce) and folder structure (grouped by a Jira field, or AI deduce), and the wizard previews counts first.

TestRail (test cases):

  • What you provide: TestRail URL, your login email, and an API key (API access must be enabled in TestRail under Admin > Site Settings > API). Credentials are used for the import and are not stored.
  • What you get: each TestRail case becomes a TestChimp scenario, a markdown file. The wizard previews the case count first, lets you choose file naming (external ID, TestChimp ID, or AI deduce) and folder structure (TestRail section hierarchy or AI deduce), and runs as an async job with progress.
  • Stories: the TestRail import brings in cases as scenarios only. If your stories are in Jira, import them first, and cases with matching Jira refs are linked to them automatically. Cases whose refs do not match an imported story are still imported, just without the link.

Both:

  • Undo: Import / Export > Revert recent import deletes everything created by the most recent import.
  • Trial limits: the docs list up to 25 imported user stories for Jira and up to 100 imported cases for TestRail on a free trial.

After the import, sync the new markdown to your repo with Sync to Git Repo, which raises a pull request you review and merge. Then link your automated tests to scenarios with the #TS-<n> annotation, and keep running the old system in parallel only until coverage overlaps. If you have an existing Playwright or other E2E suite instead, the /testchimp import workflow brings it in with one approved plan.

How does this compare with hosted test case stores?​

This compares the two models in general terms. It is not a feature-by-feature review of TestRail; check any specific vendor's current documentation.

QuestionHosted test case store (typical)Git-native (TestChimp)
Where do plans live?In the vendor's database, edited in its UIMarkdown files in your repo, reviewed by PR
How do automated tests link to cases?External references or IDs held outside the codeA scenario annotation inside the test code
Can a coding agent read the plans?Through the vendor's API or UIDirectly, as files in the repo
What does a PR reviewer see?Code, with plans in another tabCode, plans and links in one diff
What happens to your assets if you leave?Depends on the vendor's exportTests and plans are already in your repo
What stays vendor-side?EverythingOnly Verified Tests status; TestChimp supports a full download of the data that is not stored in git

Hosted stores have real strengths, especially for manual test case libraries and organisations with existing TMS mandates. If that is your situation, a hybrid is reasonable: import scenarios, sync the markdown to git, and keep the old system for historical audit while new work is git-native.

For plans and pricing, see testchimp.io/pricing. Meet-Bots usage is bundled with the Team plan (20 hours) and the Growth plan (80 hours), and AgentWatch is included in both plans. Team is $420 per month and Growth is $667 per month, billed annually (shown as the effective monthly price). For a side-by-side view, see TestChimp vs TestRail and Best TestRail alternative.

Frequently asked questions​

What is git-native test management?​

It means test plans and automated tests are files in your own Git repository, reviewed by pull request, and not records that exist only inside a vendor database. In TestChimp, plans are markdown stories and scenarios and tests are standard Playwright files.

Do my tests still run if I stop using TestChimp?​

SmartTests are ordinary Playwright spec files with no proprietary format, and npx playwright test runs them without a special runner. Optional AI steps and reporting rely on the @testchimp/playwright plugin, so expect to replace or remove those.

What data stays only in the TestChimp platform?​

Only Verified Tests status lives solely in TestChimp. Everything else is either in your git repo or covered by TestChimp's full download of data not stored in git, so nothing else is lost when you leave.

Can I migrate from TestRail or Jira to TestChimp?​

Yes. The TestRail import wizard turns each test case into a TestChimp scenario, and the Jira import wizard brings in user stories (and scenarios, if you pick a scenario issue type). Both preview the counts first and can be reverted after the import. The TestRail import brings in cases as scenarios only, so import Jira stories first to get the links.

Is there vendor lock-in with AI test generation tools?​

It depends on where the generated tests live and in what format. If they are standard framework files in your repo, you keep them when you leave; if they are stored as records in a hosted tool, check the export options before you commit.

How do plans and tests get into my repo?​

You connect GitHub or GitLab and map a tests folder and a plans folder. Sync is two-way: TestChimp raises pull requests or merge requests on branches prefixed testchimp- for changes made in the platform, which you review and merge like any other change, and commits to the mapped folders are pulled back into TestChimp.