3 posts tagged with "buildinpublic"

:::tip Updated workflow Since this post, the Manual tab also supports open-ended sessions (objective instead of scenario), Add bug on steps (deferred until session finish), and Atlas-backed screen/state selection. See the current guide: Manual test session capture. :::

Manual testing is still where teams catch the “this feels wrong” stuff:

• confusing UX
• copy bugs
• edge-case flows
• integration weirdness that doesn’t show up in clean scripted runs

But there’s a persistent problem: manual testing evidence doesn’t stay connected to what was planned.

Test plans live in one place. Notes and screenshots live in another. Pass/fail outcomes live in Slack threads or Jira comments. And the mapping back to the scenario is usually… memory.

Today we’re announcing a new workflow in TestChimp: Manual testing with traceability.

The idea​

If your team already does test planning (user stories → scenarios), then manual execution should be:

• tied to a specific scenario
• captured as step-by-step evidence
• recorded with environment + release context
• marked passed / failed
• queryable later as execution history (not a dead document)

That’s what this feature does.

Manual Test Session Capture (in the Chrome extension)​

In the TestChimp Chrome extension, there’s now a Manual tab.

It lets a tester record a manual session while they execute a planned scenario, with traceability stored directly on the record.

What gets captured:

• Steps: each interaction is captured as a step
• Screenshots: uploaded automatically
• Notes: add notes to the latest step, optionally highlighting a UI element/area
• Outcome: mark the session as passed or failed
• Context: environment + release (and optional git branch context)

How it works (quick walkthrough)​

1. Open the Chrome extension and switch to the Manual tab
2. Click Create Manual Test Record
3. Select the test scenario you’re executing (required)
4. (Optional) pick the git branch context
5. Click Start Capture and run the test as usual
6. Add notes when needed (with optional element/area attachments)
7. Click End capture, then mark passed or failed
8. Open View execution to see the full record in TestChimp

If you want the full documentation, see Manual Test Session Capture.

Why this matters (beyond “we captured a GIF”)​

Manual testing isn’t going away. But it needs to stop being unstructured.

With scenario-linked manual records, you can answer real questions without archaeology:

• Which scenarios were manually executed for this release?
• Where are we relying on manual validation because automation doesn’t exist yet?
• What’s the evidence behind a “pass” when something regresses later?
• What’s failing on a specific branch or environment?

It’s manual testing… but operationalized.

Unified coverage: manual + automated, in one view​

The bigger win is what happens after you capture manual execution.

Because manual sessions are linked to the same scenarios as your SmartTests, TestChimp can provide unified requirement coverage insights across:

• Automated runs (SmartTests in CI or in the Web IDE)
• Manual runs (scenario-linked manual sessions with evidence + pass/fail)

So instead of two separate worlds (a test management tool for manual, and CI dashboards for automation), you get one requirement-centric view:

• scenario coverage status
• recent execution history (pass/fail)
• evidence trail for manual validation
• clear gaps where scenarios have no automated coverage yet

This is the foundation for keeping your suite honest: manual validation is visible and it doesn’t get conflated with “we have automation”.

How coding agents consume this to prioritize test authoring​

Once coverage and history are unified at the scenario layer, agents can treat it as an ordered backlog—especially in workflows like /testchimp test (PR-level) and /testchimp evolve (portfolio-level).

In practice, the agent pulls:

• Requirement coverage (what scenarios are covered vs missing tests)
• Execution history (what’s failing or flaky right now)
• (Optionally) TrueCoverage signals (what real users do most, where they drop off)

Then it prioritizes authoring work where it has the highest leverage:

• uncovered high-priority scenarios first
• gaps in the exact folder/feature area the team owns
• high-traffic paths with low coverage (when TrueCoverage is enabled)

The end result is a tighter loop: manual + automated executions feed the same insights, and those insights drive agents toward the most important missing tests—rather than “write more tests” as a generic goal.

Manual vs Script Gen (important distinction)​

This workflow is intentionally not about generating automation.

• Use Manual when you want an auditable record of a tester executing a scenario (pass/fail, notes, screenshots).
• Use Script Gen when you want to capture steps and generate a Playwright SmartTest.

Try it​

• Install the Chrome extension
• Plan scenarios in Test Planning
• Run your next manual regression session through the Manual tab

If you have feedback on what would make manual execution records more useful (branch/release filtering, richer notes, better rollups), we’re actively iterating.

Q: How do you currently get requirement traceability?
Which user stories and scenarios are covered by tests, and what’s failing?

A:
For most teams, it looks something like this:

User stories live in Jira.
Test cases live somewhere else.
The mapping between them lives in an Excel sheet that someone manually maintains.

That spreadsheet is periodically uploaded to a test management tool like PractiTest. Then test execution results are pushed via an API to get a view of coverage and failures.

It works—until it doesn’t.

The problem with today’s approach​

This is how requirement traceability is typically achieved today: a hodgepodge of tools stitched together with process and hope.

• Multiple sources of truth
• Manually maintained spreadsheets that inevitably go stale
• Fragile workflows that break as teams and test suites scale

No single system actually owns the full picture. Instead, teams spend time keeping artifacts in sync rather than improving product quality.

A simpler model with TestChimp​

In TestChimp, requirement traceability isn’t an afterthought. It’s built in.

You already author detailed user stories and break them down into meaningful test scenarios directly in the platform - with AI assistance that understands your product through your existing test scripts and documentation.

Linking tests to those scenarios is intentionally simple. In your test script, add a comment:

// @Scenario: <scenario title>

That’s it! TestChimp takes care of the rest:

• Automatically links tests to scenarios
• Tracks execution results across runs
• Aggregates outcomes at scenario, story, and suite level

You get clear, real-time dashboards that let you answer business relevant questions:

• Which user stories are missing test coverage?
• Which scenarios are currently failing?
• Which tests are flaky or unstable?

All without juggling multiple tools or maintaining brittle Excel sheets.

⸻

One system, end to end​

Instead of retrofitting traceability after the fact, TestChimp treats it as a first-class concept - connecting requirements, scenarios, and executions in one place.

• No spreadsheets.
• No manual syncing.
• Just a single system that understands what you’re building, how it’s tested, and where the gaps are.

LLMs sound like humans – so we often end up instructing them as if they experience the world like us.

But there’s a subtle difference – especially when used as Agents.

👀 Humans experience a continuous stream of input and reasoning.

We build tiny hypotheses along the way:

“Let me hover over the tooltip to see what this button is for.”

It’s a loop of sense → reason → act, in continuity.

🧠 Agents, on the other hand, live in snapshots:

See screen → Decide → Act → See new screen.

They’re like a human who:

• Looks at the screen
• Writes a letter to a controller to perform an action
• Closes their eyes while it’s happening ← VERY IMPORTANT
• Opens their eyes to a new scene – with no memory of the past The only continuity? 📝

A notepad on the table – a few scribbled notes before they "blacked out".

So we asked ourselves:

“If this were me, how would I use that notepad?”

We’d been giving agents summaries of prior steps – but something was still missing.

So we made a small tweak to the prompt:

👉 “Write a note to your future self”

Result: the agent now jots down whatever it wants its future self to know, such as:

• What hypothesis it’s testing
• Why it chose this action
• What to look for in the new state

So in the next iteration when it wakes up, it knows: “What was I thinking?”

That single line — “Write a note to your future self” —

gave our agent a memory-like thread.

A small change. A big leap in clarity and navigation. 🚀

#AI #Agents #LLM #StartUp #BuildInPublic #AgenticAI