Observability in API Contract Governance

In brief: TestChimp maps runtime routes to OpenAPI operations and uses persisted daily volume, error, and latency summaries to prioritize functional coverage and performance-test upkeep without reading raw events.

Observability adds runtime evidence to the OpenAPI contract already configured in TestChimp. It answers:

• Which documented operations received traffic?
• Which observed routes do not match the OpenAPI contract?
• What were the operation's request rate, error rate, and latency?
• Which spec operations had no observed traffic during discovery?

Provider setup and credential requirements live in the authoritative Observability integrations guide.

How TestChimp uses the data

TestChimp performs read-only, provider-side aggregate queries. It does not download raw spans, logs, or request payloads from your observability provider.

1. Discover routes — TestChimp queries routes observed during the previous 24 hours.
2. Map the contract — Method and route templates are matched to operations in the selected OpenAPI service. Exact and safe normalized matches are automatic.
3. Expose drift — Observed routes that cannot be matched appear as Unmapped and can be manually assigned to an operation.
4. Collect hourly metrics — A scheduled job queries the prior completed hour and stores its aggregate result.
5. Collect daily summaries — A daily job queries the complete prior UTC day and stores precomputed RPM, error rate, status counts, and latency percentiles.
6. Render persisted values — The APIs list reads the latest completed daily summary. Until that summary exists, it reads the latest persisted hourly result. Metrics are never recalculated while loading the page.

What appears in the APIs list

Value	Meaning
RPM	Average requests per minute over the persisted daily or fallback hourly window
Error %	Provider-reported errors divided by requests for that window
Latency	Provider-reported p95 latency
Not observed	An OpenAPI operation had no route mapping from the 24-hour discovery window
Unmapped	Runtime traffic was observed, but its method and route did not match an OpenAPI operation

The mapping counts describe observed routes, not the number of operations in the spec. For example, Mapped 11 · Unmapped 32 means 43 runtime routes were discovered; it does not mean a 492-operation specification contains only 43 operations.

Operation detail

For a mapped operation, open the final Observability accordion pane to inspect its persisted runtime history. Contract coverage and observability remain separate signals:

• Coverage proves which operations and fields tests exercised.
• Observability shows which mapped operations production-like runtime traffic exercised.

Neither signal silently changes the other.

Use runtime evidence to prioritize coverage

Observability helps decide which coverage gap to address first. It does not make an operation covered by itself.

For functional API coverage:

1. Start with operations or fields that are not covered, partially covered, or not deliberately ignored.
2. Among those gaps, prioritize operations with high request count / RPM because a regression would affect more traffic.
3. Raise operations with high error rate / error count—especially 5xx responses—because they combine missing test evidence with observed failure exposure.
4. Keep business criticality and branch complexity in the decision. Authentication, authorization, state transitions, pagination, and error paths may outrank raw traffic.

TestChimp's create-tests, Run QA, and upkeep agent workflows can read the same daily summary through API operation list/detail tools. They use these values to propose the highest-impact uncovered operations, then inspect request fields, response fields, and status codes before extending an existing SmartTest or creating a focused test.

Prioritize performance coverage and upkeep

Use p95 / p99 latency to identify operations that most need a k6 journey or whose existing performance coverage should be reviewed. High volume and high error exposure strengthen that priority:

• Create performance tests prioritizes a high-latency operation when no current journey represents it.
• Performance upkeep prioritizes operations whose journey, baseline, dataset, or dependency model is missing or stale.
• Run performance tests can prefer related journeys for hot or slow operations when the requested scope contains more candidates than should run at once.

Production observations select what to test. They do not define k6 VUs, RPS, duration, dataset size, SLOs, or pass/fail thresholds. Those values must come from your capacity objectives and performance policy. Do not call a change a regression by directly comparing production p95/p99 with a k6 result unless environment, workload, time window, dataset, and dependency behavior are genuinely equivalent.

Check freshness before ranking

Read the observation window and sync status with every metric:

• The APIs list normally shows the latest completed UTC daily summary and may temporarily fall back to the latest completed hourly result.
• Missing, stale, partial, no-data, or query-failed observations mean unknown, not zero.
• requestCount is a window total; RPM is its rate view. Do not add them together.
• Compare volume, error, and latency values only across compatible windows and environments.

Configure a service mapping

1. Open Project Settings → Integrations → Observability.
2. Add and test a provider connection using its provider guide.
3. Open APIs and select an OpenAPI service.
4. Choose Map observability resource.
5. Select the provider connection and the provider resource corresponding to that service.
6. Run Sync now.
7. Filter the table by Mapped, Unmapped, or Not observed.
8. For an unmapped route, choose Map and select the correct OpenAPI operation.

Only map a provider resource to the OpenAPI service that owns those routes. Reusing one resource across unrelated services creates ambiguous mappings.

Data and credential handling

• Connection credentials are encrypted before storage.
• Provider calls are read-only.
• Stored runtime data consists of route mappings and aggregate hourly/daily metrics.
• Raw provider events and trace payloads are not copied into TestChimp.
• Removing a service mapping stops future collection for that service.

Next steps

• Observability integration overview
• Google Cloud Monitoring
• CloudWatch
• Datadog
• Honeycomb
• APIs list view
• API operation detail