Tests Passed, But the Fix Is Not Live: Verify the Running Agent Service

A green test suite proves the artifact you intended. It does not prove the workload the reader sees. The two questions are different and the answers can disagree for reasons that have nothing to do with the test itself: a wrong revision was deployed, the right revision was deployed to the wrong target, the process never restarted, the route points elsewhere, the background job never ran, the cache still serves the old payload, the migration applied to a stale database, or the dependency the reader hits is failing in a way your test never touched. The right move is not to re-run the tests louder. The right move is to gather five independent pieces of evidence before declaring a fix live.

This guide gives a platform-neutral verification recipe. The exact commands belong to your runtime; the layers and the order do not.

Why green tests are not enough

A passing test answers one question: did the defined check pass against the defined artifact? It does not answer whether that artifact is the one running. It does not answer whether the runtime serving the artifact is the one your reader will hit. It does not answer whether the reader’s request will traverse the route, the cache, the worker, and the dependency you assume. Production guidance separates these explicitly: post-deployment verification is the step where the operator independently confirms that the running system matches the intended change, not the step where the operator trusts the test report.

A useful way to think about it: source control identifies the intended revision, the build identifies whether the defined checks passed, the deploy log identifies whether the delivery mechanism reported completion, and the runtime probe identifies whether a target service responds. Treat each as evidence of its own question and as nothing more. A green test is one piece of evidence out of five you need before you tell a reader the fix is live.

Symptom table

Match what you see to the row that fits, then run the cheap-first check for that layer. The “Confirms” column tells you which evidence layer is broken.

#	Symptom	Cheap-first check	Confirms
1	Tests pass, deploy log shows success, nothing changes for the reader	Read the running artifact’s revision or build identifier and compare to the one you intended	Artifact identity layer
2	Right artifact, but the deployed target is a different environment or host than the test target	List the targets your deploy pipeline wrote to and diff against your test environment	Target layer
3	Right artifact, right target, but the long-running process is still the old one	Check the running process started at time and look for restart signals after the deploy	Process restart layer
4	Right process, but the reader’s request hits a different route, cache, or upstream	Hit the same URL the reader hits, with cache disabled, from a network outside your test rig	Route and cache layer
5	Right route, but background jobs, schedulers, queues, or migrations did not run for the new revision	Check job status, scheduler run timestamps, and migration-applied markers against the new revision	Background work layer
6	All five above green, but a downstream dependency the reader touches is failing or rate-limited	Probe the dependency directly and check its status page for the time window of the deploy	Dependency layer

If row 1 is broken, no amount of process restarts will fix it. Fix in this order.

Diagnostic decision tree

Start at the top. Each node is a single question with one cheap probe. The branch you take tells you which fix path to follow.

• Did the running artifact’s identifier match the one you intended? No — wrong artifact shipped. Redeploy the intended revision. Yes — next.
• Did the deploy pipeline write to the targets you expected? No — wrong target. Re-point the pipeline or ship to the correct environment. Yes — next.
• Is the long-running process running the new revision (started-after timestamp, version string, hash, log line)? No — process did not restart. Restart the process against the new artifact. Yes — next.
• Does a probe from outside the test rig, with caches disabled, return the new behavior? No — route or cache points elsewhere. Investigate the route, the cache key, and any CDN or edge layer between your reader and the process. Yes — next.
• Did background jobs, schedulers, queues, or migrations execute against the new revision in the deploy window? No — background work did not run. Trigger the relevant job or migration manually, then verify. Yes — next.
• Is every dependency the reader’s request touches healthy in the deploy window? No — a dependency is failing or pacing. Either wait for recovery or roll back if the dependency is in your control. Yes — the fix is live. Announce.

The point of the order is that each step rules out one whole class of failure with one probe. If you skip ahead, you can spend hours investigating the wrong layer.

Five independent checks before you declare live

The recipe is platform-neutral. The exact commands depend on your runtime; use whatever your platform exposes to answer each question.

Check 1: artifact identity

What is the running artifact? Compare its identifier — revision, commit, build hash, image digest, or version string — to the one the green report claimed. If they disagree, the wrong artifact is live. This check is your own release evidence, not a claim that one generic deployment document can prove for every runtime. The independent post-deployment checks in Microsoft’s guidance are the useful principle here: verify the user journey, background processes, integrations, and monitoring rather than treating the delivery report as the finish line.

Check 2: target identity

Where did the deploy land? List the targets your pipeline wrote to (host, cluster, environment, region, slot) and confirm they are the ones your readers hit. A green deploy to a staging target does not prove anything about production. A green deploy to a canary slot does not prove anything about the rest of the fleet.

Check 3: process restart

Did the long-running service actually restart against the new artifact? Look at the process started-at timestamp, the log banner, the version endpoint, or any signal your runtime exposes. A common failure mode is that the deploy replaced the artifact on disk but did not signal the process to reload, so the process keeps serving the previous in-memory copy.

Check 4: route and cache

Does the reader’s URL return the new behavior when you probe it from a network outside your test rig, with caches disabled at every layer between the probe and the process? Edge caches, CDNs, reverse proxies, and application-level caches can each serve the previous payload. A green probe from the test rig is not the same as a green probe from the reader’s path.

Check 5: background work

Did the things that do not run on the request path execute against the new revision? That includes scheduled jobs, queue workers, migration runs, and any cron-like task that reads from the artifact. Kubernetes documents rollout status as the way to watch completion; Google SRE’s canarying workbook describes staged exposure so that background work gets exercised on a small population before the full fleet; Microsoft Learn’s safe-deployment guidance recommends allowing time to observe workload health between phases. The platform-specific command to check job status varies; the principle does not.

When to escalate

Escalate, do not retry, when:

• Two of the five checks fail and the failure points to two different layers at once — that is usually a wider incident, not a single fix.
• The artifact identity is wrong and you cannot explain why the deploy pipeline shipped a different revision than the one your test passed against.
• The reader-visible probe returns the old behavior from outside your test rig and your own edge cache cannot explain the gap — there is probably an upstream cache or DNS path you do not own.
• Background work did not run and triggering it manually has its own blast radius that you have not analyzed.
• A dependency in the reader’s path is degraded and rolling back your change will not restore reader-visible behavior.

Done means

You are done with this verification, not with the change, when all five of the following are true at the same time, in this order, without skipping ahead:

• The running artifact’s identifier matches the intended revision.
• The deploy landed on the targets your readers hit.
• The long-running process restarted against the new artifact, and its banner or version endpoint confirms the new revision.
• A reader-path probe, with caches disabled at every layer, returns the new behavior.
• Background jobs, schedulers, queues, and migrations executed against the new revision in the deploy window, and no dependency in the reader’s path is failing.

If any one of those is unclear, the fix is not yet verified. Do not announce.

What this article does NOT cover

• The mechanics of any specific runtime. The five layers and the order are platform-neutral; the exact commands to inspect them belong to your runtime’s documentation.
• A general argument against tests. Tests are necessary. They are not sufficient for verifying that a fix is live.
• A recipe for choosing a deployment platform, a rollout strategy, or an observability vendor. The verification recipe works against whatever you already chose.
• Recovery from a real incident. Once a fix is verified live, the next question (rollback versus forward-fix) is a separate decision and a separate guide.
• The cost question. Verification has a real cost in time and probes; that cost is a separate decision from whether to do it.

Related guides

• Why every agent needs a cost cap on day one — the cost-side companion, because a fix that you cannot verify is a fix you will pay to re-run.
• The 1-line observability hook that fits any agent — the request-path evidence layer that this guide assumes you have somewhere in your stack.
• Agents and the deploy gate: when they push — the deploy-pipeline discipline that ships the artifact; this guide covers the next question, whether the shipped artifact is the one the reader is seeing.

Research basis: Safe Deployment Verification and Rollback research dossier, Johnny5/Websites/Agentic Botsitting/research/safe-deployment-verification/safe-deployment-verification-research-2026-08-31.md (verified 2026-08-31).

Sources

• Deploy cloud-native solutions
• Safe deployment practices
• kubectl rollout
• docker service update
• Canarying releases
• Release engineering