What Observational Data Can't Tell You About Work Done

Before you can pay for work, you have to be able to tell that it happened. So we built an observation harness that records an agent's session in forensic detail: every tool call intercepted before it runs, every token attributed, roughly thirty artifacts per session. The implicit bet was that a recording this close would let us score the work from it. This report settles that bet, against us. Across 119 recorded sessions the record reconstructs what the agent did with near-perfect fidelity, and tells us almost nothing about whether what it did was any good. Observation and verification look like different jobs, and this is a first measurement of how far apart they sit.

XO Labs Inc. · August 2026

TL;DR: We recorded 119 agent coding sessions. The 72 of them run against real repositories were graded offline against the maintainers' own tests, taken from the pull requests that fixed the bugs, and those 72 carry every quantitative result below. Then we asked which recorded channels predict completion. Three results. First, much of the record is inert: six instrumentation channels have zero variance across every run, including every alarm we built to catch an agent misrepresenting its own work. Second, the most promising process measure was an artifact of its own denominator. It appeared to beat the trivial baseline by 0.045 AUC, but it is undefined in 32 of 72 runs, and on the rows where it is defined the baseline scores 0.878 against its 0.661. The real sign is -0.217. Third, and most consequential for metering: the agent's own claim that it finished carries no usable information. It claims completion in 68 of 72 runs, is wrong 22 times, and read as a detector scores AUC 0.491, below chance. Verification cannot be delegated to the worker, and on this evidence it cannot be delegated to watching the worker either.

1. What we recorded

Atlas is our environment runner: it points a headless coding agent at a pinned repository and a task, runs that task across controlled conditions, and records the session rather than relying on the agent's account of it. Instrumentation is a pair of Claude Code hooks, one at session start and one that intercepts every tool call before it executes, and the agent itself is restricted to six tools: Bash, Read, Write, Edit, Glob and Grep.

The corpus is 126 environment recordings, of which 119 are study sessions: 47 on a purpose-built accounting codebase run under a deliberately starved step budget, and 72 on 18 real bug-fix tasks drawn from merged pull requests in two open-source Python projects. One model throughout (Claude Sonnet), one tool set, one enforced pacing rule of at most one tool call per assistant message.

A scoping note, because it governs how to read everything after §5. Independent ground truth exists only for the 72 real-repository runs, where the maintainers' own tests settle the question. Those 72 runs carry every quantitative result in §§6 to 8. The 47 practice sessions were scored against a synthesised acceptance battery rather than anyone's real test suite, so we use them for the channel inventory in §5 and for the degeneracy checks, and not for the detection results.

Each session leaves roughly thirty artifacts. The substantive ones:

That second-to-last channel was the ambitious part. reconcile.json attributes every fact the agent used to the channel that carried it, whether a bash result, a file read, or its own earlier thinking, and raises an alarm when the agent appears to possess something no channel delivered. We built it to catch confabulation.

In total: 11,814 files, 74.6 MB, 3,455 gate-intercepted tool calls, 294 graded acceptance criteria.

2. What we derived from it, and why we expected it to matter

The channels above are a log. The reason to keep them is the second layer we compute from each session, a set of summary measures meant to describe how the agent worked rather than only what it typed.

The main one is a phase split. Each session is divided around its first and last file edit: the stretch before the first edit is orientation, the stretch containing the edits is implementation, and everything after the last edit is verification. The harness reports seconds and tokens in each, and together they account for about 97% of wall-clock time. On top of that sit navigation measures, a tool mix, token accounting, and the gate's ordered record of every call.

Why any of this should matter. The bet is that competent work has a recognisable shape. An agent that reads the relevant files, changes them deliberately, then runs the tests and reads the output ought to look different from one that skims, guesses, and stops. If that shape is real and visible in the record, you can score work by watching it, which is cheap, continuous, and needs no test suite at all. That would be worth a great deal, because tests are expensive to build, frequently unavailable for the open-ended work people actually want to delegate, and always arrive after the fact.

That is the hypothesis §§6 and 7 test. One thing to say plainly up front: these measures were chosen after the sessions were recorded, not specified in advance, so the search is exploratory by construction. That is exactly why every result in §7 is reported against a permutation null that accounts for how wide the search was.

One detail that matters later: these derived measures are not all defined for every run. The phase split needs at least one file edit to divide on, so it is missing wherever the agent never edited anything. That sounds like bookkeeping. It turns out to be the trap in §6.

3. Preliminaries

Everything below turns on a handful of ideas. They are standard, but the report is unreadable without them, so here they are in plain terms.

What counts as done. For the real-repository tasks, the pull request that fixed the bug also added tests. Two sets matter. FAIL_TO_PASS are the tests that failed before the fix and pass after it, so they are the specific evidence the bug is gone. PASS_TO_PASS are the tests that already passed and must keep passing, so they catch a fix that breaks something else. A run is complete only if every test in both sets passes. Nobody's opinion enters this, including ours.

A false completion is a run where the agent said it had finished and the tests say it had not. That population is the one this report cares about most, because it is the one that costs money.

How we score a measure. Suppose you think some recorded quantity, say how long the agent spent editing, tells you whether the run succeeded. The AUC is the way we grade that hunch: pick one run that succeeded and one that failed at random, and ask how often your quantity ranks the successful one higher.

An AUC of 0.5 means the quantity is a coin flip. 1.0 means it is always right. A score below 0.5 is not useless, it is pointing backwards, and reading it in reverse gives you one minus the score. We always report the better of the two directions, so nothing in this report sits below 0.5 unless we say so explicitly.

AUC stands for area under the curve, meaning the receiver operating characteristic curve, and the plain-English version above is not a simplification of it. The area under that curve provably equals the probability that a randomly chosen success outranks a randomly chosen failure (Bamber, 1975; Hanley and McNeil, 1982), so we compute it straight from that definition as a Mann-Whitney statistic with midrank ties (Mann and Whitney, 1947) and never draw a curve at all. Fawcett (2006) is the clearest short introduction if you want one. One consequence matters in §6: comparing two AUCs measured on the same runs is a paired problem, not two independent measurements (DeLong et al., 1988).

Beating a baseline. A measure is only interesting if it beats something you could have got for free. Our baseline is simply how many tool calls the agent made, which requires understanding nothing about the session. Any proposed detector has to do better than that, on the same runs, or it has told us nothing.

Checking that a result is not luck. When you try many measures and report the best one, the best one will look good even if none of them work. So we shuffle the answer key at random, re-run the entire search on data we know contains no signal, and record the best score the search finds anyway. Repeating that 2,000 times, a permutation test, shows what the procedure produces from pure noise. If the real result sits inside that range, it is not a result.

Uncertainty. A confidence interval, written [a, b], is the range of values consistent with the data; a wide one means we have too few runs to be sure. We compute intervals by resampling, and we resample whole tasks rather than individual runs, because two runs of the same bug agree with each other 86% of the time and treating them as independent would make us look more certain than we are.

The rest of the vocabulary, once:

4. Ground truth had to be computed away from the observation

The harness could not grade the real-repository work, and the reason is structural rather than a defect. Ground truth for those tasks is the maintainers' own test patch from the merged pull request, and the test patch is the answer key. It cannot exist in the workspace while the agent is running, or the agent can read the tests and satisfy them directly. The harness grades during teardown, against a live workspace that has no test patch applied, and then deletes that workspace unconditionally.

So grading was rebuilt as a separate offline pass: check out the run's own baseline commit, apply the run's archived diff, reset the test tree, apply the maintainers' test patch, then run the tests the pull request made pass alongside those that must keep passing. The verdict is whatever the test runner reports. No model authors any part of it.

The result: 49 of 72 real-repository runs genuinely complete, all 72 patches applying cleanly, and, worth recording, zero runs in either condition touched the test files, which the work order placed out of scope.

5. Most of what we recorded carried nothing

With ground truth fixed, we could finally ask what the observation was worth. We classified every per-run channel by two questions: did it vary at all, and if it varied, was it already implied by something simpler?

Six channels have zero variance across the entire corpus. The probe and exposure paths are empty in all 126 recordings. The fact-trace file has 47 rows that are byte-identical to one another. The counters built to detect whether the agent read the structured context we planted read zero in every run, not because the context was absent, but because it arrived as an auto-loaded payload of roughly 10,766 tokens rather than through a tool call our counters could see. The instrumentation was watching the wrong door.

The confabulation alarms are the most instructive failure. Across all 119 sessions: 0 confabulations, 0 ordering violations, 0 unexplained possessions, 0 quarantines. The machinery runs correctly. It is simply attesting that nothing went wrong, in every run, which makes it a passed audit rather than a variable. We built an elaborate detector for a failure mode that does not occur in this corpus. Worth stating plainly: that is not the same as validating the detector. It has never had a positive case to catch, so we know it does not fire spuriously and we do not know that it fires when it should.

A second group is worse than inert, because it looks like signal. tool_calls_barriered, binarised at its own best threshold, is bit-identical to the step budget on 45 of 45 runs, which means it recovers the experimenter's own knob. tool_calls_total equals another tool-call count exactly on 72 of 72 runs, and turns_total binarises identically. In the real-repository setting, diff_nonempty and patch_bytes > 0 are constant at 1 across all 72 runs, which quietly voids any degeneracy test that uses them as a reference.

6. The measure that looked like a finding

One process measure did appear to work. Runs that falsely claimed completion spent far longer in their implementation phase, a median of 204 seconds against 66, giving an AUC of 0.661 against a trivial baseline of 0.616. It was a satisfying story: the agent that grinds is the agent that fools itself.

It does not survive contact with its own denominator.

Method. The detection target is gt_completed, established by test execution (§4). The trivial baseline is the number of tool calls, scored two-sided and stratified by condition, exactly as our prior work defines it. AUC is Mann-Whitney with midrank ties. The feature under test is the harness's implementation-phase duration. n = 72 runs; the feature is null or zero in 32 of them.

Result. The feature scores 0.661 on the 40 runs where it is defined. The baseline scores 0.616 on all 72. Scored on the same 40 rows, the baseline scores 0.878.

The apparent advantage of 0.045 becomes a deficit of 0.217 once both are read on the same rows. The feature is not a weak detector. It is a substantially worse one than counting tool calls, and it looked good only because it was quietly scored on an easier subset. Nothing about the arithmetic was wrong. The comparison was.

Widening the search does not rescue it. Across the program we scored 186 feature by dataset AUCs. Not one candidate feature, 0 of 9 on all runs and 0 of 9 on the subset of runs where the agent claimed completion, has a paired bootstrap interval for the difference against the baseline that excludes zero, once resampling respects the fact that repeated runs of the same task agree 86% of the time.

7. The bar itself was below the noise floor

If a search this wide produces a best feature that clears the baseline, the right question is how often noise would do the same. We shuffled the outcome labels and rescored the entire feature family, 2,000 times.

The median of the null distribution for the family maximum is 0.671, already above the 0.616 bar. The probability that a family this wide clears the bar on shuffled labels is 0.950, rising to 0.983 under a permutation that preserves run-to-run correlation. Beating the trivial baseline is what this procedure does by chance, essentially every time.

One feature does survive family-wise correction: a count of search calls, at joint p = 0.016. It is not a self-knowledge detector. Read within a task it scores 0.467 on 15 comparable pairs, which is chance, and it loses to a detector that reads nothing but which bug the run was assigned, 0.881 against 0.775. Remove that one feature and the family's significance collapses to p = 0.566. What it measures is task difficulty, which the agent's behaviour reflects but does not create.

8. The one informative channel is informative in the wrong direction

Against all of that, the agent's own report of its work does carry information. It points the wrong way.

Method. A completion claim is a label on the agent's final message, assigned by three independent raters blind to the test outcome. A message counts as claiming completion if it asserts the work is done; a message that flags remaining work, or that is too equivocal to read either way, does not. The three raters were unanimous on 71 of 72 messages, with a single three-way split and no majority-only cases. Ground truth is test execution. Confidence intervals are exact binomial; the κ interval is a cluster bootstrap resampling tasks, 200,000 draws.

Result. n = 72.

The agent claims completion in 68 of 72 runs. Twenty-two of those claims are false, a false-completion rate of 0.324 [0.215, 0.448]. When a run had genuinely failed, the agent withheld the claim once in 23 attempts, a specificity of 0.043. Read as a binary detector of its own success the claim scores AUC 0.491, below chance, and well below the 0.616 you get from counting tool calls. Cohen's κ is -0.023 [-0.161, 0.182].

The headline does not rest on where the ambiguous messages were placed. Four messages were equivocal, and the results above treat them as not claiming completion. Counting all four as claims instead moves the false-completion rate from 22 of 68 to 23 of 72, or 0.324 to 0.319, and changes nothing else.

The obvious explanation is that these agents skipped verification. They did not. Twenty-one of the 23 failing runs ran the test suite. Effort spent verifying after the last edit does not separate false completions from true ones. The agents looked, and reported success anyway.

We also compared two workspace conditions, with and without a structured operating scaffold, and found nothing: 24 of 36 complete against 25 of 36, five discordant pairs out of 36, exact McNemar p = 1.00. That is an absence of evidence at low power rather than evidence of absence, and we report it as such.

9. What this means for metering work

The quirq is defined as minted by verification and never self-reported. This corpus is the empirical case for why "never" is load-bearing rather than fastidious.

• The worker's report cannot contribute to the score. Not weighted, not as a prior, not as a tie-breaker. It is below chance here, and it fails in the expensive direction: confidently wrong on failures, which is precisely the population a payment system exists to catch.
• Watching the worker is not a substitute for checking the world. Thirty channels of forensic process observation, and no behavioural measure beat counting tool calls. The completion score V has to come from a state comparison, S₀ against S₁, checked by something the worker cannot reach, because on this evidence the trajectory does not contain it.
• A detector must be scored against a trivial baseline, on identical rows, under a permutation null. Two of our three near-misses came from violating one of those three conditions. Any future scoring model deserves the same three gates before it is believed.

What observation is genuinely good for is unchanged and still valuable: reconstructing what happened, attributing cost, and, through the archived diff and baseline commit, regrading offline whenever the definition of done improves. That last property is what allowed this analysis to exist at all. The recording is a provenance and audit substrate. It is not a scoring substrate, and detail alone does not make it one.

10. Limits, and what this is not

This is a first pass rather than a settled result.

The corpus is one model, one harness, and 18 real tasks, with task identity dominating outcome at AUC 0.881. The forced pacing rule of one tool call per message makes absolute counts incomparable to unconstrained sessions. The two settings differ in codebase, budget, and harness version simultaneously, so we deliberately do not report a contrast between them. Such a contrast is also fragile in a way worth recording: computed under one labelling convention it reaches p = 0.054, and under another it moves to p = 0.664. A comparison that can travel that far on a coding rule is not a finding, and cross-setting contrasts of this kind should be treated with corresponding suspicion.

We also found that our finest-grained ground truth is coarser than it looks. Within every one of 45 graded runs, the criteria that discriminate share an identical verdict, and 10 of 23 criteria never vary at all. Criterion-level analysis is the run-level result reweighted, not independent evidence.

The honest summary is that we built an observation apparatus considerably more detailed than the question could use. The next experiment should spend its budget on more tasks rather than more channels. Two things we would keep unchanged: grading offline from an archived diff against a baseline commit, and refusing to let the worker's account of its work enter the score.

11. How to reproduce this

A report arguing that unverified claims deserve suspicion should say how to check its own.

Grading is reproducible from the archive alone, without the harness. Each run keeps its diff, the commit it started from, and the instance definition naming the tests. Rebuilding a verdict means checking out that commit, applying the run's diff, resetting the test tree, applying the maintainers' test patch, and running the two test sets. That procedure is deterministic and re-runnable after any fix to the grading code, which is the property that made this analysis possible at all: a defect in the harness cannot silently define the truth it is being measured against.

The analysis is deliberately dependency-free. AUC, exact binomial intervals, Fisher, McNemar, the cluster bootstrap and the permutation nulls are implemented directly in the Python standard library rather than pulled from a statistics package, so every number can be read off the arithmetic rather than trusted to a library call. The extraction step writes three flat tables, one row per run, per tool call and per criterion, and the analysis reads only those.

Figures are generated from the same constants the prose cites, by a script that emits the SVG. No figure value is typed a second time, so a chart cannot drift away from the sentence next to it.

Two checks a reader can hold us to. Recomputing the trivial baseline from the raw tool-call log reproduces its published within-arm AUC of 0.616, which is the anchor that says the extraction is faithful. Read without stratifying by condition the same feature gives 0.614, and the two should not be quoted interchangeably. And the run counts reconcile exactly: 72 graded runs, 49 complete, 68 completion claims, 22 of them false.

The recordings, the grading records and the analysis scripts are held internally and are available to anyone inside the project who wants to re-run them. They are not published here: the recordings carry absolute machine paths, session identifiers, and contributor email addresses captured from repository history.

12. References

Verified against source. Venues, volumes and pages follow the primary record.

• Bamber, D. (1975). The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psychology 12(4), 387-415. Proves that the area under the ROC curve equals the probability a randomly chosen positive outranks a randomly chosen negative, which is the definition used throughout this report. ScienceDirect
• Hanley, J. A., and McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143(1), 29-36. The standard reference for interpreting an AUC. 10.1148/radiology.143.1.7063747
• Mann, H. B., and Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. Annals of Mathematical Statistics 18(1), 50-60. The rank statistic our AUC is computed from. 10.1214/aoms/1177730491
• Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters 27(8), 861-874. The accessible modern treatment. 10.1016/j.patrec.2005.10.010
• DeLong, E. R., DeLong, D. M., and Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44(3), 837-845. Why two AUCs measured on the same runs must be compared as a paired quantity, as in §6. Semantic Scholar
• Clopper, C. J., and Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26(4), 404-413. The exact binomial intervals reported for proportions. 10.1093/biomet/26.4.404
• Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1), 37-46. The κ statistic used in §8. 10.1177/001316446002000104
• Westfall, P. H., and Young, S. S. (1993). Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment. Wiley, New York. The maximum-statistic permutation procedure used in §7 to test a whole family of features at once. Wiley