# Twenty synthetic agent task comparisons

Measured October 1, 2026. These are exploratory comparisons of doing a task locally versus calling a known ComputeFirst tool through a prepared client.

Under the frozen grade, the service arm passed 20/20 tasks; the local arm passed 19/20. The one excluded pair had an ambiguous input wrapper: the local answer flattened the whole supplied object, while the tool and oracle flattened input.value. That difference does not establish a correctness advantage for the service. In the 19 pairs where both answers passed, service use consumed more total input tokens and took longer in every pair. Its median increase was 19654 input tokens and 7.738 seconds. This sample does not demonstrate token or time savings.

Use a local method when it is already available and cheaper to run and check. A matching tool contract and a correct answer do not by themselves show a saving.

## Method and limits

One exploratory pair per synthetic task; no within-task repetitions. Separate preliminary smoke pair retained outside these totals.

Fresh local solver with Node, csv-parse and semver; no service call.

Fresh solver with the same task plus a known public tool contract and a prepared parent-observed client. One fixed HTTP execution per service trial.

Same exact canonical JSON oracle check for both arms; incomplete, altered-packet, scope-flagged or unsuccessful-call trials fail. Differences are null unless both arms pass. The json-flatten-escapes prompt did not identify input.value explicitly. Its excluded local answer flattened the whole supplied object; this envelope-interpretation mismatch does not establish algorithmic failure or service correctness superiority. The frozen fixture and grade are retained; a future trial must clarify the wrapper.

Native CLI completion usage includes full context and verification. Reasoning tokens are a subset of output, never added again. Prepared client setup is excluded.

Positive difference means the service used less; negative means more. Cache was observed, not controlled, so uncached-input differences do not establish service-caused or cash savings.

Not organic discovery, first installation, repeated-use economics, representative demand, or a general savings rate. Native stream did not report actual response model identity.

Actual cost and cash saved are unknown.

Requested model: gpt-6.1-sol, high effort. Actual response model identity: unknown; the native stream did not report it. Each task received one pair, with launch order counterbalanced. No cash costs were measured. These tasks are synthetic; they do not show outside customer adoption.

## Measured totals for pairs where both passed

Positive differences mean the service used less; negative differences mean more. The failed local answer remains in the task table and JSON receipt; it contributes no savings difference.

| Metric | Local total | Service total | Local minus service | Median difference |
| --- | ---: | ---: | ---: | ---: |
| input_tokens | 366478 | 740393 | -373915 | -19654 |
| uncached_input_tokens | 221070 | 248745 | -27675 | -443 |
| output_tokens | 1989 | 1996 | -7 | -9 |
| completion_wall_ms | 187294 | 333518 | -146224 | -7738 |

Cached and uncached input differences are observed, not controlled or billed prices. Reasoning output is already included in output tokens.

## Every task

| Task | Local correct | Service correct | Input token difference | Completion time difference, ms |
| --- | --- | --- | ---: | ---: |
| csv-quoted-rows | yes | yes | -19643 | -7738 |
| csv-key-dedupe | yes | yes | -19732 | -5765 |
| csv-select-reorder | yes | yes | -19682 | -4156 |
| csv-group-count | yes | yes | -19709 | -6239 |
| json-pointer-escapes | yes | yes | -19624 | -7432 |
| json-merge-patch | yes | yes | -19635 | -9234 |
| json-flatten-escapes | no | yes | unknown | unknown |
| json-stable-sort | yes | yes | -19789 | -9376 |
| date-leap-crossing | yes | yes | -19652 | -4423 |
| date-month-clamp | yes | yes | -19681 | -10021 |
| date-business-holiday | yes | yes | -19851 | -7927 |
| exact-decimal-add | yes | yes | -19654 | -6845 |
| exact-decimal-subtract | yes | yes | -19627 | -7729 |
| text-line-endings | yes | yes | -19625 | -11044 |
| text-trailing-space | yes | yes | -19654 | -9963 |
| text-leading-tabs | yes | yes | -19691 | -9090 |
| semver-precedence | yes | yes | -19654 | -10590 |
| semver-caret-zero | yes | yes | -19708 | -7696 |
| url-query-order | yes | yes | -19701 | -9518 |
| utf8-base64 | yes | yes | -19603 | -1438 |

[Full numeric receipt and method](/evidence/agent-task-comparison-2026-10-01.json). The receipt contains all twenty pairs, native usage counters, outcome grades, frozen artifact hashes and parent-observed service-response hashes. Cash saved remains unknown.