News: Agent Evaluation Science Symposium 2026

• News
• Research

Excited to share that our paper, “Accounting for Every Agent-Second in LLM Teams,” will appear at the Agent Evaluation Science Symposium!

A team of m language-model agents usually finishes a task less than m times faster than a single agent. Speedup tells you how much faster the team was. It says nothing about where the rest of the time went. We propose an agent-time account that gives every agent-second of a run exactly one cause from a fixed set, including new work, rework, overwriting a teammate’s work, and four kinds of waiting, so that the causes add up to m times the run time.

Any such breakdown sums correctly by construction, so the sum on its own proves nothing. We test the account the way you would test an instrument: inject a known, logged amount of one kind of loss, then check that the matching bin moves by that amount while the others stay put. The pass criteria were fixed in a versioned analysis plan before the runs they govern. The clearest case is a straggler delay.

We then apply the account to the published benchmarking harness for LLM teams from Mieczkowski et al. (2026), with DeepSeek V4.1 Flash, Gemini 3.1 Flash-Lite, and Gemini 3 Flash, and it changes how the harness’s results read.

What we find:

• The account closes, and the waiting bins find a known delay. The bins add up to m·T on every accounted run, both when agents declare the task done and when the untouched test suite first passes. On 324 new runs, collected after the criterion was fixed, the four waiting bins together recover 95–97% of an injected straggler delay on all three model families, and 99% when measured against the delay that actually lengthened each round. Even so, the intervals exclude exact recovery, so this test fails as specified, and graph wait on its own misses part of the dose.
• The other known-answer tests are mixed. Giving agents more visibility into teammates’ files lowers overwriting step by step, as expected. A weaker teammate moves inflation as required, but its second target, rework, does not, and waiting moves with it, so a change in capability is not cleanly separated from coordination. Assignment imbalance and false dependency edges fail because their logged doses count time the agents actually spent working.
• Most of the apparent coordination loss was hidden specifications. In our rerun of the published setup, fewer than a third of the runs that declare success pass the untouched test suite, and 54 of 236 declared runs had edited the test suite or its configuration first. Most of the time the published setup attributes to coordination comes from agents guessing at specifications they cannot see. Showing them the subtask descriptions raises pre-assigned speedup from 1.05 to 1.37 with three agents and from 1.09 to 1.37 with five, as plan wait and rework shrink to a fraction. Two main coordination costs remain: waiting for the slowest reply in pre-assigned teams, and overwriting in self-organizing teams, which stay near S = 1.
• Blocking edits to other agents’ subtasks mostly relabels the loss. The rule removes overwriting on both model families, but agents still write the same files in the same round (about four of every five agents per file, with five agents). On Gemini 3.1 Flash-Lite the time moves into no-op replies almost one for one; on DeepSeek V4.1 Flash more of it becomes first-time work. The test we specified in advance passed on DeepSeek V4.1 Flash and failed to replicate on Gemini 3.1 Flash-Lite, and the pass came from the baseline: under the rule a single DeepSeek V4.1 Flash agent needs 57 s instead of 12 s to pass the tests. Measured against the unmodified one-agent time, the rule lowers DeepSeek V4.1 Flash’s speedup and raises Gemini 3.1 Flash-Lite’s slightly.
• No coordination metric we tested isolates coordination loss. We ran ten published metrics and our own summary (the gap between the best possible speedup and the observed one) on the same doses. Each either misses the injected loss or also moves when model capability changes. Activity counts such as idle rounds, replies, and coordination overhead move most under a weaker teammate; edit-conflict counts respond to overwriting but not to waiting; our summary tracks the straggler but inherits speedup’s dependence on capability.

Here are all the known-answer tests with specifications visible:

With specifications hidden, nearly all of the single-bin tests failed.

A few more results:

• Reproduction. Recomputed from the released logs, the account reproduces the published speedup table in 67 of 72 cells.
• Larger teams. With up to ten pre-assigned agents and visible specifications, DeepSeek V4.1 Flash speedup peaks at 1.41 with five agents and falls to 1.29 with ten, while Gemini 3.1 Flash-Lite creeps up to 1.52. Inflation and graph wait grow with team size, and from eight agents on, DeepSeek V4.1 Flash teams overwrite each other even under pre-assignment.
• Fewer runs needed. Measured when agents declare the task done, the edit bins detect a 0.25 agent-equivalent difference with about 10 runs per condition, where speedup needs 237.
• The optimal reference depends on units. The best possible speedup, solved exactly with CP-SAT, agrees within about 0.2 across task, token, and seconds units on two of the three dependency structures but spreads by 0.7 on the third, so coordination loss measured as the gap to it depends on the unit chosen.
• Limits. These results come from one harness, three small Python libraries, inexpensive models, and our visible-specification setting. Whether they hold for teams whose agents message each other is open.

To find out more:

• Agent Evaluation Science Symposium