What does the next dollar buy an agent?

On 21 July, METR publishes the “expenditure horizon,” an approach to comparing human and agent optimization at comparable financial budgets. Its concrete study example is the NanoGPT speedrun. Budget also includes experiments and compute; it differs from human task time. METR method. A 24 July research note discusses further agent metrics. Metrics and limitations.

Our example: training takes ten seconds to reach a specified quality target. An agent receives code, test data and a limited budget for changes and experiments. It should reach the same quality faster. A nine-second version would be interesting if the control test confirmed its quality. “Drop half the data” might also make the clock faster while failing the goal.

Successful suggestions alone are not enough. Failed experiments, repetitions and result checks also belong in the accounting. This example illustrates the measurement question; its ten and nine seconds are not METR findings.

On 21 July, Google introduces Gemini 3.5 Flash Cyber for selected government and other trusted partners, aiming to find, validate and help fix vulnerabilities. This is controlled access, not a generally available guarantee of perfect defense. Google announcement.

Both developments lead to the same practical question: which result was achieved at which budget and under which conditions? A discovered flaw, a tested fix and a successful production change are separate steps.