Smart is not finished: reading HLE and METR correctly
What do hard academic questions and human task times tell us about AI? A Friday afternoon assignment takes us through two different tests. Video in German.
An assignment at 16:55
Import a customer list: read ordinary entries, report duplicates and handle an empty file. This example is fictional. We follow it through two benchmarks to understand what their results tell us and how to check the finished assignment ourselves.
Two tests, two questions
Humanity’s Last Exam (HLE) tests difficult academic questions with known answers. Accuracy describes that question set. The original has 2,500 questions; HLE-Diamond uses a different set of 1,000. Compare benchmark versions and permitted tools.
METR measures agents completing connected tasks. Its time horizon uses human expert task time, not the AI’s actual runtime. A 50% horizon estimates a difficulty threshold across the tested task mix; it does not guarantee an individual assignment’s success. A higher reliability target usually gives a shorter horizon.
The percentages and curve in this video are illustrative examples, not measured model results.
Back to the customer list
Check three cases: an ordinary list, duplicates and an empty file. Define the correct result beforehand. Benchmarks, versions, tools and your reliability target help select a system; acceptance tests tell you whether your assignment is actually finished.
Primary sources: HLE, HLE paper, Diamond dataset, METR horizons, measurement method and Time Horizon 1.1.
My notes
Notes and highlights for this post
Stored only in this browser. Nothing is sent.
Your notes and highlights appear here.
Select a passage and choose “Highlight” or “Note”. Or write a note about the whole post.