Single-turn benchmarks reward the exact habit that makes an AI agent annoying to work with: guessing what you meant on the first try.
We tend to assume a model that scores well on a benchmark will do well in our product. With agents, that link breaks.
A one-shot test gives the model one prompt and grades the first answer. To score well, it has to fill in everything you left out and hand back a finished-looking reply immediately.
That’s the opposite of what I want from an agent. I don’t need a tool that speculates for me. I need it to run the loop: inspect actual state, ask when critical context is missing instead of guessing, and complete the task over several turns.
The multi-turn eval piece I read this week was a vendor’s own blog - and they sell the alternative, so discount it accordingly - but their list of failure modes still reads true to me: contradictions, infinite loops, false task completion. None of those surface on turn one.
Single-turn evals are still cheap, reproducible, and fine for narrow capabilities that genuinely are one-shot. Not every task needs an agent loop, and multi-turn evals take real effort to build and maintain.
So the shift I’d suggest is smaller than throwing away your eval suite: for anything that behaves like an agent, grade the end of the loop - whether the task actually got done - instead of grading turn one in isolation.
If the first reply looks complete, that may be exactly the problem.