Douwe Osinga

Automatic model routing is having a moment. Once an agent session makes dozens of model calls, using the strongest model for all of them gets expensive. A difficult refactor might deserve the best model available. Saying hello probably does not.

But switching models is a fairly blunt instrument. A different model can mean different tool-calling behavior, prompting quirks and context limits. It also means a different prompt cache. Reasoning models already expose another knob: keep the model fixed and vary how hard it thinks.

That is an attractive trade. Reasoning effort is already a range rather than a choice between unrelated models. The prompt, tools and model behavior stay the same. On APIs that support configuration updates, effort can now change without changing the cached prompt prefix. OpenAI explicitly recommends this pattern when effort changes between responses.

So we had an experiment: could goose choose the reasoning effort for each user turn automatically?

goose launched a long time ago. Dog years have nothing on Agent years. goose predates general agentic AI by a bit. The initial thought behind it all was: what happens if you give an LLM control over a shell session?

Turns out interesting things happen. The model writes a command, the command runs, the result goes back into the conversation and the model decides what to do next. Our tool calling grew naturally out of that experiment, as did the agent loop that now sits at the heart of most coding agents.

Then tool calling got formalized and MCP happened. So did approvals, context management, todo lists, top-of-mind, subagents, retries, hooks, skills, recipes and steering. But the original agent loop remained. Each feature added another branch, another local variable or another special case. A dozen different concerns might be active inside one turn, each influencing the others. Cancellation needs to know whether inference or a tool is running. Compaction needs to understand provider errors and usage. Tools can change the prompt. A stop hook can turn a completed answer back into more work. The loop started to turn into spaghetti.

In the olden days, contributing your first PR to an open source project was rather involved. Even with great instructions, getting the project to build, the app to run and the tests to pass took real work. And even if you knew exactly which bug to fix, you needed a rough understanding of the project’s architecture and more detailed knowledge of the code you were changing.

Coding agents have changed all this. The problem is not code quality. Complaints about AI slop are all over the internet, but code written by agents is often better than what you would get from a first-time contributor. The problem is that agents have changed the economics of open source.

Goodhart's law is the benchmarker's curse: when a measure becomes a target, it stops being a good measure. Coding-agent benchmarks are almost designed to trigger it. The tasks are public, the result is one number, and the leaderboard inevitably fills up with harnesses that are, often without meaning to be, overfit to the benchmark.

That does not make the go-to standard Terminal-bench useless, but it does change how the goose team uses it. The leaderboard is a noisy measure of general agent ability. The signal is a pattern of failures: places where goose keeps getting stuck or where goose fails and another harness succeeds.

That is also why we usually benchmark with Sonnet rather than the strongest model available. We are not trying to get the largest possible number. We want enough failures left on the table to see what support the agent is missing.