Coding Agents Live or Die by Their Interface
Better commands and clearer observations improved SWE-agent results with the base model unchanged. The interface is the part you build, and SWE-bench still passes patches that miss the intent.
The interface is the part of a coding agent that a team builds, and it sets the ceiling on the model inside it. SWE-agent showed the mechanism: better commands and clearer observations improved results while the base model stayed fixed.
Code Has an Oracle
Coding agents are the easiest agents to overrate and the easiest to check. Code has tests. The agent can claim it fixed the bug, and the test suite gets the last word.
Autocomplete suggests and an assistant answers. An agent inspects the repository, edits files, runs commands, reads the failure, and tries again. Each rung takes more responsibility, and only the last one runs the code.
The Agent Sees the Repo Through Its Tools
A coding agent sees the repository through a narrow surface: search, open file, edit, run tests, read output. Everything it knows about the code arrives through that surface. In a real repository the same channel carries linters, type errors, and version-control state. When those tools are clumsy, the agent makes worse decisions with less to go on.
SWE-agent gave that surface a name, the agent-computer interface, and treated it as the variable to move. Command design, observation format, and file localization are edits a team makes without touching a weight.
The loop note made the same argument for a browser agent, where the action space decides how much the model must infer. Code shows it first because a failing test names the line that broke.
A Bad Action Space Gives Too Little Power or Too Much Rope
A bad action space fails in two directions. An agent that reads one terminal line at a time burns its context on noise. Hand it the power to rewrite the repository with no guardrails and the patch comes back too big for anyone to review.
The surface that works copies how a careful engineer already moves. Search symbols, read the nearby code and its style, and make one narrow change. Run the smallest test that applies, and widen the change only when the evidence demands it.
- Search must return ranked, structured context, never a wall of text.
- Edits must be patch-shaped, easy to review and easy to revert.
- Test output must come back summarized, with the failing line kept intact.
- Every command must carry a cost label: cheap, expensive, risky, or destructive.
File Localization Is Half the Battle
Most coding tasks start as a search problem: which file owns the behavior, which function matters, which test describes the failure.
The agent that reads everything loses. The agent that narrows fast wins. Localization is the first engineering act in a run, and every later step inherits it.
The Best Runs Are Boring
A serious coding agent lives inside one cycle: orient, hypothesize, edit, run the test, read the failure, repeat. The best runs feel boring because every step has evidence behind it.
The overview drew that shape as observe, decide, act, examine, remember. A coding agent fills the examine step with a command that returns a verdict.
An agent can jump from the issue text to a confident patch. When it cannot show how it localized the behavior, it skipped the step where most engineering judgment lives.
SWE-bench Still Passes Patches That Miss the Intent
SWE-bench asks agents to fix real GitHub issues, which puts it closer to real work than multiple choice. The scoring stays messy anyway.
Tests flake, issue descriptions arrive incomplete, and dependencies stay hidden. A patch passes the suite, misses the intent, and the benchmark marks it solved.
The reasoning note put the verifier outside the model. A test that passes while the intent fails is a weak verifier. The agent cannot see that gap from inside its own loop.
Read the final report against the diff. That check catches the loud failures: the wrong file changed, unrelated work overwritten, success declared before the code ran. The other check is whether the tests exercised the changed behavior. A green suite that never ran the new path proves nothing.
The Builder Test
Ask what the agent learns after a failed test. Take one run, stop at the first red test, and read what the agent does next.
A loop that cannot turn failure output into a smaller, better patch is not a coding workflow yet. The next patch must be narrower than the last one, and the failing line must explain why.
What Carries
Carry the edit-test loop. A test run is the oracle that tells the agent whether the patch touched reality. The interface decides whether the agent can reach that oracle and read what it says.
Code gets that oracle for the price of a command. Most workplace tasks have no oracle at all. The record of what the agent did is the only thing left to read, so build the record before you need it.