Architectures of Large Language Model-Based Terminal Agents

Abstract

Large language models increasingly operate as agents: they issue shell commands, read and write files, run code, and iterate toward a goal in a real environment. A large body of work proposes scaffolding around the base model (planning, self-verification, multi-agent debate, orchestration) and reports improved success. But most of that work evaluates on a single model, usually a frontier one, and the studies that do control for compute vary one factor at a time. None varies architecture and model capability together under a controlled budget, which is what the central question needs: does the value of an agent architecture depend on the capability of the model it wraps?

We run a controlled grid of 14 architectures (single-agent ablations, pre-execution context augmentation, post-execution verification, multi-agent collaboration) across 7 open and closed models spanning a wide capability range, on the 64 “easy”-labelled tasks of TerminalBench, under a uniform carved 60-step budget shared among sub-agents rather than added to them. A separate heterogeneous supervision track has a strong model supervise a weaker worker, isolating whether capability can be injected at the coordination layer. The study comprises 7424 model–task runs.

The central result is that an architecture’s value depends on the capability of the model it wraps, and the dependence is strong enough to reverse the direction of the effect. The number of architectures that beat a model’s own baseline falls from 13 of 13 on one of the weakest models to 0 of 13 on the strongest open coder (ρ=−0.886, p = 0.033 across the open models). The ordering holds when the baseline is removed from the measure (ρ=−0.829) and under a split-half check (ρ=−0.62). The same scaffold that rescues a weak model burdens a strong one, an inversion that no single-model study can observe, so no architecture is best for every model, and for the strongest open coder the best choice is the plain baseline. The axis is coding capability rather than parameter count: the least-helped model is a 32B coder, not the largest model by parameters.

The per-run traces show why. Models most often fail by submitting a wrong answer early rather than by running out of budget, and the way they fail differs from model to model: the weakest abandon a task within a few steps, while others spend the whole budget repeating the same command. An architecture helps when it supplies something the model is missing, such as a reason to keep working or a check before submitting, and it hurts when it displaces something the model already does well or asks for an artifact the model cannot produce. The same pattern appears across task types. Scaffolding adds solves where a model still has room and loses them on tasks the model already solves, so the question worth asking is which architecture fits a given model and task.

A smaller, exploratory track asks whether capability can be injected at the coordination layer, with a strong model supervising a weaker worker. It can, through some roles and not others. A supervisor that directs the worker, by routing, planning, or guiding, helps every worker (+4, +5, +8 under orchestration). The same supervisor judging the worker’s output does not transfer capability the same way and is net negative on the strongest worker (−6); its main effect is to make a worker steadier on what it can already do.

Because the step budget is shared among sub-agents rather than added to them, none of this comes from giving scaffolded runs a larger action budget. Scaffolding should be chosen against a model’s measured capability and failure profile and against a task’s headroom, not adopted as a default improvement.