A batch LLM pipeline ran 2.7x slower and 2.5x more expensive than it needed to, on every run since it was built. The bug was one unqualified shell-out. Finding it took a day, because I spent the day measuring the wrong things.
I run a pipeline that turns my notes into a knowledge graph. It chunks the corpus and shells out to a coding agent's CLI in headless mode, one call per chunk, expecting each call to do one thing: read the chunk, return structured entities and edges as JSON. A dumb, stateless extraction worker.
During a full rebuild, chunks were taking around eight minutes each, and some never finished at all: they sat silent for exactly fifteen minutes and died with an empty log. The rebuild estimate said the whole job would take about three hours. Reality was heading past ten.
Slowness in an LLM pipeline has a lineup of usual suspects, and I worked through them in the worst possible way: by forming a theory, restarting the live run, and watching whether it felt faster. Three theories died that day, each to a test that took minutes. The day went to the runs in between.
What finally worked was embarrassingly cheap. Take one chunk, the same ~15k-token payload, and send it three times under three configurations. Three API calls, run last, after everything else had burned real quota on live reruns.
The pipeline resolves the CLI binary with a bare which and
invokes it with no isolation flags. In headless mode that binary loads the
same configuration my interactive sessions use. On this machine that
means nine MCP tool servers, covering knowledge-graph queries, news
feeds, local model serving, and automation bridges, plus the agent's
entire built-in tool set: file reading, shell execution, search.
And the model used them. Faced with "extract entities from this text," it would browse the graph it was in the middle of rebuilding, poke at files, take extra turns. Each chunk stopped being one structured extraction and became a small multi-turn agentic session with opinions.
| config | wall clock | cost | cache creation | turns |
|---|---|---|---|---|
| full config, the old default | 323s | $1.21 | 94k | 4 |
| MCP servers stripped | 166s | $0.59 | 41k | 2 |
| servers and tools stripped | 120s | $0.49 | 37k | 2 |
2.7x the wall clock and 2.5x the cost, for tools an extraction task never needs. And this was not a property of the one broken rebuild. It had been true of every headless run since the backend was adopted, on every corpus, quietly, because the runs still succeeded. Nothing fails when your batch worker is secretly an agent. It just costs more and wanders.
The fix is a ten-line shim: a wrapper script earlier in the
pipeline's PATH that execs the real binary with strict flags
for an empty tool config. Interactive sessions still resolve the real
binary; only the pipeline sees the shim. Verified after: zero MCP child
processes per extraction, where there had been nine.
A day later the rebuilt graph showed a quality shift: the share of edges the extractor marked as low-confidence jumped from 0.13% to 2.39%, against a documented "under 1%" expectation. The shim was the obvious suspect. It was the newest change, and "stripping the tools changed what the model sees" is a perfectly believable mechanism.
Instead of reverting it on vibes, I ran the accusation properly: a 2x2 factorial over prompt version and shim, two replicates each, on a fixed 12-file payload, through a path that writes nothing to the real graph. The verdict was clean. The shim's main effect on the hedged-edge share was −0.3 points, indistinguishable from zero. The real cause was a prompt patch from the same week, main effect +4.8 points, and the mechanism was the opposite of a regression: the patch suppresses cheap structural nodes, confident boilerplate edges fell from 81 to 42 per chunk, and the uncertain conceptual edges that remained now make up more of a smaller denominator. The share of hedged edges rose because the graph got better.
The newest change is the natural suspect, and the natural suspect was innocent. Attribution by recency would have reverted a working fix and kept the actual cause.
The isolation fix explained slow. It did not explain the chunks that died silent at exactly the fifteen-minute timeout. That turned out to be a second, unrelated bug, and my favourite of the two.
Measured on a real chunk, this extraction emits about 1.16 output tokens per input token: a ~15k-token chunk produces ~17.5k tokens of JSON. The pipeline's default token budget packs chunks up to 60k input tokens. At the measured ratio, a full chunk demands roughly 70k output tokens, which is past what the model can emit in one response. The request is not slow. It is impossible. It cannot converge, so it burns the full timeout and dies with nothing in the log.
I reproduced it deliberately as a control: a 63,927-token payload through the fixed, isolated path timed out at 900.0 seconds with zero edges, both of two runs. Dropping the budget to 25k input tokens, about 29k output, made every chunk complete comfortably.
The reason this stayed invisible for months is worth keeping. Incremental updates only ever process a handful of changed files, which never packs a chunk anywhere near the cap. Only a full rebuild fills chunks to the budget, and full rebuilds are rare. The bug lived exclusively in the configuration's far corner, waiting.