My local model server kept falling over, and I kept reaching for fixes that did nothing. The problem was one word, "contention," hiding two different failures with opposite fixes.
I run a local model server on a Mac that a few dozen scheduled jobs call for summarizing, redacting, classifying, and ranking. Several different models, one machine, roughly 21 GB of usable GPU memory. Periodically things went bad in ways that all got filed under the same word: contention. Jobs failed, or the whole machine crawled, and the internet's advice was a grab bag of the same three knobs: keep-alive, max loaded models, use smaller models.
What eventually helped was not a knob. It was noticing that "contention" was two different failures, that they present almost identically from a distance, and that every fix for one is useless against the other.
Two jobs want different models that cannot both fit. Each request evicts the other's resident model, and the loser either fails outright with a 500 during model load, or eats a cold reload measured in tens of seconds. The job is the casualty; the machine is fine.
The instinctive fix is keep-alive tuning: keep the model warm longer and it will not need reloading. Measured against my actual incident timeline, keep-alive would have prevented zero of these. Every reload in the logs was a demand eviction, another model actively claiming the memory, not an idle expiry. Keeping a model warm does not protect it from being thrown out by the next request.
What actually fixed it was accepting eviction as normal and making the clients survive it: a retry across the eviction window in every scheduled job that calls the server. The alternation of two models that cannot co-reside stayed, as a deliberate cost, because each model had won its role on merit and swapping one out to dodge reloads would have traded output quality for scheduling convenience.
The second class produces no errors at all. The scheduler estimates memory optimistically, decides several models can co-reside, and loads them: on the worst day, a 26B, a 12B, and a 4B totalling roughly 28 GB of weights onto a 21 GB budget. Nothing returns a 500. Every request technically proceeds. The machine just starts swapping and near-freezes, taking every unrelated process down with it.
Client-side retries cannot even see this class. No request fails, so there is nothing to retry. From the jobs' perspective everything is fine, slowly. It happened six times across three weeks before the pattern was named.
| eviction | pile-up | |
|---|---|---|
| casualty | one job | the whole machine |
| error surface | 500 on load | none |
| tell in the logs | load/evict churn | multiple models resident, swap climbing |
| client retry | works | invisible |
| cap loaded models at 1 | makes churn worse | prevents it |
| keep-alive tuning | prevents zero | irrelevant |
The pile-up fix is the cap: at most one model resident at a time, so the scheduler's optimism has nothing to be optimistic about. Note that this is exactly the wrong move for class one, where a cap of one guarantees an evict-and-reload on every model switch. The two fixes pull in opposite directions, which is why naming the class first is the entire game.
Alongside the cap I added a watcher: every five minutes, if memory pressure is at warning level while models are resident, it sends a push alert with a one-tap unload action. Silence means healthy. One design choice mattered more than expected: it alerts on pressure level, not on swap usage, because swap stays allocated long after an incident ends and would page me about yesterday's problem.
A tidy ending would be "and then the alerts stopped." The opposite happened: twenty alerts in the first three days, twelve of them in one 48-hour stretch, swap peaking above 6 GB. The taxonomy was not wrong, but the monitoring built to confirm it started producing evidence it could not interpret.
The trigger is "pressure high while models are resident," and the while is doing quiet work: it establishes correlation, not cause. One 3 AM alert landed squarely inside a cluster of backup and regeneration jobs, none of which call a model. The models were resident and innocent. Attributing those alerts to the model server because the model server's watcher raised them is exactly the recency-flavoured mistake the class taxonomy was built to avoid.
And the honest state is that a third framing is now open. Roughly 54 scheduled jobs call different models at uncoordinated times, so under a cap of one, every model switch is a full evict-and-reload cycle. The lever might not be the roster or any server knob at all. It might be the schedule: batch the callers by model so switches happen a few times a day instead of constantly. That is unresolved, and I would rather publish the open question than a fake resolution.
Before touching any knob on a shared model server, name which class the incident belongs to, from its artifacts: jobs failing with 500s and load churn in the logs is eviction; a slow machine with multiple models resident and swap climbing is a pile-up. Then replay the proposed fix against that specific incident's timeline and ask whether it would have changed anything.
That replay step is what killed keep-alive here. It is a fine knob for a real idle-expiry problem, and it does nothing for either class I actually have. Every hour I almost spent tuning it was an hour aimed at a third failure mode that never occurred.