Insight / AI and infrastructure
Published
Strata and the agent harness: where local inference actually helps
How Strata fits between large local models and coding agents, why its memory strategy can help some consumer machines, and how to compare it fairly with Ollama and vLLM.
A coding agent and an inference engine solve different problems. The agent harness reads files, proposes edits, calls tools and decides what to do next. The inference engine loads model weights, turns prompts into tokens and returns answers. Putting both on one desktop can reduce dependence on a hosted model, but only if the machine can run the chosen model at a useful speed and the harness can reliably use its API.
Strata’s project documentation describes a local inference system aimed at large models on consumer hardware. Its integration reference lists local OpenAI-style and Anthropic-style endpoints and connections for coding tools, including Claude Code and Hermes Agent. These are project-supported interfaces; test the exact version and tool workflow you intend to use. An endpoint that returns text is not proof that a multi-step agent can edit files, recover from tool errors and keep context coherently.
Three layers, three failure points
Model. The weights, quantisation, tokenizer, context settings and license determine what can be loaded and how it behaves. A model with 125 billion total parameters is not necessarily doing dense 125-billion-parameter work for each token. Strata’s documentation describes support for sparse mixture-of-experts families; the model guide gives the current formats and memory requirements. Do not infer quality from the total parameter count alone.
Engine. Strata, Ollama or vLLM decides where weights and caches live, how requests are scheduled, what accelerators are used, and how much latency a memory transfer adds. The runtime can be the difference between a model that loads and a model a person can use interactively.
Harness. An agent may send long system prompts, call tools many times, reuse context, and expect a particular response format. That can expose different bottlenecks from a single chat benchmark. Test the harness’s file permissions, terminal access, approval controls and audit trail as carefully as its token speed.
This separation is why planning local AI infrastructure is a useful starting point rather than the whole architecture. Local weights do not make an agent safe, and a compatible API does not make every tool call correct.
Why a specialist engine may win on a specific machine
The outline for this article suggests Strata is simply faster than Ollama or vLLM. That is too broad. An engine tuned for a sparse model and a particular memory hierarchy can make different trade-offs from a general runtime. If a model’s active experts and attention state can be placed or cached intelligently, the engine may avoid work or transfers that another configuration performs. Whether that helps depends on the model, quantisation, VRAM, system RAM, memory bandwidth, SSD, context size and prompt pattern.
llama.cpp supports many backends, quantisation levels and CPU–GPU hybrid inference. Ollama builds a convenient model-running experience around that family of approaches. vLLM’s serving options address memory allocation, batching and cache behaviour for serving workloads. None of those facts makes one engine universally better. vLLM may be the right choice for many concurrent requests on supported GPUs; a lean local runner may suit a single user; a specialised sparse-model engine may suit a model that otherwise spills awkwardly across memory tiers.
Claims such as a fixed 40–90 tokens per second, or a guaranteed speed-up from speculative decoding, need a reproducible comparison. Strata’s published figures are a starting point for testing, not a benchmark of your workstation. Record the exact model and quant, prompt length, generated length, context, warm or cold cache, number of concurrent requests, hardware, driver and power settings. Measure time to first token and sustained generation separately. Then check answer quality and tool-call success; a faster incorrect agent is slower business work.
A practical connection test
Start with the smallest permitted model that can complete a representative task. Install the engine from its documented release, verify the downloaded artifact and restrict the local API to the intended machine or network. Load the model and test a plain prompt, a long prompt and a request with a tool declaration. The Strata integration reference documents connection checks for supported harnesses; use the check for the tool you actually run rather than assuming an OpenAI-compatible URL is enough.
Next, give the harness a disposable repository. Ask it to read a file, make one narrow change, run a test and explain the result. Inspect the diff and log every tool action. Repeat with a failing test and a context-heavy task. If the harness loses track of files, produces malformed tool arguments or silently changes scope, first separate model behaviour from adapter and harness behaviour.
Finally, compare the local run with an alternative engine under the same task set. Use the local AI system checklist to check clock stability, memory pressure, disk placement and power before attributing every slowdown to the runtime. If the system is part of a business process, carry forward the recovery principles in our model-change runbook: version the entire setup and keep a way to pause or revert it.
Strata is worth investigating when its supported model and memory strategy match your workload. It is not a substitute for a workload test, a license review or a safe agent boundary.