Insight / AI and infrastructure
Published
Is your local AI system doing its best work? A practical checklist
A measured way to tune a local model setup across inference engines, memory, clocks, storage, model choice and repeatable benchmarks.
An expensive graphics card can feel slow when the wrong model, context size or runtime is asked to do the wrong job. Before buying another card or changing three settings at once, establish what the system does now. A useful benchmark is a repeatable task that resembles your work, not the highest number shown by a launch screen.
The checklist below is for a workstation or small local server. It applies whether the goal is private document search, coding assistance, image understanding or a small internal service. Keep the hardware, model license and data-handling requirements in view as you tune. Our infrastructure calculator helps frame demand; this checklist tests the machine that will meet it.
1. Write down the workload before the hardware
Record the number of people using the system at once, typical prompt and document size, expected answer length, model quality threshold, acceptable first-response delay and daily volume. Separate interactive use from batch processing. A local assistant that waits ten seconds for a first answer may be acceptable for overnight classification and frustrating during a live customer conversation.
Select three test cases: an ordinary request, a long-context request, and a difficult example that previously failed. Keep approved or synthetic inputs. Save the expected result and the reason it matters. Track answer quality, time to first token, output tokens per second, total completion time, memory use and energy where possible. Without that baseline, a speed gain can hide a quality loss.
2. Choose the engine for the workload
Try a supported runtime before customising one. llama.cpp covers multiple accelerator backends and quantised models; Ollama provides a straightforward way to run many local models; vLLM offers serving controls that can help with concurrency and cache use; Strata targets particular large-model and hardware combinations. Their defaults, model support and APIs differ. Choose a candidate because it serves your model and request pattern, then measure it.
Use the same model revision, quantisation, context and prompts when comparing engines. If an engine requires a different format or quant, record that difference and evaluate quality as well as speed. Our Strata and agent harness guide shows why a chat benchmark does not predict the behaviour of a tool-using coding agent.
3. Account for the whole memory budget
Weights are only one consumer of memory. The context window, KV cache, runtime buffers, image encoder, operating system, browser and other GPU tasks also need space. Watch peak VRAM and RAM during a long prompt and while generating, not only at startup. A setup that fits a short test can fail under the real context length or two simultaneous users.
Try a smaller quantisation, shorter context or a model built for the task before assuming another GPU is required. These changes can reduce memory pressure, but they may change output quality. If your engine offloads layers or experts between GPU, RAM and storage, check whether the bottleneck is transfer bandwidth rather than arithmetic. Do not treat an SSD capacity number as a promise of usable inference speed.
4. Check clocks, cooling and power without guessing
Record the GPU model, driver, power limit, memory and core clocks, utilisation and temperature during the same test. NVIDIA’s performance guidance recommends capturing those conditions because cooling and clock differences can change measured results. On other hardware, use the vendor’s equivalent telemetry.
Verify BIOS settings, PCIe link speed, memory channel configuration and firmware against the board and CPU documentation. Start with supported settings. An unstable overclock or a driver change can make an apparent speed improvement disappear under longer work. If thermal or power limits are reached, improve cooling and airflow before raising clocks. Save the old configuration so it can be restored.
5. Place models and workspaces deliberately
Fast local storage helps load large weights and can matter more when an engine actively reads data during inference. Keep enough free space for weights, caches, temporary files, logs and a second version for rollback. Check actual read throughput during model load or swap activity.
The advice to put the model, workspace and project on the same disk is conditional. One fast NVMe drive can simplify a single-user setup. On a busy machine, placing heavy model reads and repository builds on separate drives may reduce contention. A network mount may add latency or availability risk. Measure both arrangements on your tasks before reorganising data. Keep backups separate from the working disk either way.
6. Align the model with the work
More total parameters do not automatically mean better output for a defined task. A smaller coding or document model can be more useful than a much larger general model if it meets the quality threshold and responds quickly. Sparse mixture-of-experts designs activate part of their total weights per token, but still need their full artifact stored and managed. Distillation, quantisation and pruning are different interventions; pruning a model yourself is not a free switch and may require retraining and careful evaluation.
Check the model’s license, known limitations, prompt template and supported context before tuning. Build a small evaluation set from the work people actually do. Do not tune only for tokens per second: a faster model that causes extra review or missed fields raises the cost of the workflow. The model supply-chain article explains why exact revisions and verified copies matter when a platform changes.
7. Record a result someone else can reproduce
Write down hardware, BIOS and driver versions, engine build, model revision and checksum, quant, context, concurrency, prompt set, timings, memory peaks and observed failures. Repeat tests after the machine warms up, and report a range rather than one lucky run. Keep the last known-good setup and a clear rollback path.
The best local system is the one that reliably completes the required work within a known time and cost envelope. When the benchmark shows a ceiling, decide whether to adjust the workload, run a different model, add hardware or use a hosted service for selected tasks. Consumer hardware support can change over time, so include support life and replaceability in that decision.