What hardware do you need to run a local AI model?

A model that loads is not necessarily a model that serves your workload well. Start with the task and the people using it, then check memory, throughput and operating limits.

01

Start with the task, not a GPU model

Write down the task, the quality you need, the longest input you expect and how many people may use the system at once. A short document classification task has a different memory and latency profile from a long research conversation. Test a representative set of approved inputs before buying hardware.

Model size is only one input. Quantisation changes the memory needed for weights and can change output quality. Longer context and more concurrent requests need additional working memory. Leave room for the operating system, inference engine and other services rather than sizing to a model file alone.

02

Understand RAM, VRAM and the bottleneck

VRAM is memory attached to a GPU. System RAM is memory available to the CPU. A model can sometimes run partly or wholly in system RAM, but speed may differ substantially from a configuration that fits the active work in VRAM. The CPU, GPU, memory bandwidth and connection between them can all become bottlenecks.

The useful question is not simply 'will it run?' Ask how many tokens per second the complete workload produces, how quickly the first answer appears, and whether that remains acceptable when several users arrive together. Measure with the inference engine and model configuration you intend to operate.

03

Work through a real configuration

Record the CPU, installed RAM, GPU model and VRAM, then use our calculator to narrow candidate models. Treat its estimates as a shortlist, not a capacity guarantee: runtime settings, context size, quantisation and concurrent users change the result.

If this is a shared business service, carry the candidate into the infrastructure calculator to consider concurrency, power and cost. Run a small evaluation on representative tasks before deciding whether local, hosted or hybrid delivery is the right fit.

04

Plan for operation and data boundaries

Hardware ownership does not itself establish data sovereignty. Document where prompts, files, logs, search indexes, telemetry and backups go. Decide who may update models, who sees failures and how a version change is evaluated.

A private deployment needs a maintenance plan for hardware faults, operating-system updates, capacity and recovery. If the task is occasional and non-sensitive, a hosted service may be simpler. The decision should follow the workload and information involved.

External project website

Open this project site?

is hosted at . It will open in a new tab.