← All insights

Insight / AI and infrastructure

Published

If it's not running locally, it's not your model

What running a model on your own infrastructure actually gives you, what it does not, and a practical route from a first download to a dependable local setup.

Close view of a computer graphics card
Photo: Đào Hiếu / Unsplash

The title is deliberately provocative. A model running on a server you administer can be just as much under your operational control as one running on your laptop. And downloading a model does not make its intellectual property yours. The useful distinction is whether you can keep the same permitted weights and setup available for your work, on infrastructure you control, when a provider changes its hosted service.

That control matters when a workflow depends on familiar behaviour. A hosted model can change version, availability or price; even a pinned API version may eventually be retired. A local copy lets you decide when to test and adopt a replacement. It does not guarantee identical answers forever: prompts, sampling settings, the runtime and surrounding software also affect results. Keep a small set of representative tasks and record the full setup alongside the weights.

Privacy is another reason to run locally, but hosted data terms vary by product and settings. It would be inaccurate to say that every prompt or correction becomes training data; OpenAI’s published policy is one example of different defaults for individual and business services. Check the current terms for the exact service before using sensitive material.

A downloaded file is the start, not the finish

Choose a model whose card and licence permit your intended use. Some repositories also require access approval. Check the memory and storage needed for the particular model and quantisation before downloading it. If you need to reproduce a deployment, record the model’s exact revision rather than relying on a moving label; Hugging Face’s download tools support a specified revision.

For a first local trial, Ollama’s quickstart offers a desktop app and the command ollama run gemma4:e2b. It downloads that model and starts a local chat. Ollama also offers cloud models, so check which one you selected. Its FAQ explains how to disable cloud features if local-only operation is a requirement. Keep the local service private unless you have deliberately configured access controls.

For an application serving multiple requests, vLLM is another option. It is a serving engine, not a model: you provide compatible weights and it exposes an API. Check its current hardware and installation requirements before choosing it. Despite the names, no actual llama is required.

Then test the whole workflow with the network disconnected. A local model cannot protect data that a connected chat interface, search tool, logging service or fallback API sends elsewhere. Keep documents, embeddings, logs and backups inside the intended boundary, and decide who can read them. Budget for power, hardware, updates, monitoring and recovery as well as the initial download.

Make changes on your terms

Start with a clear prompt and real examples. If the model needs current private documents, consider retrieval from a local index. If it needs to learn a repeatable style or task, a small adapter may be worth testing; Hugging Face’s PEFT guide describes approaches such as LoRA that train fewer parameters than full fine-tuning. Evaluate any adapted model against the original before replacing it. Chatting with a local model does not, by itself, fine-tune its weights.

The aim is a setup you can inspect, reproduce and change deliberately. Keep the model files you are allowed to retain, their licence and revision, your prompts and settings, your evaluation cases, and a way to restore the service. That is a stronger form of control than merely having downloaded something once. For the wider question of hosted data policies and sovereignty, see what control really means for local AI.

External project website

Open this project site?

is hosted at . It will open in a new tab.