Local and Private AI Models for Developers

When Local Makes Sense

Local AI models make sense when one or more of these apply:

• NDA or IP restrictions — your contract or company policy prohibits sending code to external services; confirm that the proposed local runtime, editor integration, logs and support path are within the approved boundary.
• Regulated data — health records (HIPAA), financial data, government-classified material, or similar.
• Air-gapped environment — no outbound internet access by policy or infrastructure constraint.
• Cost control at scale — running millions of short classification calls locally can be cheaper than API costs, depending on your hardware.
• Latency requirements — local inference eliminates network round-trips, which matters for real-time tooling.

If none of these apply, compare the actual quality, cost, latency and operational burden before choosing. Model capability changes quickly; do not assume that either local or cloud is universally better for your task.

The Main Tools

Ollama — Developer-first CLI

Ollama provides a CLI for downloading and serving models, and a local HTTP API. Treat the default local endpoint as a service boundary: confirm its bind address, firewall exposure, authentication needs and outbound traffic before sending sensitive material. Its OpenAI-compatible API can simplify some integrations, but confirm the endpoint and feature compatibility in the client you use.

# Install from the current official Ollama instructions, then verify the
# downloaded installer according to your organisation's software policy.

# Pull and run a model after confirming its tag, licence and storage location.
ollama pull your-verified-model-tag
ollama run your-verified-model-tag "Explain this function: ..."

# Or run as a local API server (http://localhost:11434)
ollama serve

The OpenAI-compatible endpoint means you can point the official OpenAI SDK at Ollama and use local models from existing code:

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'http://localhost:11434/v1',
  apiKey: 'ollama',  // required by the library, not used by Ollama
});

const response = await client.chat.completions.create({
  model: 'your-verified-model-tag',
  messages: [{ role: 'user', content: 'Review this code...' }],
});

LM Studio — GUI with model browser

LM Studio provides a desktop GUI for browsing, downloading, and running models. It also exposes local REST and compatibility APIs. The documented default server address is http://localhost:1234; if you enable network access, configure authentication and verify the bind address and firewall before loading sensitive material.

Jan — Privacy-first, offline-first

Jan is an open-source desktop option with offline-oriented workflows. Downloading models, updates, extensions and any optional online features are separate network and retention questions; verify the installed version and configuration rather than treating the product label as a security guarantee.

Choose a Model by Measured Fit

A model name is not a durable recommendation. Local quality and speed depend on the exact model tag, quantisation, context length, runtime version, concurrency, GPU offload and hardware. Start with a representative task set, not a leaderboard.

Build a Small Acceptance Set

• Use real tasks — include code explanation, a constrained edit, a failing test, repository-specific conventions and one request the model should refuse or flag.
• Define pass criteria first — tests pass, required files stay untouched, citations or uncertainty are present when needed, and no secret or prohibited data leaves the approved boundary.
• Repeat runs — one good answer is not evidence of reliable behaviour. Record failures and variance as well as the best result.

Record the Whole Configuration

• Model name and immutable tag or file hash, quantisation, licence and source.
• Runtime and integration versions, context setting, sampling settings and enabled tools.
• CPU, available RAM, GPU and VRAM, plus first-token latency, throughput and peak memory on the acceptance set.
• Output quality, failed cases and the date you checked the configuration.

Connecting VS Code to a Local Model

Some coding extensions can use Ollama directly or an OpenAI-compatible endpoint. Support differs by feature: chat, edits, autocomplete and agents may use different providers. Verify each workflow rather than assuming that one base URL redirects the whole product.

With Continue

Continue supports local providers. Its current local configuration file is ~/.continue/config.yaml on macOS and Linux or %USERPROFILE%\.continue\config.yaml on Windows. A minimal chat and edit configuration is:

name: Local Ollama
version: 1.0.0
schema: v1
models:
  - name: Local coding model
    provider: ollama
    model: your-verified-model-tag
    roles:
      - chat
      - edit
      - apply

Check the current Continue schema before copying this example. Start the runtime, open a non-sensitive test repository and confirm in runtime logs that the request reached the intended local model.

With Cursor

Cursor's provider settings, endpoint routing and supported features can change independently. Use its current documentation and settings rather than a copied menu path. Test chat, edits, autocomplete and agent actions separately, and inspect the local server logs: a model appearing in the interface does not prove that every feature uses that endpoint. Repeat the test after editor upgrades.

Handling Slow Inference

Do not estimate latency from parameter count alone. Prompt length, output length, quantisation, memory bandwidth, GPU offload and concurrent users all matter. Measure the exact workload on the intended machine.

• Separate interactive and batch targets — define a first-token target for editor use and a completion-time target for queued work.
• Compare configurations — record quality and latency for each quantisation, context limit and CPU/GPU split instead of assuming the largest model is best.
• Measure cold and warm runs — model loading and cache state can change the result. Document idle-unload settings and memory pressure.
• Test concurrency — a responsive single-user demo can stall when several editor sessions share one runtime.

Private Cloud as a Middle Option

A managed service or a model hosted in your cloud account creates a different trust boundary; it is not automatically equivalent to local or air-gapped operation. Evaluate it against the same data-flow requirements.

• Map the data flow — document endpoints, regions, network paths, provider and subprocessors, logs, retention and support access.
• Verify isolation claims — distinguish a private endpoint or VPC connection from where inference, control-plane operations and logging actually occur.
• Test controls — deny unintended egress, restrict identities, rotate credentials and confirm that audit logs record the expected events without storing prohibited prompt content.
• Compare total cost — include idle capacity, operations, monitoring, upgrades and incident response, not only token or GPU-hour price.

Obtain the current service terms and architecture documents for the exact region and plan. A product family name is not evidence that your required residency, retention or access controls are enabled.

Official Sources and Review Trigger

• Ollama: OpenAI compatibility — local base URL, API-key behaviour and supported compatibility surface, checked Sep 22 2026.
• LM Studio developer documentation — local server and compatibility APIs, checked Sep 22 2026.
• LM Studio: serve on the network — network exposure and authentication boundary, checked Sep 22 2026.
• Continue: configuration — current YAML location and schema, checked Sep 22 2026.

Review this guide again by Oct 22 2026, or earlier if Ollama, LM Studio, Continue or an editor integration changes its endpoint, authentication, configuration schema or routing behaviour.

Related Guides