Local GGUF inference

How to run an LLM locally with Saient

Download Saient, add a compatible GGUF model, choose GPU layers and context length, then start the bundled local server. The model runs on your CPU or NVIDIA GPU with no Saient account, licence key or inference API key.

Download Saient freeSetup guide
Short answer: Saient scans local GGUF files and launches its own Rust inference server on the loopback interface. Chat uses that local server; it does not route ordinary inference through OpenAI, Anthropic or another hosted model provider.
Real interface

Choose a local model and start the server

This v1.0.4 screenshot shows three GGUF files discovered on disk, their sizes, GPU-layer and context controls, and the local Start server action.

Saient Chat screen listing three local GGUF models with GPU layers, context length and Start server controls

Saient local LLM screen

The listed 7B and 14B Qwen files were present on the test machine when this screenshot was captured.

Setup

Run a local LLM in four steps

  1. Install Saient. Fast setup is enough for chat and the agent; Full setup also installs image, video, TTS and LoRA dependencies.
  2. Add a GGUF. Use the built-in downloader or copy a .gguf file into the displayed models folder. Public downloads need no Saient key; a gated third-party repository can still require its own terms/token.
  3. Set memory controls. Use GPU layers -1 for the automatic/all-GPU path where supported, or reduce offload when VRAM is limited. Context length also consumes memory.
  4. Start and chat. Select the model, start the local server and send a message. Saient also exposes an OpenAI-compatible local chat-completions endpoint for its own interface and local integrations.
GGUF support

Architectures and quantisations

The current GGUF metadata validator recognises these architecture identifiers:

Architectures

llama, mistral, phi, qwen2, gemma, falcon and gpt-oss.

Documented weight types

Q4_K, Q5_0/1, Q6_K, Q8_0, BF16, F16 and F32, including the project's gpt-oss MoE path.

A GGUF file extension is not sufficient proof of compatibility. The architecture, tensor types and tokenizer metadata must all be supported; keep the model card and a smaller fallback quantisation handy.
Hardware

How much RAM or VRAM?

  • Start with the model file size. A 4.4 GB file cannot fit into less than 4.4 GB of combined storage and runtime memory; inference also needs KV cache and working buffers.
  • Context costs memory. A larger context window increases KV-cache use, so a model that loads at 4K context may fail at a much larger setting.
  • NVIDIA GPU: fastest supported desktop path. GPU layers can be reduced if the whole model does not fit in VRAM.
  • CPU: supported. Use enough system RAM for the model plus runtime overhead and expect lower token speed.
No universal minimum is claimed. “7B” or “14B” parameter count alone does not determine memory: quantisation, context, architecture and offload settings all matter.
Proof

A live local inference response

On 9 August 2026, the already-running Saient server on the development machine returned ok from its local health endpoint, identified its loaded model as Qwen2.5-Coder-14B-Instruct-Q4_K_M, and answered a deterministic request with:

SAIENT_LOCAL_OK

This is a functionality check, not a speed or quality benchmark. No benchmark number is inferred from it.

Keep the prompt and inference loop local.

Saient is free and the local server needs no hosted-inference key. You still choose and comply with each third-party model's licence.

Download Saient freeSee the all-in-one comparison