llama.cpp: The local inference engine that changed how developers ship AI

TL;DR: llama.cpp is a pure C/C++ inference engine that runs quantized GGUF models on CPU and GPU. Georgi Gerganov wrote it in 2023 to run LLaMA on a MacBook Air. By 2026, it’s the engine behind Ollama, LM Studio, and every local AI tool with over 100k GitHub stars, running on everything from Raspberry Pi to 192 GB Mac Studios. This guide covers how it works, how to use it, and which quantization won’t hurt your quality.

Key takeaways:

• llama.cpp is the runtime. Ollama, LM Studio, and text-generation-webui are wrappers around it.
• GGUF is the file format. It bundles weights, config, and tokenizer into one self-contained binary that loads via memory mapping.
• GGML is the tensor library. It provides the SIMD-optimized matrix operations that make CPU inference fast.
• Q4_K_M is the community default for 8B models. Q4_K_S has better Pareto efficiency. Q5_0 gives the highest benchmark scores.
• CLI gives you more control than any GUI. llama-cli is all you need.
• Python bindings (llama-cpp-python) add an OpenAI-compatible server in 10 lines.
• KV cache quantization (--cache-type-k q4_0) can cut memory usage by 30-40% with minimal impact on quality.

What are the three layers of llama.cpp?

Layer 1: GGUF (the file format)

GGUF is the model on disk. The llama.cpp GitHub repo defines the format. It’s a binary file that stores everything llama.cpp needs: the weight tensors, the model architecture metadata (which type of model it is: LLaMA, Mistral, Qwen, Gemma), the tokenizer vocabulary, and the quantization parameters for each tensor.

Before GGUF (2024), GGML models required separate files for weights, config, and tokenizer. GGUF unified them into one self-describing binary with a magic byte header (GGUF at offset 0). This means llama.cpp can open any GGUF file, read its header, and know exactly what it’s dealing with. No external config files needed.

The header contains:

• Magic bytes (GGUF version 3)
• Tensor count, metadata key-value pairs (model name, architecture, context length)
• Tokenizer type and vocabulary

Layer 2: GGML (the tensor library)

GGML is the compute engine, a standalone tensor library written in pure C/C++ with no external dependencies. Think of it as a minimalist PyTorch that provides tensor types, matrix operations, and automatic differentiation, but designed specifically for LLM inference workloads.

GGML’s key contribution is hand-optimized SIMD kernels for every major CPU architecture:

• AVX2 and AVX-512 on x86 (Intel/AMD)
• NEON and SVE on ARM (Apple Silicon, Qualcomm, Raspberry Pi)
• CUDA on NVIDIA GPUs
• Metal on Apple Silicon GPUs
• Vulkan for cross-platform GPU support
• SYCL for Intel GPUs
• WebGPU for browser-based inference

When you build llama.cpp, you choose which backends to compile in. The CPU backend is always included. GPU backends are optional flags:

# CPU only (default)
cmake -B build
cmake --build build --config Release

# With Apple Silicon GPU
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release

# With NVIDIA GPU
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

GGML operates on tensor computation graphs. At load time, it builds a static graph of all the operations in the model: embedding → N transformer blocks (each with self-attention + feed-forward) → output projection. During inference, this graph is executed once per token. The graph structure is fixed; only the data flowing through it changes.

Layer 3: The sampling layer

The sampling layer is the final step before a token is emitted. The model outputs a probability distribution over the entire vocabulary (32K-128K possible tokens). Sampling applies your chosen strategy to pick one token from that distribution.

The default sampler order in llama.cpp is:

1. Temperature: Scales the logits. Lower temperature (0.1-0.3) makes the distribution sharper, favoring high-probability tokens. Higher temperature (0.8-1.0) flattens it, making lower-probability tokens more likely. Above 1.0, the distribution becomes increasingly uniform. The model gets increasingly random, not creative.
2. Top-k: Only keep the K most probable tokens, discard the rest.
3. Top-p (nucleus): Keep the smallest set of tokens whose cumulative probability exceeds P.
4. Min-P: Discard tokens with probability less than min_p * probability_of_top_token. A newer alternative to top-p that adapts better to different contexts.
5. Final random draw: Pick one token from the filtered distribution.

Contrary to what many guides suggest, temperature above 1.0 doesn’t produce useful “creativity”. It breaks the probability calibration the model was trained with. Values 0.7-1.0 produce varied but coherent output. Values above 1.2 produce increasingly incoherent text.

How does GGUF loading work with mmap?

When you run llama-cli -m model.gguf, the loading process unfolds in four steps:

1. Open the file. llama.cpp opens the GGUF file and reads the header bytes (typically the first 4-16 KB). It validates the magic bytes, extracts metadata, and maps out where each tensor is stored in the file.

2. Memory-map the file. Instead of reading the entire file into RAM with read(), llama.cpp calls mmap(). This tells the OS: “treat this file as if it’s in memory.” The OS sets up virtual memory page table entries that point directly to the file on disk. No data is copied into RAM at this point.

3. Build the computation graph. GGML allocates placeholder tensors and wires up the execution graph. No weight data has been loaded yet.

4. Page fault your way through inference. The first time a layer’s weights are accessed, the CPU tries to read from a memory page that hasn’t been loaded yet. This triggers a page fault. The OS reads exactly that 4 KB or 16 KB page from disk into RAM. The next time that layer runs, the page is already resident.

Why this matters:

• Loading a 70B model takes milliseconds. You don’t slurp 42 GB through the I/O bus. You just map it.
• Cold start is instant. The model starts generating tokens immediately, even though most weights aren’t in RAM yet. The first token might be slower as pages are faulted in, but subsequent tokens benefit from the warm cache.
• Multiple processes sharing the same GGUF file share the same physical RAM pages. Run two inference servers on the same model and the RAM cost doesn’t double.
• The OS manages the working set. If other applications need memory, the OS can evict unused model pages. If you’re only using a 70B model’s first 20 layers, the other 50 layers’ pages may never be loaded.

The mmap trick is the single biggest reason llama.cpp is fast. It is not new technology: Unix has had mmap since the 1980s. The insight was recognizing that LLM inference has exactly the memory access pattern mmap was designed for: large, randomly-accessed files with predictable locality of reference.

How does quantization work and which one should I use?

The intuition

Every parameter (weight) in an LLM starts as a 16-bit or 32-bit floating-point number. That’s 2-4 bytes per weight. For a 7B-parameter model, that’s 14-28 GB. Too big for most consumer hardware.

Quantization stores each weight in fewer bits (8, 5, 4, or even 3) by mapping the original range of values to a smaller set of integer codes. Each block of weights (typically 32 or 256 weights per block) stores a shared scale factor. When the model runs, the quantized integers are dequantized back to approximate floats using that scale.

The tradeoff is simple: fewer bits means a smaller model but more quantization error. The question is how much error is acceptable.

The three quantization families

Legacy quants (Q4_0, Q4_1, Q5_0, Q5_1, Q8_0): Blocks of 32 weights, one scale factor per block. These are the simplest and fastest. Q4_0 is symmetric (no zero-point offset), Q4_1 adds an offset for better handling of asymmetric weight distributions. Q8_0 at 8 bits is essentially lossless: the quantization error is negligible.

K-quants (Q2_K, Q3_K_S/M/L, Q4_K_S/M, Q5_K_S/M, Q6_K): The K stands for “keep.” The format preserves important weights at higher precision. K-quants use a hierarchical structure:

• Super-blocks of 256 weights
• Sub-blocks within each super-block, each with its own scale factor
• Mixed precision: some sub-blocks get more bits, some get fewer, depending on their importance

The S/M/L suffix on 3-bit, 4-bit, and 5-bit K-quants controls this tradeoff:

• S (small): larger sub-blocks, more compression, slightly more quality loss
• M (medium): balance, the community favorite
• L (large): smaller sub-blocks, less compression, better quality

K-quants are the most commonly used family. Q4_K_M is the default for good reason: it captures most of the quality improvement over legacy quants at a modest size increase.

I-quants (IQ2_XXS, IQ3_S, IQ4_NL, IQ4_K_XS, IQ5_K_S): Importance-matrix quants. During quantization, each weight is scored by how much it affects the model’s output quality. The most important weights are mapped to the most precise entries in a lookup table. Less important weights share coarser codes.

I-quants give the best quality-per-bit, meaning at the same bit budget, they outperform K-quants. But they’re more complex to compute and only worth it when you’re truly memory-squeezed (e.g., fitting a 7B model into 4 GB of RAM).

Critically, “best quality-per-bit” does NOT mean “better quality at small sizes.” An IQ2_XXS model at 2.6 GB will still perform worse than a Q4_K_M model at 4.9 GB. It just performs better than a Q2_K model at a similar size.

Benchmark data from arxiv 2601.14277

The paper “Which Quantization Should I Use?” by Uygar Kurt (Jan 2026) provides the most comprehensive evaluation of llama.cpp quantization on Llama-3.1-8B-Instruct as a single modern model. All models quantized from the same FP16 GGUF baseline (15,317 MiB) and evaluated on the same hardware using the EleutherAI lm_eval harness.

Format	Size (MiB)	Reduction	GSM8K	HellaSwag	IFEval	MMLU	TQA(mc2)	Avg	PPL
FP16	15317	–	77.63	72.51	78.93	63.50	54.79	69.47	7.32
Q8_0	–	–	–	72.52	–	–	–	–	–
Q6_K	–	–	78.17	–	–	–	–	–	–
Q5_0	–	~65%	79.08	–	–	–	–	69.75	–
Q5_1	–	–	78.47	–	–	–	–	–	–
Q5_K_M	–	–	78.54	–	–	–	–	–	–
Q5_K_S	–	–	75.66	–	–	–	–	–	–
Q4_K_M	~4690	69.41%	77.41	72.35	79.06	62.43	–	–	–
Q4_K_S	~4440	71.03%	77.33	–	–	–	–	–	–
Q4_1	–	68.11%	76.04	71.29	78.45	63.17	55.01	68.79	7.72
Q4_0	–	71.03%	75.66	71.88	77.46	62.20	52.68	67.98	7.74
Q3_K_L	–	73.14%	74.07	73.54	79.14	62.31	54.84	68.78	7.81
Q3_K_M	–	75.03%	73.16	73.41	77.19	62.01	54.56	68.07	7.96
Q3_K_S	–	77.23%	68.31	71.87	73.89	59.31	54.08	65.49	8.96

Source: arxiv.org/abs/2601.14277. Blank cells indicate the paper didn’t report per-benchmark breakdowns for those configurations in the main table. Size estimates based on reduction percentages from 15,317 MiB baseline.

Key findings from the paper:

1. GSM8K (math reasoning) is the most sensitive benchmark. It shows the widest variation across quantization levels, from 68.31 (Q3_K_S) to 79.08 (Q5_0). Multi-step math is where quantization error accumulates.

2. HellaSwag (commonsense) is nearly immune to quantization. All levels stay within ~1 point of the FP16 baseline. Simple multiple-choice reasoning doesn’t stress weight precision.

3. Q5_0 has the highest average score (69.75) across all benchmarks, even slightly above FP16 (69.47). The paper notes this is likely evaluation variance, not genuine improvement, but it shows that well-chosen 5-bit quantization doesn’t degrade quality.

4. The Pareto frontier (highest compression for a given quality level):

   • Q5_0: Accuracy-favoring choice with meaningful compression
   • Q4_K_S: Natural balanced default for stronger compression
   • Q3_K_L: Further compression with moderate loss
   • Q3_K_M: Additional compression step
   • Q3_K_S: Maximum reduction endpoint

   Notable: Q4_K_M is NOT on the Pareto frontier. Q4_K_S achieves similar quality with better compression. Q5_0 achieves better quality with similar compression. Q4_K_M sits in the dominated middle.

5. 3-bit is usable but risky. Q3_K_M and Q3_K_L retain most benchmark performance, but GSM8K drops noticeably (77.63 → 73.16 for K_M). Q3_K_S shows significant degradation across the board.

Practical decision framework

Use case	Recommended quant	Rationale
Interactive coding assistant	Q4_K_M or Q4_K_S	Fast, low memory, minimal quality loss. Q4_K_S if RAM-constrained
Reasoning / math / agent loops	Q5_0 or Q5_K_M	5-bit narrows GSM8K gap to under 1% vs FP16
Long-document summarization	Q4_K_M + large context	Quality lives in context, not precision
70B on Mac Studio (192 GB)	Q4_K_M	42 GB fits comfortably, 30+ tok/s
70B on 64 GB workstation	Q4_K_M + partial GPU offload	CPU+GPU split needed for the KV cache
Raspberry Pi / edge device	Q3_K_M or IQ4_NL	3-4 GB budget, quality tradeoff accepted
Production API server	Q8_0 or FP16	Eliminates quantization as a failure mode
Fitting a model into minimum RAM	IQ4_NL then Q3_K_L	Best quality-per-bit at extreme compression

How do I install and run my first model?

Pre-built binaries (recommended for first use)

# Download for macOS ARM
curl -L https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-cli-macos-arm64.tar.gz -o llama.tar.gz
tar xzf llama.tar.gz
./llama-cli --version

# Linux (x86_64)
# curl -L https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-cli-ubuntu-x64.tar.gz -o llama.tar.gz

Build from source (for GPU support)

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

# CPU only
cmake -B build
cmake --build build --config Release -j$(nproc)

# With Metal (Apple Silicon GPU)
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release -j$(nproc)

# With CUDA (NVIDIA GPU)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

# With Vulkan (cross-platform GPU)
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -j$(nproc)

# Binary lands at: build/bin/llama-cli (or build/bin/Release/llama-cli on Windows)

Running inference

# Interactive chat with automatic model download
./llama-cli \
  --hf-repo microsoft/Phi-3-mini-4k-instruct-gguf \
  --hf-file Phi-3-mini-4k-instruct-q4_k_m.gguf \
  -cnv

# One-shot completion with a local model
./llama-cli -m models/qwen-3.5-9b-q4_k_m.gguf \
  -p "Explain async/await in Python like I'm a senior engineer" \
  -n 512

# Chat mode (auto-detects chat template from GGUF metadata)
./llama-cli -m models/gemma-4-12b-q4_k_m.gguf -cnv

The parameters that matter

# Context window. Set before loading. Default: 4096.
./llama-cli -m model.gguf -c 8192 -p "..."

# CPU threads. -1 = auto-detect all cores. On Apple Silicon, match total core count.
./llama-cli -m model.gguf -t 8 -p "..."

# GPU layers offloaded. 0 = CPU only. -1 or 99 = all layers to GPU.
./llama-cli -m model.gguf -ngl 99   # all layers on GPU
./llama-cli -m model.gguf -ngl 35   # 35 layers on GPU, rest on CPU

# Generation length in tokens. -1 = unlimited (until EOS or context full).
./llama-cli -m model.gguf -n 2048 -p "..."

# Temperature. 0.0-1.0 for coherent output. Above 1.2 produces increasingly random text.
./llama-cli -m model.gguf --temp 0.2 -p "..."   # deterministic, factual
./llama-cli -m model.gguf --temp 0.7 -p "..."   # balanced, slight variety
./llama-cli -m model.gguf --temp 1.0 -p "..."   # varied, creative

# Top-p (nucleus) + Top-k sampling. Lower values = more focused output.
./llama-cli -m model.gguf --top-p 0.9 --top-k 40 --temp 0.8

# Min-P (alternative to top-p). More adaptive threshold.
./llama-cli -m model.gguf --min-p 0.05 --temp 0.7

# Repeat penalty. 1.0 = none. 1.1 = mild. Values >1.2 can cause topic drift.
./llama-cli -m model.gguf --repeat-penalty 1.1 -p "..."

# KV cache quantization. Reduces memory by 30-40% with minimal quality loss.
./llama-cli -m model.gguf --cache-type-k q4_0 --cache-type-v q4_0

# Batch size for prompt processing. Larger = faster prefill but more memory.
./llama-cli -m model.gguf -b 2048 -ub 512

GPU offload decision tree

Does your system have a dedicated GPU?
├── Apple Silicon      → cmake -DGGML_METAL=ON, then -ngl 99
│   ├── M4 Max         → ~89 tok/s on 14B Q4 (raw llama.cpp Metal)
│   └── M3 Pro (16GB)  → 45-60 tok/s on 7-8B Q4 (estimated)
├── NVIDIA GPU         → cmake -DGGML_CUDA=ON, then -ngl 99
├── AMD GPU            → cmake -DGGML_VULKAN=ON or -DGGML_HIP=ON
├── No discrete GPU    → CPU only, -t N (N = physical core count)
└── Mixed: GPU + RAM   → -ngl 35 to split layers between GPU and system RAM

On a 16 GB Mac Mini (your setup), a 7-9B Q4_K_M model with full Metal offload runs at roughly 40-55 tok/s. The GPU is faster but the CPU alone (with NEON) is fast enough for interactive use at 15-25 tok/s.

How do I use Python bindings in production?

The llama-cpp-python library wraps llama.cpp in a pip package with an OpenAI-compatible API.

pip install llama-cpp-python

# With Metal support (Apple Silicon):
# CMAKE_ARGS="-DGGML_METAL=ON" pip install llama-cpp-python

# With CUDA support:
# CMAKE_ARGS="-DGGML_CUDA=ON" pip install llama-cpp-python

Basic usage

from llama_cpp import Llama

llm = Llama(
    model_path="./models/qwen-3.5-9b-q4_k_m.gguf",
    n_ctx=4096,          # context window
    n_threads=8,         # CPU threads
    n_gpu_layers=99,     # Metal/CUDA (99 = all)
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a senior engineer."},
        {"role": "user", "content": "What's the difference between a mutex and a spinlock?"}
    ],
    temperature=0.3,
    max_tokens=512,
)

print(response["choices"][0]["message"]["content"])

OpenAI-compatible server

In one command, you get a full REST API that speaks the same protocol as OpenAI:

python -m llama_cpp.server \
  --model ./models/qwen-3.5-9b-q4_k_m.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  --n_gpu_layers 99

Now any tool that speaks OpenAI’s API can use your local model:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

This is how Hermes, LangChain, and other frameworks connect to local models. You swap the endpoint URL and everything works the same.

How does inference work step by step?

Understanding the full pipeline helps you debug crashes, performance problems, and unexpected output.

Step 1: Parse GGUF header. llama-cli reads the binary header, validates magic bytes (GGUF), extracts architecture type (LLaMA, Mistral, Qwen, etc.), layer count, and per-tensor quantization metadata. This is fast. The header is typically 4-16 KB.

Step 2: Build the execution graph. GGML allocates tensor slots for every weight matrix and wires them into a computation graph. The graph is fixed at load time: input tokens → embedding → N transformer blocks → output projection. Each block contains self-attention (Q, K, V projections + output projection) and feed-forward (gate, up, down projections).

Step 3: Memory-map weights. The GGUF file is mmap’d. As the execution graph runs, each layer’s weights are paged in from disk on demand. A 42 GB 70B model doesn’t need 42 GB of free RAM upfront. Pages are loaded as they’re needed.

Step 4: Run the forward pass with KV caching. For each token position:

• Compute the embedding for the current token
• Run it through all N transformer blocks
• In each block, self-attention computes Query vectors for the current token and Key/Value vectors that are cached for future tokens
• The cached K and V from all previous tokens are concatenated with the current token’s K and V, enabling attention across the full context

The KV cache grows linearly: 2 × n_layers × n_kv_heads × head_dim × context_length × bytes_per_element. For a 70B model at 8K context in FP16: roughly 2 GB. At 32K context: roughly 8 GB.

Without the KV cache, generating token N would require recomputing attention for all N-1 previous tokens, making inference O(N²) instead of O(N) per token.

Step 5: Sample the next token. The output projection produces logits over the vocabulary. The sampling pipeline (temperature → top-k → top-p/min-p → random draw) selects one token. That token becomes the input for step 4, and the loop repeats until the stop condition is met (max tokens, EOS token, or stop string).

The key bottleneck: This loop is sequential by design. Token N+1 needs the KV cache from tokens 0..N. You can’t parallelize across tokens within one sequence. But you CAN parallelize across sequences (batch processing): sending 8 prompts at once amortizes the per-layer weight-loading cost across all 8 sequences. This is how inference servers achieve high throughput.

How does KV cache memory management work?

The KV cache is often the largest memory consumer during inference, especially at long context lengths.

KV cache size formula

KV_cache_bytes = 2 × n_layers × n_kv_heads × head_dim × context_length × dtype_bytes

Example: Llama 3.1 70B at 8K context in FP16:

• n_layers = 80, n_kv_heads = 8, head_dim = 128
• = 2 × 80 × 8 × 128 × 8192 × 2 = ~2.68 GB

At 128K context: ~42 GB, bigger than the model weights themselves.

Reducing KV cache memory

llama.cpp supports KV cache quantization, storing the cache in fewer bits than FP16:

# Quantize K cache to Q4_0, V cache to Q4_0
./llama-cli -m model.gguf --cache-type-k q4_0 --cache-type-v q4_0

This cuts KV cache memory by ~60% (from 2 bytes per element to ~0.5 bytes) with minimal quality impact. The K cache (keys) is more sensitive to quantization than V cache (values), so some setups use q4_0 for V and q8_0 for K.

llama.cpp also uses paged KV cache, similar to virtual memory paging. The cache is allocated in fixed-size blocks (typically 512 tokens each). Blocks are allocated on demand and can be spooled to system RAM if GPU VRAM runs out. This avoids OOM failures on long generations: instead of crashing, generation slows down as some cache pages live in system RAM.

Context window sizing rules of thumb

Model size	Q4_K_M weights	KV cache (8K ctx)	KV cache (32K ctx)	Total RAM (8K ctx)
3B	~2 GB	~0.3 GB	~1.2 GB	~2.5 GB
7-8B	~4.5 GB	~0.6 GB	~2.4 GB	~5.5 GB
12-14B	~7.5 GB	~1.0 GB	~4.0 GB	~9 GB
30B	~18 GB	~1.5 GB	~6 GB	~20 GB
70B	~42 GB	~2.7 GB	~11 GB	~46 GB
120B	~72 GB	~4 GB	~16 GB	~78 GB

What are the most common mistakes and how do I fix them?

Mistake 1: Setting context size too small.

# Wrong: default 4096 cuts off longer prompts silently
./llama-cli -m model.gguf -p "Here is a 5000-token document: [paste long text]"
# Correct: match your prompt + expected response length
./llama-cli -m model.gguf -c 8192 -p "Here is a 5000-token document: [paste long text]"

Mistake 2: Not setting thread count on Apple Silicon. Auto-detection on macOS sometimes under-uses performance cores. Set -t to the total number of cores (E + P).

# M3 Pro: 4E + 6P = 10 logical cores
./llama-cli -m model.gguf -t 10 -p "..."

Mistake 3: Ignoring KV cache memory at long context. At 32K context, the KV cache for a 70B model is ~11 GB. If you set -c 32768 without enough RAM, the OS starts swapping.

# Safe: quantize the KV cache too
./llama-cli -m model.gguf -c 32768 --cache-type-k q4_0 --cache-type-v q4_0

Mistake 4: Comparing models at different quantization levels. When evaluating two different models, always compare at the same quant level. Q4_K_M on Model A vs Q8_0 on Model B isn’t a fair model comparison. It’s a quantization comparison. State your quantization level explicitly.

Mistake 5: Using temperature > 1.0 expecting creativity. Temperature above 1.0 flattens the probability distribution, which makes the model increasingly random, not usefully creative. For varied but coherent output, stay in the 0.7-1.0 range. For deterministic output, use 0.1-0.3.

Mistake 6: Forgetting to build with the right backend. If you build without -DGGML_METAL=ON on Apple Silicon, inference runs entirely on CPU via NEON. You’ll get 15-25 tok/s instead of 45+ tok/s for a 7B model. Always check the build output for your GPU backend being enabled.

What quantization should I use for my use case?

Use case	Recommended quant	Memory (model)	Memory (total with 8K ctx)	Why this choice
Interactive coding assistant	Q4_K_M or Q4_K_S	~4.5 GB	~5.5 GB	Fast, fits 16 GB Mac Mini, minimal quality loss
Reasoning / agent loops	Q5_0 or Q5_K_M	~5.5 GB	~6.5 GB	GSM8K drop <1% vs FP16
Production API	Q8_0	~8.6 GB	~9.5 GB	Eliminates quantization surprise
70B on Mac Studio	Q4_K_M	~42 GB	~46 GB	Fits 192 GB, 30+ tok/s
70B on 64 GB workstation	Q4_K_M + partial GPU offload	~42 GB	~46 GB	May need CPU-only at full context
Embedded / edge	IQ4_NL or Q3_K_M	~3-4 GB	~3.5-4.5 GB	Aggressive compression, quality tradeoff
Long context (32K+)	Q4_K_M + KV cache quant	~4.5 GB	~7 GB	Memory dominated by KV cache, not weights

FAQ

What is the difference between llama.cpp and Ollama?

Ollama bundles llama.cpp, a model library, a version manager, and a REST API into one product. ollama run llama3.2 downloads a GGUF, loads it with defaults, and gives you a chat interface. Under the hood, it calls llama.cpp. llama.cpp is the engine. Ollama is the car. For development, the engine gives you finer control. For deployment, Ollama’s server mode is often the right abstraction. (As of 2026, Ollama has switched to MLX for Apple Silicon.)

My model generates garbage. What do I check first?

Temperature first. If you set temp > 1.0, the model produces increasingly random output. Drop it to 0.3 and check if output becomes coherent. If yes, your prompt is fine. If no, check the chat template. A model trained with LLaMA format but prompted with Phi format produces word salad. GGUF metadata usually handles this, but for fine-tuned variants you may need --chat-template explicitly.

Can I run a 70B model on my machine?

Check your available RAM. A 70B at Q4_K_M needs ~42 GB for weights plus ~2.7 GB for KV cache at 8K context. If you have less than 48 GB total system RAM, it won’t fit. A single RTX 4090 (24 GB) cannot fit a 70B. You would need CPU-GPU split or a unified memory system (Mac Studio M2 Ultra, Strix Halo).

What’s the difference between --temp, --top-p, and --min-p?

--temp scales the probability distribution before sampling. --top-p discards the least probable tokens until the remaining cumulative probability exceeds P. --min-p discards tokens with probability less than min_p × probability_of_top_token. Min-p is newer and generally more adaptive than top-p. It adjusts the threshold based on the shape of the distribution rather than using a fixed cutoff.

Should I use --cache-type-k and --cache-type-v?

Yes, if you’re running at long context lengths (>8K tokens). Quantizing the KV cache to q4_0 reduces its memory footprint by about 60% with negligible quality impact. At 32K context on a 70B model, this can save 6-7 GB of RAM. The V cache is more robust to quantization than the K cache. In practice, q8_0 for K and q4_0 for V balances quality with compression.

How do I know if my build has GPU acceleration?

Run ./llama-cli --version and look for the backend list. If you compiled with Metal, it should show Metal or Metal,BLAS in the build info. On a running model, check the startup log for lines like “ggml_metal_init” or “CUDA backend initialized.”

Related Posts

• I tested 7 local LLMs on real agent work. Two survived.. Dense models beat MoE variants on agentic tool calling. The same quantization principles apply: Q4_K_M is the floor for production agent use.
• Which local hardware should you buy for which LLM in 2026?. VRAM requirements by model size and quantization level. Complements this post: know your model before you know your hardware.
• The 2026 open-source model landscape: 5 architecture families. The model families llama.cpp supports natively. Knowing the architecture tells you which quantization flags matter.

---

This article was published on Agentic Up (https://agenticup.dev): practical guides for developers and founders building with AI agents. Reach me at [email protected]