Get the most from local inference

Local LLM inference is served by an embedded llama-server (upstream llama.cpp) that the daemon spawns and supervises over its OpenAI-compatible HTTP API. It is the engine of GGUF models, and there is no separate process to install, run, or point Apollia at. On macOS arm64 the desktop app also ships an MLX engine (vllm-mlx) for models in the MLX format: the format of a model picks its engine, and System > Engines sets the engine of each format. This page is about llama-server. It ships prebuilt inside the desktop app, and on a source build the daemon finds llama-server on your PATH (see Install and run the runtime).

Two capabilities that used to require an extra, hand-run server are now on by default, because the embedded engine is that server:

• Continuous batching. The engine decodes several sequences in the same GPU pass, so concurrent and batch requests share the hardware instead of serializing one behind another. Nothing to enable.
• Native tool calling. The engine is driven with --jinja, so tool calls go through the model's own chat template rather than a bespoke grammar path. Local models call your tools reliably, with no tuning on your side.

Tracking upstream llama.cpp also widens model coverage: newer architectures land in the engine as they land upstream.

Which engine ships with which artifact

The embedded engine is chosen when the artifact is built, one per artifact, so there is nothing to select at install time. There is one installer per system. What ships:

Artifact	Engine	Covers
Desktop app, macOS	Metal	Apple Silicon
Desktop app, Windows and Linux	Vulkan	NVIDIA, AMD and Intel
CLI, *-cpu presets	CPU	any machine, no GPU offload
CLI, *-vulkan presets	Vulkan	NVIDIA, AMD and Intel

Vulkan in the desktop bundle is a deliberate trade. One 32 MB binary reaches all three vendors, where the CUDA engine covers NVIDIA alone, weighs 238 MB plus a 373 MB runtime, and falls back to the processor on every other card. So the CUDA engine is not in any installer. It is published apart, on the engine channel, and the application downloads it on demand: on a machine with an NVIDIA card, the AI screen offers it, Speed up with your NVIDIA card, and one click installs it. The next model load uses it, and the embedded Vulkan engine stays in place as the fallback. On Windows the download carries the CUDA runtime libraries; on Linux the engine relies on those of the machine.

Nothing is run before it is verified. The channel publishes one manifest listing each archive with its SHA-256, signed in the minisign format with a dedicated distribution key. The application carries the public half of that key, requests no archive before the signature holds, and deletes an archive whose digest differs. A machine without network imports the same three kinds of file, manifest, signature and archive, from a disk, through the same checks.

The command-line archives have no CUDA preset, on any platform. If you run the daemon from the CLI on an NVIDIA card and want CUDA rather than Vulkan, point it at a build of your own, below.

Configure a local backend

Register a .gguf model as a local backend and let the daemon serve it. The provider name is llama-cpp:

apollia-os llm setup --local --model /path/to/model.gguf
apollia-os llm reload
apollia-os llm status

The daemon starts the embedded llama-server for that model on demand and routes inference to it. SSE streaming and tool calls travel the same path, already wired and tested.

Get good throughput

The engine handles the mechanics (GPU offload, batching, KV cache). The choices that move throughput are upstream of it:

• Pick a model sized to your hardware. A mixture-of-experts (MoE) model activates only a fraction of its parameters per token, so it can beat a dense model of similar quality on both speed and batch throughput. Prefer a quantization that leaves headroom for the KV cache. apollia-os model recommend ranks candidates for this machine, with the memory each one needs and how many of its layers reach the GPU.
• One model per engine process. Each loaded model runs in an engine of its own, and several stay resident when the load guardrail judges they fit beside each other; APOLLIA_LLAMA_MAX_LOADED sets an explicit ceiling.
• Slots are planned for you. Continuous batching is on by default, and the number of decode slots is planned at each start from the model and the machine's memory, up to eight, so concurrent requests decode together when there is room. APOLLIA_LLAMA_N_PARALLEL (below) pins the count instead.

Tune the embedded engine

The embedded engine reads thirteen environment variables at every start, so a knob can be turned without a source build. Set them in the environment of whatever launches the daemon.

Variable	Default	What it does
APOLLIA_LLAMA_N_CTX	(unset)	Context window in tokens, over every other source. Unset, the window is the preset's, the backend's, then the default of [llm.defaults] in apollia.toml, and 32768 last.
APOLLIA_LLAMA_N_GPU_LAYERS	planned, else 999	Layers offloaded to the GPU; 0 forces CPU. Left unset, the count is planned per model on a discrete GPU (below).
APOLLIA_LLAMA_N_BATCH	engine default	Logical batch size.
APOLLIA_LLAMA_N_UBATCH	engine default	Physical micro-batch size.
APOLLIA_LLAMA_N_PARALLEL	planned	Decode slots served concurrently. Unset, the count is planned from the model and the memory, up to 8; set, it is used as given.
APOLLIA_LLAMA_CONT_BATCHING	true	Continuous batching.
APOLLIA_LLAMA_CACHE_TYPE_K	engine default	KV cache quantization, keys.
APOLLIA_LLAMA_CACHE_TYPE_V	engine default	KV cache quantization, values.
APOLLIA_LLAMA_FLASH_ATTN	on	Flash attention mode.
APOLLIA_LLAMA_CACHE_REUSE	engine default	Prefix-reuse threshold.
APOLLIA_LLAMA_METRICS	false	Exposes the engine's metrics endpoint.
APOLLIA_LLAMA_LOG_VERBOSITY	4	Engine log level (-lv). At 4 the load names its device and its offload tally, which the daemon's journal reports as llama.server.device, llama.server.offload and llama.server.buffer.
APOLLIA_LLAMA_EXTRA_ARGS	empty	Extra flags passed through verbatim.

The slot count is what turns continuous batching into real concurrency: with a single slot, requests queue. Pin N_PARALLEL only when the planned count does not suit your load.

When APOLLIA_LLAMA_N_GPU_LAYERS is unset and the machine has a graphics card with memory of its own, the daemon no longer asks for every layer. It reads the header of the model file, takes the context cache and the graphics context off the card's memory, and offloads as many layers as the remainder holds; the rest run on the processor. Asking for every layer on a card too small for the model made the engine fail to allocate instead of placing what it could. The journal reports the decision as llama.server.ngl.planned, with n_gpu_layers against total_layers. On Apple Silicon, where the processor and the GPU share one memory, and on a machine where no GPU was detected, the daemon still passes 999: every layer belongs on a shared-memory GPU, and with no GPU the engine ignores the value. Setting the variable always wins.

To check that a model really loaded on the accelerator, read the daemon's journal at the next engine start: llama.server.offload carries the layers placed against the model's total, and llama.server.buffer the size of each buffer per device.

The full list, including the secret-storage and diagnostic variables, is in Environment variables.

Run an engine of your own

On a source build the daemon uses the llama-server it finds on your PATH rather than a bundled binary. The repository ships a recipe to start one for local testing:

just llama-server /path/to/model.gguf

That recipe runs the upstream binary, so the usual llama.cpp options apply when you experiment locally: total context (-c) split across parallel slots (-np), GPU offload (-ngl), flash attention (--flash-attn on), and a quantized KV cache (-ctk q8_0 -ctv q8_0, which needs flash attention). These are upstream llama.cpp flags, useful for probing what your hardware sustains before you settle on a model and quantization.

APOLLIA_LLAMA_SERVER_BIN points the daemon at a binary of your choosing, and it wins over the bundled one:

export APOLLIA_LLAMA_SERVER_BIN=/opt/llama.cpp/build/bin/llama-server

This is also the path for a CUDA build of your own. An operator running NVIDIA cards usually already has a llama.cpp compiled for those cards and tuned for that load; Apollia uses it rather than replacing it with another build.

Related

• Install and run the runtime for the build and the PATH requirement on a source build.
• Deploy in production for serving on a server.
• The CLI reference for the llm commands.