Self-host llama.cpp Server on a VPS

The llama.cpp HTTP server: run quantized GGUF models on plain CPUs with an OpenAI-compatible API.

levelrail-cli templates deploy llama-cpp --name my-llama-cpp

Specification

Status
Listed
Category
AI
Services
llama-cpp
Ports
8080
Volumes
1
Memory
about 4 GiB
arm64
image published
amd64
image published
Settings to fill in
none

About llama.cpp Server

The llama.cpp HTTP server: run quantized GGUF models on plain CPUs with an OpenAI-compatible API. Runs as a single container from the llama.cpp image, listening on port 8080; data is kept in named volumes.

What it deploys

llama.cpp Server runs as one service in a single Compose app, listening on port 8080, with 1 named volume mount.

ServiceImageTagPortsVolumes
llama-cpp ghcr.io/ggml-org/llama.cpp server 8080 llama_cpp_models -> /models

Before you install

  • CPU architecture. arm64 listed. Every image lists linux/arm64 in its registry manifest. That is a manifest fact, not proof the app runs on Arm.
  • Listed. It is in the catalog. Back up the data volume before changing an image tag.
  • Host ports. It publishes port 8080 directly on the host, so open them in the VPS firewall only if you want that.

Settings and environment

You do not have to fill in anything. Values are never shown here.

ServiceVariableValue
llama-cppLLAMA_CACHEPreset by the template

Install llama.cpp Server on a VPS

  1. Install Levelrail on a fresh VPS

    Run this as root on a Linux server (amd64 or arm64) with ports 80 and 443 reachable. The installer prints a setup link.

    curl -fsSL https://levelrail.com/install.sh | sudo sh
  2. Sign in

    Open the setup link, choose a password, then log the CLI in. The CLI install command is curl -fsSL https://levelrail.com/install-cli.sh | sh if the machine does not have it. See Getting started.

    levelrail-cli login --device
  3. Read the template

    Shows the entry, including its full compose.yaml.

    levelrail-cli templates get llama-cpp
  4. Deploy it

    This deploys the template compose file as an app. The app name defaults to the template id; --name sets another.

    levelrail-cli templates deploy llama-cpp --name my-llama-cpp

Without a domain the app gets an sslip.io address. You can also deploy it from the dashboard: Apps, New app, Browse templates, then llama.cpp Server.

More in AI

All AI templates