# Self-host llama.cpp Server

> The llama.cpp HTTP server: run quantized GGUF models on plain CPUs with an OpenAI-compatible API.

- Page: https://levelrail.com/templates/llama-cpp/
- JSON: https://levelrail.com/templates/llama-cpp.json
- Category: AI
- Licence: MIT (Permissive license)
- Source: https://github.com/ggml-org/llama.cpp
- Upstream docs: https://github.com/ggml-org/llama.cpp/tree/master/tools/server

## Requirements
- Memory: about 4 GiB
- amd64: published, arm64: published
- Needs configuration before start: no
- Pinned image tag: server

## Services

| Service | Image | Tag | Ports | Volumes |
| --- | --- | --- | --- | --- |
| llama-cpp | ghcr.io/ggml-org/llama.cpp | server | 8080 | llama_cpp_models -> /models |

## Environment

| Service | Variable | Kind |
| --- | --- | --- |
| llama-cpp | LLAMA_CACHE | preset |

## Deploy

### CLI

```bash
levelrail-cli templates deploy llama-cpp --name my-llama-cpp
```

### API

```bash
curl -X POST -H "Authorization: Bearer $TOKEN" \
  https://levelrail.example.com/api/v1/service-templates/llama-cpp/deploy
```

Create a token in the dashboard first. This one-click route deploys it as a trial app you can keep or remove.

## Questions

### Can I deploy llama.cpp Server on Dokploy or Dokku?

The services and pinned images on this page can be run on any Docker host. The public Dokploy templates list we compared does not have an entry with the same id for llama.cpp Server, so on Dokploy you would use a Docker Compose application with the file from this page. Dokku deploys container images with its own commands, so the pinned image ghcr.io/ggml-org/llama.cpp:server is what you deploy there.

Self-host llama.cpp Server, llama.cpp Server docker compose file, deploy llama.cpp Server on a VPS: this page lists the pinned image, ports, volumes and the commands for each.
