Self-host vLLM on a VPS
A fast, high-throughput inference server that exposes an OpenAI-compatible API for open-weight language models.
levelrail-cli templates deploy vllm --name my-vllmSpecification
- Status
- Listed
- Category
- AI
- Services
- vllm
- Ports
- 8000
- Volumes
- 1
- Memory
- about 8 GiB
- arm64
- image published
- amd64
- image published
- Settings to fill in
- none
- License
- Permissive license, Apache-2.0
About vLLM
A fast, high-throughput inference server that exposes an OpenAI-compatible API for open-weight language models. Runs as a single container from the vllm-openai image, listening on port 8000; data is kept in named volumes.
What it deploys
vLLM runs as one service in a single Compose app, listening on port 8000, with 1 named volume mount.
| Service | Image | Tag | Ports | Volumes |
|---|---|---|---|---|
| vllm | vllm/vllm-openai | v0.30.0 | 8000 | vllm_hf_cache -> /root/.cache/huggingface |
Before you install
- CPU architecture. arm64 listed. Every image lists linux/arm64 in its registry manifest. That is a manifest fact, not proof the app runs on Arm.
- Listed. It is in the catalog. Back up the data volume before changing an image tag.
- GPU. The catalog marks this template as needing an NVIDIA GPU on the node to be practical.
- Host ports. It publishes port 8000 directly on the host, so open them in the VPS firewall only if you want that.
Settings and environment
You do not have to fill in anything. Values are never shown here.
| Service | Variable | Value |
|---|---|---|
| vllm | HF_HOME | Preset by the template |
Install vLLM on a VPS
-
Install Levelrail on a fresh VPS
Run this as root on a Linux server (amd64 or arm64) with ports 80 and 443 reachable. The installer prints a setup link.
curl -fsSL https://levelrail.com/install.sh | sudo sh -
Sign in
Open the setup link, choose a password, then log the CLI in. The CLI install command is
curl -fsSL https://levelrail.com/install-cli.sh | shif the machine does not have it. See Getting started.levelrail-cli login --device -
Read the template
Shows the entry, including its full compose.yaml.
levelrail-cli templates get vllm -
Deploy it
This deploys the template compose file as an app. The app name defaults to the template id; --name sets another.
levelrail-cli templates deploy vllm --name my-vllm
Without a domain the app gets an sslip.io address. You can also deploy it from the dashboard: Apps, New app, Browse templates, then vLLM.
Docs and upstream
More in AI
- UnstructuredAn API that turns PDFs, Word files, and other documents into clean data for RAG pipelines.
- Wyoming openWakeWordLocal wake word detection over the Wyoming protocol for Home Assistant voice assistants, preloaded with the ok nabu model.
- Wyoming PiperFast local text to speech over the Wyoming protocol, used by Home Assistant voice assistants. Downloads the chosen voice on first start.
- Text Generation InferenceHugging Face's production inference server for transformer language models, with continuous batching and streaming.
- Text Embeddings InferenceA lean server that turns text into embeddings with popular open models, on CPU with no extra setup.
- Stable Diffusion WebUIA browser interface for generating and editing images with Stable Diffusion checkpoints.