Self-host Text Generation Inference on a VPS

Hugging Face's production inference server for transformer language models, with continuous batching and streaming.

levelrail-cli templates deploy text-generation-inference --name my-text-generation-inference

Specification

Status
Listed
Category
AI
Services
tgi
Ports
80
Volumes
1
Memory
about 8 GiB
arm64
not available
amd64
image published
Settings to fill in
none

This template is deprecated. Upstream repository is archived.

About Text Generation Inference

Hugging Face's production inference server for transformer language models, with continuous batching and streaming. Runs as a single container from the text-generation-inference image, listening on port 80; data is kept in named volumes.

What it deploys

Text Generation Inference runs as one service in a single Compose app, listening on port 80, with 1 named volume mount.

ServiceImageTagPortsVolumes
tgi ghcr.io/huggingface/text-generation-inference 3.3.6 80 tgi_data -> /data

Before you install

  • CPU architecture. amd64 only. At least one image lists no linux/arm64, so there is no native Arm image.
  • Listed. It is in the catalog. Back up the data volume before changing an image tag.
  • GPU. The catalog marks this template as needing an NVIDIA GPU on the node to be practical.
  • Host ports. It publishes port 80 directly on the host, so open them in the VPS firewall only if you want that.

Settings and environment

The catalog lists no environment variables for this template.

Install Text Generation Inference on a VPS

  1. Install Levelrail on a fresh VPS

    Run this as root on a Linux server (amd64 or arm64) with ports 80 and 443 reachable. The installer prints a setup link.

    curl -fsSL https://levelrail.com/install.sh | sudo sh
  2. Sign in

    Open the setup link, choose a password, then log the CLI in. The CLI install command is curl -fsSL https://levelrail.com/install-cli.sh | sh if the machine does not have it. See Getting started.

    levelrail-cli login --device
  3. Read the template

    Shows the entry, including its full compose.yaml.

    levelrail-cli templates get text-generation-inference
  4. Deploy it

    This deploys the template compose file as an app. The app name defaults to the template id; --name sets another.

    levelrail-cli templates deploy text-generation-inference --name my-text-generation-inference

Without a domain the app gets an sslip.io address. You can also deploy it from the dashboard: Apps, New app, Browse templates, then Text Generation Inference.

More in AI

All AI templates