Data to 2026-11-18
Measured on real servers
40 measurements: 29 pass, 8 slower than hoped, 3 fail. None is hidden. Each has the machine it ran on, the date, the build, the exact command and a link to the raw evidence.
Most rows so far ran on a laptop or a test box, not on a production server. The host column says which, and the note says what weakens the figure. Rows are added as they are measured, never edited to look better.
Footprint
Memory and CPU of the control plane while idle. Rows marked not comparable ran on a dev build and are kept so nobody mistakes them for server figures.
| Metric | Result | Status | Host | Run | Method | Evidence |
|---|---|---|---|---|---|---|
| Control plane idle RSS Peak 253 MB, 273 goroutines, 1.5 percent CPU; one live instance, one measurement. The provider and CPU architecture were not recorded. | 200 MB | pass | not stated, test boxnot stated, 2 vCPU, 8 GB, Linux | not recorded, build not recorded | Read from the Control plane health page of a live instance, all features at defaults, no apps under load | Raw evidence for Control plane idle RSS |
| Control plane idle RSS About 1.3 percent of one core over the same five minutes. The VM size and provider were not recorded. | 184 MiB | pass | not stated, VMarm64, size not stated, Ubuntu | not recorded, build 243a8ee00 | Resident memory of the control plane over five idle minutes, 12 containers running on the VM | Raw evidence for Control plane idle RSS |
| Boot harness peak RSS, Linux arm64, experimental features off 100 goroutines, 0.41 percent CPU. Ran in a golang:1.23 container limited to 2 CPUs and 8 GB with a cross-compiled test binary, so RSS is an upper bound for the shipped binary. | 232 MB | pass | Container on a local laptop, Apple M4 Max hostarm64, 2 vCPU, 8 GB, Linux (container on a Mac host) | not recorded, build not recorded | go test ./cmd/levelrail -run TestFootprint (60 s idle per run after a 20 s settle, no Docker daemon) | Raw evidence for Boot harness peak RSS, Linux arm64, experimental features off |
| Boot harness peak RSS, Linux arm64, experimental features on 104 goroutines, 0.40 percent CPU. Ran in a golang:1.23 container limited to 2 CPUs and 8 GB with a cross-compiled test binary, so RSS is an upper bound for the shipped binary. | 233 MB | pass | Container on a local laptop, Apple M4 Max hostarm64, 2 vCPU, 8 GB, Linux (container on a Mac host) | not recorded, build not recorded | go test ./cmd/levelrail -run TestFootprint (60 s idle per run after a 20 s settle, no Docker daemon) | Raw evidence for Boot harness peak RSS, Linux arm64, experimental features on |
| Boot harness peak RSS, macOS arm64, experimental features off 114 goroutines, 0.22 percent CPU. Test binary on the host, so RSS is an upper bound for the shipped binary. | 170 MB | pass | Local laptop, Apple M4 Max hostarm64, 14 vCPU, 36 GB, macOS 27.2 | not recorded, build not recorded | go test ./cmd/levelrail -run TestFootprint (60 s idle per run after a 20 s settle, no Docker daemon) | Raw evidence for Boot harness peak RSS, macOS arm64, experimental features off |
| Idle RSS, 0 apps (Mac dev build) Not comparable A Mac dev build, not comparable with the server figures above. The date is the commit that added the row to the doc. | 67.7 MB | pass | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS | not recorded, build dev build | Idle window of 60 s on a dev build, same-machine loopback client, then RSS read at the end | Raw evidence for Idle RSS, 0 apps (Mac dev build) |
| Idle RSS, 0 apps (dev-mode control plane) Not comparable The same run read 128 MB at 10 apps and 84 MB at 50, so RSS here follows garbage collection timing, not the app count. Dev mode on a laptop. | 248 MB | slower than hoped | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2 | not recorded, build not recorded | BENCH_COUNTS="0 10 50" BENCH_SECONDS=30 scripts/bench-idle.sh | Raw evidence for Idle RSS, 0 apps (dev-mode control plane) |
| Control plane idle RSS, v1.5.3 Range 212 to 223 MB (peak shown), about 0.4 percent of one vCPU; one host, one run. | 223 MB | pass | Oracle Cloud ARM, Ampere A1 VMaarch64, 1 vCPU, 5.8 GB, Ubuntu 24.04 | v1.5.3, build not recorded | Resident memory, 6 samples 30 s apart starting 10 min after the upgrade, 4 test containers running | Raw evidence for Control plane idle RSS, v1.5.3 |
Throughput and latency
Requests through the ingress or the load driver. Most of these ran with the load generator on the same machine, which the host column and the note say.
| Metric | Result | Status | Host | Run | Method | Evidence |
|---|---|---|---|---|---|---|
| Highest rate held through the ingress, 0 of 128000 errors No ceiling found, the run stopped at the top step. Load driver and stack shared the laptop, so this is a floor for a trivial fixture, not a server figure. | 12,800 req/s | pass | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | not recorded, build not recorded | levelrail-cli bench chaos run ingress-steady (build with -tags chaoslive), open loop, 10 s per step, one replica, TLS on loopback | Raw evidence for Highest rate held through the ingress, 0 of 128000 errors |
| p99 latency at 12800 req/s through the ingress p50 0.66 ms, p95 1.82 ms. Same single run as above. | 9.51 ms | pass | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | not recorded, build not recorded | levelrail-cli bench chaos run ingress-steady (build with -tags chaoslive), open loop, 10 s per step, one replica, TLS on loopback | Raw evidence for p99 latency at 12800 req/s through the ingress |
| Failed requests at the 6400 req/s step 0.03 percent. Not reproduced in the step above it and not investigated. | 19 failed of 64000 | slower than hoped | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | not recorded, build not recorded | levelrail-cli bench chaos run ingress-steady (build with -tags chaoslive), open loop, 10 s per step | Raw evidence for Failed requests at the 6400 req/s step |
| Highest clean step, run 1 Host load average 13 to 30 on 14 CPUs during the runs. The ceiling is unestablished. | 3,200 req/s | pass | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | post v0.8.3, build 58233c7 | levelrail-cli bench chaos run ingress-steady --repeat 2 --out results.json | Raw evidence for Highest clean step, run 1 |
| Highest clean step, run 2 The 400 and 800 steps already showed 13 and 11 errors and p95 over 800 ms, which the docs attribute to a busy host and a degrading Docker Desktop. It did not reproduce the 12800 req/s run. | 800 req/s | slower than hoped | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | post v0.8.3, build 58233c7 | levelrail-cli bench chaos run ingress-steady --repeat 2 --out results.json | Raw evidence for Highest clean step, run 2 |
| Failed requests at the 6400 req/s step, run 1 Docker Desktop stopped answering during or just after the next run. Whether the overload caused it was not investigated. | 34,486 failed of 64000 | fail | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | post v0.8.3, build 58233c7 | levelrail-cli bench chaos run ingress-steady --repeat 2 --out results.json | Raw evidence for Failed requests at the 6400 req/s step, run 1 |
| Closed loop at concurrency 32, best of 3 runs Runs read 13911, 17236 and 18538 req/s with 0 errors, p99 9.2, 8.6 and 4.9 ms. Closed loop, so it can hide stalls. | 18,538 req/s | pass | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | post v0.8.3, build 1403b01 | levelrail-cli bench chaos run ingress-concurrency --repeat 2 --out results.json | Raw evidence for Closed loop at concurrency 32, best of 3 runs |
| p99 latency at concurrency 128, worst of 3 runs p99 59.8, 174.5 and 241.5 ms, with 15, 58 and 0 errors. Client and server share the laptop, so this is not an ingress limit. | 241.5 ms | slower than hoped | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | post v0.8.3, build 1403b01 | levelrail-cli bench chaos run ingress-concurrency --repeat 2 --out results.json | Raw evidence for p99 latency at concurrency 128, worst of 3 runs |
| Load driver against the fixture app, 5000 req/s 0 of 75000 failed, p50 0.16 ms, p99 16.45 ms. This is the ceiling of the driver and fixture, a floor to subtract from real runs, not a Levelrail deployment. | 5,000 req/s | pass | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Go 1.26.9 | not recorded, build not recorded | levelrail-cli bench run --url http://127.0.0.1:PORT/ --rate 5000 --duration 15s (fixture built from source, no ingress, no Docker) | Raw evidence for Load driver against the fixture app, 5000 req/s |
| p99 latency during a redeploy at 200 rps 18000 of 18000 ok, p50 2.04, p95 3.01 ms; baseline p50 1.99, p95 2.58, p99 5.57 ms. Load generator on a second VM over the private network, one run, 1 vCPU. | 7.94 ms | pass | Oracle Cloud ARM, Ampere A1 VMaarch64, 1 vCPU, 5.8 GB, Ubuntu 24.04 | v1.5.3, build not recorded | Open loop 200 rps for 90 s through the ingress, apps deploy at t=28 s | Raw evidence for p99 latency during a redeploy at 200 rps |
Deploy
Image to first answer from the new version, and what clients saw during a rollout.
| Metric | Result | Status | Host | Run | Method | Evidence |
|---|---|---|---|---|---|---|
| Cold deploy, image to first answer, slowest of 3 runs Runs read 4678, 3653 and 3442 ms. Each includes the 3 s stability window the deploy gate waits for. | 4,678 ms | pass | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | post v0.8.3, build 1403b01 | levelrail-cli bench chaos run deploy-latency --repeat 2 --out results.json | Raw evidence for Cold deploy, image to first answer, slowest of 3 runs |
| Warm redeploy p95, slowest of 3 runs Five redeploys per run, so p95 is the maximum. Warm p95 read 4446, 4501 and 3922 ms. | 4,501 ms | pass | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | post v0.8.3, build 1403b01 | levelrail-cli bench chaos run deploy-latency --repeat 2 --out results.json | Raw evidence for Warm redeploy p95, slowest of 3 runs |
| Failed requests during a 2 replica rolling deploy 0 of 4000 in all 3 runs, p99 19.6, 25.4 and 20.8 ms, max up to 319.5 ms. | 0 failed of 4000 | pass | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | post v0.8.3, build 1403b01 | levelrail-cli bench chaos run rollout-under-traffic --repeat 2 --out results.json (100 req/s open loop, 40 s) | Raw evidence for Failed requests during a 2 replica rolling deploy |
| Failed requests while the Docker API was unreachable for 6 s mid-deploy The deploy resumed and succeeded and the old container kept serving. The first run found a real bug (185 of 1200 failed), fixed before this run. The daemon itself was not stopped. | 0 failed of 1200 | pass | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | not recorded, build not recorded | levelrail-cli bench chaos run daemon-unreachable (build with -tags chaoslive) | Raw evidence for Failed requests while the Docker API was unreachable for 6 s mid-deploy |
| Deploy to first HTTP 200, image cached Includes a fresh internal certificate on first request; one run. | 12.5 s | pass | Oracle Cloud ARM, Ampere A1 VMaarch64, 1 vCPU, 5.8 GB, Ubuntu 24.04 | v1.5.3, build not recorded | apps create to first HTTP 200 through the ingress, nginx image already on the host | Raw evidence for Deploy to first HTTP 200, image cached |
| Deploy to first HTTP 200, cold image pull Includes a fresh internal certificate on first request; one run. | 15.4 s | pass | Oracle Cloud ARM, Ampere A1 VMaarch64, 1 vCPU, 5.8 GB, Ubuntu 24.04 | v1.5.3, build not recorded | apps create to first HTTP 200 through the ingress, httpd image not on the host | Raw evidence for Deploy to first HTTP 200, cold image pull |
| Redeploy command to new container running Zero failed requests in the window; one run. | 4.6 s | pass | Oracle Cloud ARM, Ampere A1 VMaarch64, 1 vCPU, 5.8 GB, Ubuntu 24.04 | v1.5.3, build not recorded | apps deploy of a new image tag while 200 rps ran | Raw evidence for Redeploy command to new container running |
Rollback and recovery
What happens to requests when a container, a deploy or the control plane is interrupted.
| Metric | Result | Status | Host | Run | Method | Evidence |
|---|---|---|---|---|---|---|
| Time to healthy after SIGKILL of the only container, slower run of 2 142 and 147 of 1200 requests failed, healthy after 3550 and 3675 ms. Restarted in place, p95 about 2 s because requests queue during the outage. | 3,675 ms | slower than hoped | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | post v0.8.3, build 58233c7 | levelrail-cli bench chaos run kill-container --repeat 2 --out results.json (40 req/s open loop) | Raw evidence for Time to healthy after SIGKILL of the only container, slower run of 2 |
| Second replica running again after SIGKILL of 1 of 2, slowest of 3 0 of 3000 requests failed in all 3 runs. Restore read 426, 769 and 515 ms, polled at 50 ms. | 769 ms | pass | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | post v0.8.3, build fc525fa | levelrail-cli bench chaos run replica-failover --repeat 3 --out results.json (100 req/s open loop, 30 s) | Raw evidence for Second replica running again after SIGKILL of 1 of 2, slowest of 3 |
| Replica restore time, first failover run (harness defect) The run failed its own assertion: the replacement was visible before the 200 ms poll saw the dip, so restore was reported as negative. 0 of 3000 requests failed. The poll is now 50 ms and the 3 runs above are after the fix. | 0 ms | fail | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | post v0.8.3, build 1403b01 | levelrail-cli bench chaos run replica-failover --repeat 2 --out results.json (200 ms poll) | Raw evidence for Replica restore time, first failover run (harness defect) |
| Ingress outage on an in-process control plane restart, slower run of 2 8 and 3 of 1200 requests failed, outages 200 and 75 ms. The app container was not touched. This is not a systemd restart, so it is not the figure for systemctl restart. | 200 ms | pass | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | post v0.8.3, build 58233c7 | levelrail-cli bench chaos run control-plane-restart --repeat 2 --out results.json | Raw evidence for Ingress outage on an in-process control plane restart, slower run of 2 |
| Failed requests when a deploy was OOM killed during readiness The deploy failed as OOMKilledDuringReadiness and version 1 kept serving. | 0 failed of 1200 | pass | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Docker Desktop 28.3.0 (7 GiB visible to Docker) | not recorded, build not recorded | levelrail-cli bench chaos run oom-during-readiness (build with -tags chaoslive) | Raw evidence for Failed requests when a deploy was OOM killed during readiness |
| Rollback until previous release serves Command itself returned in 0.29 s; a first attempt was invalid and rerun. | 11.8 s | pass | Oracle Cloud ARM, Ampere A1 VMaarch64, 1 vCPU, 5.8 GB, Ubuntu 24.04 | v1.5.3, build not recorded | levelrail-cli rollback on an app, polled through the ingress | Raw evidence for Rollback until previous release serves |
| Reboot to local service recovery About 30 s from kernel boot; external ports stayed unreachable because the host firewall rules were not saved (host config). | 56 s | slower than hoped | Oracle Cloud ARM, Ampere A1 VMaarch64, 1 vCPU, 5.8 GB, Ubuntu 24.04 | v1.5.3, build not recorded | systemctl reboot, containers, control plane, ingress routes and Postgres checked locally | Raw evidence for Reboot to local service recovery |
Upgrade
Timing and outcome of upgrading a running install.
| Metric | Result | Status | Host | Run | Method | Evidence |
|---|---|---|---|---|---|---|
| Documented upgrade path, levelrail-cli upgrade --apply Refused: no release repository configured and no --repo flag on this path. | 0 runs completed | fail | Oracle Cloud ARM, Ampere A1 VMaarch64, 1 vCPU, 5.8 GB, Ubuntu 24.04 | v1.5.3, build not recorded | levelrail-cli upgrade --apply --version v1.5.3 on v0.9.5 | Raw evidence for Documented upgrade path, levelrail-cli upgrade --apply |
| Self-upgrade v0.9.5 to v1.5.3 10 of 10 steps ok, signature verified, table md5 unchanged; one host, one run. | 50 s | pass | Oracle Cloud ARM, Ampere A1 VMaarch64, 1 vCPU, 5.8 GB, Ubuntu 24.04 | v1.5.3, build not recorded | levelrail self-upgrade --to v1.5.3 --repo glincker/levelrail --yes | Raw evidence for Self-upgrade v0.9.5 to v1.5.3 |
| Ingress and readyz gap during self-upgrade The ingress lives in the control plane, so the app route dropped while its container kept running. | 8 s | slower than hoped | Oracle Cloud ARM, Ampere A1 VMaarch64, 1 vCPU, 5.8 GB, Ubuntu 24.04 | v1.5.3, build not recorded | 0.5 s probe of an app via the ingress and /readyz during the upgrade | Raw evidence for Ingress and readyz gap during self-upgrade |
Backups
Backup and restore timing against real data.
| Metric | Result | Status | Host | Run | Method | Evidence |
|---|---|---|---|---|---|---|
| Postgres backup to a local S3 target 38276 bytes, sealed; local S3 target, not a cloud bucket. | 3 s | pass | Oracle Cloud ARM, Ampere A1 VMaarch64, 1 vCPU, 5.8 GB, Ubuntu 24.04 | v1.5.3, build not recorded | backups trigger on a managed Postgres 16 database, S3-compatible server on the same host | Raw evidence for Postgres backup to a local S3 target |
| Postgres restore to a verified state 1000 rows back, md5 matches the original; managed Postgres only, MySQL and PITR not run. | 2.6 s | pass | Oracle Cloud ARM, Ampere A1 VMaarch64, 1 vCPU, 5.8 GB, Ubuntu 24.04 | v1.5.3, build not recorded | backups restore --latest --confirm after deleting 500 rows | Raw evidence for Postgres restore to a verified state |
Control plane at scale
Store and API latency with many apps. These are database and API figures, not end to end deploy figures.
| Metric | Result | Status | Host | Run | Method | Evidence |
|---|---|---|---|---|---|---|
| List apps, page of 50, p95 at 1000 apps Store calls in process, 200 samples. Excludes HTTP, JSON encoding and Docker, so it is a floor, not an end to end figure. | 0.62 ms | pass | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, Go 1.26.9 | not recorded, build not recorded | LEVELRAIL_SCALE_BENCH=1 go test ./internal/store -run TestScaleSeeded -v | Raw evidence for List apps, page of 50, p95 at 1000 apps |
| Unified search p95 at 1000 apps, worst of 3 runs Target under 50 ms; runs read 17 to 21 ms through the full HTTP stack with authorization. | 21 ms | pass | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, SQLite on local SSD | not recorded, build not recorded | go test ./internal/api -run TestSearch_ThousandAppsP95 | Raw evidence for Unified search p95 at 1000 apps, worst of 3 runs |
| Scoped search p95 at 5000 apps, 50 deny policies Just under the 50 ms budget, with other builds running (load average near 30). The first run at 1000 apps missed the budget (50 ms) until policy parsing was cached. | 49 ms | slower than hoped | Local laptop, Apple M4 Maxarm64, 14 vCPU, 36 GB, macOS 27.2, SQLite on local SSD | not recorded, build not recorded | go test ./internal/api -run TestSearch_ScopedCallerP95 | Raw evidence for Scoped search p95 at 5000 apps, 50 deny policies |
How we measure
- Open loop. Request
iis scheduled ati / rateseconds and sent whether or not earlier requests returned. A closed loop generator slows down when the server stalls and hides the stall, so rows run closed loop say so. - Intended send time. Latency is counted from when a request was due, not when it left the client, so a stall shows up as queueing delay.
- Nearest-rank percentiles. Percentiles are taken over every request, failures included.
- Generator on a separate machine. A server row needs a load generator on a second machine. A run where the client was the bottleneck (
client_saturatedabove zero) is invalid and is rerun. Laptop rows share one machine and are labelled that way.
The full method, the hardware disclosure table and the scenario list are in Load and failover benchmarks.
Reproduce it yourself
From the benchmark runbook. You provision the machines by hand, and destroying them afterwards is on you.
-
Build the CLI with the chaos scenarios and run them on a machine with a local Docker daemon.
go build -tags chaoslive -o levelrail-cli ./cmd/levelrail-cli -
Run every scenario three times and write the structured file and a Markdown table.
BENCH_NOISY_LOAD_RATIO=1 ./levelrail-cli bench chaos run all --repeat 3 --out results.json --md-out results.md --hardware "short note on the machine" -
For a server run, build the fixture app on the server first.
docker build -t bench-app test/fixtures/bench-app -
On a second machine in the same region, set the target. The values here are the runbook placeholders, replace them with yours.
export BENCH_HOST=203.0.113.10 BENCH_USER=root BENCH_INSTALL_URL=https://levelrail.example.com BENCH_TOKEN=... BENCH_APP_URL=https://bench.example.com/ BENCH_REPLICA_URL=http://REPLICA_IP:8080/ -
Check the plan without sending load.
scripts/bench/run-vps.sh --dry-run -
Run steady load, replica kill, node drain and control plane restart.
scripts/bench/run-vps.sh
The script writes docs/benchmark-results/<date>-vps-*.json and a Markdown fragment. Check that saturated is 0, the hardware line is complete and the date is today before comparing. Pull requests with a new run are welcome in the repository.
Not yet measured
These have no dated, hardware-labelled row, so no limit is claimed for them.
- Throughput, latency and failover through the ingress on a Linux server with a separate load generator.
- Ingress overhead against a direct path, node drain, zone loss and a control plane restart under systemd socket activation.
- Upgrade and backup timings on a server. No dated, hardware-labelled row exists for them yet, so none is shown above.
- Statistical confidence: most rows are one to three runs with no interval.
See also what is verified.