62 lines
1.8 KiB
Markdown
62 lines
1.8 KiB
Markdown
|
# GPU Lab Runbook — VM303 (atc-gpu-dev)
|
||
|
|
|
||
|
|
## Runtimes (Docker)
|
||
|
|
|
||
|
|
| Runtime ID | Compose | Image |
|
||
|
|
|---|---|---|
|
||
|
|
| `vllm-legacy-040` | `compose.vllm-legacy.yml` | `vllm/vllm-openai:v0.4.0.post1` |
|
||
|
|
| `vllm-modern-080` | `compose.vllm-modern.yml` | `vllm/vllm-openai:v0.8.5.post1` |
|
||
|
|
| `sglang-modern` | `compose.sglang.yml` | `lmsysorg/sglang:v0.5.12-cu129-runtime` |
|
||
|
|
|
||
|
|
Only **one** inference container runs at a time (port **8010**).
|
||
|
|
|
||
|
|
## Activate (UI)
|
||
|
|
|
||
|
|
GPU Lab → pick model → **Activate**. Writes `/root/gpu-lab/active/current.env`, stops all gpu-lab containers, starts the matching compose stack.
|
||
|
|
|
||
|
|
## Manual activate
|
||
|
|
|
||
|
|
```bash
|
||
|
|
cd /root/gpu-lab
|
||
|
|
# edit active/current.env (MODEL_PATH, TP_SIZE, VLLM_EXTRA_ARGS, …)
|
||
|
|
docker compose -f compose.vllm-legacy.yml down
|
||
|
|
docker compose -f compose.vllm-legacy.yml up -d
|
||
|
|
curl -s http://127.0.0.1:8010/v1/models
|
||
|
|
```
|
||
|
|
|
||
|
|
## Rollback to systemd vLLM
|
||
|
|
|
||
|
|
```bash
|
||
|
|
cd /root/gpu-lab
|
||
|
|
docker compose -f compose.vllm-legacy.yml down
|
||
|
|
docker compose -f compose.vllm-modern.yml down
|
||
|
|
docker compose -f compose.sglang.yml down
|
||
|
|
systemctl start vllm.service
|
||
|
|
systemctl restart vllm-gateway.service litellm-proxy.service
|
||
|
|
```
|
||
|
|
|
||
|
|
## Logs
|
||
|
|
|
||
|
|
```bash
|
||
|
|
docker logs -f gpu-lab-vllm-legacy
|
||
|
|
docker logs -f gpu-lab-vllm-modern
|
||
|
|
docker logs -f gpu-lab-sglang
|
||
|
|
journalctl -u model-manager -f
|
||
|
|
```
|
||
|
|
|
||
|
|
## SGLang on V100
|
||
|
|
|
||
|
|
Always use `--disable-flashinfer` and `--disable-flashinfer-sampling` (already in compose.sglang.yml).
|
||
|
|
|
||
|
|
## Dockhand
|
||
|
|
|
||
|
|
- Environment **GPU-Dev** → `10.0.20.106:2375`
|
||
|
|
- Stacks: `gpu-lab-vllm-legacy`, `gpu-lab-vllm-modern`, `gpu-lab-sglang` (compose in Dockhand data volume)
|
||
|
|
- Running workload appears as stack **`gpu-lab`** (compose project on VM303)
|
||
|
|
- API: `GET /api/stacks?env=8` (not `environmentId`)
|
||
|
|
- UI: http://atc-docker01.dell-atc.lan:8082/
|
||
|
|
|
||
|
|
## Docker data
|
||
|
|
|
||
|
|
Images stored under `/root/docker-data` (not `/var` — small partition).
|