Add GPU Lab configs, docs, and runbooks for VM303.
Mirror gpu-lab compose files and model-manager catalog from GPU-Dev; document Dockhand mapping and SGLang cutover target.
This commit is contained in:
@@ -0,0 +1,12 @@
|
||||
# Dockhand — GPU Lab stacks
|
||||
|
||||
| Dockhand env | ID | Docker host | Stacks |
|
||||
|--------------|-----|-------------|--------|
|
||||
| GPU-Dev | 8 | 10.0.20.106:2375 | gpu-lab-vllm-legacy, gpu-lab-vllm-modern, gpu-lab-sglang |
|
||||
| Spark+Trino / Lakehouse | 9 | 10.0.21.50 | lakehouse-spark-trino (Trino, Spark, kafka-connect, s3-kafka-consumer) |
|
||||
| Airflow | 10 | 10.0.21.55 | airflow |
|
||||
|
||||
- Dockhand UI: http://atc-docker01.dell-atc.lan:8082/
|
||||
- API stacks: `GET /api/stacks?env=8`
|
||||
|
||||
Only one GPU inference stack should be **running** at a time; others appear as stopped compose projects.
|
||||
@@ -0,0 +1,30 @@
|
||||
# GPU Lab Landscape
|
||||
|
||||
## Nodes
|
||||
|
||||
| Host | IP | Role |
|
||||
|------|-----|------|
|
||||
| VM303 GPU-Dev | 10.0.20.106 | GPU Lab UI (:9000), inference (:8010), gateway (:8001) |
|
||||
| atc-docker01 | 10.0.21.45 | Homepage, Dockhand (:8082) |
|
||||
| Bart-GPU | — | Reference node (documented in model-manager) |
|
||||
| Mo-GPU | — | Reference node |
|
||||
|
||||
## VM303 specs
|
||||
|
||||
See `config/model-manager/vm-specs.json`: 4× Tesla V100 32GB, Docker with NVIDIA runtime.
|
||||
|
||||
## Inference runtimes
|
||||
|
||||
Only one runtime is active at a time on port **8010**:
|
||||
|
||||
| Runtime ID | Engine | Compose file |
|
||||
|------------|--------|--------------|
|
||||
| `vllm-legacy-040` | vLLM 0.4.x | `compose.vllm-legacy.yml` |
|
||||
| `vllm-modern-080` | vLLM 0.8.x | `compose.vllm-modern.yml` |
|
||||
| `sglang-modern` | SGLang 0.5.12 | `compose.sglang.yml` |
|
||||
|
||||
Models live under `/root/models/` on VM303 (not in git).
|
||||
|
||||
## Docker data
|
||||
|
||||
Images stored under `/root/docker-data` (avoid small `/var` partition).
|
||||
@@ -0,0 +1,20 @@
|
||||
# Model catalog (VM303)
|
||||
|
||||
| ID | Name | Runtime | Backend | V100 |
|
||||
|---|---|---|---|---|
|
||||
| `llama3-70b-gptq` | Llama 3 70B GPTQ | vllm-legacy-040 | vllm | yes |
|
||||
| `qwen25-coder-32b-gptq` | Qwen 2.5 Coder 32B GPTQ | vllm-legacy-040 | vllm | yes |
|
||||
| `qwen25-32b-gptq` | Qwen 2.5 32B GPTQ | vllm-legacy-040 | vllm | yes |
|
||||
| `deepseek-coder-33b-gptq` | DeepSeek Coder 33B GPTQ | vllm-legacy-040 | vllm | yes |
|
||||
| `qwen25-coder-7b` | Qwen 2.5 Coder 7B | vllm-legacy-040 | vllm | yes |
|
||||
| `deepseek-coder-6.7b` | DeepSeek Coder 6.7B | vllm-legacy-040 | vllm | yes |
|
||||
| `deepseek-coder-7b` | DeepSeek Coder 7B v1.5 | vllm-legacy-040 | vllm | yes |
|
||||
| `starcoder2-15b` | StarCoder2 15B | vllm-legacy-040 | vllm | yes |
|
||||
| `granite-code-8b` | IBM Granite Code 8B | vllm-legacy-040 | vllm | yes |
|
||||
| `yi-coder-9b` | Yi-Coder 9B | vllm-legacy-040 | vllm | yes |
|
||||
| `deepseek-coder-v2-lite` | DeepSeek Coder V2 Lite | vllm-modern-080 | vllm | yes |
|
||||
| `qwen3-coder-next` | Qwen3 Coder Next | vllm-modern-080 | vllm | yes |
|
||||
| `qwen3-coder-30b` | Qwen3 Coder 30B A3B | vllm-modern-080 | vllm | yes |
|
||||
| `qwen36-27b` | Qwen 3.6 27B | sglang-modern | sglang | yes |
|
||||
| `qwen36-27b-awq` | Qwen 3.6 27B AWQ | sglang-modern | sglang | yes |
|
||||
| `codellama-34b` | Code Llama 34B | vllm-legacy-040 | vllm | yes |
|
||||
@@ -0,0 +1,61 @@
|
||||
# GPU Lab Runbook — VM303 (atc-gpu-dev)
|
||||
|
||||
## Runtimes (Docker)
|
||||
|
||||
| Runtime ID | Compose | Image |
|
||||
|---|---|---|
|
||||
| `vllm-legacy-040` | `compose.vllm-legacy.yml` | `vllm/vllm-openai:v0.4.0.post1` |
|
||||
| `vllm-modern-080` | `compose.vllm-modern.yml` | `vllm/vllm-openai:v0.8.5.post1` |
|
||||
| `sglang-modern` | `compose.sglang.yml` | `lmsysorg/sglang:v0.5.12-cu129-runtime` |
|
||||
|
||||
Only **one** inference container runs at a time (port **8010**).
|
||||
|
||||
## Activate (UI)
|
||||
|
||||
GPU Lab → pick model → **Activate**. Writes `/root/gpu-lab/active/current.env`, stops all gpu-lab containers, starts the matching compose stack.
|
||||
|
||||
## Manual activate
|
||||
|
||||
```bash
|
||||
cd /root/gpu-lab
|
||||
# edit active/current.env (MODEL_PATH, TP_SIZE, VLLM_EXTRA_ARGS, …)
|
||||
docker compose -f compose.vllm-legacy.yml down
|
||||
docker compose -f compose.vllm-legacy.yml up -d
|
||||
curl -s http://127.0.0.1:8010/v1/models
|
||||
```
|
||||
|
||||
## Rollback to systemd vLLM
|
||||
|
||||
```bash
|
||||
cd /root/gpu-lab
|
||||
docker compose -f compose.vllm-legacy.yml down
|
||||
docker compose -f compose.vllm-modern.yml down
|
||||
docker compose -f compose.sglang.yml down
|
||||
systemctl start vllm.service
|
||||
systemctl restart vllm-gateway.service litellm-proxy.service
|
||||
```
|
||||
|
||||
## Logs
|
||||
|
||||
```bash
|
||||
docker logs -f gpu-lab-vllm-legacy
|
||||
docker logs -f gpu-lab-vllm-modern
|
||||
docker logs -f gpu-lab-sglang
|
||||
journalctl -u model-manager -f
|
||||
```
|
||||
|
||||
## SGLang on V100
|
||||
|
||||
Always use `--disable-flashinfer` and `--disable-flashinfer-sampling` (already in compose.sglang.yml).
|
||||
|
||||
## Dockhand
|
||||
|
||||
- Environment **GPU-Dev** → `10.0.20.106:2375`
|
||||
- Stacks: `gpu-lab-vllm-legacy`, `gpu-lab-vllm-modern`, `gpu-lab-sglang` (compose in Dockhand data volume)
|
||||
- Running workload appears as stack **`gpu-lab`** (compose project on VM303)
|
||||
- API: `GET /api/stacks?env=8` (not `environmentId`)
|
||||
- UI: http://atc-docker01.dell-atc.lan:8082/
|
||||
|
||||
## Docker data
|
||||
|
||||
Images stored under `/root/docker-data` (not `/var` — small partition).
|
||||
@@ -0,0 +1,47 @@
|
||||
# GPU Lab Runbook — VM303 (atc-gpu-dev)
|
||||
|
||||
See also `config/gpu-lab/RUNBOOK.md` (synced from VM303).
|
||||
|
||||
## Activate via UI
|
||||
|
||||
GPU Lab → select model → **Activate**. Writes `/root/gpu-lab/active/current.env`, stops other gpu-lab containers, starts matching compose stack.
|
||||
|
||||
## Manual activate (SGLang + Qwen 3.6 27B AWQ)
|
||||
|
||||
```bash
|
||||
cd /root/gpu-lab
|
||||
cat > active/current.env << 'EOF'
|
||||
MODEL_PATH=/models/qwen36-27b-awq
|
||||
SERVED_NAME=qwen3.6-27b-awq
|
||||
TP_SIZE=4
|
||||
GPU_MEM_UTIL=0.9
|
||||
MAX_MODEL_LEN=8192
|
||||
EOF
|
||||
docker compose -f compose.vllm-legacy.yml down
|
||||
docker compose -f compose.vllm-modern.yml down
|
||||
docker compose -f compose.sglang.yml up -d
|
||||
curl -s http://127.0.0.1:8010/v1/models
|
||||
systemctl restart vllm-gateway.service
|
||||
```
|
||||
|
||||
## Rollback to systemd vLLM
|
||||
|
||||
```bash
|
||||
cd /root/gpu-lab
|
||||
docker compose -f compose.vllm-legacy.yml down
|
||||
docker compose -f compose.vllm-modern.yml down
|
||||
docker compose -f compose.sglang.yml down
|
||||
systemctl start vllm.service
|
||||
systemctl restart vllm-gateway.service litellm-proxy.service
|
||||
```
|
||||
|
||||
## Logs
|
||||
|
||||
```bash
|
||||
docker logs -f gpu-lab-sglang
|
||||
journalctl -u model-manager -f
|
||||
```
|
||||
|
||||
## SGLang on V100
|
||||
|
||||
Always use `--disable-flashinfer` and `--disable-flashinfer-sampling` (in `compose.sglang.yml`).
|
||||
Reference in New Issue
Block a user