ATC/atc-GPU
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add GPU Lab configs, docs, and runbooks for VM303.

Browse Source 
Mirror gpu-lab compose files and model-manager catalog from GPU-Dev; document Dockhand mapping and SGLang cutover target.
This commit is contained in:
mo
2026-06-04 02:24:57 +02:00
parent 26aec9ed04
commit c154c94c6d
14 changed files with 930 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+34 -2
View File
@@ -1,3 +1,35 @@
# atc-GPU
# atc-GPU — GPU Lab Infrastructure
GPU Lab infrastructure — VM303 model-manager, gpu-lab Docker stacks, Dockhand
Infrastructure-as-code and documentation for the ATC GPU Lab environment on **VM303 (GPU-Dev)**.
## Quick links
| Service | URL |
|---------|-----|
| GPU Lab UI | http://10.0.20.106:9000/ |
| Inference API (active runtime) | http://10.0.20.106:8010/v1/models |
| OpenAI gateway | http://10.0.20.106:8001/v1/models |
| Dockhand (GPU-Dev env) | http://atc-docker01.dell-atc.lan:8082/ |
## Repository layout
```
atc-GPU/
├── README.md
├── docs/ # Landscape, models, runbooks, Dockhand mapping
├── config/ # gpu-lab compose + model-manager catalog (no secrets)
├── compose/ # Dockhand-safe compose variants
└── scripts/ # Helper scripts (activate-model)
```
## Active target (2026-06)
- **Runtime:** SGLang (`sglang-modern`) via `compose.sglang.yml`
- **Model:** Qwen 3.6 27B AWQ (`qwen36-27b-awq` → served as `qwen3.6-27b-awq`)
- **Previous:** Llama3-70b-GPTQ on vLLM legacy (to be decommissioned after cutover)
See [docs/runbook-vm303.md](docs/runbook-vm303.md) for activate/rollback steps.
## Related repos
- [mo/Lakehouse](http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse) — Spark, Trino, Debezium, Airflow, S3 pipeline
config/gpu-lab/RUNBOOK.md
+61
View File
@@ -0,0 +1,61 @@
# GPU Lab Runbook — VM303 (atc-gpu-dev)
## Runtimes (Docker)
| Runtime ID | Compose | Image |
|---|---|---|
| `vllm-legacy-040` | `compose.vllm-legacy.yml` | `vllm/vllm-openai:v0.4.0.post1` |
| `vllm-modern-080` | `compose.vllm-modern.yml` | `vllm/vllm-openai:v0.8.5.post1` |
| `sglang-modern` | `compose.sglang.yml` | `lmsysorg/sglang:v0.5.12-cu129-runtime` |
Only **one** inference container runs at a time (port **8010**).
## Activate (UI)
GPU Lab → pick model → **Activate**. Writes `/root/gpu-lab/active/current.env`, stops all gpu-lab containers, starts the matching compose stack.
## Manual activate
```bash
cd /root/gpu-lab
# edit active/current.env (MODEL_PATH, TP_SIZE, VLLM_EXTRA_ARGS, …)
docker compose -f compose.vllm-legacy.yml down
docker compose -f compose.vllm-legacy.yml up -d
curl -s http://127.0.0.1:8010/v1/models
```
## Rollback to systemd vLLM
```bash
cd /root/gpu-lab
docker compose -f compose.vllm-legacy.yml down
docker compose -f compose.vllm-modern.yml down
docker compose -f compose.sglang.yml down
systemctl start vllm.service
systemctl restart vllm-gateway.service litellm-proxy.service
```
## Logs
```bash
docker logs -f gpu-lab-vllm-legacy
docker logs -f gpu-lab-vllm-modern
docker logs -f gpu-lab-sglang
journalctl -u model-manager -f
```
## SGLang on V100
Always use `--disable-flashinfer` and `--disable-flashinfer-sampling` (already in compose.sglang.yml).
## Dockhand
- Environment **GPU-Dev** → `10.0.20.106:2375`
- Stacks: `gpu-lab-vllm-legacy`, `gpu-lab-vllm-modern`, `gpu-lab-sglang` (compose in Dockhand data volume)
- Running workload appears as stack **`gpu-lab`** (compose project on VM303)
- API: `GET /api/stacks?env=8` (not `environmentId`)
- UI: http://atc-docker01.dell-atc.lan:8082/
## Docker data
Images stored under `/root/docker-data` (not `/var` — small partition).
config/gpu-lab/compose.sglang.yml
+38
View File
@@ -0,0 +1,38 @@
name: gpu-lab-sglang
services:
sglang:
image: lmsysorg/sglang:v0.5.12-cu129-runtime
container_name: gpu-lab-sglang
network_mode: host
ipc: host
shm_size: "32gb"
env_file:
- active/current.env
volumes:
- /root/models:/models:ro
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
entrypoint: ["python3", "-m", "sglang.launch_server"]
command:
- --model-path
- ${MODEL_PATH}
- --served-model-name
- ${SERVED_NAME}
- --host
- "0.0.0.0"
- --port
- "8010"
- --tp-size
- ${TP_SIZE}
- --mem-fraction-static
- ${GPU_MEM_UTIL}
- --context-length
- ${MAX_MODEL_LEN}
- --disable-flashinfer
- --disable-flashinfer-sampling
restart: "no"
config/gpu-lab/compose.vllm-legacy.yml
+33
View File
@@ -0,0 +1,33 @@
name: gpu-lab-vllm-legacy
services:
vllm-legacy:
image: vllm/vllm-openai:v0.4.0.post1
container_name: gpu-lab-vllm-legacy
network_mode: host
ipc: host
shm_size: "32gb"
env_file:
- active/current.env
volumes:
- /root/models:/models:ro
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
entrypoint: ["/bin/sh", "-c"]
command:
- >
exec python3 -m vllm.entrypoints.openai.api_server
--model "$$MODEL_PATH"
--served-model-name "$$SERVED_NAME"
--host 0.0.0.0 --port 8010
--tensor-parallel-size "$$TP_SIZE"
--dtype "$$DTYPE"
--max-model-len "$$MAX_MODEL_LEN"
--gpu-memory-utilization "$$GPU_MEM_UTIL"
--max-num-seqs "$$MAX_NUM_SEQS"
$$VLLM_EXTRA_ARGS
restart: "no"
config/gpu-lab/compose.vllm-modern.yml
+34
View File
@@ -0,0 +1,34 @@
name: gpu-lab-vllm-modern
services:
vllm-modern:
image: vllm/vllm-openai:v0.8.5.post1
container_name: gpu-lab-vllm-modern
network_mode: host
ipc: host
shm_size: "32gb"
env_file:
- active/current.env
volumes:
- /root/models:/models:ro
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
entrypoint: ["/bin/sh", "-c"]
command:
- >
exec python3 -m vllm.entrypoints.openai.api_server
--model "$$MODEL_PATH"
--served-model-name "$$SERVED_NAME"
--host 0.0.0.0 --port 8010
--tensor-parallel-size "$$TP_SIZE"
--dtype "$$DTYPE"
--max-model-len "$$MAX_MODEL_LEN"
--gpu-memory-utilization "$$GPU_MEM_UTIL"
--max-num-seqs "$$MAX_NUM_SEQS"
--trust-remote-code --enforce-eager
$$VLLM_EXTRA_ARGS
restart: "no"
config/gpu-lab/runtimes.json
+24
View File
@@ -0,0 +1,24 @@
{
"vllm-legacy-040": {
"engine": "vllm",
"compose_file": "compose.vllm-legacy.yml",
"image": "vllm/vllm-openai:v0.4.0.post1",
"v100_ok": true,
"label": "vLLM 0.4"
},
"vllm-modern-080": {
"engine": "vllm",
"compose_file": "compose.vllm-modern.yml",
"image": "vllm/vllm-openai:v0.8.5.post1",
"v100_ok": true,
"label": "vLLM 0.8"
},
"sglang-modern": {
"engine": "sglang",
"compose_file": "compose.sglang.yml",
"image": "lmsysorg/sglang:v0.5.12-cu129-runtime",
"v100_ok": true,
"label": "SGLang",
"extra_args": ["--disable-flashinfer", "--disable-flashinfer-sampling"]
}
}
config/model-manager/catalog.json
+515
View File
@@ -0,0 +1,515 @@
{
"models": [
{
"id": "llama3-70b-gptq",
"name": "Llama 3 70B GPTQ",
"description": "Meta Llama 3 open weights. Strong general coder (lab default). 4\u00d7 V100.",
"tags": [
"coding",
"opensource",
"v100_ready"
],
"license": "Llama 3 Community License",
"repo_id": null,
"local_path": "/root/models/llama3-70b-quantized",
"served_name": "llama3-70b-gptq",
"size_gb_estimate": 38,
"vllm_min_version": "0.4.0",
"vllm": {
"tensor_parallel_size": 4,
"quantization": "gptq",
"dtype": "float16",
"max_model_len": 8192,
"gpu_memory_utilization": 0.88,
"max_num_seqs": 4,
"extra_args": [
"--disable-custom-all-reduce",
"--enforce-eager"
]
},
"runtime_id": "vllm-legacy-040",
"inference_backend": "vllm",
"v100_activate": true,
"v100_block_reason": null
},
{
"id": "qwen25-coder-32b-gptq",
"name": "Qwen 2.5 Coder 32B GPTQ",
"description": "Alibaba Qwen open weights. Top open-source code LLM (GPT-4o class). 4\u00d7 V100.",
"tags": [
"coding",
"opensource",
"qwen",
"v100_ready"
],
"license": "Apache-2.0",
"repo_id": "Qwen/Qwen2.5-Coder-32B-Instruct-GPTQ-Int4",
"local_path": "/root/models/qwen25-coder-32b-gptq",
"served_name": "qwen2.5-coder-32b-gptq",
"size_gb_estimate": 20,
"vllm_min_version": "0.4.0",
"vllm": {
"tensor_parallel_size": 4,
"quantization": "gptq",
"dtype": "float16",
"max_model_len": 8192,
"gpu_memory_utilization": 0.9,
"max_num_seqs": 4,
"extra_args": [
"--trust-remote-code",
"--enforce-eager"
]
},
"runtime_id": "vllm-legacy-040",
"inference_backend": "vllm",
"v100_activate": true,
"v100_block_reason": null
},
{
"id": "qwen25-32b-gptq",
"name": "Qwen 2.5 32B GPTQ",
"description": "Alibaba Qwen open weights (general instruct, strong coder).",
"tags": [
"coding",
"opensource",
"qwen",
"v100_ready"
],
"license": "Apache-2.0",
"repo_id": "Qwen/Qwen2.5-32B-Instruct-GPTQ-Int4",
"local_path": "/root/models/qwen25-32b-gptq",
"served_name": "qwen2.5-32b-gptq",
"size_gb_estimate": 20,
"vllm_min_version": "0.4.0",
"vllm": {
"tensor_parallel_size": 4,
"quantization": "gptq",
"dtype": "float16",
"max_model_len": 8192,
"gpu_memory_utilization": 0.9,
"max_num_seqs": 4,
"extra_args": [
"--trust-remote-code",
"--enforce-eager"
]
},
"runtime_id": "vllm-legacy-040",
"inference_backend": "vllm",
"v100_activate": true,
"v100_block_reason": null
},
{
"id": "deepseek-coder-33b-gptq",
"name": "DeepSeek Coder 33B GPTQ",
"description": "DeepSeek open weights (TheBloke GPTQ). Top-tier coder, TP=4.",
"tags": [
"coding",
"deepseek",
"opensource",
"v100_ready"
],
"license": "DeepSeek License",
"repo_id": "TheBloke/deepseek-coder-33B-instruct-GPTQ",
"local_path": "/root/models/deepseek-coder-33b-gptq",
"served_name": "deepseek-coder-33b-gptq",
"size_gb_estimate": 18,
"vllm_min_version": "0.4.0",
"vllm": {
"tensor_parallel_size": 4,
"quantization": "gptq",
"dtype": "float16",
"max_model_len": 4096,
"gpu_memory_utilization": 0.88,
"max_num_seqs": 4,
"extra_args": [
"--trust-remote-code",
"--enforce-eager"
]
},
"runtime_id": "vllm-legacy-040",
"inference_backend": "vllm",
"v100_activate": true,
"v100_block_reason": null
},
{
"id": "qwen25-coder-7b",
"name": "Qwen 2.5 Coder 7B",
"description": "Alibaba Qwen open weights. Fast code model, 1 GPU.",
"tags": [
"coding",
"opensource",
"qwen",
"v100_ready"
],
"license": "Apache-2.0",
"repo_id": "Qwen/Qwen2.5-Coder-7B-Instruct",
"local_path": "/root/models/qwen25-coder-7b",
"served_name": "qwen2.5-coder-7b",
"size_gb_estimate": 15,
"vllm_min_version": "0.4.0",
"vllm": {
"tensor_parallel_size": 1,
"dtype": "float16",
"max_model_len": 8192,
"gpu_memory_utilization": 0.9,
"max_num_seqs": 8,
"extra_args": [
"--trust-remote-code",
"--enforce-eager"
]
},
"runtime_id": "vllm-legacy-040",
"inference_backend": "vllm",
"v100_activate": true,
"v100_block_reason": null
},
{
"id": "deepseek-coder-6.7b",
"name": "DeepSeek Coder 6.7B",
"description": "DeepSeek open weights. Fast lightweight coder, 1 GPU.",
"tags": [
"coding",
"deepseek",
"opensource",
"v100_ready"
],
"license": "DeepSeek License",
"repo_id": "deepseek-ai/deepseek-coder-6.7b-instruct",
"local_path": "/root/models/deepseek-coder-6.7b",
"served_name": "deepseek-coder-6.7b",
"size_gb_estimate": 14,
"vllm_min_version": "0.4.0",
"vllm": {
"tensor_parallel_size": 1,
"dtype": "float16",
"max_model_len": 8192,
"gpu_memory_utilization": 0.9,
"max_num_seqs": 8,
"extra_args": [
"--trust-remote-code",
"--enforce-eager"
]
},
"runtime_id": "vllm-legacy-040",
"inference_backend": "vllm",
"v100_activate": true,
"v100_block_reason": null
},
{
"id": "deepseek-coder-7b",
"name": "DeepSeek Coder 7B v1.5",
"description": "DeepSeek open weights. Balanced 7B coder.",
"tags": [
"coding",
"deepseek",
"opensource",
"v100_ready"
],
"license": "DeepSeek License",
"repo_id": "deepseek-ai/deepseek-coder-7b-instruct-v1.5",
"local_path": "/root/models/deepseek-coder-7b",
"served_name": "deepseek-coder-7b",
"size_gb_estimate": 15,
"vllm_min_version": "0.4.0",
"vllm": {
"tensor_parallel_size": 1,
"dtype": "float16",
"max_model_len": 8192,
"gpu_memory_utilization": 0.9,
"max_num_seqs": 8,
"extra_args": [
"--trust-remote-code",
"--enforce-eager"
]
},
"runtime_id": "vllm-legacy-040",
"inference_backend": "vllm",
"v100_activate": true,
"v100_block_reason": null
},
{
"id": "starcoder2-15b",
"name": "StarCoder2 15B",
"description": "BigCode/Hugging Face. Apache-2.0 code specialist, 1\u20132 GPU.",
"tags": [
"coding",
"opensource",
"v100_ready"
],
"license": "Apache-2.0",
"repo_id": "bigcode/starcoder2-15b",
"local_path": "/root/models/starcoder2-15b",
"served_name": "starcoder2-15b",
"size_gb_estimate": 30,
"vllm_min_version": "0.4.0",
"vllm": {
"tensor_parallel_size": 2,
"dtype": "float16",
"max_model_len": 8192,
"gpu_memory_utilization": 0.9,
"max_num_seqs": 4,
"extra_args": [
"--trust-remote-code",
"--enforce-eager"
]
},
"runtime_id": "vllm-legacy-040",
"inference_backend": "vllm",
"v100_activate": true,
"v100_block_reason": null
},
{
"id": "granite-code-8b",
"name": "IBM Granite Code 8B",
"description": "IBM open weights. Enterprise-friendly Apache-2.0 coder.",
"tags": [
"coding",
"opensource",
"v100_ready"
],
"license": "Apache-2.0",
"repo_id": "ibm-granite/granite-3.3-8b-instruct",
"local_path": "/root/models/granite-code-8b",
"served_name": "granite-code-8b",
"size_gb_estimate": 16,
"vllm_min_version": "0.4.0",
"vllm": {
"tensor_parallel_size": 1,
"dtype": "float16",
"max_model_len": 8192,
"gpu_memory_utilization": 0.9,
"max_num_seqs": 8,
"extra_args": [
"--trust-remote-code",
"--enforce-eager"
]
},
"runtime_id": "vllm-legacy-040",
"inference_backend": "vllm",
"v100_activate": true,
"v100_block_reason": null
},
{
"id": "yi-coder-9b",
"name": "Yi-Coder 9B",
"description": "01.AI open weights. Strong 9B code model, 1 GPU.",
"tags": [
"coding",
"opensource",
"v100_ready"
],
"license": "Apache-2.0",
"repo_id": "01-ai/Yi-Coder-9B-Chat",
"local_path": "/root/models/yi-coder-9b",
"served_name": "yi-coder-9b",
"size_gb_estimate": 18,
"vllm_min_version": "0.4.0",
"vllm": {
"tensor_parallel_size": 1,
"dtype": "float16",
"max_model_len": 8192,
"gpu_memory_utilization": 0.9,
"max_num_seqs": 8,
"extra_args": [
"--trust-remote-code",
"--enforce-eager"
]
},
"runtime_id": "vllm-legacy-040",
"inference_backend": "vllm",
"v100_activate": true,
"v100_block_reason": null
},
{
"id": "deepseek-coder-v2-lite",
"name": "DeepSeek Coder V2 Lite",
"description": "DeepSeek MoE coder (16B/2.4B active). vLLM 0.8 on V100.",
"tags": [
"coding",
"deepseek",
"opensource",
"v100_ready"
],
"license": "DeepSeek License",
"repo_id": "deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct",
"local_path": "/root/models/deepseek-coder-v2-lite",
"served_name": "deepseek-coder-v2-lite",
"size_gb_estimate": 32,
"vllm_min_version": "0.6.0",
"vllm": {
"tensor_parallel_size": 2,
"dtype": "bfloat16",
"max_model_len": 8192,
"gpu_memory_utilization": 0.9,
"max_num_seqs": 4,
"extra_args": [
"--trust-remote-code",
"--enforce-eager"
]
},
"runtime_id": "vllm-modern-080",
"inference_backend": "vllm",
"v100_activate": true,
"v100_block_reason": null
},
{
"id": "qwen3-coder-next",
"name": "Qwen3 Coder Next",
"description": "Alibaba agentic coder (80B MoE, 3B active). vLLM 0.8 on V100.",
"tags": [
"coding",
"opensource",
"qwen",
"v100_ready"
],
"license": "Apache-2.0",
"repo_id": "Qwen/Qwen3-Coder-Next",
"local_path": "/root/models/qwen3-coder-next",
"served_name": "qwen3-coder-next",
"size_gb_estimate": 45,
"vllm_min_version": "0.8.0",
"vllm": {
"tensor_parallel_size": 4,
"dtype": "bfloat16",
"max_model_len": 8192,
"gpu_memory_utilization": 0.9,
"max_num_seqs": 4,
"extra_args": [
"--trust-remote-code",
"--enforce-eager"
]
},
"runtime_id": "vllm-modern-080",
"inference_backend": "vllm",
"v100_activate": true,
"v100_block_reason": null
},
{
"id": "qwen3-coder-30b",
"name": "Qwen3 Coder 30B A3B",
"description": "Alibaba MoE agentic coder. vLLM 0.8 on V100.",
"tags": [
"coding",
"opensource",
"qwen",
"v100_ready"
],
"license": "Apache-2.0",
"repo_id": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
"local_path": "/root/models/qwen3-coder-30b",
"served_name": "qwen3-coder-30b",
"size_gb_estimate": 60,
"vllm_min_version": "0.8.0",
"vllm": {
"tensor_parallel_size": 4,
"dtype": "bfloat16",
"max_model_len": 8192,
"gpu_memory_utilization": 0.9,
"max_num_seqs": 4,
"extra_args": [
"--trust-remote-code",
"--enforce-eager"
]
},
"runtime_id": "vllm-modern-080",
"inference_backend": "vllm",
"v100_activate": true,
"v100_block_reason": null
},
{
"id": "qwen36-27b",
"name": "Qwen 3.6 27B",
"description": "Qwen open weights (dense). On V100 via SGLang \u2014 prefer AWQ variant.",
"tags": [
"coding",
"opensource",
"qwen",
"v100_via_sglang"
],
"license": "Apache-2.0",
"repo_id": "Qwen/Qwen3.6-27B",
"local_path": "/root/models/qwen36-27b",
"served_name": "qwen3.6-27b",
"size_gb_estimate": 54,
"vllm_min_version": "0.19.0",
"vllm": {
"tensor_parallel_size": 4,
"dtype": "bfloat16",
"max_model_len": 8192,
"gpu_memory_utilization": 0.9,
"max_num_seqs": 4,
"extra_args": [
"--trust-remote-code",
"--enforce-eager"
]
},
"runtime_id": "sglang-modern",
"inference_backend": "sglang",
"v100_activate": true,
"v100_block_reason": null
},
{
"id": "qwen36-27b-awq",
"name": "Qwen 3.6 27B AWQ",
"description": "Qwen open weights (AWQ). Primary Qwen 3.6 path on V100 via SGLang.",
"tags": [
"coding",
"opensource",
"qwen",
"v100_via_sglang"
],
"license": "Apache-2.0",
"repo_id": "QuantTrio/Qwen3.6-27B-AWQ",
"local_path": "/root/models/qwen36-27b-awq",
"served_name": "qwen3.6-27b-awq",
"size_gb_estimate": 16,
"vllm_min_version": "0.19.0",
"vllm": {
"tensor_parallel_size": 4,
"quantization": "awq",
"dtype": "float16",
"max_model_len": 8192,
"gpu_memory_utilization": 0.9,
"max_num_seqs": 8,
"extra_args": [
"--trust-remote-code",
"--enforce-eager"
]
},
"runtime_id": "sglang-modern",
"inference_backend": "sglang",
"v100_activate": true,
"v100_block_reason": null
},
{
"id": "codellama-34b",
"name": "Code Llama 34B",
"description": "Meta open weights (custom license). Classic code LLM \u2014 downloadable via HF.",
"tags": [
"coding",
"opensource",
"v100_ready"
],
"license": "Llama 3 Community License",
"repo_id": "codellama/CodeLlama-34b-Instruct-hf",
"local_path": "/root/models/codellama-34b",
"served_name": "codellama-34b",
"size_gb_estimate": 68,
"vllm_min_version": "0.4.0",
"vllm": {
"tensor_parallel_size": 4,
"dtype": "float16",
"max_model_len": 4096,
"gpu_memory_utilization": 0.9,
"max_num_seqs": 2,
"extra_args": [
"--enforce-eager"
]
},
"runtime_id": "vllm-legacy-040",
"inference_backend": "vllm",
"v100_activate": true,
"v100_block_reason": null
}
]
}
config/model-manager/vm-specs.json
+14
View File
@@ -0,0 +1,14 @@
{
"vmid": 303,
"name": "atc-gpu-dev",
"proxmox_host": "atc-gpu",
"ip": "10.0.20.106",
"cpu_model": "Intel Xeon Gold 6148 @ 2.40GHz",
"vcpu": 32,
"sockets": 2,
"cores_per_socket": 16,
"memory_gib": 128,
"disk_gib": 250,
"gpu": "4× Tesla V100 PCIe 32GB",
"os": "Rocky Linux 9"
}
docs/dockhand.md
+12
View File
@@ -0,0 +1,12 @@
# Dockhand — GPU Lab stacks
| Dockhand env | ID | Docker host | Stacks |
|--------------|-----|-------------|--------|
| GPU-Dev | 8 | 10.0.20.106:2375 | gpu-lab-vllm-legacy, gpu-lab-vllm-modern, gpu-lab-sglang |
| Spark+Trino / Lakehouse | 9 | 10.0.21.50 | lakehouse-spark-trino (Trino, Spark, kafka-connect, s3-kafka-consumer) |
| Airflow | 10 | 10.0.21.55 | airflow |
- Dockhand UI: http://atc-docker01.dell-atc.lan:8082/
- API stacks: `GET /api/stacks?env=8`
Only one GPU inference stack should be **running** at a time; others appear as stopped compose projects.
docs/landscape.md
+30
View File
@@ -0,0 +1,30 @@
# GPU Lab Landscape
## Nodes
| Host | IP | Role |
|------|-----|------|
| VM303 GPU-Dev | 10.0.20.106 | GPU Lab UI (:9000), inference (:8010), gateway (:8001) |
| atc-docker01 | 10.0.21.45 | Homepage, Dockhand (:8082) |
| Bart-GPU | — | Reference node (documented in model-manager) |
| Mo-GPU | — | Reference node |
## VM303 specs
See `config/model-manager/vm-specs.json`: 4× Tesla V100 32GB, Docker with NVIDIA runtime.
## Inference runtimes
Only one runtime is active at a time on port **8010**:
| Runtime ID | Engine | Compose file |
|------------|--------|--------------|
| `vllm-legacy-040` | vLLM 0.4.x | `compose.vllm-legacy.yml` |
| `vllm-modern-080` | vLLM 0.8.x | `compose.vllm-modern.yml` |
| `sglang-modern` | SGLang 0.5.12 | `compose.sglang.yml` |
Models live under `/root/models/` on VM303 (not in git).
## Docker data
Images stored under `/root/docker-data` (avoid small `/var` partition).
docs/models.md
+20
View File
@@ -0,0 +1,20 @@
# Model catalog (VM303)
| ID | Name | Runtime | Backend | V100 |
|---|---|---|---|---|
| `llama3-70b-gptq` | Llama 3 70B GPTQ | vllm-legacy-040 | vllm | yes |
| `qwen25-coder-32b-gptq` | Qwen 2.5 Coder 32B GPTQ | vllm-legacy-040 | vllm | yes |
| `qwen25-32b-gptq` | Qwen 2.5 32B GPTQ | vllm-legacy-040 | vllm | yes |
| `deepseek-coder-33b-gptq` | DeepSeek Coder 33B GPTQ | vllm-legacy-040 | vllm | yes |
| `qwen25-coder-7b` | Qwen 2.5 Coder 7B | vllm-legacy-040 | vllm | yes |
| `deepseek-coder-6.7b` | DeepSeek Coder 6.7B | vllm-legacy-040 | vllm | yes |
| `deepseek-coder-7b` | DeepSeek Coder 7B v1.5 | vllm-legacy-040 | vllm | yes |
| `starcoder2-15b` | StarCoder2 15B | vllm-legacy-040 | vllm | yes |
| `granite-code-8b` | IBM Granite Code 8B | vllm-legacy-040 | vllm | yes |
| `yi-coder-9b` | Yi-Coder 9B | vllm-legacy-040 | vllm | yes |
| `deepseek-coder-v2-lite` | DeepSeek Coder V2 Lite | vllm-modern-080 | vllm | yes |
| `qwen3-coder-next` | Qwen3 Coder Next | vllm-modern-080 | vllm | yes |
| `qwen3-coder-30b` | Qwen3 Coder 30B A3B | vllm-modern-080 | vllm | yes |
| `qwen36-27b` | Qwen 3.6 27B | sglang-modern | sglang | yes |
| `qwen36-27b-awq` | Qwen 3.6 27B AWQ | sglang-modern | sglang | yes |
| `codellama-34b` | Code Llama 34B | vllm-legacy-040 | vllm | yes |
docs/runbook-vm303-source.md
+61
View File
@@ -0,0 +1,61 @@
# GPU Lab Runbook — VM303 (atc-gpu-dev)
## Runtimes (Docker)
| Runtime ID | Compose | Image |
|---|---|---|
| `vllm-legacy-040` | `compose.vllm-legacy.yml` | `vllm/vllm-openai:v0.4.0.post1` |
| `vllm-modern-080` | `compose.vllm-modern.yml` | `vllm/vllm-openai:v0.8.5.post1` |
| `sglang-modern` | `compose.sglang.yml` | `lmsysorg/sglang:v0.5.12-cu129-runtime` |
Only **one** inference container runs at a time (port **8010**).
## Activate (UI)
GPU Lab → pick model → **Activate**. Writes `/root/gpu-lab/active/current.env`, stops all gpu-lab containers, starts the matching compose stack.
## Manual activate
```bash
cd /root/gpu-lab
# edit active/current.env (MODEL_PATH, TP_SIZE, VLLM_EXTRA_ARGS, …)
docker compose -f compose.vllm-legacy.yml down
docker compose -f compose.vllm-legacy.yml up -d
curl -s http://127.0.0.1:8010/v1/models
```
## Rollback to systemd vLLM
```bash
cd /root/gpu-lab
docker compose -f compose.vllm-legacy.yml down
docker compose -f compose.vllm-modern.yml down
docker compose -f compose.sglang.yml down
systemctl start vllm.service
systemctl restart vllm-gateway.service litellm-proxy.service
```
## Logs
```bash
docker logs -f gpu-lab-vllm-legacy
docker logs -f gpu-lab-vllm-modern
docker logs -f gpu-lab-sglang
journalctl -u model-manager -f
```
## SGLang on V100
Always use `--disable-flashinfer` and `--disable-flashinfer-sampling` (already in compose.sglang.yml).
## Dockhand
- Environment **GPU-Dev** → `10.0.20.106:2375`
- Stacks: `gpu-lab-vllm-legacy`, `gpu-lab-vllm-modern`, `gpu-lab-sglang` (compose in Dockhand data volume)
- Running workload appears as stack **`gpu-lab`** (compose project on VM303)
- API: `GET /api/stacks?env=8` (not `environmentId`)
- UI: http://atc-docker01.dell-atc.lan:8082/
## Docker data
Images stored under `/root/docker-data` (not `/var` — small partition).
docs/runbook-vm303.md
+47
View File
@@ -0,0 +1,47 @@
# GPU Lab Runbook — VM303 (atc-gpu-dev)
See also `config/gpu-lab/RUNBOOK.md` (synced from VM303).
## Activate via UI
GPU Lab → select model → **Activate**. Writes `/root/gpu-lab/active/current.env`, stops other gpu-lab containers, starts matching compose stack.
## Manual activate (SGLang + Qwen 3.6 27B AWQ)
```bash
cd /root/gpu-lab
cat > active/current.env << 'EOF'
MODEL_PATH=/models/qwen36-27b-awq
SERVED_NAME=qwen3.6-27b-awq
TP_SIZE=4
GPU_MEM_UTIL=0.9
MAX_MODEL_LEN=8192
EOF
docker compose -f compose.vllm-legacy.yml down
docker compose -f compose.vllm-modern.yml down
docker compose -f compose.sglang.yml up -d
curl -s http://127.0.0.1:8010/v1/models
systemctl restart vllm-gateway.service
```
## Rollback to systemd vLLM
```bash
cd /root/gpu-lab
docker compose -f compose.vllm-legacy.yml down
docker compose -f compose.vllm-modern.yml down
docker compose -f compose.sglang.yml down
systemctl start vllm.service
systemctl restart vllm-gateway.service litellm-proxy.service
```
## Logs
```bash
docker logs -f gpu-lab-sglang
journalctl -u model-manager -f
```
## SGLang on V100
Always use `--disable-flashinfer` and `--disable-flashinfer-sampling` (in `compose.sglang.yml`).
scripts/activate-model.sh
Executable
+7
View File
@@ -0,0 +1,7 @@
#!/usr/bin/env bash
# Activate a model via GPU Lab API (run on VM303 or with curl to :9000)
set -euo pipefail
MODEL_ID="${1:?Usage: activate-model.sh <model-id>}"
curl -sf -X POST "http://127.0.0.1:9000/api/activate/${MODEL_ID}"
echo
curl -sf "http://127.0.0.1:8010/v1/models" | python3 -m json.tool