diff --git a/README.md b/README.md index 42cd427..bb59956 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,35 @@ -# atc-GPU +# atc-GPU — GPU Lab Infrastructure -GPU Lab infrastructure — VM303 model-manager, gpu-lab Docker stacks, Dockhand \ No newline at end of file +Infrastructure-as-code and documentation for the ATC GPU Lab environment on **VM303 (GPU-Dev)**. + +## Quick links + +| Service | URL | +|---------|-----| +| GPU Lab UI | http://10.0.20.106:9000/ | +| Inference API (active runtime) | http://10.0.20.106:8010/v1/models | +| OpenAI gateway | http://10.0.20.106:8001/v1/models | +| Dockhand (GPU-Dev env) | http://atc-docker01.dell-atc.lan:8082/ | + +## Repository layout + +``` +atc-GPU/ +├── README.md +├── docs/ # Landscape, models, runbooks, Dockhand mapping +├── config/ # gpu-lab compose + model-manager catalog (no secrets) +├── compose/ # Dockhand-safe compose variants +└── scripts/ # Helper scripts (activate-model) +``` + +## Active target (2026-06) + +- **Runtime:** SGLang (`sglang-modern`) via `compose.sglang.yml` +- **Model:** Qwen 3.6 27B AWQ (`qwen36-27b-awq` → served as `qwen3.6-27b-awq`) +- **Previous:** Llama3-70b-GPTQ on vLLM legacy (to be decommissioned after cutover) + +See [docs/runbook-vm303.md](docs/runbook-vm303.md) for activate/rollback steps. + +## Related repos + +- [mo/Lakehouse](http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse) — Spark, Trino, Debezium, Airflow, S3 pipeline diff --git a/config/gpu-lab/RUNBOOK.md b/config/gpu-lab/RUNBOOK.md new file mode 100644 index 0000000..13537ac --- /dev/null +++ b/config/gpu-lab/RUNBOOK.md @@ -0,0 +1,61 @@ +# GPU Lab Runbook — VM303 (atc-gpu-dev) + +## Runtimes (Docker) + +| Runtime ID | Compose | Image | +|---|---|---| +| `vllm-legacy-040` | `compose.vllm-legacy.yml` | `vllm/vllm-openai:v0.4.0.post1` | +| `vllm-modern-080` | `compose.vllm-modern.yml` | `vllm/vllm-openai:v0.8.5.post1` | +| `sglang-modern` | `compose.sglang.yml` | `lmsysorg/sglang:v0.5.12-cu129-runtime` | + +Only **one** inference container runs at a time (port **8010**). + +## Activate (UI) + +GPU Lab → pick model → **Activate**. Writes `/root/gpu-lab/active/current.env`, stops all gpu-lab containers, starts the matching compose stack. + +## Manual activate + +```bash +cd /root/gpu-lab +# edit active/current.env (MODEL_PATH, TP_SIZE, VLLM_EXTRA_ARGS, …) +docker compose -f compose.vllm-legacy.yml down +docker compose -f compose.vllm-legacy.yml up -d +curl -s http://127.0.0.1:8010/v1/models +``` + +## Rollback to systemd vLLM + +```bash +cd /root/gpu-lab +docker compose -f compose.vllm-legacy.yml down +docker compose -f compose.vllm-modern.yml down +docker compose -f compose.sglang.yml down +systemctl start vllm.service +systemctl restart vllm-gateway.service litellm-proxy.service +``` + +## Logs + +```bash +docker logs -f gpu-lab-vllm-legacy +docker logs -f gpu-lab-vllm-modern +docker logs -f gpu-lab-sglang +journalctl -u model-manager -f +``` + +## SGLang on V100 + +Always use `--disable-flashinfer` and `--disable-flashinfer-sampling` (already in compose.sglang.yml). + +## Dockhand + +- Environment **GPU-Dev** → `10.0.20.106:2375` +- Stacks: `gpu-lab-vllm-legacy`, `gpu-lab-vllm-modern`, `gpu-lab-sglang` (compose in Dockhand data volume) +- Running workload appears as stack **`gpu-lab`** (compose project on VM303) +- API: `GET /api/stacks?env=8` (not `environmentId`) +- UI: http://atc-docker01.dell-atc.lan:8082/ + +## Docker data + +Images stored under `/root/docker-data` (not `/var` — small partition). diff --git a/config/gpu-lab/compose.sglang.yml b/config/gpu-lab/compose.sglang.yml new file mode 100644 index 0000000..d6ae859 --- /dev/null +++ b/config/gpu-lab/compose.sglang.yml @@ -0,0 +1,38 @@ +name: gpu-lab-sglang +services: + sglang: + image: lmsysorg/sglang:v0.5.12-cu129-runtime + container_name: gpu-lab-sglang + network_mode: host + ipc: host + shm_size: "32gb" + env_file: + - active/current.env + volumes: + - /root/models:/models:ro + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: [gpu] + entrypoint: ["python3", "-m", "sglang.launch_server"] + command: + - --model-path + - ${MODEL_PATH} + - --served-model-name + - ${SERVED_NAME} + - --host + - "0.0.0.0" + - --port + - "8010" + - --tp-size + - ${TP_SIZE} + - --mem-fraction-static + - ${GPU_MEM_UTIL} + - --context-length + - ${MAX_MODEL_LEN} + - --disable-flashinfer + - --disable-flashinfer-sampling + restart: "no" diff --git a/config/gpu-lab/compose.vllm-legacy.yml b/config/gpu-lab/compose.vllm-legacy.yml new file mode 100644 index 0000000..7d90ad5 --- /dev/null +++ b/config/gpu-lab/compose.vllm-legacy.yml @@ -0,0 +1,33 @@ +name: gpu-lab-vllm-legacy +services: + vllm-legacy: + image: vllm/vllm-openai:v0.4.0.post1 + container_name: gpu-lab-vllm-legacy + network_mode: host + ipc: host + shm_size: "32gb" + env_file: + - active/current.env + volumes: + - /root/models:/models:ro + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: [gpu] + entrypoint: ["/bin/sh", "-c"] + command: + - > + exec python3 -m vllm.entrypoints.openai.api_server + --model "$$MODEL_PATH" + --served-model-name "$$SERVED_NAME" + --host 0.0.0.0 --port 8010 + --tensor-parallel-size "$$TP_SIZE" + --dtype "$$DTYPE" + --max-model-len "$$MAX_MODEL_LEN" + --gpu-memory-utilization "$$GPU_MEM_UTIL" + --max-num-seqs "$$MAX_NUM_SEQS" + $$VLLM_EXTRA_ARGS + restart: "no" diff --git a/config/gpu-lab/compose.vllm-modern.yml b/config/gpu-lab/compose.vllm-modern.yml new file mode 100644 index 0000000..bff14f7 --- /dev/null +++ b/config/gpu-lab/compose.vllm-modern.yml @@ -0,0 +1,34 @@ +name: gpu-lab-vllm-modern +services: + vllm-modern: + image: vllm/vllm-openai:v0.8.5.post1 + container_name: gpu-lab-vllm-modern + network_mode: host + ipc: host + shm_size: "32gb" + env_file: + - active/current.env + volumes: + - /root/models:/models:ro + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: [gpu] + entrypoint: ["/bin/sh", "-c"] + command: + - > + exec python3 -m vllm.entrypoints.openai.api_server + --model "$$MODEL_PATH" + --served-model-name "$$SERVED_NAME" + --host 0.0.0.0 --port 8010 + --tensor-parallel-size "$$TP_SIZE" + --dtype "$$DTYPE" + --max-model-len "$$MAX_MODEL_LEN" + --gpu-memory-utilization "$$GPU_MEM_UTIL" + --max-num-seqs "$$MAX_NUM_SEQS" + --trust-remote-code --enforce-eager + $$VLLM_EXTRA_ARGS + restart: "no" diff --git a/config/gpu-lab/runtimes.json b/config/gpu-lab/runtimes.json new file mode 100644 index 0000000..8b5eaae --- /dev/null +++ b/config/gpu-lab/runtimes.json @@ -0,0 +1,24 @@ +{ + "vllm-legacy-040": { + "engine": "vllm", + "compose_file": "compose.vllm-legacy.yml", + "image": "vllm/vllm-openai:v0.4.0.post1", + "v100_ok": true, + "label": "vLLM 0.4" + }, + "vllm-modern-080": { + "engine": "vllm", + "compose_file": "compose.vllm-modern.yml", + "image": "vllm/vllm-openai:v0.8.5.post1", + "v100_ok": true, + "label": "vLLM 0.8" + }, + "sglang-modern": { + "engine": "sglang", + "compose_file": "compose.sglang.yml", + "image": "lmsysorg/sglang:v0.5.12-cu129-runtime", + "v100_ok": true, + "label": "SGLang", + "extra_args": ["--disable-flashinfer", "--disable-flashinfer-sampling"] + } +} diff --git a/config/model-manager/catalog.json b/config/model-manager/catalog.json new file mode 100644 index 0000000..6951fba --- /dev/null +++ b/config/model-manager/catalog.json @@ -0,0 +1,515 @@ +{ + "models": [ + { + "id": "llama3-70b-gptq", + "name": "Llama 3 70B GPTQ", + "description": "Meta Llama 3 open weights. Strong general coder (lab default). 4\u00d7 V100.", + "tags": [ + "coding", + "opensource", + "v100_ready" + ], + "license": "Llama 3 Community License", + "repo_id": null, + "local_path": "/root/models/llama3-70b-quantized", + "served_name": "llama3-70b-gptq", + "size_gb_estimate": 38, + "vllm_min_version": "0.4.0", + "vllm": { + "tensor_parallel_size": 4, + "quantization": "gptq", + "dtype": "float16", + "max_model_len": 8192, + "gpu_memory_utilization": 0.88, + "max_num_seqs": 4, + "extra_args": [ + "--disable-custom-all-reduce", + "--enforce-eager" + ] + }, + "runtime_id": "vllm-legacy-040", + "inference_backend": "vllm", + "v100_activate": true, + "v100_block_reason": null + }, + { + "id": "qwen25-coder-32b-gptq", + "name": "Qwen 2.5 Coder 32B GPTQ", + "description": "Alibaba Qwen open weights. Top open-source code LLM (GPT-4o class). 4\u00d7 V100.", + "tags": [ + "coding", + "opensource", + "qwen", + "v100_ready" + ], + "license": "Apache-2.0", + "repo_id": "Qwen/Qwen2.5-Coder-32B-Instruct-GPTQ-Int4", + "local_path": "/root/models/qwen25-coder-32b-gptq", + "served_name": "qwen2.5-coder-32b-gptq", + "size_gb_estimate": 20, + "vllm_min_version": "0.4.0", + "vllm": { + "tensor_parallel_size": 4, + "quantization": "gptq", + "dtype": "float16", + "max_model_len": 8192, + "gpu_memory_utilization": 0.9, + "max_num_seqs": 4, + "extra_args": [ + "--trust-remote-code", + "--enforce-eager" + ] + }, + "runtime_id": "vllm-legacy-040", + "inference_backend": "vllm", + "v100_activate": true, + "v100_block_reason": null + }, + { + "id": "qwen25-32b-gptq", + "name": "Qwen 2.5 32B GPTQ", + "description": "Alibaba Qwen open weights (general instruct, strong coder).", + "tags": [ + "coding", + "opensource", + "qwen", + "v100_ready" + ], + "license": "Apache-2.0", + "repo_id": "Qwen/Qwen2.5-32B-Instruct-GPTQ-Int4", + "local_path": "/root/models/qwen25-32b-gptq", + "served_name": "qwen2.5-32b-gptq", + "size_gb_estimate": 20, + "vllm_min_version": "0.4.0", + "vllm": { + "tensor_parallel_size": 4, + "quantization": "gptq", + "dtype": "float16", + "max_model_len": 8192, + "gpu_memory_utilization": 0.9, + "max_num_seqs": 4, + "extra_args": [ + "--trust-remote-code", + "--enforce-eager" + ] + }, + "runtime_id": "vllm-legacy-040", + "inference_backend": "vllm", + "v100_activate": true, + "v100_block_reason": null + }, + { + "id": "deepseek-coder-33b-gptq", + "name": "DeepSeek Coder 33B GPTQ", + "description": "DeepSeek open weights (TheBloke GPTQ). Top-tier coder, TP=4.", + "tags": [ + "coding", + "deepseek", + "opensource", + "v100_ready" + ], + "license": "DeepSeek License", + "repo_id": "TheBloke/deepseek-coder-33B-instruct-GPTQ", + "local_path": "/root/models/deepseek-coder-33b-gptq", + "served_name": "deepseek-coder-33b-gptq", + "size_gb_estimate": 18, + "vllm_min_version": "0.4.0", + "vllm": { + "tensor_parallel_size": 4, + "quantization": "gptq", + "dtype": "float16", + "max_model_len": 4096, + "gpu_memory_utilization": 0.88, + "max_num_seqs": 4, + "extra_args": [ + "--trust-remote-code", + "--enforce-eager" + ] + }, + "runtime_id": "vllm-legacy-040", + "inference_backend": "vllm", + "v100_activate": true, + "v100_block_reason": null + }, + { + "id": "qwen25-coder-7b", + "name": "Qwen 2.5 Coder 7B", + "description": "Alibaba Qwen open weights. Fast code model, 1 GPU.", + "tags": [ + "coding", + "opensource", + "qwen", + "v100_ready" + ], + "license": "Apache-2.0", + "repo_id": "Qwen/Qwen2.5-Coder-7B-Instruct", + "local_path": "/root/models/qwen25-coder-7b", + "served_name": "qwen2.5-coder-7b", + "size_gb_estimate": 15, + "vllm_min_version": "0.4.0", + "vllm": { + "tensor_parallel_size": 1, + "dtype": "float16", + "max_model_len": 8192, + "gpu_memory_utilization": 0.9, + "max_num_seqs": 8, + "extra_args": [ + "--trust-remote-code", + "--enforce-eager" + ] + }, + "runtime_id": "vllm-legacy-040", + "inference_backend": "vllm", + "v100_activate": true, + "v100_block_reason": null + }, + { + "id": "deepseek-coder-6.7b", + "name": "DeepSeek Coder 6.7B", + "description": "DeepSeek open weights. Fast lightweight coder, 1 GPU.", + "tags": [ + "coding", + "deepseek", + "opensource", + "v100_ready" + ], + "license": "DeepSeek License", + "repo_id": "deepseek-ai/deepseek-coder-6.7b-instruct", + "local_path": "/root/models/deepseek-coder-6.7b", + "served_name": "deepseek-coder-6.7b", + "size_gb_estimate": 14, + "vllm_min_version": "0.4.0", + "vllm": { + "tensor_parallel_size": 1, + "dtype": "float16", + "max_model_len": 8192, + "gpu_memory_utilization": 0.9, + "max_num_seqs": 8, + "extra_args": [ + "--trust-remote-code", + "--enforce-eager" + ] + }, + "runtime_id": "vllm-legacy-040", + "inference_backend": "vllm", + "v100_activate": true, + "v100_block_reason": null + }, + { + "id": "deepseek-coder-7b", + "name": "DeepSeek Coder 7B v1.5", + "description": "DeepSeek open weights. Balanced 7B coder.", + "tags": [ + "coding", + "deepseek", + "opensource", + "v100_ready" + ], + "license": "DeepSeek License", + "repo_id": "deepseek-ai/deepseek-coder-7b-instruct-v1.5", + "local_path": "/root/models/deepseek-coder-7b", + "served_name": "deepseek-coder-7b", + "size_gb_estimate": 15, + "vllm_min_version": "0.4.0", + "vllm": { + "tensor_parallel_size": 1, + "dtype": "float16", + "max_model_len": 8192, + "gpu_memory_utilization": 0.9, + "max_num_seqs": 8, + "extra_args": [ + "--trust-remote-code", + "--enforce-eager" + ] + }, + "runtime_id": "vllm-legacy-040", + "inference_backend": "vllm", + "v100_activate": true, + "v100_block_reason": null + }, + { + "id": "starcoder2-15b", + "name": "StarCoder2 15B", + "description": "BigCode/Hugging Face. Apache-2.0 code specialist, 1\u20132 GPU.", + "tags": [ + "coding", + "opensource", + "v100_ready" + ], + "license": "Apache-2.0", + "repo_id": "bigcode/starcoder2-15b", + "local_path": "/root/models/starcoder2-15b", + "served_name": "starcoder2-15b", + "size_gb_estimate": 30, + "vllm_min_version": "0.4.0", + "vllm": { + "tensor_parallel_size": 2, + "dtype": "float16", + "max_model_len": 8192, + "gpu_memory_utilization": 0.9, + "max_num_seqs": 4, + "extra_args": [ + "--trust-remote-code", + "--enforce-eager" + ] + }, + "runtime_id": "vllm-legacy-040", + "inference_backend": "vllm", + "v100_activate": true, + "v100_block_reason": null + }, + { + "id": "granite-code-8b", + "name": "IBM Granite Code 8B", + "description": "IBM open weights. Enterprise-friendly Apache-2.0 coder.", + "tags": [ + "coding", + "opensource", + "v100_ready" + ], + "license": "Apache-2.0", + "repo_id": "ibm-granite/granite-3.3-8b-instruct", + "local_path": "/root/models/granite-code-8b", + "served_name": "granite-code-8b", + "size_gb_estimate": 16, + "vllm_min_version": "0.4.0", + "vllm": { + "tensor_parallel_size": 1, + "dtype": "float16", + "max_model_len": 8192, + "gpu_memory_utilization": 0.9, + "max_num_seqs": 8, + "extra_args": [ + "--trust-remote-code", + "--enforce-eager" + ] + }, + "runtime_id": "vllm-legacy-040", + "inference_backend": "vllm", + "v100_activate": true, + "v100_block_reason": null + }, + { + "id": "yi-coder-9b", + "name": "Yi-Coder 9B", + "description": "01.AI open weights. Strong 9B code model, 1 GPU.", + "tags": [ + "coding", + "opensource", + "v100_ready" + ], + "license": "Apache-2.0", + "repo_id": "01-ai/Yi-Coder-9B-Chat", + "local_path": "/root/models/yi-coder-9b", + "served_name": "yi-coder-9b", + "size_gb_estimate": 18, + "vllm_min_version": "0.4.0", + "vllm": { + "tensor_parallel_size": 1, + "dtype": "float16", + "max_model_len": 8192, + "gpu_memory_utilization": 0.9, + "max_num_seqs": 8, + "extra_args": [ + "--trust-remote-code", + "--enforce-eager" + ] + }, + "runtime_id": "vllm-legacy-040", + "inference_backend": "vllm", + "v100_activate": true, + "v100_block_reason": null + }, + { + "id": "deepseek-coder-v2-lite", + "name": "DeepSeek Coder V2 Lite", + "description": "DeepSeek MoE coder (16B/2.4B active). vLLM 0.8 on V100.", + "tags": [ + "coding", + "deepseek", + "opensource", + "v100_ready" + ], + "license": "DeepSeek License", + "repo_id": "deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct", + "local_path": "/root/models/deepseek-coder-v2-lite", + "served_name": "deepseek-coder-v2-lite", + "size_gb_estimate": 32, + "vllm_min_version": "0.6.0", + "vllm": { + "tensor_parallel_size": 2, + "dtype": "bfloat16", + "max_model_len": 8192, + "gpu_memory_utilization": 0.9, + "max_num_seqs": 4, + "extra_args": [ + "--trust-remote-code", + "--enforce-eager" + ] + }, + "runtime_id": "vllm-modern-080", + "inference_backend": "vllm", + "v100_activate": true, + "v100_block_reason": null + }, + { + "id": "qwen3-coder-next", + "name": "Qwen3 Coder Next", + "description": "Alibaba agentic coder (80B MoE, 3B active). vLLM 0.8 on V100.", + "tags": [ + "coding", + "opensource", + "qwen", + "v100_ready" + ], + "license": "Apache-2.0", + "repo_id": "Qwen/Qwen3-Coder-Next", + "local_path": "/root/models/qwen3-coder-next", + "served_name": "qwen3-coder-next", + "size_gb_estimate": 45, + "vllm_min_version": "0.8.0", + "vllm": { + "tensor_parallel_size": 4, + "dtype": "bfloat16", + "max_model_len": 8192, + "gpu_memory_utilization": 0.9, + "max_num_seqs": 4, + "extra_args": [ + "--trust-remote-code", + "--enforce-eager" + ] + }, + "runtime_id": "vllm-modern-080", + "inference_backend": "vllm", + "v100_activate": true, + "v100_block_reason": null + }, + { + "id": "qwen3-coder-30b", + "name": "Qwen3 Coder 30B A3B", + "description": "Alibaba MoE agentic coder. vLLM 0.8 on V100.", + "tags": [ + "coding", + "opensource", + "qwen", + "v100_ready" + ], + "license": "Apache-2.0", + "repo_id": "Qwen/Qwen3-Coder-30B-A3B-Instruct", + "local_path": "/root/models/qwen3-coder-30b", + "served_name": "qwen3-coder-30b", + "size_gb_estimate": 60, + "vllm_min_version": "0.8.0", + "vllm": { + "tensor_parallel_size": 4, + "dtype": "bfloat16", + "max_model_len": 8192, + "gpu_memory_utilization": 0.9, + "max_num_seqs": 4, + "extra_args": [ + "--trust-remote-code", + "--enforce-eager" + ] + }, + "runtime_id": "vllm-modern-080", + "inference_backend": "vllm", + "v100_activate": true, + "v100_block_reason": null + }, + { + "id": "qwen36-27b", + "name": "Qwen 3.6 27B", + "description": "Qwen open weights (dense). On V100 via SGLang \u2014 prefer AWQ variant.", + "tags": [ + "coding", + "opensource", + "qwen", + "v100_via_sglang" + ], + "license": "Apache-2.0", + "repo_id": "Qwen/Qwen3.6-27B", + "local_path": "/root/models/qwen36-27b", + "served_name": "qwen3.6-27b", + "size_gb_estimate": 54, + "vllm_min_version": "0.19.0", + "vllm": { + "tensor_parallel_size": 4, + "dtype": "bfloat16", + "max_model_len": 8192, + "gpu_memory_utilization": 0.9, + "max_num_seqs": 4, + "extra_args": [ + "--trust-remote-code", + "--enforce-eager" + ] + }, + "runtime_id": "sglang-modern", + "inference_backend": "sglang", + "v100_activate": true, + "v100_block_reason": null + }, + { + "id": "qwen36-27b-awq", + "name": "Qwen 3.6 27B AWQ", + "description": "Qwen open weights (AWQ). Primary Qwen 3.6 path on V100 via SGLang.", + "tags": [ + "coding", + "opensource", + "qwen", + "v100_via_sglang" + ], + "license": "Apache-2.0", + "repo_id": "QuantTrio/Qwen3.6-27B-AWQ", + "local_path": "/root/models/qwen36-27b-awq", + "served_name": "qwen3.6-27b-awq", + "size_gb_estimate": 16, + "vllm_min_version": "0.19.0", + "vllm": { + "tensor_parallel_size": 4, + "quantization": "awq", + "dtype": "float16", + "max_model_len": 8192, + "gpu_memory_utilization": 0.9, + "max_num_seqs": 8, + "extra_args": [ + "--trust-remote-code", + "--enforce-eager" + ] + }, + "runtime_id": "sglang-modern", + "inference_backend": "sglang", + "v100_activate": true, + "v100_block_reason": null + }, + { + "id": "codellama-34b", + "name": "Code Llama 34B", + "description": "Meta open weights (custom license). Classic code LLM \u2014 downloadable via HF.", + "tags": [ + "coding", + "opensource", + "v100_ready" + ], + "license": "Llama 3 Community License", + "repo_id": "codellama/CodeLlama-34b-Instruct-hf", + "local_path": "/root/models/codellama-34b", + "served_name": "codellama-34b", + "size_gb_estimate": 68, + "vllm_min_version": "0.4.0", + "vllm": { + "tensor_parallel_size": 4, + "dtype": "float16", + "max_model_len": 4096, + "gpu_memory_utilization": 0.9, + "max_num_seqs": 2, + "extra_args": [ + "--enforce-eager" + ] + }, + "runtime_id": "vllm-legacy-040", + "inference_backend": "vllm", + "v100_activate": true, + "v100_block_reason": null + } + ] +} diff --git a/config/model-manager/vm-specs.json b/config/model-manager/vm-specs.json new file mode 100644 index 0000000..ce4c337 --- /dev/null +++ b/config/model-manager/vm-specs.json @@ -0,0 +1,14 @@ +{ + "vmid": 303, + "name": "atc-gpu-dev", + "proxmox_host": "atc-gpu", + "ip": "10.0.20.106", + "cpu_model": "Intel Xeon Gold 6148 @ 2.40GHz", + "vcpu": 32, + "sockets": 2, + "cores_per_socket": 16, + "memory_gib": 128, + "disk_gib": 250, + "gpu": "4× Tesla V100 PCIe 32GB", + "os": "Rocky Linux 9" +} diff --git a/docs/dockhand.md b/docs/dockhand.md new file mode 100644 index 0000000..3f35932 --- /dev/null +++ b/docs/dockhand.md @@ -0,0 +1,12 @@ +# Dockhand — GPU Lab stacks + +| Dockhand env | ID | Docker host | Stacks | +|--------------|-----|-------------|--------| +| GPU-Dev | 8 | 10.0.20.106:2375 | gpu-lab-vllm-legacy, gpu-lab-vllm-modern, gpu-lab-sglang | +| Spark+Trino / Lakehouse | 9 | 10.0.21.50 | lakehouse-spark-trino (Trino, Spark, kafka-connect, s3-kafka-consumer) | +| Airflow | 10 | 10.0.21.55 | airflow | + +- Dockhand UI: http://atc-docker01.dell-atc.lan:8082/ +- API stacks: `GET /api/stacks?env=8` + +Only one GPU inference stack should be **running** at a time; others appear as stopped compose projects. diff --git a/docs/landscape.md b/docs/landscape.md new file mode 100644 index 0000000..37820c3 --- /dev/null +++ b/docs/landscape.md @@ -0,0 +1,30 @@ +# GPU Lab Landscape + +## Nodes + +| Host | IP | Role | +|------|-----|------| +| VM303 GPU-Dev | 10.0.20.106 | GPU Lab UI (:9000), inference (:8010), gateway (:8001) | +| atc-docker01 | 10.0.21.45 | Homepage, Dockhand (:8082) | +| Bart-GPU | — | Reference node (documented in model-manager) | +| Mo-GPU | — | Reference node | + +## VM303 specs + +See `config/model-manager/vm-specs.json`: 4× Tesla V100 32GB, Docker with NVIDIA runtime. + +## Inference runtimes + +Only one runtime is active at a time on port **8010**: + +| Runtime ID | Engine | Compose file | +|------------|--------|--------------| +| `vllm-legacy-040` | vLLM 0.4.x | `compose.vllm-legacy.yml` | +| `vllm-modern-080` | vLLM 0.8.x | `compose.vllm-modern.yml` | +| `sglang-modern` | SGLang 0.5.12 | `compose.sglang.yml` | + +Models live under `/root/models/` on VM303 (not in git). + +## Docker data + +Images stored under `/root/docker-data` (avoid small `/var` partition). diff --git a/docs/models.md b/docs/models.md new file mode 100644 index 0000000..0326b66 --- /dev/null +++ b/docs/models.md @@ -0,0 +1,20 @@ +# Model catalog (VM303) + +| ID | Name | Runtime | Backend | V100 | +|---|---|---|---|---| +| `llama3-70b-gptq` | Llama 3 70B GPTQ | vllm-legacy-040 | vllm | yes | +| `qwen25-coder-32b-gptq` | Qwen 2.5 Coder 32B GPTQ | vllm-legacy-040 | vllm | yes | +| `qwen25-32b-gptq` | Qwen 2.5 32B GPTQ | vllm-legacy-040 | vllm | yes | +| `deepseek-coder-33b-gptq` | DeepSeek Coder 33B GPTQ | vllm-legacy-040 | vllm | yes | +| `qwen25-coder-7b` | Qwen 2.5 Coder 7B | vllm-legacy-040 | vllm | yes | +| `deepseek-coder-6.7b` | DeepSeek Coder 6.7B | vllm-legacy-040 | vllm | yes | +| `deepseek-coder-7b` | DeepSeek Coder 7B v1.5 | vllm-legacy-040 | vllm | yes | +| `starcoder2-15b` | StarCoder2 15B | vllm-legacy-040 | vllm | yes | +| `granite-code-8b` | IBM Granite Code 8B | vllm-legacy-040 | vllm | yes | +| `yi-coder-9b` | Yi-Coder 9B | vllm-legacy-040 | vllm | yes | +| `deepseek-coder-v2-lite` | DeepSeek Coder V2 Lite | vllm-modern-080 | vllm | yes | +| `qwen3-coder-next` | Qwen3 Coder Next | vllm-modern-080 | vllm | yes | +| `qwen3-coder-30b` | Qwen3 Coder 30B A3B | vllm-modern-080 | vllm | yes | +| `qwen36-27b` | Qwen 3.6 27B | sglang-modern | sglang | yes | +| `qwen36-27b-awq` | Qwen 3.6 27B AWQ | sglang-modern | sglang | yes | +| `codellama-34b` | Code Llama 34B | vllm-legacy-040 | vllm | yes | diff --git a/docs/runbook-vm303-source.md b/docs/runbook-vm303-source.md new file mode 100644 index 0000000..13537ac --- /dev/null +++ b/docs/runbook-vm303-source.md @@ -0,0 +1,61 @@ +# GPU Lab Runbook — VM303 (atc-gpu-dev) + +## Runtimes (Docker) + +| Runtime ID | Compose | Image | +|---|---|---| +| `vllm-legacy-040` | `compose.vllm-legacy.yml` | `vllm/vllm-openai:v0.4.0.post1` | +| `vllm-modern-080` | `compose.vllm-modern.yml` | `vllm/vllm-openai:v0.8.5.post1` | +| `sglang-modern` | `compose.sglang.yml` | `lmsysorg/sglang:v0.5.12-cu129-runtime` | + +Only **one** inference container runs at a time (port **8010**). + +## Activate (UI) + +GPU Lab → pick model → **Activate**. Writes `/root/gpu-lab/active/current.env`, stops all gpu-lab containers, starts the matching compose stack. + +## Manual activate + +```bash +cd /root/gpu-lab +# edit active/current.env (MODEL_PATH, TP_SIZE, VLLM_EXTRA_ARGS, …) +docker compose -f compose.vllm-legacy.yml down +docker compose -f compose.vllm-legacy.yml up -d +curl -s http://127.0.0.1:8010/v1/models +``` + +## Rollback to systemd vLLM + +```bash +cd /root/gpu-lab +docker compose -f compose.vllm-legacy.yml down +docker compose -f compose.vllm-modern.yml down +docker compose -f compose.sglang.yml down +systemctl start vllm.service +systemctl restart vllm-gateway.service litellm-proxy.service +``` + +## Logs + +```bash +docker logs -f gpu-lab-vllm-legacy +docker logs -f gpu-lab-vllm-modern +docker logs -f gpu-lab-sglang +journalctl -u model-manager -f +``` + +## SGLang on V100 + +Always use `--disable-flashinfer` and `--disable-flashinfer-sampling` (already in compose.sglang.yml). + +## Dockhand + +- Environment **GPU-Dev** → `10.0.20.106:2375` +- Stacks: `gpu-lab-vllm-legacy`, `gpu-lab-vllm-modern`, `gpu-lab-sglang` (compose in Dockhand data volume) +- Running workload appears as stack **`gpu-lab`** (compose project on VM303) +- API: `GET /api/stacks?env=8` (not `environmentId`) +- UI: http://atc-docker01.dell-atc.lan:8082/ + +## Docker data + +Images stored under `/root/docker-data` (not `/var` — small partition). diff --git a/docs/runbook-vm303.md b/docs/runbook-vm303.md new file mode 100644 index 0000000..98781a9 --- /dev/null +++ b/docs/runbook-vm303.md @@ -0,0 +1,47 @@ +# GPU Lab Runbook — VM303 (atc-gpu-dev) + +See also `config/gpu-lab/RUNBOOK.md` (synced from VM303). + +## Activate via UI + +GPU Lab → select model → **Activate**. Writes `/root/gpu-lab/active/current.env`, stops other gpu-lab containers, starts matching compose stack. + +## Manual activate (SGLang + Qwen 3.6 27B AWQ) + +```bash +cd /root/gpu-lab +cat > active/current.env << 'EOF' +MODEL_PATH=/models/qwen36-27b-awq +SERVED_NAME=qwen3.6-27b-awq +TP_SIZE=4 +GPU_MEM_UTIL=0.9 +MAX_MODEL_LEN=8192 +EOF +docker compose -f compose.vllm-legacy.yml down +docker compose -f compose.vllm-modern.yml down +docker compose -f compose.sglang.yml up -d +curl -s http://127.0.0.1:8010/v1/models +systemctl restart vllm-gateway.service +``` + +## Rollback to systemd vLLM + +```bash +cd /root/gpu-lab +docker compose -f compose.vllm-legacy.yml down +docker compose -f compose.vllm-modern.yml down +docker compose -f compose.sglang.yml down +systemctl start vllm.service +systemctl restart vllm-gateway.service litellm-proxy.service +``` + +## Logs + +```bash +docker logs -f gpu-lab-sglang +journalctl -u model-manager -f +``` + +## SGLang on V100 + +Always use `--disable-flashinfer` and `--disable-flashinfer-sampling` (in `compose.sglang.yml`). diff --git a/scripts/activate-model.sh b/scripts/activate-model.sh new file mode 100755 index 0000000..c10ff25 --- /dev/null +++ b/scripts/activate-model.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +# Activate a model via GPU Lab API (run on VM303 or with curl to :9000) +set -euo pipefail +MODEL_ID="${1:?Usage: activate-model.sh }" +curl -sf -X POST "http://127.0.0.1:9000/api/activate/${MODEL_ID}" +echo +curl -sf "http://127.0.0.1:8010/v1/models" | python3 -m json.tool