164 lines
6.6 KiB
Markdown
164 lines
6.6 KiB
Markdown
|
# Architecture
|
|||
|
|
|
|||
|
|
## 1. System context
|
|||
|
|
|
|||
|
|
OME Cockpit is a **BFF + SPA** running on `atc-portal01` (Docker). It does **not** embed OME; it continuously aggregates OME REST data, enriches with GPU telemetry and LLM chat, and pushes live snapshots to browsers over WebSocket.
|
|||
|
|
|
|||
|
|
### Actors
|
|||
|
|
|
|||
|
|
| Actor | Role |
|
|||
|
|
|-------|------|
|
|||
|
|
| ATC admin (browser) | Operates fleet via UI |
|
|||
|
|
| OpenManage Enterprise | Source of truth for devices/alerts/inventory |
|
|||
|
|
| vLLM (`atc-gpu-prod:8000`) | LLM inference for Cockpit chat |
|
|||
|
|
| Open WebUI (`:3080`) | Model catalog + virtual models (`ome-copilot`, `arena-model`) |
|
|||
|
|
| GPU metrics sidecar (`:9110`) | NVIDIA V100 utilization/power |
|
|||
|
|
| Target servers / iDRACs | SSH endpoints opened from the browser via cockpit bridge |
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 2. Runtime components
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
┌─────────────────────────────────────────────────────────────┐
|
|||
|
|
│ Docker: ome-cockpit │
|
|||
|
|
│ uvicorn → FastAPI (main.py) :8090 → host :3090 │
|
|||
|
|
│ │
|
|||
|
|
│ Background tasks: │
|
|||
|
|
│ • poll_loop() ~POLL_INTERVAL (default 12–15s) │
|
|||
|
|
│ • gpu_loop() ~2s │
|
|||
|
|
│ │
|
|||
|
|
│ In-memory STATE + WebSocket fan-out │
|
|||
|
|
│ SQLite tickets → /data/ops.db │
|
|||
|
|
│ Static UI → /ui (bind-mount) │
|
|||
|
|
└─────────────────────────────────────────────────────────────┘
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Process model
|
|||
|
|
|
|||
|
|
- **Single uvicorn worker** (important): fleet `STATE`, WebSocket client sets, and SSH session registry are in-process memory. Do not scale to multiple workers without an external shared store.
|
|||
|
|
- UI assets and `main.py` are bind-mounted for lab hot-reload; image rebuild needed when Python dependencies change (`asyncssh`, etc.).
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 3. Data flows
|
|||
|
|
|
|||
|
|
### 3.1 Fleet snapshot (OME → UI)
|
|||
|
|
|
|||
|
|
1. `poll_loop` authenticates to OME `SessionService/Sessions`
|
|||
|
|
2. Fetches devices, groups, alerts (paged `$top`)
|
|||
|
|
3. For a capped set of connected servers, fetches `/Power`
|
|||
|
|
4. Normalizes into cockpit device nodes (`is_server`, `is_idrac`, subnet, watts, …)
|
|||
|
|
5. Diffs against `PREV_DEVICES` → `EVENT_FEED` (connection/power/status + **device_new** / **device_removed**)
|
|||
|
|
6. Writes `STATE`, broadcasts `{type:"snapshot", data: STATE}` to all `/ws/fleet` clients
|
|||
|
|
|
|||
|
|
Browser `app.js` applies snapshot → layout → canvas draw loop; `ops.js` updates GPU, tickets, notifications.
|
|||
|
|
|
|||
|
|
### 3.2 Device inventory (on demand)
|
|||
|
|
|
|||
|
|
`GET /api/devices/{id}`:
|
|||
|
|
|
|||
|
|
1. Resolves node from current `STATE`
|
|||
|
|
2. Opens OME session
|
|||
|
|
3. Reads `InventoryTypes` for that device and pulls each `InventoryDetails('…')`
|
|||
|
|
4. Builds normalized `landscape` (OS, firmware, drivers, applications, management, licenses)
|
|||
|
|
5. Caches in `DETAIL_CACHE` (~300s)
|
|||
|
|
|
|||
|
|
### 3.3 Chat
|
|||
|
|
|
|||
|
|
`POST /api/chat`:
|
|||
|
|
|
|||
|
|
1. Builds compact fleet system prompt (`build_fleet_context`, char-budgeted for 4096-token models)
|
|||
|
|
2. Routes:
|
|||
|
|
- `llama3-70b-gptq` (and similar) → direct vLLM `/v1/chat/completions`
|
|||
|
|
- `ome-copilot` / `arena-model` → Open WebUI `/api/chat/completions` (service login)
|
|||
|
|
3. Retries with tighter context on vLLM context-length 400 errors
|
|||
|
|
|
|||
|
|
### 3.4 SSH bridge
|
|||
|
|
|
|||
|
|
Browser `ssh.js` (xterm) → `WS /ws/ssh`:
|
|||
|
|
|
|||
|
|
1. First message: JSON `{type:"auth", host, port, username, password, cols, rows}`
|
|||
|
|
2. Server allowlists `host` against current fleet IPs only
|
|||
|
|
3. `asyncssh.connect` + interactive shell process
|
|||
|
|
4. Bidirectional pump; resize via JSON control messages
|
|||
|
|
5. Concurrency: global semaphore (25) + max 3 sessions per client IP; idle timeout 30 min
|
|||
|
|
|
|||
|
|
### 3.5 GPU telemetry
|
|||
|
|
|
|||
|
|
`gpu_loop` polls `GPU_METRICS_URL/api/gpu` every ~2s and broadcasts `{type:"gpu"}` (and embeds GPU in fleet snapshots).
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 4. Frontend architecture
|
|||
|
|
|
|||
|
|
| File | Responsibility |
|
|||
|
|
|------|----------------|
|
|||
|
|
| `index.html` | App chrome, rails, stage, drawers, modals (KPI, triage, connect, SSH, tickets) |
|
|||
|
|
| `app.js` | Canvas topology, camera, filters, KPIs, inspector, inventory render, KPI popup |
|
|||
|
|
| `ops.js` | Ops desk CRUD, chat drawer, triage modal, live-feed collapse/drag, GPU matrix |
|
|||
|
|
| `ssh.js` | SSH modal + xterm session lifecycle |
|
|||
|
|
| `styles.css` | Design tokens + component styles |
|
|||
|
|
| `vendor/xterm/*` | Local terminal assets (CDN-independent) |
|
|||
|
|
|
|||
|
|
### State
|
|||
|
|
|
|||
|
|
- Primary client state lives in `app.js` `state` (data, filters, camera, layoutMode, inventoryCache, …)
|
|||
|
|
- Shared bridge: `window.cockpit` and `window.cockpitSsh`
|
|||
|
|
|
|||
|
|
### Rendering
|
|||
|
|
|
|||
|
|
- Topology is an **immediate-mode canvas** (`requestAnimationFrame`), not DOM nodes for devices
|
|||
|
|
- Camera = translate + scale; layouts write target coordinates (`tx`/`ty`) with lerp for motion
|
|||
|
|
- Pulse packets on links are decorative; toggle via **Animate link pulse**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 5. Persistence
|
|||
|
|
|
|||
|
|
| Store | Path | Contents |
|
|||
|
|
|-------|------|----------|
|
|||
|
|
| SQLite | `/data/ops.db` (`COCKPIT_DATA`) | Tickets + ticket_messages |
|
|||
|
|
| Browser localStorage | client | `feed_collapsed`, `feed_bottom`, `cockpit_ssh_user`, `cockpit_chat_model`, actor |
|
|||
|
|
| In-memory | process | Fleet STATE, DETAIL_CACHE, SSH_SESSIONS, EVENT_FEED |
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 6. External contracts
|
|||
|
|
|
|||
|
|
### OME (OpenManage Enterprise 4.7)
|
|||
|
|
|
|||
|
|
- Session auth header `X-Auth-Token`
|
|||
|
|
- Device types: server `Type == 1000`; iDRAC via `SubDeviceType == "iDRAC"`
|
|||
|
|
- Inventory via per-device `InventoryTypes` + `InventoryDetails`
|
|||
|
|
|
|||
|
|
### vLLM OpenAI-compatible API
|
|||
|
|
|
|||
|
|
- `GET /v1/models`, `POST /v1/chat/completions`
|
|||
|
|
- Lab model `llama3-70b-gptq` with **max context 4096** → prompt budgeting is mandatory
|
|||
|
|
|
|||
|
|
### Open WebUI
|
|||
|
|
|
|||
|
|
- Sign-in → Bearer token (cached ~1h)
|
|||
|
|
- `GET /api/models`, `POST /api/chat/completions`
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 7. Failure domains
|
|||
|
|
|
|||
|
|
| Failure | UX impact | Mitigation |
|
|||
|
|
|---------|-----------|------------|
|
|||
|
|
| OME down | Stale/empty fleet | Health + last `updated_at`; poll retries |
|
|||
|
|
| vLLM 400 context | Chat errors | Slim context + retry |
|
|||
|
|
| Open WebUI auth missing | Virtual models unavailable | Falls back to vLLM list / error message |
|
|||
|
|
| SSH to non-fleet IP | Rejected | Allowlist |
|
|||
|
|
| Multi-user SSH overload | Queue/reject | Semaphore + per-IP cap |
|
|||
|
|
| Soft inspector refresh | Inventory preserved | `softRefreshInspector` + inventory cache |
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 8. Related systems (out of repo)
|
|||
|
|
|
|||
|
|
- **OpenManage AI** (`mo/openmanage-ai`): Open WebUI branding + OME REST tool for demos
|
|||
|
|
- Native OME MCP at `https://ome/mcp` exists but session stickiness is unreliable in this lab — Cockpit does not depend on it
|