ome-cockpit/docs/ARCHITECTURE.md

Architecture

1. System context

OME Cockpit is a BFF + SPA running on atc-portal01 (Docker). It does not embed OME; it continuously aggregates OME REST data, enriches with GPU telemetry and LLM chat, and pushes live snapshots to browsers over WebSocket.

Actors

Actor	Role
ATC admin (browser)	Operates fleet via UI
OpenManage Enterprise	Source of truth for devices/alerts/inventory
vLLM (atc-gpu-prod:8000)	LLM inference for Cockpit chat
Open WebUI (:3080)	Model catalog + virtual models (ome-copilot, arena-model)
GPU metrics sidecar (:9110)	NVIDIA V100 utilization/power
Target servers / iDRACs	SSH endpoints opened from the browser via cockpit bridge

---

2. Runtime components

┌─────────────────────────────────────────────────────────────┐
│ Docker: ome-cockpit                                         │
│  uvicorn → FastAPI (main.py) :8090  → host :3090            │
│                                                             │
│  Background tasks:                                          │
│   • poll_loop()   ~POLL_INTERVAL (default 12–15s)           │
│   • gpu_loop()    ~2s                                       │
│                                                             │
│  In-memory STATE + WebSocket fan-out                        │
│  SQLite tickets  → /data/ops.db                             │
│  Static UI       → /ui (bind-mount)                         │
└─────────────────────────────────────────────────────────────┘

Process model

• Single uvicorn worker (important): fleet STATE, WebSocket client sets, and SSH session registry are in-process memory. Do not scale to multiple workers without an external shared store.
• UI assets and main.py are bind-mounted for lab hot-reload; image rebuild needed when Python dependencies change (asyncssh, etc.).

---

3. Data flows

3.1 Fleet snapshot (OME → UI)

1. poll_loop authenticates to OME SessionService/Sessions
2. Fetches devices, groups, alerts (paged $top)
3. For a capped set of connected servers, fetches /Power
4. Normalizes into cockpit device nodes (is_server, is_idrac, subnet, watts, …)
5. Diffs against PREV_DEVICES → EVENT_FEED (connection/power/status + device_new / device_removed)
6. Writes STATE, broadcasts {type:"snapshot", data: STATE} to all /ws/fleet clients

Browser app.js applies snapshot → layout → canvas draw loop; ops.js updates GPU, tickets, notifications.

3.2 Device inventory (on demand)

GET /api/devices/{id}:

1. Resolves node from current STATE
2. Opens OME session
3. Reads InventoryTypes for that device and pulls each InventoryDetails('…')
4. Builds normalized landscape (OS, firmware, drivers, applications, management, licenses)
5. Caches in DETAIL_CACHE (~300s)

3.3 Chat

POST /api/chat:

1. Builds compact fleet system prompt (build_fleet_context, char-budgeted for 4096-token models)
2. Routes:
   • llama3-70b-gptq (and similar) → direct vLLM /v1/chat/completions
   • ome-copilot / arena-model → Open WebUI /api/chat/completions (service login)
3. Retries with tighter context on vLLM context-length 400 errors

3.4 SSH bridge

Browser ssh.js (xterm) → WS /ws/ssh:

1. First message: JSON {type:"auth", host, port, username, password, cols, rows}
2. Server allowlists host against current fleet IPs only
3. asyncssh.connect + interactive shell process
4. Bidirectional pump; resize via JSON control messages
5. Concurrency: global semaphore (25) + max 3 sessions per client IP; idle timeout 30 min

3.5 GPU telemetry

gpu_loop polls GPU_METRICS_URL/api/gpu every ~2s and broadcasts {type:"gpu"} (and embeds GPU in fleet snapshots).

---

4. Frontend architecture

File	Responsibility
index.html	App chrome, rails, stage, drawers, modals (KPI, triage, connect, SSH, tickets)
app.js	Canvas topology, camera, filters, KPIs, inspector, inventory render, KPI popup
ops.js	Ops desk CRUD, chat drawer, triage modal, live-feed collapse/drag, GPU matrix
ssh.js	SSH modal + xterm session lifecycle
styles.css	Design tokens + component styles
vendor/xterm/*	Local terminal assets (CDN-independent)

State

• Primary client state lives in app.js state (data, filters, camera, layoutMode, inventoryCache, …)
• Shared bridge: window.cockpit and window.cockpitSsh

Rendering

• Topology is an immediate-mode canvas (requestAnimationFrame), not DOM nodes for devices
• Camera = translate + scale; layouts write target coordinates (tx/ty) with lerp for motion
• Pulse packets on links are decorative; toggle via Animate link pulse

---

5. Persistence

Store	Path	Contents
SQLite	/data/ops.db (COCKPIT_DATA)	Tickets + ticket_messages
Browser localStorage	client	feed_collapsed, feed_bottom, cockpit_ssh_user, cockpit_chat_model, actor
In-memory	process	Fleet STATE, DETAIL_CACHE, SSH_SESSIONS, EVENT_FEED

---

6. External contracts

OME (OpenManage Enterprise 4.7)

• Session auth header X-Auth-Token
• Device types: server Type == 1000; iDRAC via SubDeviceType == "iDRAC"
• Inventory via per-device InventoryTypes + InventoryDetails

vLLM OpenAI-compatible API

• GET /v1/models, POST /v1/chat/completions
• Lab model llama3-70b-gptq with max context 4096 → prompt budgeting is mandatory

Open WebUI

• Sign-in → Bearer token (cached ~1h)
• GET /api/models, POST /api/chat/completions

---

7. Failure domains

Failure	UX impact	Mitigation
OME down	Stale/empty fleet	Health + last updated_at; poll retries
vLLM 400 context	Chat errors	Slim context + retry
Open WebUI auth missing	Virtual models unavailable	Falls back to vLLM list / error message
SSH to non-fleet IP	Rejected	Allowlist
Multi-user SSH overload	Queue/reject	Semaphore + per-IP cap
Soft inspector refresh	Inventory preserved	softRefreshInspector + inventory cache

---

8. Related systems (out of repo)

• OpenManage AI (mo/openmanage-ai): Open WebUI branding + OME REST tool for demos
• Native OME MCP at https://ome/mcp exists but session stickiness is unreliable in this lab — Cockpit does not depend on it