ATC/ome-cockpit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document full architecture, design, API, features, deployment, and security.

Browse Source 
Adds diagrams and expands the ATC demo path so the cockpit is fully described in-repo.
This commit is contained in:
mo
2026-07-17 04:04:32 +02:00
parent b73ff9465c
commit c2bb0465d0
9 changed files with 951 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DEMO.md
+24 -8
View File
@@ -1,11 +1,27 @@
# OME Cockpit — demo path
# OME Cockpit — ATC demo path
URL: http://atc-portal01.dell-atc.lan:3090
**URL:** http://atc-portal01.dell-atc.lan:3090
**AI companion:** http://atc-portal01.dell-atc.lan:3080 (OpenManage AI)
1. Watch live pulse lines from OME hub to connected iDRACs/servers
2. Filter a subnet (10.0.40 / 10.0.41) in the left rail
3. Click a glowing node → inspector (IP, model, service tag, power)
4. Click **OpenManage AI** → chat drawer with OME Copilot
5. Use **Connected only** to focus the live fabric
## 5-minute walkthrough
OpenManage AI standalone remains at :3080
1. **Topology** — Watch pulse lines from OME hub to connected nodes; try Orbit → Galaxy → Clusters.
2. **KPI popup** — Click **Offline** or **Live power**; search a host; use **Focus on map** / **SSH terminal**.
3. **Filters** — Enable **Connected only**; confirm empty-map behavior if Servers+iDRACs both off; **Reset filters**.
4. **Inspector** — Select a connected node → **Load full inventory + app landscape** (firmware/software).
5. **SSH** — Quick Connect → SSH terminal → log in with **your** username (not root).
6. **Chat** — Cockpit chat → pick `llama3-70b-gptq` or `OME Copilot` → “List critical alerts with IP”.
7. **Ops desk** — Create a ticket as Jody, accept as Laurens.
8. **New node story** — When OME gains a device after cockpit baseline, a toast/card appears.
## Talking points
- Single pane for OME fleet + GPU + LLM + remote shell
- Context stays live via WebSocket snapshots (~12–15s OME poll, ~2s GPU)
- Design is Dell-branded mission control, not a spreadsheet
## Related docs
- [ARCHITECTURE.md](docs/ARCHITECTURE.md)
- [DESIGN.md](docs/DESIGN.md)
- [FEATURES.md](docs/FEATURES.md)
README.md
+85 -14
View File
@@ -1,23 +1,94 @@
# Dell OpenManage Cockpit (ATC)
Interactive fleet operations UI for Dell OpenManage Enterprise.
Interactive operations cockpit for **Dell OpenManage Enterprise** in the Dell ATC lab.
## Features
- Live topology (Orbit / Galaxy / Clusters / Lanes / Helix)
- KPI context popups, triage, ops desk tickets
- In-browser SSH terminal (own username/password)
- Cockpit chat against vLLM / OpenManage AI models
- Full inventory + application/firmware landscape from OME
**Live URL (lab):** http://atc-portal01.dell-atc.lan:3090
**Forgejo:** http://atc-mgt01.dell-atc.lan:3001/mo/ome-cockpit
**Companion AI stack:** http://atc-mgt01.dell-atc.lan:3001/mo/openmanage-ai · http://atc-portal01.dell-atc.lan:3080
---
## What it is
A single-pane ops UI for ATC admins (Jody van Dongen, Laurens Rammers, Mohamed El Kadi) that:
- Polls OME for devices, power, alerts, and groups
- Renders a live topology with multiple layouts and color modes
- Surfaces KPI context popups, realtime triage, and an ops ticket desk
- Embeds an in-browser SSH terminal (user-chosen credentials)
- Chats against vLLM / OpenManage AI models with a live fleet context pack
- Loads full OME inventory + firmware/software landscape per device
---
## Documentation map
| Doc | Contents |
|-----|----------|
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | System context, components, data flows, protocols |
| [docs/DESIGN.md](docs/DESIGN.md) | Visual language, layout, interaction design |
| [docs/API.md](docs/API.md) | HTTP + WebSocket API reference |
| [docs/FEATURES.md](docs/FEATURES.md) | Feature catalog (UI + backend behaviors) |
| [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) | Install, env, ports, ops, troubleshooting |
| [docs/SECURITY.md](docs/SECURITY.md) | Secrets, SSH allowlist, multi-user limits |
| [DEMO.md](DEMO.md) | Short demo script for ATC walkthroughs |
---
## Quick start
## Run
```bash
cp .env.example .env # fill OME + vLLM settings
cp .env.example .env
# Fill OME_URL / OME_USER / OME_PASSWORD, VLLM_URL, GPU_METRICS_URL, OPENWEBUI_*
docker compose up -d --build
```
UI: `http://<host>:3090`
Open http://localhost:3090
## Layout
- `api/main.py` — FastAPI (OME poll, chat, SSH bridge, tickets)
- `ui/` — static frontend
- `docker-compose.yml` / `Dockerfile`
Hot-reload: `ui/` and `api/main.py` are bind-mounted read-only into the container.
---
## Repository layout
```
ome-cockpit/
├── api/
│ ├── main.py # FastAPI app (OME poll, WS, chat, SSH, tickets)
│ ├── requirements.txt
│ └── Dockerfile # (legacy/alternate; root Dockerfile is used)
├── ui/
│ ├── index.html # Shell + modals
│ ├── app.js # Topology canvas, KPIs, filters, inspector
│ ├── ops.js # Ops desk, chat drawer, triage, feed collapse
│ ├── ssh.js # In-browser SSH client (xterm)
│ ├── styles.css
│ ├── vendor/xterm/ # Vendored xterm + fit addon (no CDN dependency)
│ └── dell*.png|svg # Branding
├── data/ # Runtime SQLite (gitignored)
├── docker-compose.yml
├── Dockerfile
├── .env.example
└── docs/
```
---
## Lab topology (reference)
```
┌────────────────────┐ poll/REST ┌──────────────────────────┐
│ ome-cockpit :3090 │ ───────────────► │ cov-omeprod01 (OME 4.7) │
│ atc-portal01 │ │ 10.0.21.104 │
└────────┬───────────┘ └──────────────────────────┘
│
├─ chat/completions ──► vLLM atc-gpu-prod :8000 (llama3-70b-gptq)
├─ /api/gpu ──────────► GPU metrics sidecar :9110
└─ optional models ───► OpenManage AI Open WebUI :3080
```
---
## License / ownership
Internal Dell ATC lab tooling. Not for public redistribution of credentials or production secrets.
docs/API.md
+203
View File
@@ -0,0 +1,203 @@
# API reference
Base URL (lab): `http://atc-portal01.dell-atc.lan:3090`
Internal container port: `8090`
All JSON unless noted. No auth middleware in lab build (protect at network edge).
---
## Health & fleet
### `GET /api/health`
```json
{ "status": "ok", "updated_at": 1784252683.4, "pulse": 12 }
```
### `GET /api/fleet`
Full in-memory snapshot:
| Field | Description |
|-------|-------------|
| `ome` | Appliance name/version/url |
| `summary` | totals, connected, offline, watts, samples |
| `devices[]` | normalized nodes |
| `subnets[]` | /24 aggregates |
| `groups[]` | OME groups (names) |
| `models[]` | server model counts |
| `alerts[]` | recent alert feed |
| `events[]` | fleet deltas (incl. device_new/removed) |
| `context` | alert_severity, hottest, notifications, … |
| `gpu` | latest GPU matrix |
| `pulse` | monotonic refresh counter |
| `openwebui_url` | companion AI URL |
### `WS /ws/fleet`
- On connect: immediate `{type:"snapshot", data}`
- Then server pushes `{type:"snapshot"}` / `{type:"gpu"}` / `{type:"error"}`
- Client may send any text ping; server mainly broadcasts
---
## Devices
### `GET /api/devices/{device_id}`
On-demand OME inventory enrichment.
Response highlights:
```json
{
"device": { "...fleet node..." },
"power": { "watts": 320, "avg_watts": 300, "peak_watts": 400, "energy_kwh": 12.3 },
"inventory_types": ["serverProcessors", "deviceSoftware", "..."],
"inventory": { "deviceSoftware": [ /* raw OME rows */ ], "...": [] },
"landscape": {
"os": [],
"software": [],
"firmware": [],
"drivers": [],
"applications": [],
"management": [],
"licenses": []
},
"_ts": 1784252700.0
}
```
Cached ~300 seconds per device.
---
## GPU
### `GET /api/gpu`
Latest GPU sidecar payload (also present on fleet snapshot).
---
## Chat & models
### `GET /api/models`
```json
{
"default": "llama3-70b-gptq",
"models": [
{ "id": "llama3-70b-gptq", "name": "llama3-70b-gptq", "source": "openwebui|vllm" },
{ "id": "ome-copilot", "name": "OME Copilot", "source": "openwebui" }
]
}
```
### `POST /api/chat`
```json
{
"message": "List critical systems",
"history": [{ "role": "user", "content": "..." }],
"focus_device_id": 50374,
"model": "llama3-70b-gptq"
}
```
Response:
```json
{
"reply": "...",
"model": "llama3-70b-gptq",
"backend": "vllm|openwebui",
"gpu": {},
"context_bytes": 4104
}
```
Errors: `502` with upstream detail (context length, auth, etc.).
---
## Tickets (Ops desk)
Admins (fixed lab list): `jody`, `laurens` (+ names in API).
| Method | Path | Purpose |
|--------|------|---------|
| GET | `/api/admins` | Admin directory |
| GET | `/api/tickets` | List tickets |
| POST | `/api/tickets` | Create |
| GET | `/api/tickets/{id}` | Detail + messages |
| POST | `/api/tickets/{id}/messages` | Add note |
| PATCH | `/api/tickets/{id}` | status / assignee / priority / accepted_by |
| DELETE | `/api/tickets/{id}` | Delete |
SQLite file: `$COCKPIT_DATA/ops.db` (default `/data/ops.db`).
---
## SSH
### `WS /ws/ssh`
**Message 1 (client → server):**
```json
{
"type": "auth",
"host": "10.0.41.50",
"port": 22,
"username": "jody",
"password": "***",
"cols": 120,
"rows": 40,
"term": "xterm-256color"
}
```
**Server control messages:** `{type:"status|ready|error|pong", message?: string}`
**Thereafter:** raw terminal bytes as WebSocket text.
**Client control (optional JSON):**
```json
{ "type": "resize", "cols": 100, "rows": 30 }
{ "type": "ping" }
```
Constraints:
- `host` must appear as a management IP in the current fleet snapshot
- Max 25 concurrent bridges; max 3 per client IP
- Idle timeout ~30 minutes
### `GET /api/ssh/sessions`
Ops visibility (no secrets):
```json
{ "active": 2, "max": 25, "sessions": [{ "id": "...", "client": "10.0.10.65", "host": "10.0.41.50", "user": "jody", "age_sec": 42 }] }
```
---
## Static assets
| Path | File |
|------|------|
| `/` | `ui/index.html` (no-store cache) |
| `/app.js` `/ops.js` `/ssh.js` `/styles.css` | UI |
| `/vendor/xterm/*` | Terminal library |
| `/dell.png` etc. | Branding |
---
## Environment (settings)
Mapped via `pydantic-settings` from `.env` (see `.env.example`):
`OME_URL`, `OME_USER`, `OME_PASSWORD`, `POLL_INTERVAL`, `OPENWEBUI_URL`, `OPENWEBUI_EMAIL`, `OPENWEBUI_PASSWORD`, `GPU_METRICS_URL`, `VLLM_URL`, `VLLM_MODEL`, `VLLM_MAX_TOKENS`, `CHAT_SYSTEM_CHARS`, `COCKPIT_DATA`, `CORS_ORIGINS`
docs/ARCHITECTURE.md
+163
View File
@@ -0,0 +1,163 @@
# Architecture
## 1. System context
OME Cockpit is a **BFF + SPA** running on `atc-portal01` (Docker). It does **not** embed OME; it continuously aggregates OME REST data, enriches with GPU telemetry and LLM chat, and pushes live snapshots to browsers over WebSocket.
### Actors
| Actor | Role |
|-------|------|
| ATC admin (browser) | Operates fleet via UI |
| OpenManage Enterprise | Source of truth for devices/alerts/inventory |
| vLLM (`atc-gpu-prod:8000`) | LLM inference for Cockpit chat |
| Open WebUI (`:3080`) | Model catalog + virtual models (`ome-copilot`, `arena-model`) |
| GPU metrics sidecar (`:9110`) | NVIDIA V100 utilization/power |
| Target servers / iDRACs | SSH endpoints opened from the browser via cockpit bridge |
---
## 2. Runtime components
```
┌─────────────────────────────────────────────────────────────┐
│ Docker: ome-cockpit │
│ uvicorn → FastAPI (main.py) :8090 → host :3090 │
│ │
│ Background tasks: │
│ • poll_loop() ~POLL_INTERVAL (default 12–15s) │
│ • gpu_loop() ~2s │
│ │
│ In-memory STATE + WebSocket fan-out │
│ SQLite tickets → /data/ops.db │
│ Static UI → /ui (bind-mount) │
└─────────────────────────────────────────────────────────────┘
```
### Process model
- **Single uvicorn worker** (important): fleet `STATE`, WebSocket client sets, and SSH session registry are in-process memory. Do not scale to multiple workers without an external shared store.
- UI assets and `main.py` are bind-mounted for lab hot-reload; image rebuild needed when Python dependencies change (`asyncssh`, etc.).
---
## 3. Data flows
### 3.1 Fleet snapshot (OME → UI)
1. `poll_loop` authenticates to OME `SessionService/Sessions`
2. Fetches devices, groups, alerts (paged `$top`)
3. For a capped set of connected servers, fetches `/Power`
4. Normalizes into cockpit device nodes (`is_server`, `is_idrac`, subnet, watts, …)
5. Diffs against `PREV_DEVICES` → `EVENT_FEED` (connection/power/status + **device_new** / **device_removed**)
6. Writes `STATE`, broadcasts `{type:"snapshot", data: STATE}` to all `/ws/fleet` clients
Browser `app.js` applies snapshot → layout → canvas draw loop; `ops.js` updates GPU, tickets, notifications.
### 3.2 Device inventory (on demand)
`GET /api/devices/{id}`:
1. Resolves node from current `STATE`
2. Opens OME session
3. Reads `InventoryTypes` for that device and pulls each `InventoryDetails('…')`
4. Builds normalized `landscape` (OS, firmware, drivers, applications, management, licenses)
5. Caches in `DETAIL_CACHE` (~300s)
### 3.3 Chat
`POST /api/chat`:
1. Builds compact fleet system prompt (`build_fleet_context`, char-budgeted for 4096-token models)
2. Routes:
- `llama3-70b-gptq` (and similar) → direct vLLM `/v1/chat/completions`
- `ome-copilot` / `arena-model` → Open WebUI `/api/chat/completions` (service login)
3. Retries with tighter context on vLLM context-length 400 errors
### 3.4 SSH bridge
Browser `ssh.js` (xterm) → `WS /ws/ssh`:
1. First message: JSON `{type:"auth", host, port, username, password, cols, rows}`
2. Server allowlists `host` against current fleet IPs only
3. `asyncssh.connect` + interactive shell process
4. Bidirectional pump; resize via JSON control messages
5. Concurrency: global semaphore (25) + max 3 sessions per client IP; idle timeout 30 min
### 3.5 GPU telemetry
`gpu_loop` polls `GPU_METRICS_URL/api/gpu` every ~2s and broadcasts `{type:"gpu"}` (and embeds GPU in fleet snapshots).
---
## 4. Frontend architecture
| File | Responsibility |
|------|----------------|
| `index.html` | App chrome, rails, stage, drawers, modals (KPI, triage, connect, SSH, tickets) |
| `app.js` | Canvas topology, camera, filters, KPIs, inspector, inventory render, KPI popup |
| `ops.js` | Ops desk CRUD, chat drawer, triage modal, live-feed collapse/drag, GPU matrix |
| `ssh.js` | SSH modal + xterm session lifecycle |
| `styles.css` | Design tokens + component styles |
| `vendor/xterm/*` | Local terminal assets (CDN-independent) |
### State
- Primary client state lives in `app.js` `state` (data, filters, camera, layoutMode, inventoryCache, …)
- Shared bridge: `window.cockpit` and `window.cockpitSsh`
### Rendering
- Topology is an **immediate-mode canvas** (`requestAnimationFrame`), not DOM nodes for devices
- Camera = translate + scale; layouts write target coordinates (`tx`/`ty`) with lerp for motion
- Pulse packets on links are decorative; toggle via **Animate link pulse**
---
## 5. Persistence
| Store | Path | Contents |
|-------|------|----------|
| SQLite | `/data/ops.db` (`COCKPIT_DATA`) | Tickets + ticket_messages |
| Browser localStorage | client | `feed_collapsed`, `feed_bottom`, `cockpit_ssh_user`, `cockpit_chat_model`, actor |
| In-memory | process | Fleet STATE, DETAIL_CACHE, SSH_SESSIONS, EVENT_FEED |
---
## 6. External contracts
### OME (OpenManage Enterprise 4.7)
- Session auth header `X-Auth-Token`
- Device types: server `Type == 1000`; iDRAC via `SubDeviceType == "iDRAC"`
- Inventory via per-device `InventoryTypes` + `InventoryDetails`
### vLLM OpenAI-compatible API
- `GET /v1/models`, `POST /v1/chat/completions`
- Lab model `llama3-70b-gptq` with **max context 4096** → prompt budgeting is mandatory
### Open WebUI
- Sign-in → Bearer token (cached ~1h)
- `GET /api/models`, `POST /api/chat/completions`
---
## 7. Failure domains
| Failure | UX impact | Mitigation |
|---------|-----------|------------|
| OME down | Stale/empty fleet | Health + last `updated_at`; poll retries |
| vLLM 400 context | Chat errors | Slim context + retry |
| Open WebUI auth missing | Virtual models unavailable | Falls back to vLLM list / error message |
| SSH to non-fleet IP | Rejected | Allowlist |
| Multi-user SSH overload | Queue/reject | Semaphore + per-IP cap |
| Soft inspector refresh | Inventory preserved | `softRefreshInspector` + inventory cache |
---
## 8. Related systems (out of repo)
- **OpenManage AI** (`mo/openmanage-ai`): Open WebUI branding + OME REST tool for demos
- Native OME MCP at `https://ome/mcp` exists but session stickiness is unreliable in this lab — Cockpit does not depend on it
docs/DEPLOYMENT.md
+114
View File
@@ -0,0 +1,114 @@
# Deployment
## Lab hosts
| Host | Role | Notes |
|------|------|-------|
| `atc-portal01` `10.0.21.49` | Runs cockpit + OpenManage AI containers | Primary deploy target |
| `cov-omeprod01` `10.0.21.104` | OME 4.7.0 | HTTPS, lab admin account |
| `atc-gpu-prod` `10.0.10.106` | vLLM `:8000`, GPU metrics `:9110` | Model `llama3-70b-gptq` |
| `atc-mgt01` | Forgejo `:3001` | Source of truth for git |
**Paths on portal**
- Cockpit: `/home/lab/ome-cockpit` (symlink/convention; compose here)
- OpenManage AI: `/home/lab/openmanage`
**Published ports**
- Cockpit UI/API: **3090 → 8090**
- OpenManage AI: **3080**
---
## Install / upgrade
```bash
cd /home/lab/ome-cockpit
cp .env.example .env # first time
# edit secrets
docker compose up -d --build
docker compose ps
curl -sS http://127.0.0.1:3090/api/health
```
Dependency changes (e.g. `asyncssh`) require **image rebuild**.
UI/`main.py` edits apply via bind mounts; hard-refresh browsers (`Ctrl+Shift+R`). Index is served `Cache-Control: no-store`.
---
## Environment reference
See `.env.example`. Critical keys:
| Variable | Purpose |
|----------|---------|
| `OME_URL` | e.g. `https://cov-omeprod01.dell-atc.lan` |
| `OME_USER` / `OME_PASSWORD` | API session |
| `POLL_INTERVAL` | Seconds between OME polls |
| `VLLM_URL` | e.g. `http://10.0.10.106:8000/v1` |
| `VLLM_MODEL` | Default chat model id |
| `VLLM_MAX_TOKENS` | Completion cap (lab ~700) |
| `CHAT_SYSTEM_CHARS` | System prompt budget (~5500) |
| `GPU_METRICS_URL` | e.g. `http://10.0.10.106:9110` |
| `OPENWEBUI_URL` | Companion AI |
| `OPENWEBUI_EMAIL` / `OPENWEBUI_PASSWORD` | Service login for model list + virtual models |
| `COCKPIT_DATA` | SQLite directory (default `/data`) |
**Never commit `.env`.**
---
## Compose volumes
```yaml
volumes:
- ./ui:/ui:ro
- ./api/main.py:/app/main.py:ro
- ./data:/data
```
---
## Health checks
```bash
curl -sS http://127.0.0.1:3090/api/health
curl -sS http://127.0.0.1:3090/api/fleet | python3 -c "import sys,json;d=json.load(sys.stdin);print(d['summary'])"
curl -sS http://127.0.0.1:3090/api/ssh/sessions
docker logs ome-cockpit --tail 100
```
---
## Git workflow
Remote: `http://atc-mgt01.dell-atc.lan:3001/mo/ome-cockpit.git`
```bash
cd /home/lab/ome-cockpit
git status
# commit after reviewing diffs — never stage .env / data
git push origin main # requires Forgejo credentials/token for user mo
```
---
## Troubleshooting
| Symptom | Check |
|---------|-------|
| Empty topology | Filters (Servers/iDRACs), OME creds, `docker logs` |
| Chat 400/502 | Context size; vLLM up; try Reset chat / smaller history |
| SSH modal blank | `/vendor/xterm/xterm.min.js` 200; hard refresh |
| SSH “not in fleet” | Device has no management IP in snapshot |
| Inventory vanishes | Should be fixed via soft refresh; hard reload if old JS cached |
| GPU empty | Sidecar `:9110`, firewall from portal → GPU host |
---
## Backup
- Code: Forgejo
- Tickets: backup `/home/lab/ome-cockpit/data/ops.db`
- Secrets: out-of-band password manager — not git
docs/DESIGN.md
+160
View File
@@ -0,0 +1,160 @@
# Design
## 1. Product intent
OME Cockpit is a **mission-control surface**, not a generic admin table. The first viewport must read as one composition: brand + live fleet fabric + actionable chrome.
Primary users: ATC admins who need to **see**, **triage**, **connect**, and **handoff** quickly during demos and live ops.
---
## 2. Visual language
### Brand
- Dell mark (`dell.png`) is a first-class signal in the top bar and modal headers
- Brand click returns to cockpit home (`/`), never external dell.com
### Palette (CSS variables in `styles.css`)
| Token | Role |
|-------|------|
| Dell blue / bright | Primary actions, brand accents |
| Cyan | Live/connected signals, KPI values |
| Amber | Warning / offline emphasis |
| Red | Critical |
| Deep navy panels | Rails, drawers, modals over dark stage |
### Typography
- **IBM Plex Sans** — UI chrome, headings
- **IBM Plex Mono** — IPs, KPIs, telemetry, terminal-adjacent metadata
### Motion
Intentional, not decorative noise:
1. Link pulse packets on connected OME edges
2. Node lerp when layouts/filters change
3. Notification cards slide in for new devices
Pulse can be disabled (**Animate link pulse**) for calm viewing.
---
## 3. Spatial layout
```
┌──────────────────────────────────────────────────────────────┐
│ TOPBAR: brand | KPI strip | Reset / OME / Ops / Chat │
├──────────┬───────────────────────────────┬───────────────────┤
│ LEFT │ STAGE │ RIGHT │
│ RAIL │ ctx-bar │ INSPECTOR │
│ viz │ canvas topology │ │
│ filters │ live feed (dockable) │ │
│ color │ legend / hints │ │
│ gpu │ │ │
│ alerts │ │ │
│ nets… │ │ │
├──────────┴───────────────────────────────┴───────────────────┤
│ TICKER: OME · pulse · watts · connected · alerts · visible │
└──────────────────────────────────────────────────────────────┘
```
Drawers (chat, ops, AI iframe) slide from the right over a scrim. Modals center for connect / SSH / triage / KPI context / tickets.
---
## 4. Interaction model
### Everything clickable
| Surface | Click behavior |
|---------|----------------|
| KPI tiles | Open **Fleet context** popup (list + detail + actions) |
| Topology node | Inspector; double-click → Quick Connect |
| Alerts / feed items | Triage modal / focus device |
| Ticker chips | Focus modes / triage |
| SSH actions | In-app terminal login (no hardcoded `root@`) |
### Filters (include semantics)
- **Servers** / **iDRACs** are include toggles
- **Both off ⇒ empty map** with on-canvas guidance (never silently show all)
- **Reset filters** restores defaults (Servers on, pulse on)
- **Reset view** re-layouts and fits camera to visible nodes
### Inspector inventory
Button **Load full inventory + app landscape** fetches OME inventory types and renders:
- OS, firmware landscape, software/components, management agents, licenses
- Hardware sections (CPU, memory, disks, RAID, NIC, PSU, FRU, location, health, …)
Inventory is **cached per device** and survives soft fleet refreshes (`softRefreshInspector`).
---
## 5. Topology design
### Layouts
| Mode | Idea |
|------|------|
| Orbit | Hub + concentric subnet rings |
| Galaxy | Spiral arms per subnet |
| Clusters | Subnet islands around OME |
| Lanes | Horizontal subnet highways |
| Helix | Depth coil sorted by connectivity/power |
### Color modes
Status · Power (W) · Connection · Model · Subnet
### Camera
Pan (drag), zoom (wheel), fit (`Reset view` / auto-fit on filter & layout changes).
---
## 6. Modals & drawers (design contracts)
### KPI Fleet context
- Summary pills (counts, watts, subnets)
- Search + sort
- Left list / right detail
- Actions: Focus, SSH, Quick Connect, Inventory, Chat, iDRAC Web
- Apply / clear map filter
### SSH terminal
- Login form first (host prefilled from fleet IP, username/password user-owned)
- Then xterm session
- Host must be in current OME fleet allowlist
### Ops desk
- Acting-as identity (Jody / Laurens)
- Ticket list filters, accept/assign/status/notes/delete
### Notifications
New/removed server or iDRAC → toast + stacked notify cards (click focuses device).
---
## 7. Accessibility & ops UX notes
- Modals use `aria-hidden` and Escape-to-close
- Scrim closes the active overlay mode (`kpi`, `ssh`, `connect`, drawers, …)
- Orange accent is reserved for **SSH / iDRAC connection affordances** (`.conn-link`), not general chrome
- Index served with `Cache-Control: no-store` to reduce stale-UI issues after deploys
---
## 8. Non-goals (design)
- Not a full CMDB or change-management suite
- Not a replacement for the OME HTML console (linked out)
- Not multi-region HA UI (single lab instance)
docs/DIAGRAMS.md
+75
View File
@@ -0,0 +1,75 @@
# Diagrams
## End-to-end context
```mermaid
flowchart LR
Admin[ATC Admin Browser]
Cockpit[OME Cockpit :3090]
OME[OpenManage Enterprise]
VLLM[vLLM llama3-70b]
GPU[GPU metrics :9110]
OWUI[OpenManage AI :3080]
Node[Server / iDRAC SSH]
Admin -->|HTTPS/HTTP + WS| Cockpit
Cockpit -->|REST session| OME
Cockpit -->|chat/completions| VLLM
Cockpit -->|models + virtual chat| OWUI
Cockpit -->|/api/gpu| GPU
Admin -->|xterm WS /ws/ssh| Cockpit
Cockpit -->|asyncssh| Node
```
## Fleet poll sequence
```mermaid
sequenceDiagram
participant P as poll_loop
participant O as OME
participant S as STATE
participant WS as Browser WS clients
P->>O: Session login
P->>O: Devices / Groups / Alerts / Power samples
P->>P: Normalize + diff PREV_DEVICES
P->>S: Update snapshot + events
P->>WS: broadcast snapshot
P->>O: Session delete
```
## SSH session
```mermaid
sequenceDiagram
participant UI as ssh.js / xterm
participant API as /ws/ssh
participant Target as Device SSHD
UI->>API: auth JSON (user/pass/host)
API->>API: allowlist host in fleet
API->>Target: asyncssh connect + shell
API-->>UI: ready
loop Terminal I/O
UI->>API: keystrokes
API->>Target: stdin
Target-->>API: stdout
API-->>UI: terminal text
end
```
## Frontend module map
```mermaid
flowchart TB
HTML[index.html shell]
APP[app.js topology + KPI + inspector]
OPS[ops.js tickets + chat + triage + feed]
SSH[ssh.js terminal]
HTML --> APP
HTML --> OPS
HTML --> SSH
APP <-->|window.cockpit| OPS
APP --> SSH
OPS --> SSH
```
docs/FEATURES.md
+81
View File
@@ -0,0 +1,81 @@
# Features
## Topology & visualization
- Layouts: Orbit, Galaxy, Clusters, Lanes, Helix
- Color modes: Status, Power, Connection, Model, Subnet
- Live pulse packets on connected OME links (toggleable)
- Pan / zoom / Reset view (fit-to-fleet)
- Hover tips; click = inspector; double-click = Quick Connect
- Ghost offline links optional
## Filters & focus
- Include toggles: Servers, iDRACs (both off → empty map + message)
- Connected only, Powered on, Has power reading, Min watts, Search
- Network / model / group rails
- Clear / Reset filters restores defaults
- KPI **Apply as map filter** aligns map to the KPI cohort
## KPI fleet context popups
Every top KPI tile opens an interactive modal:
- Nodes, Servers, iDRACs, Connected, Offline, Powered on, Live power, Avg/node, Power samples
- Summary pills, search, sort
- Per-device actions: Focus, SSH, Quick Connect, Inventory, Chat, iDRAC Web
## Inspector & inventory
- Live identity, connection, power fields
- Soft-refresh on fleet poll (does not wipe loaded inventory)
- Full inventory + application/firmware landscape from OME `InventoryTypes`
- Inventory cache per device id
## Live feed & notifications
- Collapsible / lightly draggable live feed
- Merged events + alerts + hottest power
- **New device / removed device** toasts + notify cards when OME inventory changes after baseline poll
## Triage
- Critical / Warning / Events / Hottest / All feed
- Search, detail pane, jump to connect / chat / tickets
## Quick Connect
- iDRAC Web (new tab)
- SSH terminal (in-app)
- Ask AI / Inspector shortcuts
- No hardcoded `ssh://root@…` links
## In-browser SSH
- Username + password login popup (username remembered locally)
- xterm.js vendored locally
- Fleet IP allowlist; multi-user session caps
## Cockpit chat
- Model selector (Open WebUI catalog + vLLM)
- Fleet-aware system context (budgeted for 4096-token models)
- Quick prompts; focus device context
- Backends: vLLM direct or Open WebUI virtual models
## Ops desk
- Tickets for Jody / Laurens handoff
- Create, accept, assign, status, notes, delete
- Persisted in SQLite
## GPU matrix
- Live V100 cards from metrics sidecar
- Shown in left rail + available to chat context
## Branding / chrome
- Dell logo → cockpit home
- Ticker with pulse, power, alerts, visible count
- Drawers: Chat, Ops, OpenManage AI iframe
docs/SECURITY.md
+46
View File
@@ -0,0 +1,46 @@
# Security
Lab-oriented controls. This is **not** a hardened public multi-tenant SaaS.
## Secrets
| Secret | Location | Git |
|--------|----------|-----|
| OME password | `.env` | **Excluded** |
| Open WebUI service password | `.env` | **Excluded** |
| Ticket DB | `/data/ops.db` | **Excluded** |
| SSH user passwords | Never stored server-side; transient WS auth only | N/A |
Use `.env.example` as a template with `CHANGE_ME` placeholders.
## Network exposure
- Cockpit has **no application auth** in the current lab build — rely on ATC network segmentation / VPN / reverse proxy if exposed beyond trusted LAN.
- Prefer binding publish ports to trusted interfaces only in stricter environments.
## SSH bridge
- Destination host must match a **current fleet management IP**
- Concurrency caps: 25 global, 3 per client IP
- `known_hosts=None` for lab self-signed/changing keys — tighten for production
- Idle timeout terminates abandoned sessions
- Passwords are not logged; avoid verbose debug of auth payloads
## Chat / LLM
- Fleet context is sent to vLLM / Open WebUI — treat as sensitive operational data
- Service account for Open WebUI should be least-privilege if possible
## Multi-user stability (related)
- Single worker process shares STATE safely for reads with asyncio
- SSH sessions are isolated per WebSocket
- OME poll uses short-lived API sessions and deletes them after fetch
## Recommendations for hardening (future)
1. Add OIDC / reverse-proxy auth in front of `:3090`
2. Persist SSH host key policy / CA trust
3. Audit log for SSH session start/stop (user, target, client IP)
4. Encrypt ticket DB at rest if tickets contain sensitive notes
5. Rotate lab service passwords regularly