Full lab documentation and infrastructure as code

- Trino catalogs, Grafana, Spark jobs, LDAP LDIF, NPM compose - Airflow DAG scripts, Proxmox VM inventory, network docs - Ansible playbook, Gitea CI validate workflow - Backup and health-check scripts, cron documentation - Homepage DOCS tab with links to all documentation - Extended collect-fleet-config.sh and populate-repo.py
2026-05-19 23:12:41 +02:00
parent 3c8993a41e
commit df5ec93dc3
46 changed files with 1809 additions and 74 deletions
@@ -0,0 +1,64 @@
+# Network — ATC Lakehouse lab
+
+## Subnets
+
+```mermaid
+flowchart TB
+  subgraph mgmt [10.0.10.0/24 — Management]
+    PVE[pve01 · 10.0.10.65]
+    IDRAC[iDRAC · 10.0.41.102]
+  end
+
+  subgraph data20 [10.0.20.0/24 — Storage & legacy]
+    OS[ObjectScale · 10.0.20.111]
+    DB1[atc-db01 · 10.0.20.112]
+    MGT[atc-mgt01 · 10.0.20.104]
+    GF[atc-grafana · 10.0.20.103]
+  end
+
+  subgraph lake21 [10.0.21.0/24 — Lakehouse compute]
+    D1[atc-docker01 · 10.0.21.45]
+    D2[atc-docker02 · 10.0.21.47]
+    KF[atc-kafka01 · 10.0.21.36]
+    LK[atc-lake01 · 10.0.21.50]
+    DB2[atc-db02 · 10.0.21.51]
+    AF[atc-airflow01 · 10.0.21.55]
+    EL[atc-elastic01 · 10.0.21.46]
+  end
+
+  PVE --> D1 & D2 & KF & LK & DB2 & OS & DB1 & MGT
+```
+
+## DNS
+
+Internal names: `*.dell-atc.lan` (see `config/hosts/atc-lab.hosts`).
+
+| Pattern | Example |
+|---------|---------|
+| Short hostname | `atc-lake01` |
+| FQDN | `atc-lake01.dell-atc.lan` |
+
+## Key ports (east-west)
+
+| Service | Port | Protocol |
+|---------|------|----------|
+| Kafka | 9092 | TCP |
+| Debezium Connect | 8083 | HTTP |
+| Spark UI | 8080 | HTTP |
+| Trino | 8089 | HTTP |
+| PostgreSQL | 5432 | TCP |
+| ObjectScale S3 | 9020 | HTTP |
+| ObjectScale UI | 443 | HTTPS |
+| Homepage | 80 | HTTP |
+
+## SSH mesh
+
+All lab VMs: `/root/.ssh/atc_cluster` — see [ssh-mesh.md](ssh-mesh.md).
+
+ObjectScale uses `admin@` for initial bootstrap; `root@` after key install.
+
+## Firewall notes
+
+- Lab assumes flat L2/L3 trust within `10.0.10/20/21.x`.
+- ObjectScale `management_clients: 0.0.0.0/0` in deploy.yml — tighten for production.
+- Proxmox API `:8006` reachable from docker01 for homepage widget.