Document full application landscape and collect Docker configs

- Add docs/landscape.md and docker-inventory.md with Mermaid architecture - Add config/docker per host (db02, lake01, kafka01) and objectscale/elastic refs - Add live architecture page at icons/docs/architecture.html - Homepage OPS tab: Lakehouse Architecture group with diagram link - Add scripts/collect/collect-fleet-config.sh to refresh configs from fleet
2026-05-19 23:01:47 +02:00
parent e1180e14c9
commit 007f6cedc5
15 changed files with 881 additions and 42 deletions
@@ -0,0 +1,65 @@
+# Docker inventory
+
+Containers per host (reference). Regenerate with `./scripts/collect/collect-fleet-config.sh`.
+
+## atc-docker01 (10.0.21.45)
+
+| Container | Image | Ports |
+|-----------|-------|-------|
+| homepage | gethomepage/homepage | 80→3000 |
+| rss-proxy | rss-proxy:latest | 8090 |
+| icons-server | nginx:alpine | 8080→80 |
+| superset | superset-superset | 8088 |
+| superset_redis | redis:7 | 6379 |
+| forgejo | forgejo:14 | 4002, 222 |
+| lam-app-1 | ldapaccountmanager/lam | 4001 |
+
+Compose: `compose/`, `deploy/docker-compose.homepage.yml`
+
+## atc-db02 (10.0.21.51)
+
+| Container | Image | Ports |
+|-----------|-------|-------|
+| postgres_sales | postgres:15 | 5432 |
+| mysql_hr | mysql:8.0 | 3306 |
+| mongodb_supplychain | mongo:4.4 | 27017 |
+| cassandra_telemetry | cassandra:4.1 | 9042 |
+| neo4j_graph | neo4j:4.4 | 7474, 7687 |
+| mongo_express | mongo-express | 8081 |
+
+Compose: `config/docker/atc-db02/docker-compose.yml`
+
+## atc-lake01 (10.0.21.50)
+
+| Container | Image | Ports |
+|-----------|-------|-------|
+| kafka-connect | debezium/connect:2.5.4 | 8083 |
+| spark-master | bitnami/spark | 7077, 8080 |
+| spark-worker | bitnami/spark | (internal) |
+| trino | trinodb/trino:405 | 8089→8080 |
+| spark-temp | apache/spark:3.4.0 | — |
+
+Compose: `config/docker/atc-lake01/docker-compose.yml`
+
+## atc-kafka01 (10.0.21.36)
+
+| Container | Image | Ports |
+|-----------|-------|-------|
+| kafka-ui | provectuslabs/kafka-ui | 9000→8080 |
+
+Kafka broker: native on `:9092` (not containerized in current lab).
+
+## atc-mgt01 (10.0.20.104)
+
+| Container | Image | Ports |
+|-----------|-------|-------|
+| forgejo | forgejo:14 | 3001, 222 |
+| npm-app-1 | nginx-proxy-manager | 80, 443, 81 |
+
+## atc-objectscale (10.0.20.111)
+
+| Container | Image | Notes |
+|-----------|-------|-------|
+| ecs-storageos | emccorp/ecs-software | Dell ECS / ObjectScale |
+
+Config: `config/objectscale/deploy.yml`
@@ -0,0 +1,129 @@
+# ATC Lakehouse — Application Landscape
+
+High-level view of the Dell ATC Forward Deployed Engineering lab.  
+**Live diagram:** http://atc-docker01.dell-atc.lan:8080/docs/architecture.html
+
+## Architecture diagram
+
+```mermaid
+flowchart TB
+  subgraph users [Users]
+    FDE[FDE Engineers]
+  end
+
+  subgraph ops [Operations Layer]
+    HP[Homepage Dashboard<br/>atc-docker01:80]
+    GF[Grafana<br/>atc-grafana:3000]
+    PVE[Proxmox pve01<br/>10.0.10.65:8006]
+    IDRAC[iDRAC<br/>10.0.41.102]
+  end
+
+  subgraph git [Source Control]
+    FG[Forgejo Git<br/>atc-mgt01:3001]
+  end
+
+  subgraph ingest [Ingest & Streaming]
+    KF[Kafka<br/>atc-kafka01:9092]
+    KUI[Kafka UI<br/>:9000]
+    DEB[Debezium Connect<br/>atc-lake01:8083]
+  end
+
+  subgraph process [Processing]
+    SP[Spark Master/Worker<br/>atc-lake01:8080]
+    AF[Airflow<br/>atc-airflow01:8080]
+    TR[Trino<br/>atc-lake01:8089]
+  end
+
+  subgraph storage [Storage]
+  OS[ObjectScale ECS<br/>atc-objectscale 10.0.20.111]
+  end
+
+  subgraph sources [Source Databases — atc-db02]
+    PG[(PostgreSQL)]
+    MY[(MySQL)]
+    MG[(MongoDB)]
+    CA[(Cassandra)]
+    N4[(Neo4j)]
+  end
+
+  subgraph analytics [Analytics & Search]
+    SS[Superset<br/>atc-docker01:8088]
+    ES[Elasticsearch<br/>atc-elastic01:9200]
+    KB[Kibana<br/>:5601]
+  end
+
+  FDE --> HP
+  FDE --> SS
+  FDE --> KB
+  HP --> KUI
+  HP --> DEB
+  HP --> SP
+  HP --> TR
+  HP --> OS
+  HP --> FG
+
+  PG & MY & MG & CA --> DEB
+  DEB --> KF
+  KF --> SP
+  SP --> OS
+  TR --> PG
+  TR --> KF
+  KF --> SS
+  ES --> KB
+  AF --> SP
+```
+
+## Host roles
+
+| Host | IP | Role | Key services |
+|------|-----|------|--------------|
+| **atc-docker01** | 10.0.21.45 | Ops & BI hub | Homepage, Superset, RSS-proxy, icons, Forgejo (local), LAM |
+| **atc-docker02** | 10.0.21.47 | Secondary Docker | Standby / experiments |
+| **atc-mgt01** | 10.0.20.104 | Management | Forgejo (:3001), Nginx Proxy Manager, LDAP |
+| **atc-kafka01** | 10.0.21.36 | Streaming | Kafka broker (:9092), Kafka UI (:9000) |
+| **atc-lake01** | 10.0.21.50 | Lakehouse compute | Spark, Trino, Debezium Connect |
+| **atc-airflow01** | 10.0.21.55 | Orchestration | Apache Airflow (systemd) |
+| **atc-elastic01** | 10.0.21.46 | Search & observability | Elasticsearch, Kibana |
+| **atc-db01** | 10.0.20.112 | Source DBs | PostgreSQL, MySQL (native or container) |
+| **atc-db02** | 10.0.21.51 | Polyglot sources | PG, MySQL, Mongo, Cassandra, Neo4j (Docker) |
+| **atc-grafana** | 10.0.20.103 | Monitoring | Grafana |
+| **atc-objectscale** | 10.0.20.111 | Object storage | Dell ECS / S3 API (Luna OS) |
+| **pve01** | 10.0.10.65 | Hypervisor | Proxmox VE |
+
+## Data flow
+
+1. **Operational data** lives on **atc-db02** (and db01) — sales, HR, supply chain, telemetry, graph.
+2. **Debezium** on lake01 reads CDC from PostgreSQL → **Kafka** on kafka01.
+3. **Spark** processes streams/batch; results can land in **ObjectScale** (S3-compatible).
+4. **Trino** federates queries across catalogs (Postgres, Kafka, Hive, etc.).
+5. **Superset** on docker01 is the primary BI UI; **Kibana** for search/logs.
+6. **Airflow** orchestrates pipelines on airflow01.
+7. **Homepage** aggregates links, health, RSS intel, and Proxmox status.
+
+## Docker vs native
+
+| Component | Deployment | Config in git |
+|-----------|------------|---------------|
+| Homepage, Superset, RSS | Docker (docker01) | `compose/`, `deploy/` |
+| Spark, Trino, Debezium | Docker (lake01) | `config/docker/atc-lake01/` |
+| Source DBs | Docker (db02) | `config/docker/atc-db02/` |
+| Kafka UI | Docker (kafka01) | `config/docker/atc-kafka01/` |
+| Kafka broker | Native/systemd (kafka01) | `config/kafka/` |
+| Elasticsearch | Native (elastic01) | `config/elastic/` |
+| Airflow | Native (airflow01) | `config/airflow/` |
+| ObjectScale | ECS appliance | `config/objectscale/` |
+| Forgejo | Docker (mgt01) | `compose/forgejo/` |
+
+## Network zones
+
+- **Management:** 10.0.10.x — Proxmox, iDRAC
+- **Storage / legacy:** 10.0.20.x — ObjectScale, db01, mgt01, grafana, elastic
+- **Lakehouse compute:** 10.0.21.x — docker, kafka, lake, db02, airflow
+
+## Related docs
+
+- [Hosts & ports](hosts.md)
+- [Architecture layers](architecture.md)
+- [SSH mesh](ssh-mesh.md)
+- [Disaster recovery](disaster-recovery.md)
+- [Docker inventory](docker-inventory.md)