Docs-as-code: full deploy configs, compose stacks, and DR guide

- Add root README and docs/ (architecture, hosts, disaster-recovery, operations) - Add compose/ for superset, forgejo, lam, cadvisor; unified deploy compose - Add scripts/deploy (deploy-atc-docker01.sh, container inventory export) - Add inventory/containers-atc-docker01.json snapshot - Add config READMEs for kafka and airflow; .env.example; expand .gitignore - Sync atclab helper scripts; update homepage/superset compose with env support
2026-05-19 22:28:01 +02:00
parent c63181abac
commit f5d30824fc
30 changed files with 890 additions and 3 deletions
@@ -0,0 +1,67 @@
+# Architectuur — ATC Lakehouse
+
+## Dataflow (conceptueel)
+
+```mermaid
+flowchart LR
+  subgraph sources [Bronnen]
+    DB1[(atc-db01)]
+    DB2[(atc-db02)]
+  end
+
+  subgraph ingest [Ingest]
+    DEB[Debezium]
+    KAFKA[Kafka]
+  end
+
+  subgraph lake [Lake]
+    SPARK[Spark]
+    TRINO[Trino]
+    ICE[Iceberg tables]
+  end
+
+  subgraph consume [Consumptie]
+    SS[Superset]
+    KB[Kibana]
+    GF[Grafana]
+  end
+
+  DB1 --> DEB
+  DB2 --> DEB
+  DEB --> KAFKA
+  KAFKA --> SPARK
+  SPARK --> ICE
+  TRINO --> ICE
+  TRINO --> SS
+  KAFKA --> SS
+```
+
+## Lagen
+
+| Laag | Componenten | Doel |
+|------|-------------|------|
+| **Bron** | PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j | OLTP / events / graph demo data |
+| **Ingest** | Debezium, Kafka Connect | CDC naar Kafka topics |
+| **Processing** | Spark, Airflow | ETL, batch, orchestratie |
+| **Query** | Trino | Federated SQL over lake + bronnen |
+| **Storage** | ObjectScale / MinIO (indien actief) | Object storage |
+| **Analytics** | Superset, Kibana | BI en search UI |
+| **Platform** | Proxmox, Docker hosts | VMs en containers |
+| **Ops** | Homepage dashboard, Grafana | Zichtbaarheid en links |
+
+## atc-docker01 — centrale dashboard-host
+
+Draait de **gethomepage**-dashboard (poort 80), **RSS-proxy** (8090), **icon server** (8080), **Apache Superset** (8088), en optioneel lokale **Forgejo** (4002) / **LAM** (4001).
+
+Configuratiepad op de host: `/root/lakehouse/config/homepage` → gemount in container `/app/config`.
+
+## Configuratie in git
+
+| Pad | Beschrijving |
+|-----|--------------|
+| `config/kafka/*.json` | Debezium/Kafka Connect connector definities |
+| `config/trino/config.properties` | Trino coordinator (single-node) |
+| `config/spark/spark-defaults.conf` | Spark defaults |
+| `config/airflow/` | Airflow DAG + `airflow.cfg` |
+| `config/superset/` | Superset Python config + compose |
+| `config/homepage/` | Volledige dashboardconfiguratie |
@@ -0,0 +1,84 @@
+# Disaster recovery
+
+Doel: volledige **atc-docker01** dashboard-stack en configuratie terugzetten vanuit git.
+
+## Vereisten
+
+- VM `atc-docker01` met Docker en git
+- Netwerk naar interne `.dell-atc.lan` hosts
+- Secrets: `.env` of bestaande `config/homepage/proxmox.yaml`
+
+## Stap 1 — Repository
+
+```bash
+dnf install -y git docker-compose-plugin   # of equivalent
+git clone http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse.git /root/lakehouse
+cd /root/lakehouse
+cp .env.example .env
+# Bewerk .env: SUPERSET_SECRET_KEY, eventueel Proxmox token
+```
+
+## Stap 2 — Homepage + RSS + icons
+
+```bash
+docker compose -f deploy/docker-compose.homepage.yml up -d --build
+docker exec homepage wget -qO- http://127.0.0.1:3000/api/revalidate
+```
+
+## Stap 3 — Superset
+
+```bash
+docker compose -f compose/superset/docker-compose.yaml up -d --build
+```
+
+## Stap 4 — Optioneel: Forgejo, LAM, cAdvisor
+
+```bash
+docker compose -f compose/forgejo/compose.yaml up -d
+docker compose -f compose/lam/compose.yml up -d
+# docker compose -f compose/cadvisor/compose.yaml up -d
+```
+
+Of alles tegelijk:
+
+```bash
+./scripts/deploy/deploy-atc-docker01.sh
+```
+
+## Stap 5 — Andere hosts (handmatig)
+
+Deze repo bevat **config-referenties**, geen volledige VM-images:
+
+| Host | Herstel uit git |
+|------|-----------------|
+| Kafka connectors | `config/kafka/*.json` → deploy op Connect API |
+| Trino | `config/trino/config.properties` → `/etc/trino/` op lake01 |
+| Spark | `config/spark/spark-defaults.conf` |
+| Airflow | `config/airflow/` |
+| Data scripts | `config/scripts/data-generation/` |
+
+Zie `inventory/containers-atc-docker01.json` voor laatst geëxporteerde containerstate.
+
+## Volumes (niet in git)
+
+Back-up apart:
+
+- `forgejo_forgejo` — Git data
+- `superset_superset_home` — Superset metadata
+- `lam_lam` — LAM configuratie
+
+```bash
+docker run --rm -v forgejo_forgejo:/data -v /backup:/backup alpine \
+  tar czf /backup/forgejo_forgejo.tgz -C /data .
+```
+
+## Verificatie
+
+1. http://atc-docker01.dell-atc.lan/ — homepage laadt, 4 tabs
+2. http://atc-docker01.dell-atc.lan:8090/health — RSS proxy
+3. http://atc-docker01.dell-atc.lan:8088/health — Superset
+4. FEEDS-tab toont RSS-widgets
+
+## Proxmox-widget
+
+Token staat in `config/homepage/proxmox.yaml`. Na restore: Proxmox → Permissions → API Tokens → controleer `root@pam!homepage`.
@@ -0,0 +1,38 @@
+# Hosts & poorten
+
+## VM-overzicht
+
+| Hostname | IP (indien bekend) | Rol |
+|----------|-------------------|-----|
+| `atc-docker01.dell-atc.lan` | `10.0.21.45` | Docker: homepage, superset, rss-proxy, icons |
+| `atc-mgt01.dell-atc.lan` | — | Forgejo Git `:3001`, LDAP/LAM |
+| `atc-kafka01.dell-atc.lan` | — | Kafka UI `:9000` |
+| `atc-lake01.dell-atc.lan` | — | Spark `:8080`, Trino `:8089`, Debezium `:8083` |
+| `atc-elastic01.dell-atc.lan` | — | ES `:9200`, Kibana `:5601` |
+| `atc-db01.dell-atc.lan` | `10.0.20.112` | PostgreSQL `5432`, MySQL `3306` |
+| `atc-db02.dell-atc.lan` | `10.0.21.51` | PG, MySQL, Mongo `27017`, Cassandra `9042`, Neo4j `7474`/`7687` |
+| `atc-grafana.dell-atc.lan` | `10.0.20.103` | Grafana `:3000` |
+| `pve01` / Proxmox | `10.0.10.65` | Hypervisor `:8006` |
+| iDRAC | `10.0.41.102` | Out-of-band management |
+| ObjectScale | `10.0.20.111` | Object storage HTTPS |
+
+## atc-docker01 — Docker services
+
+| Service | Poort | URL |
+|---------|-------|-----|
+| Homepage | 80 | http://atc-docker01.dell-atc.lan/ |
+| Icons (nginx) | 8080 | http://atc-docker01.dell-atc.lan:8080/ |
+| RSS proxy | 8090 | http://atc-docker01.dell-atc.lan:8090/health |
+| Superset | 8088 | http://atc-docker01.dell-atc.lan:8088/ |
+| Superset Redis | 6379 | intern |
+| Forgejo (lokaal) | 4002, 222 | http://atc-docker01.dell-atc.lan:4002/ |
+| LAM (lokaal) | 4001 | http://atc-docker01.dell-atc.lan:4001/ |
+| cAdvisor | — | optioneel, niet standaard gestart |
+
+## Paden op atc-docker01
+
+| Pad | Inhoud |
+|-----|--------|
+| `/root/lakehouse` | Deze git clone |
+| `/root/compose/superset` | Legacy symlink-werkdir (sync met `compose/superset`) |
+| `/var/lib/docker/volumes/` | Persistente data (niet in git) |
@@ -0,0 +1,43 @@
+# Operaties
+
+## Config wijzigen en deployen
+
+```bash
+cd /root/lakehouse
+# Bewerk config/homepage/*.yaml of rss-proxy/server.py
+git add -A && git commit -m "beschrijving" && git push origin master
+
+# Homepage herladen
+docker exec homepage wget -qO- http://127.0.0.1:3000/api/revalidate
+
+# RSS proxy rebuild
+docker compose -f deploy/docker-compose.homepage.yml build rss-proxy
+docker compose -f deploy/docker-compose.homepage.yml up -d rss-proxy
+```
+
+## Logs
+
+```bash
+docker logs -f homepage
+docker logs -f rss-proxy
+docker logs -f superset
+```
+
+Homepage applicatielog staat buiten git: `config/homepage/logs/` (genegeerd door `.gitignore`).
+
+## Container inventory bijwerken
+
+```bash
+./scripts/deploy/export-inventory.sh
+git add inventory/ && git commit -m "chore: refresh container inventory"
+```
+
+## Stack herstarten
+
+```bash
+./scripts/deploy/deploy-atc-docker01.sh
+```
+
+## Git op de server
+
+Remote staat op `atc-docker01` in `/root/lakehouse`. Gebruik je eigen credentials voor push naar Forgejo op `atc-mgt01`.