Docs-as-code: full deploy configs, compose stacks, and DR guide

- Add root README and docs/ (architecture, hosts, disaster-recovery, operations)
- Add compose/ for superset, forgejo, lam, cadvisor; unified deploy compose
- Add scripts/deploy (deploy-atc-docker01.sh, container inventory export)
- Add inventory/containers-atc-docker01.json snapshot
- Add config READMEs for kafka and airflow; .env.example; expand .gitignore
- Sync atclab helper scripts; update homepage/superset compose with env support
This commit is contained in:
Lakehouse Admin
2026-05-19 22:28:01 +02:00
parent c63181abac
commit f5d30824fc
30 changed files with 890 additions and 3 deletions
+67
View File
@@ -0,0 +1,67 @@
# Architectuur — ATC Lakehouse
## Dataflow (conceptueel)
```mermaid
flowchart LR
subgraph sources [Bronnen]
DB1[(atc-db01)]
DB2[(atc-db02)]
end
subgraph ingest [Ingest]
DEB[Debezium]
KAFKA[Kafka]
end
subgraph lake [Lake]
SPARK[Spark]
TRINO[Trino]
ICE[Iceberg tables]
end
subgraph consume [Consumptie]
SS[Superset]
KB[Kibana]
GF[Grafana]
end
DB1 --> DEB
DB2 --> DEB
DEB --> KAFKA
KAFKA --> SPARK
SPARK --> ICE
TRINO --> ICE
TRINO --> SS
KAFKA --> SS
```
## Lagen
| Laag | Componenten | Doel |
|------|-------------|------|
| **Bron** | PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j | OLTP / events / graph demo data |
| **Ingest** | Debezium, Kafka Connect | CDC naar Kafka topics |
| **Processing** | Spark, Airflow | ETL, batch, orchestratie |
| **Query** | Trino | Federated SQL over lake + bronnen |
| **Storage** | ObjectScale / MinIO (indien actief) | Object storage |
| **Analytics** | Superset, Kibana | BI en search UI |
| **Platform** | Proxmox, Docker hosts | VMs en containers |
| **Ops** | Homepage dashboard, Grafana | Zichtbaarheid en links |
## atc-docker01 — centrale dashboard-host
Draait de **gethomepage**-dashboard (poort 80), **RSS-proxy** (8090), **icon server** (8080), **Apache Superset** (8088), en optioneel lokale **Forgejo** (4002) / **LAM** (4001).
Configuratiepad op de host: `/root/lakehouse/config/homepage` → gemount in container `/app/config`.
## Configuratie in git
| Pad | Beschrijving |
|-----|--------------|
| `config/kafka/*.json` | Debezium/Kafka Connect connector definities |
| `config/trino/config.properties` | Trino coordinator (single-node) |
| `config/spark/spark-defaults.conf` | Spark defaults |
| `config/airflow/` | Airflow DAG + `airflow.cfg` |
| `config/superset/` | Superset Python config + compose |
| `config/homepage/` | Volledige dashboardconfiguratie |