Files

T

Lakehouse Admin f5d30824fc Docs-as-code: full deploy configs, compose stacks, and DR guide

- Add root README and docs/ (architecture, hosts, disaster-recovery, operations)
- Add compose/ for superset, forgejo, lam, cadvisor; unified deploy compose
- Add scripts/deploy (deploy-atc-docker01.sh, container inventory export)
- Add inventory/containers-atc-docker01.json snapshot
- Add config READMEs for kafka and airflow; .env.example; expand .gitignore
- Sync atclab helper scripts; update homepage/superset compose with env support

2026-05-19 22:28:01 +02:00

1.9 KiB

Raw Blame History

Architectuur — ATC Lakehouse

Dataflow (conceptueel)

flowchart LR
  subgraph sources [Bronnen]
    DB1[(atc-db01)]
    DB2[(atc-db02)]
  end

  subgraph ingest [Ingest]
    DEB[Debezium]
    KAFKA[Kafka]
  end

  subgraph lake [Lake]
    SPARK[Spark]
    TRINO[Trino]
    ICE[Iceberg tables]
  end

  subgraph consume [Consumptie]
    SS[Superset]
    KB[Kibana]
    GF[Grafana]
  end

  DB1 --> DEB
  DB2 --> DEB
  DEB --> KAFKA
  KAFKA --> SPARK
  SPARK --> ICE
  TRINO --> ICE
  TRINO --> SS
  KAFKA --> SS

Lagen

Laag	Componenten	Doel
Bron	PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j	OLTP / events / graph demo data
Ingest	Debezium, Kafka Connect	CDC naar Kafka topics
Processing	Spark, Airflow	ETL, batch, orchestratie
Query	Trino	Federated SQL over lake + bronnen
Storage	ObjectScale / MinIO (indien actief)	Object storage
Analytics	Superset, Kibana	BI en search UI
Platform	Proxmox, Docker hosts	VMs en containers
Ops	Homepage dashboard, Grafana	Zichtbaarheid en links

atc-docker01 — centrale dashboard-host

Draait de gethomepage-dashboard (poort 80), RSS-proxy (8090), icon server (8080), Apache Superset (8088), en optioneel lokale Forgejo (4002) / LAM (4001).

Configuratiepad op de host: /root/lakehouse/config/homepage → gemount in container /app/config.

Configuratie in git

Pad	Beschrijving
`config/kafka/*.json`	Debezium/Kafka Connect connector definities
`config/trino/config.properties`	Trino coordinator (single-node)
`config/spark/spark-defaults.conf`	Spark defaults
`config/airflow/`	Airflow DAG + `airflow.cfg`
`config/superset/`	Superset Python config + compose
`config/homepage/`	Volledige dashboardconfiguratie

1.9 KiB Raw Blame History