68 lines
1.9 KiB
Markdown
68 lines
1.9 KiB
Markdown
|
# Architectuur — ATC Lakehouse
|
||
|
|
|
||
|
|
## Dataflow (conceptueel)
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
flowchart LR
|
||
|
|
subgraph sources [Bronnen]
|
||
|
|
DB1[(atc-db01)]
|
||
|
|
DB2[(atc-db02)]
|
||
|
|
end
|
||
|
|
|
||
|
|
subgraph ingest [Ingest]
|
||
|
|
DEB[Debezium]
|
||
|
|
KAFKA[Kafka]
|
||
|
|
end
|
||
|
|
|
||
|
|
subgraph lake [Lake]
|
||
|
|
SPARK[Spark]
|
||
|
|
TRINO[Trino]
|
||
|
|
ICE[Iceberg tables]
|
||
|
|
end
|
||
|
|
|
||
|
|
subgraph consume [Consumptie]
|
||
|
|
SS[Superset]
|
||
|
|
KB[Kibana]
|
||
|
|
GF[Grafana]
|
||
|
|
end
|
||
|
|
|
||
|
|
DB1 --> DEB
|
||
|
|
DB2 --> DEB
|
||
|
|
DEB --> KAFKA
|
||
|
|
KAFKA --> SPARK
|
||
|
|
SPARK --> ICE
|
||
|
|
TRINO --> ICE
|
||
|
|
TRINO --> SS
|
||
|
|
KAFKA --> SS
|
||
|
|
```
|
||
|
|
|
||
|
|
## Lagen
|
||
|
|
|
||
|
|
| Laag | Componenten | Doel |
|
||
|
|
|------|-------------|------|
|
||
|
|
| **Bron** | PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j | OLTP / events / graph demo data |
|
||
|
|
| **Ingest** | Debezium, Kafka Connect | CDC naar Kafka topics |
|
||
|
|
| **Processing** | Spark, Airflow | ETL, batch, orchestratie |
|
||
|
|
| **Query** | Trino | Federated SQL over lake + bronnen |
|
||
|
|
| **Storage** | ObjectScale / MinIO (indien actief) | Object storage |
|
||
|
|
| **Analytics** | Superset, Kibana | BI en search UI |
|
||
|
|
| **Platform** | Proxmox, Docker hosts | VMs en containers |
|
||
|
|
| **Ops** | Homepage dashboard, Grafana | Zichtbaarheid en links |
|
||
|
|
|
||
|
|
## atc-docker01 — centrale dashboard-host
|
||
|
|
|
||
|
|
Draait de **gethomepage**-dashboard (poort 80), **RSS-proxy** (8090), **icon server** (8080), **Apache Superset** (8088), en optioneel lokale **Forgejo** (4002) / **LAM** (4001).
|
||
|
|
|
||
|
|
Configuratiepad op de host: `/root/lakehouse/config/homepage` → gemount in container `/app/config`.
|
||
|
|
|
||
|
|
## Configuratie in git
|
||
|
|
|
||
|
|
| Pad | Beschrijving |
|
||
|
|
|-----|--------------|
|
||
|
|
| `config/kafka/*.json` | Debezium/Kafka Connect connector definities |
|
||
|
|
| `config/trino/config.properties` | Trino coordinator (single-node) |
|
||
|
|
| `config/spark/spark-defaults.conf` | Spark defaults |
|
||
|
|
| `config/airflow/` | Airflow DAG + `airflow.cfg` |
|
||
|
|
| `config/superset/` | Superset Python config + compose |
|
||
|
|
| `config/homepage/` | Volledige dashboardconfiguratie |
|