ATC/Lakehouse
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

English docs and passwordless SSH mesh for lab fleet

Browse Source 
- Translate all documentation to English
- Add docs/ssh-mesh.md and scripts/setup/setup-ssh-mesh.sh
- Add config/hosts/atc-lab.hosts for /etc/hosts on all VMs
- Ignore atc_cluster private key in git; mesh deployed on 12 hosts
This commit is contained in:
Lakehouse Admin
2026-05-19 22:46:40 +02:00
parent f5d30824fc
commit f3d24634e4
12 changed files with 330 additions and 118 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+26 -22
View File
@@ -1,10 +1,10 @@
# Architectuur — ATC Lakehouse
# Architecture — ATC Lakehouse
## Dataflow (conceptueel)
## Data flow (conceptual)
```mermaid
flowchart LR
subgraph sources [Bronnen]
subgraph sources [Sources]
DB1[(atc-db01)]
DB2[(atc-db02)]
end
@@ -20,7 +20,7 @@ flowchart LR
ICE[Iceberg tables]
end
subgraph consume [Consumptie]
subgraph consume [Consumption]
SS[Superset]
KB[Kibana]
GF[Grafana]
@@ -36,32 +36,36 @@ flowchart LR
KAFKA --> SS
```
## Lagen
## Layers
| Laag | Componenten | Doel |
|------|-------------|------|
| **Bron** | PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j | OLTP / events / graph demo data |
| **Ingest** | Debezium, Kafka Connect | CDC naar Kafka topics |
| **Processing** | Spark, Airflow | ETL, batch, orchestratie |
| **Query** | Trino | Federated SQL over lake + bronnen |
| **Storage** | ObjectScale / MinIO (indien actief) | Object storage |
| **Analytics** | Superset, Kibana | BI en search UI |
| **Platform** | Proxmox, Docker hosts | VMs en containers |
| **Ops** | Homepage dashboard, Grafana | Zichtbaarheid en links |
| Layer | Components | Purpose |
|-------|------------|---------|
| **Source** | PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j | OLTP / events / graph demo data |
| **Ingest** | Debezium, Kafka Connect | CDC to Kafka topics |
| **Processing** | Spark, Airflow | ETL, batch, orchestration |
| **Query** | Trino | Federated SQL over lake + sources |
| **Storage** | ObjectScale / MinIO (when active) | Object storage |
| **Analytics** | Superset, Kibana | BI and search UI |
| **Platform** | Proxmox, Docker hosts | VMs and containers |
| **Ops** | Homepage dashboard, Grafana | Visibility and links |
## atc-docker01 — centrale dashboard-host
## atc-docker01 — central dashboard host
Draait de **gethomepage**-dashboard (poort 80), **RSS-proxy** (8090), **icon server** (8080), **Apache Superset** (8088), en optioneel lokale **Forgejo** (4002) / **LAM** (4001).
Runs **gethomepage** (port 80), **RSS-proxy** (8090), **icon server** (8080), **Apache Superset** (8088), and optional local **Forgejo** (4002) / **LAM** (4001).
Configuratiepad op de host: `/root/lakehouse/config/homepage` → gemount in container `/app/config`.
Config path on host: `/root/lakehouse/config/homepage` → mounted at `/app/config` in the container.
## Configuratie in git
## SSH mesh
| Pad | Beschrijving |
All lab VMs use a shared cluster key for root-to-root access. See [ssh-mesh.md](ssh-mesh.md).
## Configuration in git
| Path | Description |
|-----|--------------|
| `config/kafka/*.json` | Debezium/Kafka Connect connector definities |
| `config/kafka/*.json` | Debezium/Kafka Connect connector definitions |
| `config/trino/config.properties` | Trino coordinator (single-node) |
| `config/spark/spark-defaults.conf` | Spark defaults |
| `config/airflow/` | Airflow DAG + `airflow.cfg` |
| `config/superset/` | Superset Python config + compose |
| `config/homepage/` | Volledige dashboardconfiguratie |
| `config/homepage/` | Full dashboard configuration |