Lakehouse Admin
f3d24634e4
- Translate all documentation to English - Add docs/ssh-mesh.md and scripts/setup/setup-ssh-mesh.sh - Add config/hosts/atc-lab.hosts for /etc/hosts on all VMs - Ignore atc_cluster private key in git; mesh deployed on 12 hosts
72 lines
2.0 KiB
Markdown
72 lines
2.0 KiB
Markdown
# Architecture — ATC Lakehouse
|
|
|
|
## Data flow (conceptual)
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
subgraph sources [Sources]
|
|
DB1[(atc-db01)]
|
|
DB2[(atc-db02)]
|
|
end
|
|
|
|
subgraph ingest [Ingest]
|
|
DEB[Debezium]
|
|
KAFKA[Kafka]
|
|
end
|
|
|
|
subgraph lake [Lake]
|
|
SPARK[Spark]
|
|
TRINO[Trino]
|
|
ICE[Iceberg tables]
|
|
end
|
|
|
|
subgraph consume [Consumption]
|
|
SS[Superset]
|
|
KB[Kibana]
|
|
GF[Grafana]
|
|
end
|
|
|
|
DB1 --> DEB
|
|
DB2 --> DEB
|
|
DEB --> KAFKA
|
|
KAFKA --> SPARK
|
|
SPARK --> ICE
|
|
TRINO --> ICE
|
|
TRINO --> SS
|
|
KAFKA --> SS
|
|
```
|
|
|
|
## Layers
|
|
|
|
| Layer | Components | Purpose |
|
|
|-------|------------|---------|
|
|
| **Source** | PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j | OLTP / events / graph demo data |
|
|
| **Ingest** | Debezium, Kafka Connect | CDC to Kafka topics |
|
|
| **Processing** | Spark, Airflow | ETL, batch, orchestration |
|
|
| **Query** | Trino | Federated SQL over lake + sources |
|
|
| **Storage** | ObjectScale / MinIO (when active) | Object storage |
|
|
| **Analytics** | Superset, Kibana | BI and search UI |
|
|
| **Platform** | Proxmox, Docker hosts | VMs and containers |
|
|
| **Ops** | Homepage dashboard, Grafana | Visibility and links |
|
|
|
|
## atc-docker01 — central dashboard host
|
|
|
|
Runs **gethomepage** (port 80), **RSS-proxy** (8090), **icon server** (8080), **Apache Superset** (8088), and optional local **Forgejo** (4002) / **LAM** (4001).
|
|
|
|
Config path on host: `/root/lakehouse/config/homepage` → mounted at `/app/config` in the container.
|
|
|
|
## SSH mesh
|
|
|
|
All lab VMs use a shared cluster key for root-to-root access. See [ssh-mesh.md](ssh-mesh.md).
|
|
|
|
## Configuration in git
|
|
|
|
| Path | Description |
|
|
|-----|--------------|
|
|
| `config/kafka/*.json` | Debezium/Kafka Connect connector definitions |
|
|
| `config/trino/config.properties` | Trino coordinator (single-node) |
|
|
| `config/spark/spark-defaults.conf` | Spark defaults |
|
|
| `config/airflow/` | Airflow DAG + `airflow.cfg` |
|
|
| `config/superset/` | Superset Python config + compose |
|
|
| `config/homepage/` | Full dashboard configuration |
|