Files

T

Lakehouse Admin f3d24634e4 English docs and passwordless SSH mesh for lab fleet

- Translate all documentation to English
- Add docs/ssh-mesh.md and scripts/setup/setup-ssh-mesh.sh
- Add config/hosts/atc-lab.hosts for /etc/hosts on all VMs
- Ignore atc_cluster private key in git; mesh deployed on 12 hosts

2026-05-19 22:46:40 +02:00

2.0 KiB

Raw Blame History

Architecture — ATC Lakehouse

Data flow (conceptual)

flowchart LR
  subgraph sources [Sources]
    DB1[(atc-db01)]
    DB2[(atc-db02)]
  end

  subgraph ingest [Ingest]
    DEB[Debezium]
    KAFKA[Kafka]
  end

  subgraph lake [Lake]
    SPARK[Spark]
    TRINO[Trino]
    ICE[Iceberg tables]
  end

  subgraph consume [Consumption]
    SS[Superset]
    KB[Kibana]
    GF[Grafana]
  end

  DB1 --> DEB
  DB2 --> DEB
  DEB --> KAFKA
  KAFKA --> SPARK
  SPARK --> ICE
  TRINO --> ICE
  TRINO --> SS
  KAFKA --> SS

Layers

Layer	Components	Purpose
Source	PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j	OLTP / events / graph demo data
Ingest	Debezium, Kafka Connect	CDC to Kafka topics
Processing	Spark, Airflow	ETL, batch, orchestration
Query	Trino	Federated SQL over lake + sources
Storage	ObjectScale / MinIO (when active)	Object storage
Analytics	Superset, Kibana	BI and search UI
Platform	Proxmox, Docker hosts	VMs and containers
Ops	Homepage dashboard, Grafana	Visibility and links

atc-docker01 — central dashboard host

Runs gethomepage (port 80), RSS-proxy (8090), icon server (8080), Apache Superset (8088), and optional local Forgejo (4002) / LAM (4001).

Config path on host: /root/lakehouse/config/homepage → mounted at /app/config in the container.

SSH mesh

All lab VMs use a shared cluster key for root-to-root access. See ssh-mesh.md.

Configuration in git

Path	Description
`config/kafka/*.json`	Debezium/Kafka Connect connector definitions
`config/trino/config.properties`	Trino coordinator (single-node)
`config/spark/spark-defaults.conf`	Spark defaults
`config/airflow/`	Airflow DAG + `airflow.cfg`
`config/superset/`	Superset Python config + compose
`config/homepage/`	Full dashboard configuration

2.0 KiB Raw Blame History