Lakehouse/docs/architecture.md

# Architecture — ATC Lakehouse

## Data flow (conceptual)

```mermaid
flowchart LR
  subgraph sources [Sources]
    DB1[(atc-db01)]
    DB2[(atc-db02)]
  end

  subgraph ingest [Ingest]
    DEB[Debezium]
    KAFKA[Kafka]
  end

  subgraph lake [Lake]
    SPARK[Spark]
    TRINO[Trino]
    ICE[Iceberg tables]
  end

  subgraph consume [Consumption]
    SS[Superset]
    KB[Kibana]
    GF[Grafana]
  end

  DB1 --> DEB
  DB2 --> DEB
  DEB --> KAFKA
  KAFKA --> SPARK
  SPARK --> ICE
  TRINO --> ICE
  TRINO --> SS
  KAFKA --> SS
```

## Layers

| Layer | Components | Purpose |
|-------|------------|---------|
| **Source** | PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j | OLTP / events / graph demo data |
| **Ingest** | Debezium, Kafka Connect | CDC to Kafka topics |
| **Processing** | Spark, Airflow | ETL, batch, orchestration |
| **Query** | Trino | Federated SQL over lake + sources |
| **Storage** | ObjectScale / MinIO (when active) | Object storage |
| **Analytics** | Superset, Kibana | BI and search UI |
| **Platform** | Proxmox, Docker hosts | VMs and containers |
| **Ops** | Homepage dashboard, Grafana | Visibility and links |

## atc-docker01 — central dashboard host

Runs **gethomepage** (port 80), **RSS-proxy** (8090), **icon server** (8080), **Apache Superset** (8088), and optional local **Forgejo** (4002) / **LAM** (4001).

Config path on host: `/root/lakehouse/config/homepage` → mounted at `/app/config` in the container.

## SSH mesh

All lab VMs use a shared cluster key for root-to-root access. See [ssh-mesh.md](ssh-mesh.md).

## Configuration in git

| Path | Description |
|-----|--------------|
| `config/kafka/*.json` | Debezium/Kafka Connect connector definitions |
| `config/trino/config.properties` | Trino coordinator (single-node) |
| `config/spark/spark-defaults.conf` | Spark defaults |
| `config/airflow/` | Airflow DAG + `airflow.cfg` |
| `config/superset/` | Superset Python config + compose |
| `config/homepage/` | Full dashboard configuration |