Lakehouse/docs/landscape.md

# ATC Lakehouse — Application Landscape

High-level view of the Dell ATC Forward Deployed Engineering lab.  
**Live diagram:** http://atc-docker01.dell-atc.lan:8080/docs/architecture.html

## Architecture diagram

```mermaid
flowchart TB
  subgraph users [Users]
    FDE[FDE Engineers]
  end

  subgraph ops [Operations Layer]
    HP[Homepage Dashboard<br/>atc-docker01:80]
    GF[Grafana<br/>atc-grafana:3000]
    PVE[Proxmox pve01<br/>10.0.10.65:8006]
    IDRAC[iDRAC<br/>10.0.41.102]
  end

  subgraph git [Source Control]
    FG[Forgejo Git<br/>atc-mgt01:3001]
  end

  subgraph ingest [Ingest & Streaming]
    KF[Kafka<br/>atc-kafka01:9092]
    KUI[Kafka UI<br/>:9000]
    DEB[Debezium Connect<br/>atc-lake01:8083]
  end

  subgraph process [Processing]
    SP[Spark Master/Worker<br/>atc-lake01:8080]
    AF[Airflow<br/>atc-airflow01:8080]
    TR[Trino<br/>atc-lake01:8089]
  end

  subgraph storage [Storage]
  OS[ObjectScale ECS<br/>atc-objectscale 10.0.20.111]
  end

  subgraph sources [Source Databases — atc-db02]
    PG[(PostgreSQL)]
    MY[(MySQL)]
    MG[(MongoDB)]
    CA[(Cassandra)]
    N4[(Neo4j)]
  end

  subgraph analytics [Analytics & Search]
    SS[Superset<br/>atc-docker01:8088]
    ES[Elasticsearch<br/>atc-elastic01:9200]
    KB[Kibana<br/>:5601]
  end

  FDE --> HP
  FDE --> SS
  FDE --> KB
  HP --> KUI
  HP --> DEB
  HP --> SP
  HP --> TR
  HP --> OS
  HP --> FG

  PG & MY & MG & CA --> DEB
  DEB --> KF
  KF --> SP
  SP --> OS
  TR --> PG
  TR --> KF
  KF --> SS
  ES --> KB
  AF --> SP
```

## Host roles

| Host | IP | Role | Key services |
|------|-----|------|--------------|
| **atc-docker01** | 10.0.21.45 | Ops & BI hub | Homepage, Superset, RSS-proxy, icons, Forgejo (local), LAM |
| **atc-docker02** | 10.0.21.47 | Secondary Docker | Standby / experiments |
| **atc-mgt01** | 10.0.20.104 | Management | Forgejo (:3001), Nginx Proxy Manager, LDAP |
| **atc-kafka01** | 10.0.21.36 | Streaming | Kafka broker (:9092), Kafka UI (:9000) |
| **atc-lake01** | 10.0.21.50 | Lakehouse compute | Spark, Trino, Debezium Connect |
| **atc-airflow01** | 10.0.21.55 | Orchestration | Apache Airflow (systemd) |
| **atc-elastic01** | 10.0.21.46 | Search & observability | Elasticsearch, Kibana |
| **atc-db01** | 10.0.20.112 | Source DBs | PostgreSQL, MySQL (native or container) |
| **atc-db02** | 10.0.21.51 | Polyglot sources | PG, MySQL, Mongo, Cassandra, Neo4j (Docker) |
| **atc-grafana** | 10.0.20.103 | Monitoring | Grafana |
| **atc-objectscale** | 10.0.20.111 | Object storage | Dell ECS / S3 API (Luna OS) |
| **pve01** | 10.0.10.65 | Hypervisor | Proxmox VE |

## Data flow

1. **Operational data** lives on **atc-db02** (and db01) — sales, HR, supply chain, telemetry, graph.
2. **Debezium** on lake01 reads CDC from PostgreSQL → **Kafka** on kafka01.
3. **Spark** processes streams/batch; results can land in **ObjectScale** (S3-compatible).
4. **Trino** federates queries across catalogs (Postgres, Kafka, Hive, etc.).
5. **Superset** on docker01 is the primary BI UI; **Kibana** for search/logs.
6. **Airflow** orchestrates pipelines on airflow01.
7. **Homepage** aggregates links, health, RSS intel, and Proxmox status.

## Docker vs native

| Component | Deployment | Config in git |
|-----------|------------|---------------|
| Homepage, Superset, RSS | Docker (docker01) | `compose/`, `deploy/` |
| Spark, Trino, Debezium | Docker (lake01) | `config/docker/atc-lake01/` |
| Source DBs | Docker (db02) | `config/docker/atc-db02/` |
| Kafka UI | Docker (kafka01) | `config/docker/atc-kafka01/` |
| Kafka broker | Native KRaft (kafka01) | `config/kafka/kraft-server.properties` |
| Elasticsearch | Native (elastic01) | `config/elastic/` |
| Airflow | Native (airflow01) | `config/airflow/airflow.cfg`, DAGs |
| ObjectScale | ECS appliance | `config/objectscale/` |
| Forgejo | Docker (mgt01) | `compose/forgejo/` |

## Network zones

- **Management:** 10.0.10.x — Proxmox, iDRAC
- **Storage / legacy:** 10.0.20.x — ObjectScale, db01, mgt01, grafana, elastic
- **Lakehouse compute:** 10.0.21.x — docker, kafka, lake, db02, airflow

## Related docs

- [Hosts & ports](hosts.md)
- [Architecture layers](architecture.md)
- [SSH mesh](ssh-mesh.md)
- [Disaster recovery](disaster-recovery.md)
- [Docker inventory](docker-inventory.md)