Lakehouse/docs/landscape.md

ATC Lakehouse — Application Landscape

High-level view of the Dell ATC Forward Deployed Engineering lab.
Live diagram: http://atc-docker01.dell-atc.lan:8080/docs/architecture.html

Architecture diagram

flowchart TB
  subgraph users [Users]
    FDE[FDE Engineers]
  end

  subgraph ops [Operations Layer]
    HP[Homepage Dashboard<br/>atc-docker01:80]
    GF[Grafana<br/>atc-grafana:3000]
    PVE[Proxmox pve01<br/>10.0.10.65:8006]
    IDRAC[iDRAC<br/>10.0.41.102]
  end

  subgraph git [Source Control]
    FG[Forgejo Git<br/>atc-mgt01:3001]
  end

  subgraph ingest [Ingest & Streaming]
    KF[Kafka<br/>atc-kafka01:9092]
    KUI[Kafka UI<br/>:9000]
    DEB[Debezium Connect<br/>atc-lake01:8083]
  end

  subgraph process [Processing]
    SP[Spark Master/Worker<br/>atc-lake01:8080]
    AF[Airflow<br/>atc-airflow01:8080]
    TR[Trino<br/>atc-lake01:8089]
  end

  subgraph storage [Storage]
  OS[ObjectScale ECS<br/>atc-objectscale 10.0.20.111]
  end

  subgraph sources [Source Databases — atc-db02]
    PG[(PostgreSQL)]
    MY[(MySQL)]
    MG[(MongoDB)]
    CA[(Cassandra)]
    N4[(Neo4j)]
  end

  subgraph analytics [Analytics & Search]
    SS[Superset<br/>atc-docker01:8088]
    ES[Elasticsearch<br/>atc-elastic01:9200]
    KB[Kibana<br/>:5601]
  end

  FDE --> HP
  FDE --> SS
  FDE --> KB
  HP --> KUI
  HP --> DEB
  HP --> SP
  HP --> TR
  HP --> OS
  HP --> FG

  PG & MY & MG & CA --> DEB
  DEB --> KF
  KF --> SP
  SP --> OS
  TR --> PG
  TR --> KF
  KF --> SS
  ES --> KB
  AF --> SP

Host roles

Host	IP	Role	Key services
atc-docker01	10.0.21.45	Ops & BI hub	Homepage, Superset, RSS-proxy, icons, Forgejo (local), LAM
atc-docker02	10.0.21.47	Secondary Docker	Standby / experiments
atc-mgt01	10.0.20.104	Management	Forgejo (:3001), Nginx Proxy Manager, LDAP
atc-kafka01	10.0.21.36	Streaming	Kafka broker (:9092), Kafka UI (:9000)
atc-lake01	10.0.21.50	Lakehouse compute	Spark, Trino, Debezium Connect
atc-airflow01	10.0.21.55	Orchestration	Apache Airflow (systemd)
atc-elastic01	10.0.21.46	Search & observability	Elasticsearch, Kibana
atc-db01	10.0.20.112	Source DBs	PostgreSQL, MySQL (native or container)
atc-db02	10.0.21.51	Polyglot sources	PG, MySQL, Mongo, Cassandra, Neo4j (Docker)
atc-grafana	10.0.20.103	Monitoring	Grafana
atc-objectscale	10.0.20.111	Object storage	Dell ECS / S3 API (Luna OS)
pve01	10.0.10.65	Hypervisor	Proxmox VE

Data flow

1. Operational data lives on atc-db02 (and db01) — sales, HR, supply chain, telemetry, graph.
2. Debezium on lake01 reads CDC from PostgreSQL → Kafka on kafka01.
3. Spark processes streams/batch; results can land in ObjectScale (S3-compatible).
4. Trino federates queries across catalogs (Postgres, Kafka, Hive, etc.).
5. Superset on docker01 is the primary BI UI; Kibana for search/logs.
6. Airflow orchestrates pipelines on airflow01.
7. Homepage aggregates links, health, RSS intel, and Proxmox status.

Docker vs native

Component	Deployment	Config in git
Homepage, Superset, RSS	Docker (docker01)	compose/, deploy/
Spark, Trino, Debezium	Docker (lake01)	config/docker/atc-lake01/
Source DBs	Docker (db02)	config/docker/atc-db02/
Kafka UI	Docker (kafka01)	config/docker/atc-kafka01/
Kafka broker	Native KRaft (kafka01)	config/kafka/kraft-server.properties
Elasticsearch	Native (elastic01)	config/elastic/
Airflow	Native (airflow01)	config/airflow/airflow.cfg, DAGs
ObjectScale	ECS appliance	config/objectscale/
Forgejo	Docker (mgt01)	compose/forgejo/

Network zones

• Management: 10.0.10.x — Proxmox, iDRAC
• Storage / legacy: 10.0.20.x — ObjectScale, db01, mgt01, grafana, elastic
• Lakehouse compute: 10.0.21.x — docker, kafka, lake, db02, airflow

Related docs

• Hosts & ports
• Architecture layers
• SSH mesh
• Disaster recovery
• Docker inventory