ATC Lakehouse

Infrastructure-as-code and documentation for the Dell ATC Lakehouse lab — Forward Deployed Engineering (Bart & Mo).

Repository layout

├── config/           # Application config (homepage, kafka, trino, spark, airflow, superset)
├── compose/          # Docker Compose per stack (source of truth for containers)
├── deploy/           # Combined compose files per host
├── docs/             # Documentation as code
├── inventory/        # Exported container state (reference)
├── rss-proxy/        # RSS feed proxy for homepage widgets
└── scripts/          # Deploy and setup scripts

Hosts (summary)

Host	Role	Key services
atc-docker01	Docker hub	Homepage, Superset, Forgejo (local), LAM, RSS-proxy, icons
atc-mgt01	Management	Forgejo (Git), LDAP/LAM
atc-kafka01	Streaming	Kafka, Kafka UI
atc-lake01	Lakehouse	Spark, Trino, Debezium, Airflow
atc-elastic01	Search	Elasticsearch, Kibana
atc-db01 / atc-db02	Databases	PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j
atc-grafana	Monitoring	Grafana
pve01 (10.0.10.65)	Hypervisor	Proxmox VE

See docs/hosts.md for ports and IPs.
See docs/ssh-mesh.md for passwordless SSH between VMs.

Quick start — atc-docker01

cd /root/lakehouse
cp .env.example .env   # fill in secrets
./scripts/deploy/deploy-atc-docker01.sh

Documentation

• Architecture
• Hosts & ports
• SSH mesh (passwordless fleet)
• Disaster recovery
• Day-to-day operations
• Homepage dashboard

Git remote

Forgejo: http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse

git clone http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse.git /root/lakehouse

Secrets

Files with credentials (proxmox.yaml, .env, kafka connectors) are in git for lab disaster recovery. For production, use .env + *.example and keep secrets off public remotes.