Lakehouse/README.md

# ATC Lakehouse

Infrastructure-as-code and documentation for the Dell ATC Lakehouse lab — Forward Deployed Engineering (Bart & Mo).

## Repository layout

```
├── config/           # Application config (homepage, kafka, trino, spark, airflow, superset)
├── compose/          # Docker Compose per stack (source of truth for containers)
├── deploy/           # Combined compose files per host
├── docs/             # Documentation as code
├── inventory/        # Exported container state (reference)
├── rss-proxy/        # RSS feed proxy for homepage widgets
└── scripts/          # Deploy and setup scripts
```

## Hosts (summary)

| Host | Role | Key services |
|------|------|----------------|
| `atc-docker01` | Docker hub | Homepage, Superset, Forgejo (local), LAM, RSS-proxy, icons |
| `atc-mgt01` | Management | Forgejo (Git), LDAP/LAM |
| `atc-kafka01` | Streaming | Kafka, Kafka UI |
| `atc-lake01` | Lakehouse | Spark, Trino, Debezium, Airflow |
| `atc-elastic01` | Search | Elasticsearch, Kibana |
| `atc-db01` / `atc-db02` | Databases | PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j |
| `atc-grafana` | Monitoring | Grafana |
| `pve01` (`10.0.10.65`) | Hypervisor | Proxmox VE |

See [docs/hosts.md](docs/hosts.md) for ports and IPs.  
See [docs/ssh-mesh.md](docs/ssh-mesh.md) for passwordless SSH between VMs.

## Quick start — atc-docker01

```bash
cd /root/lakehouse
cp .env.example .env   # fill in secrets
./scripts/deploy/deploy-atc-docker01.sh
```

## Documentation

- [Architecture](docs/architecture.md)
- [Hosts & ports](docs/hosts.md)
- [SSH mesh (passwordless fleet)](docs/ssh-mesh.md)
- [Disaster recovery](docs/disaster-recovery.md)
- [Day-to-day operations](docs/operations.md)
- [Homepage dashboard](config/homepage/README.md)

## Git remote

Forgejo: `http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse`

```bash
git clone http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse.git /root/lakehouse
```

## Secrets

Files with credentials (`proxmox.yaml`, `.env`, kafka connectors) are in git for **lab disaster recovery**. For production, use `.env` + `*.example` and keep secrets off public remotes.