Lakehouse Admin
f3d24634e4
- Translate all documentation to English - Add docs/ssh-mesh.md and scripts/setup/setup-ssh-mesh.sh - Add config/hosts/atc-lab.hosts for /etc/hosts on all VMs - Ignore atc_cluster private key in git; mesh deployed on 12 hosts
61 lines
2.1 KiB
Markdown
61 lines
2.1 KiB
Markdown
# ATC Lakehouse
|
|
|
|
Infrastructure-as-code and documentation for the Dell ATC Lakehouse lab — Forward Deployed Engineering (Bart & Mo).
|
|
|
|
## Repository layout
|
|
|
|
```
|
|
├── config/ # Application config (homepage, kafka, trino, spark, airflow, superset)
|
|
├── compose/ # Docker Compose per stack (source of truth for containers)
|
|
├── deploy/ # Combined compose files per host
|
|
├── docs/ # Documentation as code
|
|
├── inventory/ # Exported container state (reference)
|
|
├── rss-proxy/ # RSS feed proxy for homepage widgets
|
|
└── scripts/ # Deploy and setup scripts
|
|
```
|
|
|
|
## Hosts (summary)
|
|
|
|
| Host | Role | Key services |
|
|
|------|------|----------------|
|
|
| `atc-docker01` | Docker hub | Homepage, Superset, Forgejo (local), LAM, RSS-proxy, icons |
|
|
| `atc-mgt01` | Management | Forgejo (Git), LDAP/LAM |
|
|
| `atc-kafka01` | Streaming | Kafka, Kafka UI |
|
|
| `atc-lake01` | Lakehouse | Spark, Trino, Debezium, Airflow |
|
|
| `atc-elastic01` | Search | Elasticsearch, Kibana |
|
|
| `atc-db01` / `atc-db02` | Databases | PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j |
|
|
| `atc-grafana` | Monitoring | Grafana |
|
|
| `pve01` (`10.0.10.65`) | Hypervisor | Proxmox VE |
|
|
|
|
See [docs/hosts.md](docs/hosts.md) for ports and IPs.
|
|
See [docs/ssh-mesh.md](docs/ssh-mesh.md) for passwordless SSH between VMs.
|
|
|
|
## Quick start — atc-docker01
|
|
|
|
```bash
|
|
cd /root/lakehouse
|
|
cp .env.example .env # fill in secrets
|
|
./scripts/deploy/deploy-atc-docker01.sh
|
|
```
|
|
|
|
## Documentation
|
|
|
|
- [Architecture](docs/architecture.md)
|
|
- [Hosts & ports](docs/hosts.md)
|
|
- [SSH mesh (passwordless fleet)](docs/ssh-mesh.md)
|
|
- [Disaster recovery](docs/disaster-recovery.md)
|
|
- [Day-to-day operations](docs/operations.md)
|
|
- [Homepage dashboard](config/homepage/README.md)
|
|
|
|
## Git remote
|
|
|
|
Forgejo: `http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse`
|
|
|
|
```bash
|
|
git clone http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse.git /root/lakehouse
|
|
```
|
|
|
|
## Secrets
|
|
|
|
Files with credentials (`proxmox.yaml`, `.env`, kafka connectors) are in git for **lab disaster recovery**. For production, use `.env` + `*.example` and keep secrets off public remotes.
|