ATC/Lakehouse
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document full application landscape and collect Docker configs

Browse Source 
- Add docs/landscape.md and docker-inventory.md with Mermaid architecture
- Add config/docker per host (db02, lake01, kafka01) and objectscale/elastic refs
- Add live architecture page at icons/docs/architecture.html
- Homepage OPS tab: Lakehouse Architecture group with diagram link
- Add scripts/collect/collect-fleet-config.sh to refresh configs from fleet
This commit is contained in:
Lakehouse Admin
2026-05-19 23:01:47 +02:00
parent e1180e14c9
commit 007f6cedc5
15 changed files with 881 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+20 -42
View File
@@ -1,60 +1,38 @@
# ATC Lakehouse
Infrastructure-as-code and documentation for the Dell ATC Lakehouse lab — Forward Deployed Engineering (Bart & Mo).
Infrastructure-as-code and documentation for the Dell ATC Forward Deployed Engineering lab.
## Start here
| Resource | Link |
|----------|------|
| **Live architecture diagram** | http://atc-docker01.dell-atc.lan:8080/docs/architecture.html |
| **Dashboard** | http://atc-docker01.dell-atc.lan/ |
| **Application landscape (doc)** | [docs/landscape.md](docs/landscape.md) |
| **Docker inventory** | [docs/docker-inventory.md](docs/docker-inventory.md) |
## Repository layout
```
├── config/ # Application config (homepage, kafka, trino, spark, airflow, superset)
├── compose/ # Docker Compose per stack (source of truth for containers)
├── deploy/ # Combined compose files per host
├── config/ # App config: homepage, kafka, docker per host, objectscale, elastic
├── compose/ # Docker Compose stacks
├── deploy/ # Combined deploy compose files
├── docs/ # Documentation as code
├── inventory/ # Exported container state (reference)
├── rss-proxy/ # RSS feed proxy for homepage widgets
└── scripts/ # Deploy and setup scripts
├── inventory/ # Container snapshots
├── rss-proxy/ # RSS proxy for homepage
└── scripts/ # deploy/, setup/, collect/
```
## Hosts (summary)
See [docs/landscape.md](docs/landscape.md) for the full application map.
| Host | Role | Key services |
|------|------|----------------|
| `atc-docker01` | Docker hub | Homepage, Superset, Forgejo (local), LAM, RSS-proxy, icons |
| `atc-mgt01` | Management | Forgejo (Git), LDAP/LAM |
| `atc-kafka01` | Streaming | Kafka, Kafka UI |
| `atc-lake01` | Lakehouse | Spark, Trino, Debezium, Airflow |
| `atc-elastic01` | Search | Elasticsearch, Kibana |
| `atc-db01` / `atc-db02` | Databases | PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j |
| `atc-grafana` | Monitoring | Grafana |
| `pve01` (`10.0.10.65`) | Hypervisor | Proxmox VE |
See [docs/hosts.md](docs/hosts.md) for ports and IPs.
See [docs/ssh-mesh.md](docs/ssh-mesh.md) for passwordless SSH between VMs.
## Quick start — atc-docker01
## Quick deploy (atc-docker01)
```bash
cd /root/lakehouse
cp .env.example .env # fill in secrets
cp .env.example .env
./scripts/deploy/deploy-atc-docker01.sh
```
## Documentation
- [Architecture](docs/architecture.md)
- [Hosts & ports](docs/hosts.md)
- [SSH mesh (passwordless fleet)](docs/ssh-mesh.md)
- [Disaster recovery](docs/disaster-recovery.md)
- [Day-to-day operations](docs/operations.md)
- [Homepage dashboard](config/homepage/README.md)
## Git remote
Forgejo: `http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse`
```bash
git clone http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse.git /root/lakehouse
```
## Secrets
Files with credentials (`proxmox.yaml`, `.env`, kafka connectors) are in git for **lab disaster recovery**. For production, use `.env` + `*.example` and keep secrets off public remotes.
http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse