Full lab documentation and infrastructure as code
- Trino catalogs, Grafana, Spark jobs, LDAP LDIF, NPM compose - Airflow DAG scripts, Proxmox VM inventory, network docs - Ansible playbook, Gitea CI validate workflow - Backup and health-check scripts, cron documentation - Homepage DOCS tab with links to all documentation - Extended collect-fleet-config.sh and populate-repo.py
This commit is contained in:
Lakehouse Admin
@@ -0,0 +1,64 @@
|
||||
# Network — ATC Lakehouse lab
|
||||
|
||||
## Subnets
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph mgmt [10.0.10.0/24 — Management]
|
||||
PVE[pve01 · 10.0.10.65]
|
||||
IDRAC[iDRAC · 10.0.41.102]
|
||||
end
|
||||
|
||||
subgraph data20 [10.0.20.0/24 — Storage & legacy]
|
||||
OS[ObjectScale · 10.0.20.111]
|
||||
DB1[atc-db01 · 10.0.20.112]
|
||||
MGT[atc-mgt01 · 10.0.20.104]
|
||||
GF[atc-grafana · 10.0.20.103]
|
||||
end
|
||||
|
||||
subgraph lake21 [10.0.21.0/24 — Lakehouse compute]
|
||||
D1[atc-docker01 · 10.0.21.45]
|
||||
D2[atc-docker02 · 10.0.21.47]
|
||||
KF[atc-kafka01 · 10.0.21.36]
|
||||
LK[atc-lake01 · 10.0.21.50]
|
||||
DB2[atc-db02 · 10.0.21.51]
|
||||
AF[atc-airflow01 · 10.0.21.55]
|
||||
EL[atc-elastic01 · 10.0.21.46]
|
||||
end
|
||||
|
||||
PVE --> D1 & D2 & KF & LK & DB2 & OS & DB1 & MGT
|
||||
```
|
||||
|
||||
## DNS
|
||||
|
||||
Internal names: `*.dell-atc.lan` (see `config/hosts/atc-lab.hosts`).
|
||||
|
||||
| Pattern | Example |
|
||||
|---------|---------|
|
||||
| Short hostname | `atc-lake01` |
|
||||
| FQDN | `atc-lake01.dell-atc.lan` |
|
||||
|
||||
## Key ports (east-west)
|
||||
|
||||
| Service | Port | Protocol |
|
||||
|---------|------|----------|
|
||||
| Kafka | 9092 | TCP |
|
||||
| Debezium Connect | 8083 | HTTP |
|
||||
| Spark UI | 8080 | HTTP |
|
||||
| Trino | 8089 | HTTP |
|
||||
| PostgreSQL | 5432 | TCP |
|
||||
| ObjectScale S3 | 9020 | HTTP |
|
||||
| ObjectScale UI | 443 | HTTPS |
|
||||
| Homepage | 80 | HTTP |
|
||||
|
||||
## SSH mesh
|
||||
|
||||
All lab VMs: `/root/.ssh/atc_cluster` — see [ssh-mesh.md](ssh-mesh.md).
|
||||
|
||||
ObjectScale uses `admin@` for initial bootstrap; `root@` after key install.
|
||||
|
||||
## Firewall notes
|
||||
|
||||
- Lab assumes flat L2/L3 trust within `10.0.10/20/21.x`.
|
||||
- ObjectScale `management_clients: 0.0.0.0/0` in deploy.yml — tighten for production.
|
||||
- Proxmox API `:8006` reachable from docker01 for homepage widget.
|
||||
+29
-42
@@ -1,46 +1,33 @@
|
||||
# Recommendations — next steps for documentation & IaC
|
||||
# Recommendations — status
|
||||
|
||||
Prioritized ideas to make the lab fully reproducible and operable.
|
||||
## Completed
|
||||
|
||||
## High priority
|
||||
- [x] Trino catalog properties
|
||||
- [x] Grafana ini + dashboard provisioning
|
||||
- [x] Spark jobs (`/opt/spark-jobs`)
|
||||
- [x] Airflow `airflow.cfg` + DAG scripts
|
||||
- [x] Kafka KRaft `server.properties`
|
||||
- [x] ObjectScale `deploy.yml`
|
||||
- [x] Elasticsearch `elasticsearch.yml`
|
||||
- [x] Docker compose per host
|
||||
- [x] LDAP LDIF exports (mgt01)
|
||||
- [x] NPM compose (mgt01)
|
||||
- [x] Proxmox VM inventory JSON
|
||||
- [x] SSH mesh (12 hosts + ObjectScale)
|
||||
- [x] Architecture diagram page
|
||||
- [x] `collect-fleet-config.sh` + `populate-repo.py`
|
||||
- [x] Backup script
|
||||
- [x] Health check script
|
||||
- [x] Cron documentation
|
||||
- [x] Ansible playbook + inventory
|
||||
- [x] Gitea CI validate workflow
|
||||
- [x] Network documentation
|
||||
- [x] Homepage DOCS tab
|
||||
|
||||
| Item | Host | Why |
|
||||
|------|------|-----|
|
||||
| **Trino catalog properties** | atc-lake01 | `etc/catalog/*.properties` — documents federated queries |
|
||||
| **Grafana `grafana.ini` + datasources** | atc-grafana | Monitoring as code |
|
||||
| **Kibana / ES keystore note** | atc-elastic01 | Passwords in keystore — document enrollment, not files |
|
||||
| **Proxmox VM notes** | pve01 | VMID → hostname → IP table (API export script) |
|
||||
| **Backup script** | docker01 | Nightly `git pull` + volume tarballs to ObjectScale |
|
||||
## Optional next steps
|
||||
|
||||
## Medium priority
|
||||
|
||||
| Item | Host | Why |
|
||||
|------|------|-----|
|
||||
| **Spark jobs** | atc-lake01 | `/opt/spark-jobs/` in git |
|
||||
| **Airflow `dags/scripts/`** | atc-airflow01 | Supporting Python for DAGs |
|
||||
| **Nginx Proxy Manager** | atc-mgt01 | Export NPM config (if used for TLS) |
|
||||
| **LDAP/LDIF exports** | atc-mgt01 | `*.ldif` already on host — useful for LDAP rebuild |
|
||||
| **MinIO / S3 buckets** | objectscale | Bucket layout + IAM policy docs |
|
||||
| **Network diagram** | docs | VLAN / firewall rules (10.0.10/20/21.x) |
|
||||
|
||||
## Automation
|
||||
|
||||
| Item | Description |
|
||||
|------|-------------|
|
||||
| **CI on Forgejo** | Lint YAML, validate compose, dry-run `docker compose config` |
|
||||
| **Ansible playbook** | `ansible-playbook deploy-lakehouse.yml` from git |
|
||||
| **Health check script** | Cron: curl all `siteMonitor` URLs, alert via Grafana |
|
||||
| **Monthly collect cron** | `collect-fleet-config.sh` + auto-commit branch |
|
||||
|
||||
## Security hardening (lab → prod path)
|
||||
|
||||
- Move all secrets to `.env` / Vault; git only `*.example`
|
||||
- Rotate `atc_cluster` SSH key periodically
|
||||
- Restrict ObjectScale `management_clients` from `0.0.0.0/0`
|
||||
- Enable TLS on Kafka (`SASL_SSL`) if exposed beyond lab VLAN
|
||||
|
||||
## Dashboard enhancements
|
||||
|
||||
- Homepage widget: link to architecture diagram in header logo
|
||||
- Add **DOCS** tab with bookmarks to all `docs/*.md` on Forgejo
|
||||
- Version badge in footer (git commit SHA from build arg)
|
||||
- [ ] Kibana `kibana.yml` export
|
||||
- [ ] TLS certificates inventory (NPM letsencrypt paths)
|
||||
- [ ] Automated weekly git commit via cron (install from `scripts/cron/README.md`)
|
||||
- [ ] Vault/external secrets instead of redacted files
|
||||
- [ ] Reach remaining hosts: `10.0.21.52`, `.37`, `.38` (no SSH key yet)
|
||||
|
||||
Reference in New Issue
Block a user