Full lab documentation and infrastructure as code

- Trino catalogs, Grafana, Spark jobs, LDAP LDIF, NPM compose - Airflow DAG scripts, Proxmox VM inventory, network docs - Ansible playbook, Gitea CI validate workflow - Backup and health-check scripts, cron documentation - Homepage DOCS tab with links to all documentation - Extended collect-fleet-config.sh and populate-repo.py
2026-05-19 23:12:41 +02:00
parent 3c8993a41e
commit df5ec93dc3
46 changed files with 1809 additions and 74 deletions
@@ -1,46 +1,33 @@
-# Recommendations — next steps for documentation & IaC
+# Recommendations — status

-Prioritized ideas to make the lab fully reproducible and operable.
+## Completed

-## High priority
+- [x] Trino catalog properties
+- [x] Grafana ini + dashboard provisioning
+- [x] Spark jobs (`/opt/spark-jobs`)
+- [x] Airflow `airflow.cfg` + DAG scripts
+- [x] Kafka KRaft `server.properties`
+- [x] ObjectScale `deploy.yml`
+- [x] Elasticsearch `elasticsearch.yml`
+- [x] Docker compose per host
+- [x] LDAP LDIF exports (mgt01)
+- [x] NPM compose (mgt01)
+- [x] Proxmox VM inventory JSON
+- [x] SSH mesh (12 hosts + ObjectScale)
+- [x] Architecture diagram page
+- [x] `collect-fleet-config.sh` + `populate-repo.py`
+- [x] Backup script
+- [x] Health check script
+- [x] Cron documentation
+- [x] Ansible playbook + inventory
+- [x] Gitea CI validate workflow
+- [x] Network documentation
+- [x] Homepage DOCS tab

-| Item | Host | Why |
-|------|------|-----|
-| **Trino catalog properties** | atc-lake01 | `etc/catalog/*.properties` — documents federated queries |
-| **Grafana `grafana.ini` + datasources** | atc-grafana | Monitoring as code |
-| **Kibana / ES keystore note** | atc-elastic01 | Passwords in keystore — document enrollment, not files |
-| **Proxmox VM notes** | pve01 | VMID → hostname → IP table (API export script) |
-| **Backup script** | docker01 | Nightly `git pull` + volume tarballs to ObjectScale |
+## Optional next steps

-## Medium priority
-
-| Item | Host | Why |
-|------|------|-----|
-| **Spark jobs** | atc-lake01 | `/opt/spark-jobs/` in git |
-| **Airflow `dags/scripts/`** | atc-airflow01 | Supporting Python for DAGs |
-| **Nginx Proxy Manager** | atc-mgt01 | Export NPM config (if used for TLS) |
-| **LDAP/LDIF exports** | atc-mgt01 | `*.ldif` already on host — useful for LDAP rebuild |
-| **MinIO / S3 buckets** | objectscale | Bucket layout + IAM policy docs |
-| **Network diagram** | docs | VLAN / firewall rules (10.0.10/20/21.x) |
-
-## Automation
-
-| Item | Description |
-|------|-------------|
-| **CI on Forgejo** | Lint YAML, validate compose, dry-run `docker compose config` |
-| **Ansible playbook** | `ansible-playbook deploy-lakehouse.yml` from git |
-| **Health check script** | Cron: curl all `siteMonitor` URLs, alert via Grafana |
-| **Monthly collect cron** | `collect-fleet-config.sh` + auto-commit branch |
-
-## Security hardening (lab → prod path)
-
- Move all secrets to `.env` / Vault; git only `*.example`
- Rotate `atc_cluster` SSH key periodically
- Restrict ObjectScale `management_clients` from `0.0.0.0/0`
- Enable TLS on Kafka (`SASL_SSL`) if exposed beyond lab VLAN
-
-## Dashboard enhancements
-
- Homepage widget: link to architecture diagram in header logo
- Add **DOCS** tab with bookmarks to all `docs/*.md` on Forgejo
- Version badge in footer (git commit SHA from build arg)
+- [ ] Kibana `kibana.yml` export
+- [ ] TLS certificates inventory (NPM letsencrypt paths)
+- [ ] Automated weekly git commit via cron (install from `scripts/cron/README.md`)
+- [ ] Vault/external secrets instead of redacted files
+- [ ] Reach remaining hosts: `10.0.21.52`, `.37`, `.38` (no SSH key yet)