Lakehouse Admin
7de4f5d73a
- config/airflow/airflow.cfg (fernet/API keys redacted) + live DAG sync - config/kafka/kraft-server.properties (live KRaft broker on kafka01) - Extend collect-fleet-config.sh for airflow/kafka collection - Add docs/recommendations.md with prioritized next steps
2.0 KiB
2.0 KiB
Recommendations — next steps for documentation & IaC
Prioritized ideas to make the lab fully reproducible and operable.
High priority
| Item | Host | Why |
|---|---|---|
| Trino catalog properties | atc-lake01 | etc/catalog/*.properties — documents federated queries |
Grafana grafana.ini + datasources |
atc-grafana | Monitoring as code |
| Kibana / ES keystore note | atc-elastic01 | Passwords in keystore — document enrollment, not files |
| Proxmox VM notes | pve01 | VMID → hostname → IP table (API export script) |
| Backup script | docker01 | Nightly git pull + volume tarballs to ObjectScale |
Medium priority
| Item | Host | Why |
|---|---|---|
| Spark jobs | atc-lake01 | /opt/spark-jobs/ in git |
Airflow dags/scripts/ |
atc-airflow01 | Supporting Python for DAGs |
| Nginx Proxy Manager | atc-mgt01 | Export NPM config (if used for TLS) |
| LDAP/LDIF exports | atc-mgt01 | *.ldif already on host — useful for LDAP rebuild |
| MinIO / S3 buckets | objectscale | Bucket layout + IAM policy docs |
| Network diagram | docs | VLAN / firewall rules (10.0.10/20/21.x) |
Automation
| Item | Description |
|---|---|
| CI on Forgejo | Lint YAML, validate compose, dry-run docker compose config |
| Ansible playbook | ansible-playbook deploy-lakehouse.yml from git |
| Health check script | Cron: curl all siteMonitor URLs, alert via Grafana |
| Monthly collect cron | collect-fleet-config.sh + auto-commit branch |
Security hardening (lab → prod path)
- Move all secrets to
.env/ Vault; git only*.example - Rotate
atc_clusterSSH key periodically - Restrict ObjectScale
management_clientsfrom0.0.0.0/0 - Enable TLS on Kafka (
SASL_SSL) if exposed beyond lab VLAN
Dashboard enhancements
- Homepage widget: link to architecture diagram in header logo
- Add DOCS tab with bookmarks to all
docs/*.mdon Forgejo - Version badge in footer (git commit SHA from build arg)