Lakehouse Admin
7de4f5d73a
- config/airflow/airflow.cfg (fernet/API keys redacted) + live DAG sync - config/kafka/kraft-server.properties (live KRaft broker on kafka01) - Extend collect-fleet-config.sh for airflow/kafka collection - Add docs/recommendations.md with prioritized next steps
47 lines
2.0 KiB
Markdown
47 lines
2.0 KiB
Markdown
# Recommendations — next steps for documentation & IaC
|
|
|
|
Prioritized ideas to make the lab fully reproducible and operable.
|
|
|
|
## High priority
|
|
|
|
| Item | Host | Why |
|
|
|------|------|-----|
|
|
| **Trino catalog properties** | atc-lake01 | `etc/catalog/*.properties` — documents federated queries |
|
|
| **Grafana `grafana.ini` + datasources** | atc-grafana | Monitoring as code |
|
|
| **Kibana / ES keystore note** | atc-elastic01 | Passwords in keystore — document enrollment, not files |
|
|
| **Proxmox VM notes** | pve01 | VMID → hostname → IP table (API export script) |
|
|
| **Backup script** | docker01 | Nightly `git pull` + volume tarballs to ObjectScale |
|
|
|
|
## Medium priority
|
|
|
|
| Item | Host | Why |
|
|
|------|------|-----|
|
|
| **Spark jobs** | atc-lake01 | `/opt/spark-jobs/` in git |
|
|
| **Airflow `dags/scripts/`** | atc-airflow01 | Supporting Python for DAGs |
|
|
| **Nginx Proxy Manager** | atc-mgt01 | Export NPM config (if used for TLS) |
|
|
| **LDAP/LDIF exports** | atc-mgt01 | `*.ldif` already on host — useful for LDAP rebuild |
|
|
| **MinIO / S3 buckets** | objectscale | Bucket layout + IAM policy docs |
|
|
| **Network diagram** | docs | VLAN / firewall rules (10.0.10/20/21.x) |
|
|
|
|
## Automation
|
|
|
|
| Item | Description |
|
|
|------|-------------|
|
|
| **CI on Forgejo** | Lint YAML, validate compose, dry-run `docker compose config` |
|
|
| **Ansible playbook** | `ansible-playbook deploy-lakehouse.yml` from git |
|
|
| **Health check script** | Cron: curl all `siteMonitor` URLs, alert via Grafana |
|
|
| **Monthly collect cron** | `collect-fleet-config.sh` + auto-commit branch |
|
|
|
|
## Security hardening (lab → prod path)
|
|
|
|
- Move all secrets to `.env` / Vault; git only `*.example`
|
|
- Rotate `atc_cluster` SSH key periodically
|
|
- Restrict ObjectScale `management_clients` from `0.0.0.0/0`
|
|
- Enable TLS on Kafka (`SASL_SSL`) if exposed beyond lab VLAN
|
|
|
|
## Dashboard enhancements
|
|
|
|
- Homepage widget: link to architecture diagram in header logo
|
|
- Add **DOCS** tab with bookmarks to all `docs/*.md` on Forgejo
|
|
- Version badge in footer (git commit SHA from build arg)
|