Add Airflow and Kafka broker configs to git

- config/airflow/airflow.cfg (fernet/API keys redacted) + live DAG sync
- config/kafka/kraft-server.properties (live KRaft broker on kafka01)
- Extend collect-fleet-config.sh for airflow/kafka collection
- Add docs/recommendations.md with prioritized next steps
This commit is contained in:
Lakehouse Admin
2026-05-19 23:08:04 +02:00
parent 3f85ec8034
commit 7de4f5d73a
10 changed files with 351 additions and 36 deletions
+1 -1
View File
@@ -47,7 +47,7 @@ Compose: `config/docker/atc-lake01/docker-compose.yml`
|-----------|-------|-------|
| kafka-ui | provectuslabs/kafka-ui | 9000→8080 |
Kafka broker: native on `:9092` (not containerized in current lab).
Kafka broker: **KRaft** native on `:9092` — config: `config/kafka/kraft-server.properties`.
## atc-mgt01 (10.0.20.104)
+2 -2
View File
@@ -108,9 +108,9 @@ flowchart TB
| Spark, Trino, Debezium | Docker (lake01) | `config/docker/atc-lake01/` |
| Source DBs | Docker (db02) | `config/docker/atc-db02/` |
| Kafka UI | Docker (kafka01) | `config/docker/atc-kafka01/` |
| Kafka broker | Native/systemd (kafka01) | `config/kafka/` |
| Kafka broker | Native KRaft (kafka01) | `config/kafka/kraft-server.properties` |
| Elasticsearch | Native (elastic01) | `config/elastic/` |
| Airflow | Native (airflow01) | `config/airflow/` |
| Airflow | Native (airflow01) | `config/airflow/airflow.cfg`, DAGs |
| ObjectScale | ECS appliance | `config/objectscale/` |
| Forgejo | Docker (mgt01) | `compose/forgejo/` |
+46
View File
@@ -0,0 +1,46 @@
# Recommendations — next steps for documentation & IaC
Prioritized ideas to make the lab fully reproducible and operable.
## High priority
| Item | Host | Why |
|------|------|-----|
| **Trino catalog properties** | atc-lake01 | `etc/catalog/*.properties` — documents federated queries |
| **Grafana `grafana.ini` + datasources** | atc-grafana | Monitoring as code |
| **Kibana / ES keystore note** | atc-elastic01 | Passwords in keystore — document enrollment, not files |
| **Proxmox VM notes** | pve01 | VMID → hostname → IP table (API export script) |
| **Backup script** | docker01 | Nightly `git pull` + volume tarballs to ObjectScale |
## Medium priority
| Item | Host | Why |
|------|------|-----|
| **Spark jobs** | atc-lake01 | `/opt/spark-jobs/` in git |
| **Airflow `dags/scripts/`** | atc-airflow01 | Supporting Python for DAGs |
| **Nginx Proxy Manager** | atc-mgt01 | Export NPM config (if used for TLS) |
| **LDAP/LDIF exports** | atc-mgt01 | `*.ldif` already on host — useful for LDAP rebuild |
| **MinIO / S3 buckets** | objectscale | Bucket layout + IAM policy docs |
| **Network diagram** | docs | VLAN / firewall rules (10.0.10/20/21.x) |
## Automation
| Item | Description |
|------|-------------|
| **CI on Forgejo** | Lint YAML, validate compose, dry-run `docker compose config` |
| **Ansible playbook** | `ansible-playbook deploy-lakehouse.yml` from git |
| **Health check script** | Cron: curl all `siteMonitor` URLs, alert via Grafana |
| **Monthly collect cron** | `collect-fleet-config.sh` + auto-commit branch |
## Security hardening (lab → prod path)
- Move all secrets to `.env` / Vault; git only `*.example`
- Rotate `atc_cluster` SSH key periodically
- Restrict ObjectScale `management_clients` from `0.0.0.0/0`
- Enable TLS on Kafka (`SASL_SSL`) if exposed beyond lab VLAN
## Dashboard enhancements
- Homepage widget: link to architecture diagram in header logo
- Add **DOCS** tab with bookmarks to all `docs/*.md` on Forgejo
- Version badge in footer (git commit SHA from build arg)