Add Airflow and Kafka broker configs to git

- config/airflow/airflow.cfg (fernet/API keys redacted) + live DAG sync
- config/kafka/kraft-server.properties (live KRaft broker on kafka01)
- Extend collect-fleet-config.sh for airflow/kafka collection
- Add docs/recommendations.md with prioritized next steps
This commit is contained in:
Lakehouse Admin
2026-05-19 23:08:04 +02:00
parent 3f85ec8034
commit 7de4f5d73a
10 changed files with 351 additions and 36 deletions
+22 -17
View File
@@ -1,25 +1,30 @@
# Kafka Connect — Debezium connectors
# Kafka — atc-kafka01
JSON-definities voor CDC van databases naar Kafka.
| Item | Value |
|------|-------|
| Host | `atc-kafka01` / `10.0.21.36` |
| Mode | **KRaft** (no ZooKeeper) |
| Broker | `10.0.21.36:9092` |
| Controller | `localhost:9093` |
| UI | Kafka UI Docker — port `9000` |
| Data dir | `/home/kafka/data` |
| Systemd | `kafka.service``kraft/server.properties` |
## Connectors
## Files
| Bestand | Bron | Topic prefix |
|---------|------|--------------|
| `postgres-connector.json` | PostgreSQL op db02 | `postgres-sales` |
| `mysql-connector.json` | MySQL | — |
| `mongodb-connector.json` | MongoDB | — |
| File | Purpose |
|------|---------|
| `kraft-server.properties` | **Live** broker config (systemd) |
| `server.properties` | Legacy ZK template (not used by current service) |
## Deploy
## Connect
Vervang `PASSWORD_PLACEHOLDER` en pas host/poort aan. Registreer via Kafka Connect REST API op `atc-lake01` (Debezium `:8083`).
Debezium Connect runs on **atc-lake01:8083** (`BOOTSTRAP_SERVERS=10.0.21.36:9092`).
Connector JSON: `../kafka/*.json` (postgres, mysql, mongodb).
## Refresh
```bash
curl -X POST -H "Content-Type: application/json" \
--data @postgres-connector.json \
http://atc-lake01.dell-atc.lan:8083/connectors/
./scripts/collect/collect-fleet-config.sh
```
## Referentie
- Kafka UI: http://atc-kafka01.dell-atc.lan:9000/