ATC/Lakehouse
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Airflow and Kafka broker configs to git

Browse Source 
- config/airflow/airflow.cfg (fernet/API keys redacted) + live DAG sync
- config/kafka/kraft-server.properties (live KRaft broker on kafka01)
- Extend collect-fleet-config.sh for airflow/kafka collection
- Add docs/recommendations.md with prioritized next steps
This commit is contained in:
Lakehouse Admin
2026-05-19 23:08:04 +02:00
parent 3f85ec8034
commit 7de4f5d73a
10 changed files with 351 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
config/airflow/README.md
+30 -9
View File
@@ -1,14 +1,35 @@
# Airflow
# Airflow — atc-airflow01
| Bestand | Beschrijving |
|---------|--------------|
| `airflow.cfg` | Referentie-configuratie (deploy naar Airflow home op lake01) |
| `generate_data_dag.py` | DAG voor demo data-generatie |
| Item | Value |
|------|-------|
| Host | `atc-airflow01` / `10.0.21.55` |
| URL | http://atc-airflow01:8080/ or http://10.0.21.55:8080/ |
| Home | `/root/airflow/` |
| Config | `/root/airflow/airflow.cfg` |
| DAGs | `/root/airflow/dags/` |
| DB | SQLite (`airflow.db` — not in git) |
## URL
## Files in git
http://atc-lake01.dell-atc.lan:8080/ (of `10.0.21.55:8080` volgens netwerk)
| File | Notes |
|------|-------|
| `airflow.cfg` | Reference copy; `fernet_key` and `internal_api_secret_key` redacted |
| `generate_data_dag.py` | Synced from live DAGs folder |
| `../airflow/airflow.cfg` | Older path (legacy); prefer this directory |
## Deploy
## Restore
Kopieer `generate_data_dag.py` naar de Airflow `dags/` folder op de lake-host en herstart de scheduler/webserver.
```bash
# On atc-airflow01
mkdir -p /root/airflow/dags
cp airflow.cfg /root/airflow/
cp generate_data_dag.py /root/airflow/dags/
# Set real fernet_key in airflow.cfg or via env
systemctl restart airflow-webserver airflow-scheduler # if using systemd
```
## Refresh
```bash
./scripts/collect/collect-fleet-config.sh
```
config/airflow/airflow.cfg
+3 -3
View File
@@ -169,7 +169,7 @@ execute_tasks_new_python_interpreter = False
#
# Variable: AIRFLOW__CORE__FERNET_KEY
#
fernet_key = f4n7uT66HPrl8yxqKHVtme_qINN9YlZ1xUQcZy761MY=
fernet_key = REDACTED
# Whether to disable pickling dags
#
@@ -445,7 +445,7 @@ database_access_isolation = False
#
# Variable: AIRFLOW__CORE__INTERNAL_API_SECRET_KEY
#
internal_api_secret_key = 010RL08807/JBjH4cWzNaw==
internal_api_secret_key = REDACTED
# The ability to allow testing connections across Airflow UI, API and CLI.
# Supported options: ``Disabled``, ``Enabled``, ``Hidden``. Default: Disabled
@@ -1193,7 +1193,7 @@ enable_swagger_ui = True
#
# Variable: AIRFLOW__API__SECRET_KEY
#
secret_key = 010RL08807/JBjH4cWzNaw==
secret_key = REDACTED
# Expose the configuration file in the web server. Set to ``non-sensitive-only`` to show all values
# except those that have security implications. ``True`` shows all values. ``False`` hides the