ATC/Lakehouse
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document lakehouse Debezium connectors and updated docker inventory.

Browse Source 
Add atc-lake01 compose with s3-kafka-consumer; redact DB passwords in connector JSON.
This commit is contained in:
mo
2026-06-04 02:27:23 +02:00
parent 8f2bb06348
commit 8eb64381e3
6 changed files with 136 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/docker-inventory.md
+13 -2
View File
@@ -34,12 +34,23 @@ Compose: `config/docker/atc-db02/docker-compose.yml`
| Container | Image | Ports |
|-----------|-------|-------|
| kafka-connect | debezium/connect:2.5.4 | 8083 |
| spark-master | bitnami/spark | 7077, 8080 |
| s3-kafka-consumer | python:3.11-slim | (internal) |
| spark-master | bitnami/spark | 7077, 8081→8080 |
| spark-worker | bitnami/spark | (internal) |
| trino | trinodb/trino:405 | 8089→8080 |
| spark-temp | apache/spark:3.4.0 | — |
Compose: `config/docker/atc-lake01/docker-compose.yml`
Debezium connectors: `postgres-sales-connector`, `mysql-hr-connector`, `mongodb-supplychain-connector` (see `config/debezium/connectors/`).
Compose: `compose/atc-lake01/docker-compose.yml`
## atc-airflow01 (10.0.21.55)
| Container | Image | Ports |
|-----------|-------|-------|
| airflow | atc-airflow:3.0.6 | 8080 |
DAG `generate_data_all_databases` — data generators for atc-db02. Compose: `/opt/airflow/docker-compose.yml` on host.
## atc-kafka01 (10.0.21.36)