Widen Trino ingestion to every catalog (postgres_sales/mysql_hr/mongodb/iceberg)
and add messaging (Kafka), storage (S3 object store), dashboards (Superset) and
search (Elasticsearch) service ingestion configs so OpenMetadata reflects the
whole running platform. Secrets are injected at run time, not committed.
- deploy/airflow/mask_to_curated_dag.py: Trino-SQL masking (hash/redact/generalize)
of PII from postgres/mysql sources into iceberg.curated_masked.* (verified:
14k+11k masked rows, all PII masked).
- OM source->masked lineage edges created; curated tables cataloged.
- Version OM compose + ingestion configs + om_api helper under deploy/openmetadata.
Source generators (postgres/mysql/mongodb) now emit realistic PII
(name/email/phone/ip/iban/address/national_id/dob). deploy/mask_pii_setup.py
ALTERs the source tables and backfills a bounded sample so Debezium CDC and
OpenMetadata PII auto-classification see real sensitive values.
mo