Assign owners to all 177 tables and descriptions + tier tags to the 18
business tables, then trigger SearchIndexing + DataInsights apps so the
platform Data Insights page shows real totals, description coverage and
tier distribution instead of empty plots.
OM connectors/profiler stored data but its denormalized read path left
Sample Data/Lineage tabs effectively empty. This script populates OM directly:
- real 50-row sample data for source + iceberg curated tables
- table/column profiles (column profiles read back correctly in UI)
- full traceable lineage: generator -> source -> Debezium/Kafka CDC topic
-> S3 archive + Iceberg curated -> Trino query layer
Profiler:
- pg/mysql: table-level metrics via DB statistics (exact-ish row counts) only;
source tables hold 24-53M rows and OM's column profiler full-scans per column
(no TABLESAMPLE pushdown), which would hammer the live CDC source.
- trino/iceberg: full column metrics + sample data on the small (20-25k row)
curated_masked + hadoop tables.
Data quality: 16 test cases across 5 tables (row counts on big sources; row
count + uniqueness/not-null/range on curated/masked + historical), all passing.
This populates OM's Data Quality dashboard (coverage, healthy assets,
dimensions, test results).
The OM MySQL datadir was bind-mounted on the 448MB /opt partition and kept
filling up. Moved db-data to /var/lib/docker/openmetadata-db-data (32GB, ~31GB
free) and kept binlog disabled + redo log capped. /opt back to 7% used.
OM 1.13 cannot deserialize Airflow 3.0.6 serialized DAGs, so pipelines had no
tasks. af_tasks.py parses serialized_dag and PATCHes task lists; af_lineage.py
adds generator->table and mask_to_curated/hadoop_to_trino lineage edges.
Note: OM MySQL datadir lives on the 448MB /opt partition which had filled up
(103MB binlog). Disabled binary logging + shrank redo log capacity in the OM
docker-compose to restore headroom.
- ES: connect over https with verify_certs=false (self-signed CA); 16 indexes incl source_data
- Airflow: ingest 8 DAGs as pipelines from SQLite metadata DB snapshot
- es_run.py: standalone runner that pins the working ES TLS config
- Command Center API now wired with ELASTIC_PASSWORD
Widen Trino ingestion to every catalog (postgres_sales/mysql_hr/mongodb/iceberg)
and add messaging (Kafka), storage (S3 object store), dashboards (Superset) and
search (Elasticsearch) service ingestion configs so OpenMetadata reflects the
whole running platform. Secrets are injected at run time, not committed.
- deploy/airflow/mask_to_curated_dag.py: Trino-SQL masking (hash/redact/generalize)
of PII from postgres/mysql sources into iceberg.curated_masked.* (verified:
14k+11k masked rows, all PII masked).
- OM source->masked lineage edges created; curated tables cataloged.
- Version OM compose + ingestion configs + om_api helper under deploy/openmetadata.
Source generators (postgres/mysql/mongodb) now emit realistic PII
(name/email/phone/ip/iban/address/national_id/dob). deploy/mask_pii_setup.py
ALTERs the source tables and backfills a bounded sample so Debezium CDC and
OpenMetadata PII auto-classification see real sensitive values.
mo