Assign owners to all 177 tables and descriptions + tier tags to the 18
business tables, then trigger SearchIndexing + DataInsights apps so the
platform Data Insights page shows real totals, description coverage and
tier distribution instead of empty plots.
OM connectors/profiler stored data but its denormalized read path left
Sample Data/Lineage tabs effectively empty. This script populates OM directly:
- real 50-row sample data for source + iceberg curated tables
- table/column profiles (column profiles read back correctly in UI)
- full traceable lineage: generator -> source -> Debezium/Kafka CDC topic
-> S3 archive + Iceberg curated -> Trino query layer
Profiler:
- pg/mysql: table-level metrics via DB statistics (exact-ish row counts) only;
source tables hold 24-53M rows and OM's column profiler full-scans per column
(no TABLESAMPLE pushdown), which would hammer the live CDC source.
- trino/iceberg: full column metrics + sample data on the small (20-25k row)
curated_masked + hadoop tables.
Data quality: 16 test cases across 5 tables (row counts on big sources; row
count + uniqueness/not-null/range on curated/masked + historical), all passing.
This populates OM's Data Quality dashboard (coverage, healthy assets,
dimensions, test results).
The OM MySQL datadir was bind-mounted on the 448MB /opt partition and kept
filling up. Moved db-data to /var/lib/docker/openmetadata-db-data (32GB, ~31GB
free) and kept binlog disabled + redo log capped. /opt back to 7% used.
OM 1.13 cannot deserialize Airflow 3.0.6 serialized DAGs, so pipelines had no
tasks. af_tasks.py parses serialized_dag and PATCHes task lists; af_lineage.py
adds generator->table and mask_to_curated/hadoop_to_trino lineage edges.
Note: OM MySQL datadir lives on the 448MB /opt partition which had filled up
(103MB binlog). Disabled binary logging + shrank redo log capacity in the OM
docker-compose to restore headroom.
- ES: connect over https with verify_certs=false (self-signed CA); 16 indexes incl source_data
- Airflow: ingest 8 DAGs as pipelines from SQLite metadata DB snapshot
- es_run.py: standalone runner that pins the working ES TLS config
- Command Center API now wired with ELASTIC_PASSWORD
Widen Trino ingestion to every catalog (postgres_sales/mysql_hr/mongodb/iceberg)
and add messaging (Kafka), storage (S3 object store), dashboards (Superset) and
search (Elasticsearch) service ingestion configs so OpenMetadata reflects the
whole running platform. Secrets are injected at run time, not committed.
Align nodes into pipeline columns (producers -> sources -> CDC -> lakehouse ->
Trino) with governance centred at the bottom, so lineage reads in order instead
of Iceberg/Trino floating mid-canvas.
Query postgres/mysql/mongo directly (fast, early LIMIT) instead of full Trino
table scans; Trino remains the path for the curated lakehouse + as a fallback.
- pii_catalog: persistent per-column masking policy (default masked); GET/POST
/api/pii/policy and POST /api/pii/lookup which redacts masked values server-side.
- get_pii masked flag now reflects the policy; dataflow exposes the dataset key.
- Data Flow PII inspector: per-column lock/unlock toggles + mask-all/unmask-all,
so operators control exactly which data the assistant may reveal.
- decide_approval now executes the underlying data movement when an approved
request carries an executor=movement payload (real human-gated executor).
- rag-api service gets OPENMETADATA_* (via atc.env) + COMMAND_CENTER_URL so it can
sync the catalog and call platform tools.
- Knowledge Chat gains an Agent-mode toggle (SSE tool-loop with step chips) and a
'Sync catalog' button.
lab_context now includes an in-process governance section so the LLM knows the
live CDC volume, data movements + last-run state, the PII catalog (OM+heuristic)
with masked/unmasked status, and source->masked lineage.
- deploy/airflow/mask_to_curated_dag.py: Trino-SQL masking (hash/redact/generalize)
of PII from postgres/mysql sources into iceberg.curated_masked.* (verified:
14k+11k masked rows, all PII masked).
- OM source->masked lineage edges created; curated tables cataloged.
- Version OM compose + ingestion configs + om_api helper under deploy/openmetadata.
Source generators (postgres/mysql/mongodb) now emit realistic PII
(name/email/phone/ip/iban/address/national_id/dob). deploy/mask_pii_setup.py
ALTERs the source tables and backfills a bounded sample so Debezium CDC and
OpenMetadata PII auto-classification see real sensitive values.
Data Flow graph now shows OpenMetadata as a governance node linked to all
sources and Trino (catalog edges); node inspector exposes an 'Open in
OpenMetadata' deep link and PII-columns-cataloged metric.
Add OpenMetadata node (atc-docker02 .47) to node_registry with UI links.
pii_catalog now reads OM column PII tags (Presidio auto-classification) as the
authoritative source, merged with the name heuristic; OPENMETADATA_URL wired
into the api service (token via atc.env).
- DAG hadoop_to_trino (deployed to Airflow) + worker hadoop_to_trino.py (on the
Hadoop master) move HDFS historical_sales -> iceberg.hadoop.historical_sales_hdfs.
- api/movements.py: movement registry, Airflow trigger+watch, run tracking
(state/duration/rows), endpoints /api/movements, /{id}/run, /runs.
- agent_ops.py: ETL-agent loop autonomously triggers movements on an interval
and logs each run; /api/agent-ops/etl/toggle + etl status.
Add api/cdc_consumer.py: aiokafka background consumer subscribes to the CDC
topics (postgres_sales/mysql_hr/mongodb_supplychain + cassandra/neo4j), parses
Debezium before/after envelopes, keeps a ring buffer and publishes each change
live as type=cdc_change. Endpoints /api/changes, /api/changes/stats, /status.
Deploys Apache Hive 3.1.3 (Derby metastore + external table over the HDFS
historical CSV, MapReduce exec) on the Hadoop master, so the engine comparison
shows a REAL measured Hive latency (~5.2s) next to live Trino (~0.3s); Impala
stays clearly-labelled representative. The API re-measures Hive over SSH on a
30-min TTL (cached + persisted, with a committed seed). Adds filters
(year/region/category/channel), a region×category heatmap, Trino exec stats,
and a "where is this data read from" lineage panel (Trino->S3/Iceberg/Parquet
with snapshot+files, Hive->HDFS/CSV with namenode+files). Mounts host SSH key
read-only into the api container for the live Hive benchmark.
Adds an Analytics sub-tab to the Hadoop view with live KPIs and revenue
breakdowns (by year/region/category/channel) queried from Trino over
iceberg.hadoop.historical_sales, plus a query-engine comparison panel. Trino
latency is measured live; Impala and Hive are shown as clearly-labelled
representative figures (those engines are not deployed). New cached endpoints
/api/hadoop/analytics and /api/hadoop/engines.
Talisman forced Secure on the session cookie while Superset is served over
plain HTTP, so browsers never returned the cookie and every login failed CSRF
validation ("CSRF session token is missing"). Set session_cookie_secure=False
(+ SESSION_COOKIE_SECURE/WTF_CSRF_SSL_STRICT). Also rename S3 lakehouse prefix
iceberg-warehouse -> hadoop (Trino iceberg.hadoop.historical_sales) and repoint
the Superset dataset. Adds sanitized infra backup scripts.
mo