Source generators (postgres/mysql/mongodb) now emit realistic PII
(name/email/phone/ip/iban/address/national_id/dob). deploy/mask_pii_setup.py
ALTERs the source tables and backfills a bounded sample so Debezium CDC and
OpenMetadata PII auto-classification see real sensitive values.
Data Flow graph now shows OpenMetadata as a governance node linked to all
sources and Trino (catalog edges); node inspector exposes an 'Open in
OpenMetadata' deep link and PII-columns-cataloged metric.
Add OpenMetadata node (atc-docker02 .47) to node_registry with UI links.
pii_catalog now reads OM column PII tags (Presidio auto-classification) as the
authoritative source, merged with the name heuristic; OPENMETADATA_URL wired
into the api service (token via atc.env).
- DAG hadoop_to_trino (deployed to Airflow) + worker hadoop_to_trino.py (on the
Hadoop master) move HDFS historical_sales -> iceberg.hadoop.historical_sales_hdfs.
- api/movements.py: movement registry, Airflow trigger+watch, run tracking
(state/duration/rows), endpoints /api/movements, /{id}/run, /runs.
- agent_ops.py: ETL-agent loop autonomously triggers movements on an interval
and logs each run; /api/agent-ops/etl/toggle + etl status.
Add api/cdc_consumer.py: aiokafka background consumer subscribes to the CDC
topics (postgres_sales/mysql_hr/mongodb_supplychain + cassandra/neo4j), parses
Debezium before/after envelopes, keeps a ring buffer and publishes each change
live as type=cdc_change. Endpoints /api/changes, /api/changes/stats, /status.
Deploys Apache Hive 3.1.3 (Derby metastore + external table over the HDFS
historical CSV, MapReduce exec) on the Hadoop master, so the engine comparison
shows a REAL measured Hive latency (~5.2s) next to live Trino (~0.3s); Impala
stays clearly-labelled representative. The API re-measures Hive over SSH on a
30-min TTL (cached + persisted, with a committed seed). Adds filters
(year/region/category/channel), a region×category heatmap, Trino exec stats,
and a "where is this data read from" lineage panel (Trino->S3/Iceberg/Parquet
with snapshot+files, Hive->HDFS/CSV with namenode+files). Mounts host SSH key
read-only into the api container for the live Hive benchmark.
Adds an Analytics sub-tab to the Hadoop view with live KPIs and revenue
breakdowns (by year/region/category/channel) queried from Trino over
iceberg.hadoop.historical_sales, plus a query-engine comparison panel. Trino
latency is measured live; Impala and Hive are shown as clearly-labelled
representative figures (those engines are not deployed). New cached endpoints
/api/hadoop/analytics and /api/hadoop/engines.
Talisman forced Secure on the session cookie while Superset is served over
plain HTTP, so browsers never returned the cookie and every login failed CSRF
validation ("CSRF session token is missing"). Set session_cookie_secure=False
(+ SESSION_COOKIE_SECURE/WTF_CSRF_SSL_STRICT). Also rename S3 lakehouse prefix
iceberg-warehouse -> hadoop (Trino iceberg.hadoop.historical_sales) and repoint
the Superset dataset. Adds sanitized infra backup scripts.
Clicking agents opens a dedicated terminal panel; topology PostgreSQL/Trino nodes open SQL workbench with ten demo queries and Postgres vs Trino benchmark.
mo