Trino Federation tab (3 sub-views):
- Federated: catalog landscape + a single cross-source SQL that joins
PostgreSQL + MySQL + MongoDB (region scorecard) — the federation proof,
computed in the background and cached (large full scans take ~2 min).
- Hadoop Lake: all federated business data materialized as external Iceberg
tables on HDFS (iceberg.hadoop.*_ext, ~120k rows each) with live, fast
business analytics (revenue by region/channel, top customers, HR by
department, supply by type, telemetry averages). Includes a one-click
"rebuild external tables" job.
- Data Dictionary: every business table + column with masked / visible PII
badges and categories.
Backend trino_federated.py: /catalogs, /marquee(+refresh), /lake,
/materialize(+status), /dictionary. Name-based PII detection flags raw PII
in derived/lake tables as visible vs physically-masked curated layer.
LLM context: platform_context now emits a full BUSINESS DATA CATALOG section
(tables, columns, types, source row counts, federated scorecard) with exact
per-column masked/visible status, so the assistant knows the data in detail
and what is masked vs not.
Add a dedicated business-data view (separate from infra/search):
- New "Data Explorer" tab: KPIs (customers, employees, products, source
row totals, sample revenue) + charts driven by Elasticsearch aggregations
(revenue over time, by region/channel/status, top customers & products,
HR by department/role/event, supply chain by type, telemetry averages),
with region + free-text filters.
- Backend /api/search/business endpoint: battery of ES aggregations with
numeric/date index template (so amount sums and date histograms work),
source totals via cheap planner estimates; biased the orders sample to
rows carrying customer names so people are searchable/visible.
Manual data entry on source systems:
- "Insert row" action in the Data Hub browser opens a column-aware form;
POST /api/sql/row/insert writes to Postgres/MySQL/Mongo/Cassandra/Neo4j.
- Inserts into CDC sources (PG/MySQL/Mongo) are captured by Debezium and
streamed to Kafka in real time; UI flags this and pulses the data flow.
Build dedicated de-duplicated entity indices (atc-customers ~39k,
atc-employees ~39k, atc-products) from Trino so users can search by
customer/employee name, id and email directly instead of scanning raw
transaction rows. Raise per-table sample caps for the big source tables
(sales_orders, employee_events, supplychain events, device_metrics) and
all iceberg/Hadoop tables for broader row-level coverage.
UI: add Datasets quick-filter chips (Customers, Employees, Products,
Sales orders, HR events, Supply chain, Telemetry, Lakehouse) and default
the Search tab to business data instead of an infra-looking empty view.
- Each pipeline node is now a button; clicking shows a detail card (what the step does + the LangChain/tech component)
- Stronger active-stage pulse (expanding ring + brighter glow)
- Chat auto-opens the How-it-works panel so live per-stage pulsing is always visible while answering
- rag-api: new POST /chat/stream SSE endpoint emits live pipeline stages (embed/retrieve/context/llm/answer) and streams LLM tokens
- Knowledge Chat: document chat now streams answers token-by-token and pulses each pipeline stage in real time as it executes
- How-it-works panel: active stage glows/scales, completed stages settle, agent mode also drives the pulse
- LangChain made visible: orchestrated-by-LangChain badge + LC markers on LangChain-native nodes (TextSplitter, Embeddings, as_retriever, ChatOpenAI, Chroma)
Add a collapsible, pulsing architecture diagram to the Knowledge Chat that
visualizes the real RAG pipeline in two rows: Index (Upload -> Docling ->
Split -> Embed/MiniLM -> ChromaDB) and Query (Question -> Embed -> Retrieve
-> Context -> LLM -> Answer), badged with the actual stack (LangChain,
ChromaDB, Docling, sentence-transformers).
Connectors have a flowing dashed track plus a traveling glow dot and nodes
pulse; the whole flow speeds up while the chat/ingest is busy. Honors
prefers-reduced-motion.
When switching source engine, the sample effect fired with the previous
engine selected object (e.g. public.sales_orders against MySQL), surfacing
"Table hr.sales_orders doesn't exist". Track which engine the selected
object belongs to and only sample when it matches the active engine, plus
guard against stale responses overwriting newer ones.
Row-count for the browser used an exact count(*) which full-scanned huge
tables (~30s on 24M rows). Use planner/statistics estimates and only run a
time-bounded exact count for small (<=50k) tables.
Add a /preview endpoint that range-reads the first chunk of an object and
returns it as text. Browser gets an eye action (list + gallery) opening a
modal that pretty-prints JSON, renders CSV/TSV as a table, and shows logs/
text/yaml/xml verbatim, with a truncation note and download link.
Add a List/Gallery toggle to the bucket browser that auto-switches to a
thumbnail grid when a listing is mostly images, and a keyboard-navigable
lightbox (prev/next/esc) for full-size previews. Download endpoint serves
inline with the correct image/* content-type (ECS stores octet-stream) so
previews render instead of forcing a download.
Analytics now walks every top-level prefix with its own budget instead of a
flat alphabetical scan, so a single huge prefix (kafka/ CDC json) no longer
hides the rest — composition now correctly reflects images, logs, csv,
parquet and avro, and reports true total size.
Browser gains recursive (whole-subtree) listing, free-text name search,
file-type filter chips, type column and Load-more pagination so every object
across all folders is discoverable.
Add /api/storage/s3/analytics endpoint that scans buckets (bounded +
cached) to compute total size/objects, per-bucket distribution, file-type
and size-class breakdowns, cumulative data-growth timeline and largest /
recent objects. Track every S3 op routed through the API for a live
storage-activity timeline.
Rebuild StorageView into a tabbed view: Overview (KPI cards + SVG charts:
growth area, bucket donut, type/size bars, activity sparkline, top folders,
largest & recent objects) and the original bucket Browser.
Add Graph sub-tab in Data Sources UI for Neo4j: force-directed SVG view of
Product-Supplier nodes and SUPPLIES/RELATED_TO/PART_OF/COMPATIBLE_WITH edges.
Backend GET /api/sql/graph/neo4j with rel_type filter and edge limit.
New "Data Sources UI" tab with per-database Browser (catalog + sample data),
Query Console (SqlWorkbench) and embedded interactive Shell (DbShell via SSH).
Backend:
- Extend sql_console.py with Cassandra (CQL) + Neo4j (Cypher) engines
- Add GET /api/sql/catalog/{engine} and GET /api/sql/sample/{engine}
- ssh_terminal: optional initial_command for auto-launching DB CLIs
Frontend:
- DataSourcesView with 5-DB rail, health dots, Browser/Console/Shell sub-tabs
- DbShell embedded xterm terminal with docker exec CLI per engine
- Deep-link topology DB nodes to Data Sources UI (no SQL dock on platform)
- WorkbenchPanel restricted to agent mode only — frees dashboard space
Assign owners to all 177 tables and descriptions + tier tags to the 18
business tables, then trigger SearchIndexing + DataInsights apps so the
platform Data Insights page shows real totals, description coverage and
tier distribution instead of empty plots.
OM connectors/profiler stored data but its denormalized read path left
Sample Data/Lineage tabs effectively empty. This script populates OM directly:
- real 50-row sample data for source + iceberg curated tables
- table/column profiles (column profiles read back correctly in UI)
- full traceable lineage: generator -> source -> Debezium/Kafka CDC topic
-> S3 archive + Iceberg curated -> Trino query layer
Profiler:
- pg/mysql: table-level metrics via DB statistics (exact-ish row counts) only;
source tables hold 24-53M rows and OM's column profiler full-scans per column
(no TABLESAMPLE pushdown), which would hammer the live CDC source.
- trino/iceberg: full column metrics + sample data on the small (20-25k row)
curated_masked + hadoop tables.
Data quality: 16 test cases across 5 tables (row counts on big sources; row
count + uniqueness/not-null/range on curated/masked + historical), all passing.
This populates OM's Data Quality dashboard (coverage, healthy assets,
dimensions, test results).
The OM MySQL datadir was bind-mounted on the 448MB /opt partition and kept
filling up. Moved db-data to /var/lib/docker/openmetadata-db-data (32GB, ~31GB
free) and kept binlog disabled + redo log capped. /opt back to 7% used.
OM 1.13 cannot deserialize Airflow 3.0.6 serialized DAGs, so pipelines had no
tasks. af_tasks.py parses serialized_dag and PATCHes task lists; af_lineage.py
adds generator->table and mask_to_curated/hadoop_to_trino lineage edges.
Note: OM MySQL datadir lives on the 448MB /opt partition which had filled up
(103MB binlog). Disabled binary logging + shrank redo log capacity in the OM
docker-compose to restore headroom.
- ES: connect over https with verify_certs=false (self-signed CA); 16 indexes incl source_data
- Airflow: ingest 8 DAGs as pipelines from SQLite metadata DB snapshot
- es_run.py: standalone runner that pins the working ES TLS config
- Command Center API now wired with ELASTIC_PASSWORD
Widen Trino ingestion to every catalog (postgres_sales/mysql_hr/mongodb/iceberg)
and add messaging (Kafka), storage (S3 object store), dashboards (Superset) and
search (Elasticsearch) service ingestion configs so OpenMetadata reflects the
whole running platform. Secrets are injected at run time, not committed.
Align nodes into pipeline columns (producers -> sources -> CDC -> lakehouse ->
Trino) with governance centred at the bottom, so lineage reads in order instead
of Iceberg/Trino floating mid-canvas.
Query postgres/mysql/mongo directly (fast, early LIMIT) instead of full Trino
table scans; Trino remains the path for the curated lakehouse + as a fallback.
- pii_catalog: persistent per-column masking policy (default masked); GET/POST
/api/pii/policy and POST /api/pii/lookup which redacts masked values server-side.
- get_pii masked flag now reflects the policy; dataflow exposes the dataset key.
- Data Flow PII inspector: per-column lock/unlock toggles + mask-all/unmask-all,
so operators control exactly which data the assistant may reveal.
- decide_approval now executes the underlying data movement when an approved
request carries an executor=movement payload (real human-gated executor).
- rag-api service gets OPENMETADATA_* (via atc.env) + COMMAND_CENTER_URL so it can
sync the catalog and call platform tools.
- Knowledge Chat gains an Agent-mode toggle (SSE tool-loop with step chips) and a
'Sync catalog' button.
lab_context now includes an in-process governance section so the LLM knows the
live CDC volume, data movements + last-run state, the PII catalog (OM+heuristic)
with masked/unmasked status, and source->masked lineage.
- deploy/airflow/mask_to_curated_dag.py: Trino-SQL masking (hash/redact/generalize)
of PII from postgres/mysql sources into iceberg.curated_masked.* (verified:
14k+11k masked rows, all PII masked).
- OM source->masked lineage edges created; curated tables cataloged.
- Version OM compose + ingestion configs + om_api helper under deploy/openmetadata.
Source generators (postgres/mysql/mongodb) now emit realistic PII
(name/email/phone/ip/iban/address/national_id/dob). deploy/mask_pii_setup.py
ALTERs the source tables and backfills a bounded sample so Debezium CDC and
OpenMetadata PII auto-classification see real sensitive values.
Data Flow graph now shows OpenMetadata as a governance node linked to all
sources and Trino (catalog edges); node inspector exposes an 'Open in
OpenMetadata' deep link and PII-columns-cataloged metric.
Add OpenMetadata node (atc-docker02 .47) to node_registry with UI links.
pii_catalog now reads OM column PII tags (Presidio auto-classification) as the
authoritative source, merged with the name heuristic; OPENMETADATA_URL wired
into the api service (token via atc.env).
- DAG hadoop_to_trino (deployed to Airflow) + worker hadoop_to_trino.py (on the
Hadoop master) move HDFS historical_sales -> iceberg.hadoop.historical_sales_hdfs.
- api/movements.py: movement registry, Airflow trigger+watch, run tracking
(state/duration/rows), endpoints /api/movements, /{id}/run, /runs.
- agent_ops.py: ETL-agent loop autonomously triggers movements on an interval
and logs each run; /api/agent-ops/etl/toggle + etl status.
Add api/cdc_consumer.py: aiokafka background consumer subscribes to the CDC
topics (postgres_sales/mysql_hr/mongodb_supplychain + cassandra/neo4j), parses
Debezium before/after envelopes, keeps a ring buffer and publishes each change
live as type=cdc_change. Endpoints /api/changes, /api/changes/stats, /status.
Deploys Apache Hive 3.1.3 (Derby metastore + external table over the HDFS
historical CSV, MapReduce exec) on the Hadoop master, so the engine comparison
shows a REAL measured Hive latency (~5.2s) next to live Trino (~0.3s); Impala
stays clearly-labelled representative. The API re-measures Hive over SSH on a
30-min TTL (cached + persisted, with a committed seed). Adds filters
(year/region/category/channel), a region×category heatmap, Trino exec stats,
and a "where is this data read from" lineage panel (Trino->S3/Iceberg/Parquet
with snapshot+files, Hive->HDFS/CSV with namenode+files). Mounts host SSH key
read-only into the api container for the live Hive benchmark.
Adds an Analytics sub-tab to the Hadoop view with live KPIs and revenue
breakdowns (by year/region/category/channel) queried from Trino over
iceberg.hadoop.historical_sales, plus a query-engine comparison panel. Trino
latency is measured live; Impala and Hive are shown as clearly-labelled
representative figures (those engines are not deployed). New cached endpoints
/api/hadoop/analytics and /api/hadoop/engines.
Talisman forced Secure on the session cookie while Superset is served over
plain HTTP, so browsers never returned the cookie and every login failed CSRF
validation ("CSRF session token is missing"). Set session_cookie_secure=False
(+ SESSION_COOKIE_SECURE/WTF_CSRF_SSL_STRICT). Also rename S3 lakehouse prefix
iceberg-warehouse -> hadoop (Trino iceberg.hadoop.historical_sales) and repoint
the Superset dataset. Adds sanitized infra backup scripts.
mo