- New "Live" tab: auto-polls /api/federated/live every 2.5s with
animated counters, ingestion throughput sparkline, per-source
write-rate bars, a region scorecard matrix (heat-shaded) and live
business breakdown charts (region/channel/status/customers/
telemetry/supply)
- Backend /api/federated/live: instant source estimates (Postgres),
monotonic max(event_id) for MySQL and Mongo estimated count for
immediate movement, Cassandra from cached matrix; business aggs
cached over the small Hadoop lake tables (short TTL)
- Fix embedded Trino panels being squeezed with internal scrollbars
by making panels/grids shrink-0 so the page scrolls instead
The marquee panel only joined 3 region-keyed sources. Add a
"one SQL across every database" reach matrix that fans a single
Trino query out to PostgreSQL, MySQL, MongoDB, Cassandra and the
Hadoop/Iceberg lake in one UNION ALL (telemetry has no region, so
a per-source summary is used instead of a misleading join).
- New MATRIX_SQL + concurrent execution alongside the region
scorecard so total latency stays ~ the slower query
- Federated tab shows the 5-source matrix (records + headline
metric per engine) above the relabelled 3-source region scorecard
Trino Federation tab (3 sub-views):
- Federated: catalog landscape + a single cross-source SQL that joins
PostgreSQL + MySQL + MongoDB (region scorecard) — the federation proof,
computed in the background and cached (large full scans take ~2 min).
- Hadoop Lake: all federated business data materialized as external Iceberg
tables on HDFS (iceberg.hadoop.*_ext, ~120k rows each) with live, fast
business analytics (revenue by region/channel, top customers, HR by
department, supply by type, telemetry averages). Includes a one-click
"rebuild external tables" job.
- Data Dictionary: every business table + column with masked / visible PII
badges and categories.
Backend trino_federated.py: /catalogs, /marquee(+refresh), /lake,
/materialize(+status), /dictionary. Name-based PII detection flags raw PII
in derived/lake tables as visible vs physically-masked curated layer.
LLM context: platform_context now emits a full BUSINESS DATA CATALOG section
(tables, columns, types, source row counts, federated scorecard) with exact
per-column masked/visible status, so the assistant knows the data in detail
and what is masked vs not.
mo