Live dashboard now feels truly real-time:
- Background generator streams randomly-sized bursts of real rows into
PostgreSQL, MySQL, MongoDB & Cassandra every ~4s (CDC picks them up).
Throughput rises and falls; counters move in lock-step (base snapshot +
generated). Runs only while the Live tab is polling (heartbeat-gated) so
source tables do not grow unbounded; on/off toggle exposed in the UI.
- New /api/federated/live/generator toggle; /live returns per-tick activity
(last burst sizes, orders by region/status, event feed).
- LiveDashboard: live-activity panel, orders-per-tick sparkline, event
stream feed, burst-by-region/status charts, generator status + control.
Data Flow graph now explains how data reaches the assistant:
- Added ChromaDB -> RAG (LangChain) -> vLLM Gateway -> Knowledge Chat lane,
with Trino / OpenMetadata / curated-masked feeding LLM context. Live model
& embed metrics pulled from the RAG /config. New node/edge kinds + legend.
Align nodes into pipeline columns (producers -> sources -> CDC -> lakehouse ->
Trino) with governance centred at the bottom, so lineage reads in order instead
of Iceberg/Trino floating mid-canvas.
- pii_catalog: persistent per-column masking policy (default masked); GET/POST
/api/pii/policy and POST /api/pii/lookup which redacts masked values server-side.
- get_pii masked flag now reflects the policy; dataflow exposes the dataset key.
- Data Flow PII inspector: per-column lock/unlock toggles + mask-all/unmask-all,
so operators control exactly which data the assistant may reveal.
Data Flow graph now shows OpenMetadata as a governance node linked to all
sources and Trino (catalog edges); node inspector exposes an 'Open in
OpenMetadata' deep link and PII-columns-cataloged metric.
mo