Add Project Nessie on lake01 with Command Center Iceberg tab (snapshots,
manifests, data files, time-travel SQL). Data Flow pulses only when endpoints
are reachable; offline nodes/edges render red.
- etl_offload.py: autonomous agent backfills/tails source DBs (PG/MySQL/
Mongo/Cassandra) to S3 as Parquet in small chunks, accumulates a live
federated business matrix (/api/etl/status, /api/etl/business, /run, /config).
- storage_s3.py: buffer generated CDC + masked curated rows to S3, overlay
live last-write into analytics; put_object_bytes for Parquet parts.
- trino_federated.py: capture generated rows + archive to S3; generator_active.
- dataflow.py: pulse generate + kafka/spark->S3 archive edges when active.
- StorageView: realtime ETL ingest panel; TrinoFederationView: realtime
business KPIs/charts from /api/etl/business.
- ChangesView: top KPIs/charts now overlay the live WS stream on server stats
so they update in lock-step with the bottom feed; faster 2.5s refresh.
- useCommandCenter: retain 800 live CDC changes.
Generated data is now visible flowing into the source systems on the Data
Flow graph: a manual "Generate data" burst (and the continuous live loop)
stamps a last_tick, and a new generator_active() helper lights up the
generator -> postgres/mysql/mongodb/cassandra edges for ~12s. Neo4j stays
dark since the live generator does not write to it.
Redesigned the Changes tab into a "New & Changed Data" dashboard driven by
the live CDC stream: animated KPI cards (new/updated/deleted + throughput),
a change-volume area chart, an operation-mix donut, per-system and
top-table breakdown bars, plus the existing filterable change feed.
Data Flow tab:
- Prominent "Generate data" button (500 / 2K / 10K) that inserts a fresh
burst of business rows into all source DBs on demand via a new
POST /api/federated/generate (fresh connections, safe alongside the
background streamer); result toast shows what was inserted, CDC streams it.
- "Scripts" button + a "View generation scripts" action on the Data Generator
node open a modal listing every generator script with full source, served by
GET /api/dataflow/scripts. Sources are the real files: the live streaming
generator (sliced live out of trino_federated.py) and the Airflow per-source
DAGs + Faker scripts (mounted read-only from infra/airflow into the API).
Knowledge Chat:
- New "Vector DB" explorer modal: shows the ChromaDB chunking config
(RecursiveCharacterTextSplitter 800/120, all-MiniLM-L6-v2, 384-dim, HNSW),
collections & documents, and the actual stored chunks with text, metadata and
an embedding preview (bars + values) so you can see exactly how files are
split and written as vectors.
Refactor: generator row-builders shared by the streamer and the on-demand burst.
Live dashboard now feels truly real-time:
- Background generator streams randomly-sized bursts of real rows into
PostgreSQL, MySQL, MongoDB & Cassandra every ~4s (CDC picks them up).
Throughput rises and falls; counters move in lock-step (base snapshot +
generated). Runs only while the Live tab is polling (heartbeat-gated) so
source tables do not grow unbounded; on/off toggle exposed in the UI.
- New /api/federated/live/generator toggle; /live returns per-tick activity
(last burst sizes, orders by region/status, event feed).
- LiveDashboard: live-activity panel, orders-per-tick sparkline, event
stream feed, burst-by-region/status charts, generator status + control.
Data Flow graph now explains how data reaches the assistant:
- Added ChromaDB -> RAG (LangChain) -> vLLM Gateway -> Knowledge Chat lane,
with Trino / OpenMetadata / curated-masked feeding LLM context. Live model
& embed metrics pulled from the RAG /config. New node/edge kinds + legend.
Align nodes into pipeline columns (producers -> sources -> CDC -> lakehouse ->
Trino) with governance centred at the bottom, so lineage reads in order instead
of Iceberg/Trino floating mid-canvas.
- pii_catalog: persistent per-column masking policy (default masked); GET/POST
/api/pii/policy and POST /api/pii/lookup which redacts masked values server-side.
- get_pii masked flag now reflects the policy; dataflow exposes the dataset key.
- Data Flow PII inspector: per-column lock/unlock toggles + mask-all/unmask-all,
so operators control exactly which data the assistant may reveal.
Data Flow graph now shows OpenMetadata as a governance node linked to all
sources and Trino (catalog edges); node inspector exposes an 'Open in
OpenMetadata' deep link and PII-columns-cataloged metric.
mo