feat: pulse generator edges on data gen + New & Changed Data dashboard
Generated data is now visible flowing into the source systems on the Data Flow graph: a manual "Generate data" burst (and the continuous live loop) stamps a last_tick, and a new generator_active() helper lights up the generator -> postgres/mysql/mongodb/cassandra edges for ~12s. Neo4j stays dark since the live generator does not write to it. Redesigned the Changes tab into a "New & Changed Data" dashboard driven by the live CDC stream: animated KPI cards (new/updated/deleted + throughput), a change-volume area chart, an operation-mix donut, per-system and top-table breakdown bars, plus the existing filterable change feed.
This commit is contained in:
+11
-1
@@ -173,6 +173,11 @@ async def _build() -> dict[str, Any]:
|
||||
cdc = cdc_snapshot(15)
|
||||
except Exception:
|
||||
cdc = {"connected": False, "consumed": 0, "by_source": {}, "window_total": 0}
|
||||
try:
|
||||
from trino_federated import generator_active
|
||||
gen_active = generator_active()
|
||||
except Exception:
|
||||
gen_active = False
|
||||
try:
|
||||
from pii_catalog import get_pii
|
||||
pii = get_pii()
|
||||
@@ -255,7 +260,12 @@ async def _build() -> dict[str, Any]:
|
||||
edge["last_rows"] = lr.get("rows")
|
||||
edge["last_duration_s"] = lr.get("duration_s")
|
||||
edge["active"] = lr.get("state") == "running"
|
||||
if e["kind"] == "cdc":
|
||||
if e["kind"] == "generate":
|
||||
# The live generator (continuous loop or the 'Generate data' burst)
|
||||
# writes into these four sources; pulse the edge while it is active.
|
||||
if e["to"] in ("postgres", "mysql", "mongodb", "cassandra"):
|
||||
edge["active"] = gen_active or bool(edge.get("active"))
|
||||
elif e["kind"] == "cdc":
|
||||
edge["active"] = cdc.get("by_source", {}).get(e["from"], 0) > 0
|
||||
elif e.get("from") == "hdfs" and e.get("to") == "kafka":
|
||||
edge["active"] = bool(edge_live.get("hdfs→kafka"))
|
||||
|
||||
Reference in New Issue
Block a user