feat: realtime ETL offload to S3 + live business dashboards

- etl_offload.py: autonomous agent backfills/tails source DBs (PG/MySQL/
  Mongo/Cassandra) to S3 as Parquet in small chunks, accumulates a live
  federated business matrix (/api/etl/status, /api/etl/business, /run, /config).
- storage_s3.py: buffer generated CDC + masked curated rows to S3, overlay
  live last-write into analytics; put_object_bytes for Parquet parts.
- trino_federated.py: capture generated rows + archive to S3; generator_active.
- dataflow.py: pulse generate + kafka/spark->S3 archive edges when active.
- StorageView: realtime ETL ingest panel; TrinoFederationView: realtime
  business KPIs/charts from /api/etl/business.
- ChangesView: top KPIs/charts now overlay the live WS stream on server stats
  so they update in lock-step with the bottom feed; faster 2.5s refresh.
- useCommandCenter: retain 800 live CDC changes.
This commit is contained in:
mo
2026-06-28 23:33:21 +00:00
parent dfd5d4da8a
commit b6d7d3dc74
11 changed files with 1062 additions and 45 deletions
+9 -1
View File
@@ -178,6 +178,11 @@ async def _build() -> dict[str, Any]:
gen_active = generator_active()
except Exception:
gen_active = False
try:
from storage_s3 import archive_active
arch_active = archive_active()
except Exception:
arch_active = False
try:
from pii_catalog import get_pii
pii = get_pii()
@@ -274,7 +279,10 @@ async def _build() -> dict[str, Any]:
elif e.get("from") == "spark" and e.get("to") == "iceberg_curated":
edge["active"] = bool(edge_live.get("spark→iceberg")) or edge.get("active")
elif e.get("from") == "spark" and e.get("to") == "s3_cdc":
edge["active"] = bool(edge_live.get("spark→s3")) or edge.get("active")
edge["active"] = bool(edge_live.get("spark→s3")) or edge.get("active") or arch_active
elif e["kind"] == "archive" and e.get("from") == "kafka" and e.get("to") == "s3_cdc":
# Kafka → S3 CDC archive pulses while the pipeline lands objects in S3
edge["active"] = arch_active
elif e["kind"] in ("context", "retrieve", "prompt", "answer"):
# AI serving lane pulses while governed data is being served to the LLM
edge["active"] = bool(_rag_info())