feat: realtime ETL offload to S3 + live business dashboards

- etl_offload.py: autonomous agent backfills/tails source DBs (PG/MySQL/
  Mongo/Cassandra) to S3 as Parquet in small chunks, accumulates a live
  federated business matrix (/api/etl/status, /api/etl/business, /run, /config).
- storage_s3.py: buffer generated CDC + masked curated rows to S3, overlay
  live last-write into analytics; put_object_bytes for Parquet parts.
- trino_federated.py: capture generated rows + archive to S3; generator_active.
- dataflow.py: pulse generate + kafka/spark->S3 archive edges when active.
- StorageView: realtime ETL ingest panel; TrinoFederationView: realtime
  business KPIs/charts from /api/etl/business.
- ChangesView: top KPIs/charts now overlay the live WS stream on server stats
  so they update in lock-step with the bottom feed; faster 2.5s refresh.
- useCommandCenter: retain 800 live CDC changes.
This commit is contained in:
mo
2026-06-28 23:33:21 +00:00
parent dfd5d4da8a
commit b6d7d3dc74
11 changed files with 1062 additions and 45 deletions
+2
View File
@@ -54,6 +54,7 @@ from streaming_ops import router as streaming_router
from spark_workbench import router as spark_workbench_router
from pii_catalog import router as pii_router
from trino_federated import router as federated_router
from etl_offload import router as etl_offload_router
from ssh_terminal import ssh_session
from node_registry import NODE_IDS, NODE_AGENT, NODE_REGISTRY, is_node_id
from node_ops import build_node_detail, probe_node, run_node_probe_task
@@ -759,6 +760,7 @@ app.include_router(streaming_router)
app.include_router(spark_workbench_router)
app.include_router(pii_router)
app.include_router(federated_router)
app.include_router(etl_offload_router)
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],