Commit Graph

9 Commits

Author SHA1 Message Date
mo d5fba208a3 Merge origin/main into running Command Center
Keep the deployed tree on conflict; integrate the remote Dockhand/CDC tip.
2026-07-21 23:21:23 +00:00
mo 9008fbd512 feat: Authentik login + switchable GPU prod target
Add OIDC auth for Command Center and runtime GPU endpoint selection
pointed at atc-gpu-prod (10.0.10.106), matching what is currently deployed.
2026-07-21 23:20:24 +00:00
root b46e7f01dd Fix Dockhand auth for topology flow and accurate CDC change stats.
Pass DOCKHAND_API_TOKEN to container inventory calls so pipeline_active
and topology animation work after Authentik. Replace ring-buffer-only CDC
stats with minute rollups (no 1000 cap), add 15m/1h/6h/24h window selector
on the Live Changes tab, and poll recent events on an interval.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-01 15:15:15 +02:00
mo d066def8b4 feat(cdc): add Re-sync sources button + auto-heal for Debezium connectors
- streaming_ops: add connector status discovery, manual resync endpoint
  (POST /api/pipeline/streaming/resync), connectors status endpoint, and a
  background connector_autoheal_loop that restarts FAILED tasks automatically
- main: wire connector_autoheal_loop into app lifespan
- api.ts: add resyncSources() helper
- ChangesView: add "Re-sync sources" header button with live status
2026-06-29 13:31:11 +00:00
mo 46b9c50e73 feat: Spark Workbench everywhere, autonomous Hadoop offload & LLM masking-aware
- Data Hub with Hadoop tab (HDFS/Iceberg browser, Spark, pipeline)
- Databricks-style Lakehouse Workbench (Trino engine, live exec matrix,
  materialize to Iceberg/S3); reused & embedded in every source-DB UI
- HDFS -> Kafka -> Spark -> Iceberg/S3 pipeline; WebHDFS hostname resolver
- Data Flow master pulse switch (Run/Pause/Stop) gating animated edges
- Data Custodian autonomous Hadoop offload loop (batch counterpart to CDC),
  pulsing source -> HDFS edges; toggle in Data Flow
- LLM now autonomously aware of all latest platform changes (live platform
  context) and enforces masking policy: never reveals masked PII, still
  answers helpfully with aggregates/explanations
2026-06-27 19:37:50 +00:00
mo 215ce111f1 feat(pii): self-service per-column masking policy enforced for the LLM
- pii_catalog: persistent per-column masking policy (default masked); GET/POST
  /api/pii/policy and POST /api/pii/lookup which redacts masked values server-side.
- get_pii masked flag now reflects the policy; dataflow exposes the dataset key.
- Data Flow PII inspector: per-column lock/unlock toggles + mask-all/unmask-all,
  so operators control exactly which data the assistant may reveal.
2026-06-27 11:36:36 +02:00
mo 6f20e24b8b feat(ui): Data Flow tab — live lineage graph + PII overlay + movement triggers
New DataFlowView (topology-style): nodes positioned from /api/dataflow with
measured-anchor SVG edges and animated particles (active CDC + running
movements). PII overlay shows shield badges + per-column detail inspector.
Bottom strip triggers ETL movements and toggles the ETL agent. Wired into
SideNav (Data Flow) and App routing; added types + api helpers.
2026-06-27 02:09:38 +02:00
mo 921342442f feat(ui): Live Changes (CDC) tab — real-time Debezium stream with filters, volume bars and before/after diff 2026-06-27 01:44:26 +02:00
mo a11621b21f Add Command Center v2: DQ/RAG integration, S3 browser, Jupyter, GPU matrix.
Mirror mo/atc-GPU layout with config/, docs/, scripts/ for Gitea deploy.
2026-06-25 00:28:23 +00:00