46b9c50e73
feat: Spark Workbench everywhere, autonomous Hadoop offload & LLM masking-aware
...
- Data Hub with Hadoop tab (HDFS/Iceberg browser, Spark, pipeline)
- Databricks-style Lakehouse Workbench (Trino engine, live exec matrix,
materialize to Iceberg/S3); reused & embedded in every source-DB UI
- HDFS -> Kafka -> Spark -> Iceberg/S3 pipeline; WebHDFS hostname resolver
- Data Flow master pulse switch (Run/Pause/Stop) gating animated edges
- Data Custodian autonomous Hadoop offload loop (batch counterpart to CDC),
pulsing source -> HDFS edges; toggle in Data Flow
- LLM now autonomously aware of all latest platform changes (live platform
context) and enforces masking policy: never reveals masked PII, still
answers helpfully with aggregates/explanations
2026-06-27 19:37:50 +00:00
215ce111f1
feat(pii): self-service per-column masking policy enforced for the LLM
...
- pii_catalog: persistent per-column masking policy (default masked); GET/POST
/api/pii/policy and POST /api/pii/lookup which redacts masked values server-side.
- get_pii masked flag now reflects the policy; dataflow exposes the dataset key.
- Data Flow PII inspector: per-column lock/unlock toggles + mask-all/unmask-all,
so operators control exactly which data the assistant may reveal.
2026-06-27 11:36:36 +02:00
c70639c46d
feat(dataflow): add OpenMetadata governance node + catalog edges + UI link
...
Data Flow graph now shows OpenMetadata as a governance node linked to all
sources and Trino (catalog edges); node inspector exposes an 'Open in
OpenMetadata' deep link and PII-columns-cataloged metric.
2026-06-27 02:43:22 +02:00
6f20e24b8b
feat(ui): Data Flow tab — live lineage graph + PII overlay + movement triggers
...
New DataFlowView (topology-style): nodes positioned from /api/dataflow with
measured-anchor SVG edges and animated particles (active CDC + running
movements). PII overlay shows shield badges + per-column detail inspector.
Bottom strip triggers ETL movements and toggles the ETL agent. Wired into
SideNav (Data Flow) and App routing; added types + api helpers.
2026-06-27 02:09:38 +02:00
921342442f
feat(ui): Live Changes (CDC) tab — real-time Debezium stream with filters, volume bars and before/after diff
2026-06-27 01:44:26 +02:00
46b51a891c
Add HDFS browser, SSH terminal, presentation editor, lab health panel
...
- HDFS WebHDFS file browser (api/hdfs_api.py + HdfsView)
- In-browser SSH terminal via paramiko WebSocket bridge (api/ssh_terminal.py + SshTerminal, xterm.js)
- Presentation deck editor (text + image upload) and CRUD endpoints
- Collapsible GPU matrix + new LabHealthPanel in SideNav
- Topology fixes (edge alignment, Hadoop node, compact nodes)
- nginx ws timeout bump for long-lived SSH sessions
2026-06-26 00:47:49 +00:00
a11621b21f
Add Command Center v2: DQ/RAG integration, S3 browser, Jupyter, GPU matrix.
...
Mirror mo/atc-GPU layout with config/, docs/, scripts/ for Gitea deploy.
2026-06-25 00:28:23 +00:00
fb9cc21c9a
Add ATC Command Center v1 with light UI theme.
...
Agent hub dashboard, FastAPI backend, and Docker stack for VM 304 MCP.
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-06-23 15:07:51 +02:00
mo