46b9c50e73
feat: Spark Workbench everywhere, autonomous Hadoop offload & LLM masking-aware
...
- Data Hub with Hadoop tab (HDFS/Iceberg browser, Spark, pipeline)
- Databricks-style Lakehouse Workbench (Trino engine, live exec matrix,
materialize to Iceberg/S3); reused & embedded in every source-DB UI
- HDFS -> Kafka -> Spark -> Iceberg/S3 pipeline; WebHDFS hostname resolver
- Data Flow master pulse switch (Run/Pause/Stop) gating animated edges
- Data Custodian autonomous Hadoop offload loop (batch counterpart to CDC),
pulsing source -> HDFS edges; toggle in Data Flow
- LLM now autonomously aware of all latest platform changes (live platform
context) and enforces masking policy: never reveals masked PII, still
answers helpfully with aggregates/explanations
2026-06-27 19:37:50 +00:00
215ce111f1
feat(pii): self-service per-column masking policy enforced for the LLM
...
- pii_catalog: persistent per-column masking policy (default masked); GET/POST
/api/pii/policy and POST /api/pii/lookup which redacts masked values server-side.
- get_pii masked flag now reflects the policy; dataflow exposes the dataset key.
- Data Flow PII inspector: per-column lock/unlock toggles + mask-all/unmask-all,
so operators control exactly which data the assistant may reveal.
2026-06-27 11:36:36 +02:00
6f20e24b8b
feat(ui): Data Flow tab — live lineage graph + PII overlay + movement triggers
...
New DataFlowView (topology-style): nodes positioned from /api/dataflow with
measured-anchor SVG edges and animated particles (active CDC + running
movements). PII overlay shows shield badges + per-column detail inspector.
Bottom strip triggers ETL movements and toggles the ETL agent. Wired into
SideNav (Data Flow) and App routing; added types + api helpers.
2026-06-27 02:09:38 +02:00
921342442f
feat(ui): Live Changes (CDC) tab — real-time Debezium stream with filters, volume bars and before/after diff
2026-06-27 01:44:26 +02:00
a11621b21f
Add Command Center v2: DQ/RAG integration, S3 browser, Jupyter, GPU matrix.
...
Mirror mo/atc-GPU layout with config/, docs/, scripts/ for Gitea deploy.
2026-06-25 00:28:23 +00:00
mo