Commit Graph

6 Commits

Author SHA1 Message Date
mo cdc8aa4bf7 feat(pii): masking for Cassandra & Neo4j (graph nodes + live property read) + MongoDB payload/free-text 2026-06-27 21:10:16 +00:00
mo 46b9c50e73 feat: Spark Workbench everywhere, autonomous Hadoop offload & LLM masking-aware
- Data Hub with Hadoop tab (HDFS/Iceberg browser, Spark, pipeline)
- Databricks-style Lakehouse Workbench (Trino engine, live exec matrix,
  materialize to Iceberg/S3); reused & embedded in every source-DB UI
- HDFS -> Kafka -> Spark -> Iceberg/S3 pipeline; WebHDFS hostname resolver
- Data Flow master pulse switch (Run/Pause/Stop) gating animated edges
- Data Custodian autonomous Hadoop offload loop (batch counterpart to CDC),
  pulsing source -> HDFS edges; toggle in Data Flow
- LLM now autonomously aware of all latest platform changes (live platform
  context) and enforces masking policy: never reveals masked PII, still
  answers helpfully with aggregates/explanations
2026-06-27 19:37:50 +00:00
mo ca05e04f8c fix(dataflow): clean left-to-right stage layout for the topology
Align nodes into pipeline columns (producers -> sources -> CDC -> lakehouse ->
Trino) with governance centred at the bottom, so lineage reads in order instead
of Iceberg/Trino floating mid-canvas.
2026-06-27 12:07:25 +02:00
mo 215ce111f1 feat(pii): self-service per-column masking policy enforced for the LLM
- pii_catalog: persistent per-column masking policy (default masked); GET/POST
  /api/pii/policy and POST /api/pii/lookup which redacts masked values server-side.
- get_pii masked flag now reflects the policy; dataflow exposes the dataset key.
- Data Flow PII inspector: per-column lock/unlock toggles + mask-all/unmask-all,
  so operators control exactly which data the assistant may reveal.
2026-06-27 11:36:36 +02:00
mo c70639c46d feat(dataflow): add OpenMetadata governance node + catalog edges + UI link
Data Flow graph now shows OpenMetadata as a governance node linked to all
sources and Trino (catalog edges); node inspector exposes an 'Open in
OpenMetadata' deep link and PII-columns-cataloged metric.
2026-06-27 02:43:22 +02:00
mo ee9357aa31 feat(dataflow): live Data Flow graph API + PII catalog
api/dataflow.py: node-link landscape (generators->sources->CDC/Kafka->sinks,
HDFS->Iceberg, sources->curated_masked) with live overlays (movement run state,
CDC volume, Trino counts) and PII overlay. api/pii_catalog.py: column PII
classification via Trino information_schema (OM-ready). Endpoints /api/dataflow,
/api/dataflow/{id}/run, /api/pii.
2026-06-27 02:04:46 +02:00