46b9c50e73
feat: Spark Workbench everywhere, autonomous Hadoop offload & LLM masking-aware
...
- Data Hub with Hadoop tab (HDFS/Iceberg browser, Spark, pipeline)
- Databricks-style Lakehouse Workbench (Trino engine, live exec matrix,
materialize to Iceberg/S3); reused & embedded in every source-DB UI
- HDFS -> Kafka -> Spark -> Iceberg/S3 pipeline; WebHDFS hostname resolver
- Data Flow master pulse switch (Run/Pause/Stop) gating animated edges
- Data Custodian autonomous Hadoop offload loop (batch counterpart to CDC),
pulsing source -> HDFS edges; toggle in Data Flow
- LLM now autonomously aware of all latest platform changes (live platform
context) and enforces masking policy: never reveals masked PII, still
answers helpfully with aggregates/explanations
2026-06-27 19:37:50 +00:00
a724615a9a
feat(etl): Hadoop->Trino movement DAG + movements registry + autonomous ETL-agents
...
- DAG hadoop_to_trino (deployed to Airflow) + worker hadoop_to_trino.py (on the
Hadoop master) move HDFS historical_sales -> iceberg.hadoop.historical_sales_hdfs.
- api/movements.py: movement registry, Airflow trigger+watch, run tracking
(state/duration/rows), endpoints /api/movements, /{id}/run, /runs.
- agent_ops.py: ETL-agent loop autonomously triggers movements on an interval
and logs each run; /api/agent-ops/etl/toggle + etl status.
2026-06-27 01:59:35 +02:00
3c9661e7f9
feat(agents): autonomous guard-railed DML on source DBs for Debezium CDC
...
Add api/agent_ops.py: background loop performs small INSERT/UPDATE/DELETE on
public.sales_orders (PG), hr.employee_events (MySQL) and supplychain.events
(Mongo). Agent rows are tagged (notes/atc_agent); UPDATE/DELETE only ever touch
agent-created rows. Env kill-switch + interval + per-tick row cap. Endpoints
/api/agent-ops/{status,toggle,run-once}. Loop started in lifespan.
2026-06-27 01:25:08 +02:00
mo