2 Commits

Author	SHA1	Message	Date
mo	46b9c50e73	feat: Spark Workbench everywhere, autonomous Hadoop offload & LLM masking-aware ... - Data Hub with Hadoop tab (HDFS/Iceberg browser, Spark, pipeline) - Databricks-style Lakehouse Workbench (Trino engine, live exec matrix, materialize to Iceberg/S3); reused & embedded in every source-DB UI - HDFS -> Kafka -> Spark -> Iceberg/S3 pipeline; WebHDFS hostname resolver - Data Flow master pulse switch (Run/Pause/Stop) gating animated edges - Data Custodian autonomous Hadoop offload loop (batch counterpart to CDC), pulsing source -> HDFS edges; toggle in Data Flow - LLM now autonomously aware of all latest platform changes (live platform context) and enforces masking policy: never reveals masked PII, still answers helpfully with aggregates/explanations	2026-06-27 19:37:50 +00:00
mo	a724615a9a	feat(etl): Hadoop->Trino movement DAG + movements registry + autonomous ETL-agents ... - DAG hadoop_to_trino (deployed to Airflow) + worker hadoop_to_trino.py (on the Hadoop master) move HDFS historical_sales -> iceberg.hadoop.historical_sales_hdfs. - api/movements.py: movement registry, Airflow trigger+watch, run tracking (state/duration/rows), endpoints /api/movements, /{id}/run, /runs. - agent_ops.py: ETL-agent loop autonomously triggers movements on an interval and logs each run; /api/agent-ops/etl/toggle + etl status.	2026-06-27 01:59:35 +02:00