Agent terminals were idle (one-shot probe) while agents were busy in the
background. Now every agent streams what it is actually doing:
- agent_terminal: emit_threadsafe() so background threads can stream lines.
- agent_ops: Data Custodian DML loop logs the real INSERT/UPDATE/DELETE SQL
(+ Mongo ops) and Hadoop-offload Trino CTAS/INSERT to its terminal; ETL
Guardian announces each orchestrated movement.
- movements: trigger_and_watch streams the Airflow DAG / API call, conf,
before/after Trino counts and result to the owning agent terminal.
- etl_offload: per-dataset read + pyarrow->S3 parquet writes and cycle
summaries stream to the ETL Guardian terminal.
- agent_activity (new): round-robin live probes for Lakehouse Ops, Hadoop
Ranger (NameNode JMX + YARN), Infra Sentinel (Dockhand inventory + host
load) and Network Watcher (VLAN 20/21 path checks).
- fix: YARN ResourceManager runs on 10.0.21.62:8088 (was .61).
- ui: terminal dock merges the selected agent ops stream with the node probe.
- DAG hadoop_to_trino (deployed to Airflow) + worker hadoop_to_trino.py (on the
Hadoop master) move HDFS historical_sales -> iceberg.hadoop.historical_sales_hdfs.
- api/movements.py: movement registry, Airflow trigger+watch, run tracking
(state/duration/rows), endpoints /api/movements, /{id}/run, /runs.
- agent_ops.py: ETL-agent loop autonomously triggers movements on an interval
and logs each run; /api/agent-ops/etl/toggle + etl status.
mo