Agent terminals were idle (one-shot probe) while agents were busy in the
background. Now every agent streams what it is actually doing:
- agent_terminal: emit_threadsafe() so background threads can stream lines.
- agent_ops: Data Custodian DML loop logs the real INSERT/UPDATE/DELETE SQL
(+ Mongo ops) and Hadoop-offload Trino CTAS/INSERT to its terminal; ETL
Guardian announces each orchestrated movement.
- movements: trigger_and_watch streams the Airflow DAG / API call, conf,
before/after Trino counts and result to the owning agent terminal.
- etl_offload: per-dataset read + pyarrow->S3 parquet writes and cycle
summaries stream to the ETL Guardian terminal.
- agent_activity (new): round-robin live probes for Lakehouse Ops, Hadoop
Ranger (NameNode JMX + YARN), Infra Sentinel (Dockhand inventory + host
load) and Network Watcher (VLAN 20/21 path checks).
- fix: YARN ResourceManager runs on 10.0.21.62:8088 (was .61).
- ui: terminal dock merges the selected agent ops stream with the node probe.
- etl_offload.py: autonomous agent backfills/tails source DBs (PG/MySQL/
Mongo/Cassandra) to S3 as Parquet in small chunks, accumulates a live
federated business matrix (/api/etl/status, /api/etl/business, /run, /config).
- storage_s3.py: buffer generated CDC + masked curated rows to S3, overlay
live last-write into analytics; put_object_bytes for Parquet parts.
- trino_federated.py: capture generated rows + archive to S3; generator_active.
- dataflow.py: pulse generate + kafka/spark->S3 archive edges when active.
- StorageView: realtime ETL ingest panel; TrinoFederationView: realtime
business KPIs/charts from /api/etl/business.
- ChangesView: top KPIs/charts now overlay the live WS stream on server stats
so they update in lock-step with the bottom feed; faster 2.5s refresh.
- useCommandCenter: retain 800 live CDC changes.
mo