feat: live per-agent terminal activity with real scripts/SQL

Agent terminals were idle (one-shot probe) while agents were busy in the
background. Now every agent streams what it is actually doing:

- agent_terminal: emit_threadsafe() so background threads can stream lines.
- agent_ops: Data Custodian DML loop logs the real INSERT/UPDATE/DELETE SQL
  (+ Mongo ops) and Hadoop-offload Trino CTAS/INSERT to its terminal; ETL
  Guardian announces each orchestrated movement.
- movements: trigger_and_watch streams the Airflow DAG / API call, conf,
  before/after Trino counts and result to the owning agent terminal.
- etl_offload: per-dataset read + pyarrow->S3 parquet writes and cycle
  summaries stream to the ETL Guardian terminal.
- agent_activity (new): round-robin live probes for Lakehouse Ops, Hadoop
  Ranger (NameNode JMX + YARN), Infra Sentinel (Dockhand inventory + host
  load) and Network Watcher (VLAN 20/21 path checks).
- fix: YARN ResourceManager runs on 10.0.21.62:8088 (was .61).
- ui: terminal dock merges the selected agent ops stream with the node probe.
This commit is contained in:
mo
2026-06-29 12:54:46 +00:00
parent b6d7d3dc74
commit 1e2cfe80f2
9 changed files with 388 additions and 52 deletions
+4
View File
@@ -46,6 +46,7 @@ from hadoop_analytics import router as hadoop_router
from elasticsearch_api import router as elasticsearch_router
from sql_console import router as sql_router
from agent_ops import router as agent_ops_router, agent_dml_loop, etl_agent_loop, custodian_offload_loop
from agent_activity import agent_activity_loop
from cdc_consumer import router as cdc_router, cdc_consumer_loop
from movements import router as movements_router
from movements import MOVEMENT_BY_ID, trigger_and_watch
@@ -735,12 +736,15 @@ async def lifespan(app: FastAPI):
cdc_task = asyncio.create_task(cdc_consumer_loop())
etl_task = asyncio.create_task(etl_agent_loop())
cust_task = asyncio.create_task(custodian_offload_loop())
act_task = asyncio.create_task(agent_activity_loop())
add_feed("infra-sentinel", "ATC Command Center API online", "info")
yield
task.cancel()
dml_task.cancel()
cdc_task.cancel()
etl_task.cancel()
cust_task.cancel()
act_task.cancel()
if redis_client:
await redis_client.close()