Commit Graph

17 Commits

Author SHA1 Message Date
mo 5147538b05 fix(agents): use in-memory PK pools for DML (avoid full-table scans on 54M-row tables) 2026-06-27 01:27:56 +02:00
mo 3c9661e7f9 feat(agents): autonomous guard-railed DML on source DBs for Debezium CDC
Add api/agent_ops.py: background loop performs small INSERT/UPDATE/DELETE on
public.sales_orders (PG), hr.employee_events (MySQL) and supplychain.events
(Mongo). Agent rows are tagged (notes/atc_agent); UPDATE/DELETE only ever touch
agent-created rows. Env kill-switch + interval + per-tick row cap. Endpoints
/api/agent-ops/{status,toggle,run-once}. Loop started in lifespan.
2026-06-27 01:25:08 +02:00
mo 96d490807a feat(hadoop): real Hive engine + filterable, detailed analytics with lineage
Deploys Apache Hive 3.1.3 (Derby metastore + external table over the HDFS
historical CSV, MapReduce exec) on the Hadoop master, so the engine comparison
shows a REAL measured Hive latency (~5.2s) next to live Trino (~0.3s); Impala
stays clearly-labelled representative. The API re-measures Hive over SSH on a
30-min TTL (cached + persisted, with a committed seed). Adds filters
(year/region/category/channel), a region×category heatmap, Trino exec stats,
and a "where is this data read from" lineage panel (Trino->S3/Iceberg/Parquet
with snapshot+files, Hive->HDFS/CSV with namenode+files). Mounts host SSH key
read-only into the api container for the live Hive benchmark.
2026-06-26 17:08:57 +00:00
mo 71a64d5a21 feat(hadoop): embedded lakehouse analytics + Impala/Hive vs Trino comparison
Adds an Analytics sub-tab to the Hadoop view with live KPIs and revenue
breakdowns (by year/region/category/channel) queried from Trino over
iceberg.hadoop.historical_sales, plus a query-engine comparison panel. Trino
latency is measured live; Impala and Hive are shown as clearly-labelled
representative figures (those engines are not deployed). New cached endpoints
/api/hadoop/analytics and /api/hadoop/engines.
2026-06-26 15:44:43 +00:00
mo 88ca338f5a feat: neo4j in topology + agent-driven datagen with activity log; fix all-sources generate 2026-06-26 08:51:39 +00:00
mo 897ccaa9f3 api: non-blocking /sync via background refresher (full-scan counts cached) 2026-06-26 01:39:53 +00:00
mo bac3a90775 api: pipeline_ops router (per-source generate trigger, run status, Trino sync counts) 2026-06-26 01:12:07 +00:00
mo 46b51a891c Add HDFS browser, SSH terminal, presentation editor, lab health panel
- HDFS WebHDFS file browser (api/hdfs_api.py + HdfsView)
- In-browser SSH terminal via paramiko WebSocket bridge (api/ssh_terminal.py + SshTerminal, xterm.js)
- Presentation deck editor (text + image upload) and CRUD endpoints
- Collapsible GPU matrix + new LabHealthPanel in SideNav
- Topology fixes (edge alignment, Hadoop node, compact nodes)
- nginx ws timeout bump for long-lived SSH sessions
2026-06-26 00:47:49 +00:00
mo ff2c31a6b5 Fix remaining Dutch string in DataQualityView 2026-06-25 10:00:07 +00:00
mo 4ff9f5589e Fix MySQL demo query for existing hr tables 2026-06-25 01:34:07 +00:00
mo 7da2dd60f0 Add MySQL/MongoDB consoles, fit topology without scroll, longer flow lines.
Compact workbench panel, collapsible infra bar, narrower topology nodes with wider inter-stage gaps for visible connection lines.
2026-06-25 01:33:48 +00:00
mo 4e1cf9ef6c Use equivalent 5M sin benchmark query on Trino 2026-06-25 01:24:58 +00:00
mo 2cf90c9e14 Fix Trino benchmark and sample queries for valid syntax and parallel demo 2026-06-25 01:24:35 +00:00
mo 170eb2418b Add agent workbench terminal and live PostgreSQL/Trino SQL consoles.
Clicking agents opens a dedicated terminal panel; topology PostgreSQL/Trino nodes open SQL workbench with ten demo queries and Postgres vs Trino benchmark.
2026-06-25 01:10:57 +00:00
mo 9deda7fcb1 Full stack visibility: mo S3 buckets, Elasticsearch/Kibana UI and topology 2026-06-25 00:53:11 +00:00
mo a11621b21f Add Command Center v2: DQ/RAG integration, S3 browser, Jupyter, GPU matrix.
Mirror mo/atc-GPU layout with config/, docs/, scripts/ for Gitea deploy.
2026-06-25 00:28:23 +00:00
mo fb9cc21c9a Add ATC Command Center v1 with light UI theme.
Agent hub dashboard, FastAPI backend, and Docker stack for VM 304 MCP.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-23 15:07:51 +02:00