Pass DOCKHAND_API_TOKEN to container inventory calls so pipeline_active
and topology animation work after Authentik. Replace ring-buffer-only CDC
stats with minute rollups (no 1000 cap), add 15m/1h/6h/24h window selector
on the Live Changes tab, and poll recent events on an interval.
Co-authored-by: Cursor <cursoragent@cursor.com>
Adds native Command Center features (no new containers) integrated as sub-tabs
in the existing Data Explorer and Data Quality views:
- Continuous Data Quality (dq_monitor.py): live completeness/uniqueness/validity/
freshness scorecards via Trino with rolling trends → DataQuality "Live Monitoring".
- Ownership & stewardship (catalog_governance.py): owner/steward/tier matrix,
orphan detection, business glossary; local store best-effort synced to
OpenMetadata (owner PATCH) → Data Explorer "Ownership".
- Access & policy posture: per-dataset compliance combining PII masking, ownership,
live DQ and observability alerts vs data contracts → Data Explorer "Access & Policies".
- Lineage (lineage.py): staged source→CDC→Spark→S3→Iceberg→Trino→serving graph with
live row counts and column-level PII/masking tracing → Data Explorer "Lineage".
- Observability (observability.py): volume/freshness/schema-drift monitoring with
alerts → Data Explorer "Observability".
- Shared lake_meta.py dataset registry + bounded Trino client; fast native row-count
and PK-indexed freshness so monitors stay cheap on 25-54M-row tables.
- LLM context (lab_context.py) enriched with DQ scores, ownership and active alerts.
- streaming_ops: add connector status discovery, manual resync endpoint
(POST /api/pipeline/streaming/resync), connectors status endpoint, and a
background connector_autoheal_loop that restarts FAILED tasks automatically
- main: wire connector_autoheal_loop into app lifespan
- api.ts: add resyncSources() helper
- ChangesView: add "Re-sync sources" header button with live status
Agent terminals were idle (one-shot probe) while agents were busy in the
background. Now every agent streams what it is actually doing:
- agent_terminal: emit_threadsafe() so background threads can stream lines.
- agent_ops: Data Custodian DML loop logs the real INSERT/UPDATE/DELETE SQL
(+ Mongo ops) and Hadoop-offload Trino CTAS/INSERT to its terminal; ETL
Guardian announces each orchestrated movement.
- movements: trigger_and_watch streams the Airflow DAG / API call, conf,
before/after Trino counts and result to the owning agent terminal.
- etl_offload: per-dataset read + pyarrow->S3 parquet writes and cycle
summaries stream to the ETL Guardian terminal.
- agent_activity (new): round-robin live probes for Lakehouse Ops, Hadoop
Ranger (NameNode JMX + YARN), Infra Sentinel (Dockhand inventory + host
load) and Network Watcher (VLAN 20/21 path checks).
- fix: YARN ResourceManager runs on 10.0.21.62:8088 (was .61).
- ui: terminal dock merges the selected agent ops stream with the node probe.
- etl_offload.py: autonomous agent backfills/tails source DBs (PG/MySQL/
Mongo/Cassandra) to S3 as Parquet in small chunks, accumulates a live
federated business matrix (/api/etl/status, /api/etl/business, /run, /config).
- storage_s3.py: buffer generated CDC + masked curated rows to S3, overlay
live last-write into analytics; put_object_bytes for Parquet parts.
- trino_federated.py: capture generated rows + archive to S3; generator_active.
- dataflow.py: pulse generate + kafka/spark->S3 archive edges when active.
- StorageView: realtime ETL ingest panel; TrinoFederationView: realtime
business KPIs/charts from /api/etl/business.
- ChangesView: top KPIs/charts now overlay the live WS stream on server stats
so they update in lock-step with the bottom feed; faster 2.5s refresh.
- useCommandCenter: retain 800 live CDC changes.
Generated data is now visible flowing into the source systems on the Data
Flow graph: a manual "Generate data" burst (and the continuous live loop)
stamps a last_tick, and a new generator_active() helper lights up the
generator -> postgres/mysql/mongodb/cassandra edges for ~12s. Neo4j stays
dark since the live generator does not write to it.
Redesigned the Changes tab into a "New & Changed Data" dashboard driven by
the live CDC stream: animated KPI cards (new/updated/deleted + throughput),
a change-volume area chart, an operation-mix donut, per-system and
top-table breakdown bars, plus the existing filterable change feed.
Previously Run/Pause/Stop only drove the pulse animation, so on the Data Flow
tab nothing was generated (the streamer is heartbeat-gated and only the Live
dashboard was sending heartbeats). Now:
- New POST /api/federated/live/heartbeat keep-alive endpoint.
- Run seeds an immediate burst into the sources and keeps the generator alive
(5s heartbeat) while the tab is open; Pause/Stop halts generation.
Data Flow tab:
- Prominent "Generate data" button (500 / 2K / 10K) that inserts a fresh
burst of business rows into all source DBs on demand via a new
POST /api/federated/generate (fresh connections, safe alongside the
background streamer); result toast shows what was inserted, CDC streams it.
- "Scripts" button + a "View generation scripts" action on the Data Generator
node open a modal listing every generator script with full source, served by
GET /api/dataflow/scripts. Sources are the real files: the live streaming
generator (sliced live out of trino_federated.py) and the Airflow per-source
DAGs + Faker scripts (mounted read-only from infra/airflow into the API).
Knowledge Chat:
- New "Vector DB" explorer modal: shows the ChromaDB chunking config
(RecursiveCharacterTextSplitter 800/120, all-MiniLM-L6-v2, 384-dim, HNSW),
collections & documents, and the actual stored chunks with text, metadata and
an embedding preview (bars + values) so you can see exactly how files are
split and written as vectors.
Refactor: generator row-builders shared by the streamer and the on-demand burst.
Live dashboard now feels truly real-time:
- Background generator streams randomly-sized bursts of real rows into
PostgreSQL, MySQL, MongoDB & Cassandra every ~4s (CDC picks them up).
Throughput rises and falls; counters move in lock-step (base snapshot +
generated). Runs only while the Live tab is polling (heartbeat-gated) so
source tables do not grow unbounded; on/off toggle exposed in the UI.
- New /api/federated/live/generator toggle; /live returns per-tick activity
(last burst sizes, orders by region/status, event feed).
- LiveDashboard: live-activity panel, orders-per-tick sparkline, event
stream feed, burst-by-region/status charts, generator status + control.
Data Flow graph now explains how data reaches the assistant:
- Added ChromaDB -> RAG (LangChain) -> vLLM Gateway -> Knowledge Chat lane,
with Trino / OpenMetadata / curated-masked feeding LLM context. Live model
& embed metrics pulled from the RAG /config. New node/edge kinds + legend.
- New "Live" tab: auto-polls /api/federated/live every 2.5s with
animated counters, ingestion throughput sparkline, per-source
write-rate bars, a region scorecard matrix (heat-shaded) and live
business breakdown charts (region/channel/status/customers/
telemetry/supply)
- Backend /api/federated/live: instant source estimates (Postgres),
monotonic max(event_id) for MySQL and Mongo estimated count for
immediate movement, Cassandra from cached matrix; business aggs
cached over the small Hadoop lake tables (short TTL)
- Fix embedded Trino panels being squeezed with internal scrollbars
by making panels/grids shrink-0 so the page scrolls instead
The marquee panel only joined 3 region-keyed sources. Add a
"one SQL across every database" reach matrix that fans a single
Trino query out to PostgreSQL, MySQL, MongoDB, Cassandra and the
Hadoop/Iceberg lake in one UNION ALL (telemetry has no region, so
a per-source summary is used instead of a misleading join).
- New MATRIX_SQL + concurrent execution alongside the region
scorecard so total latency stays ~ the slower query
- Federated tab shows the 5-source matrix (records + headline
metric per engine) above the relabelled 3-source region scorecard
Trino Federation tab (3 sub-views):
- Federated: catalog landscape + a single cross-source SQL that joins
PostgreSQL + MySQL + MongoDB (region scorecard) — the federation proof,
computed in the background and cached (large full scans take ~2 min).
- Hadoop Lake: all federated business data materialized as external Iceberg
tables on HDFS (iceberg.hadoop.*_ext, ~120k rows each) with live, fast
business analytics (revenue by region/channel, top customers, HR by
department, supply by type, telemetry averages). Includes a one-click
"rebuild external tables" job.
- Data Dictionary: every business table + column with masked / visible PII
badges and categories.
Backend trino_federated.py: /catalogs, /marquee(+refresh), /lake,
/materialize(+status), /dictionary. Name-based PII detection flags raw PII
in derived/lake tables as visible vs physically-masked curated layer.
LLM context: platform_context now emits a full BUSINESS DATA CATALOG section
(tables, columns, types, source row counts, federated scorecard) with exact
per-column masked/visible status, so the assistant knows the data in detail
and what is masked vs not.
Add a dedicated business-data view (separate from infra/search):
- New "Data Explorer" tab: KPIs (customers, employees, products, source
row totals, sample revenue) + charts driven by Elasticsearch aggregations
(revenue over time, by region/channel/status, top customers & products,
HR by department/role/event, supply chain by type, telemetry averages),
with region + free-text filters.
- Backend /api/search/business endpoint: battery of ES aggregations with
numeric/date index template (so amount sums and date histograms work),
source totals via cheap planner estimates; biased the orders sample to
rows carrying customer names so people are searchable/visible.
Manual data entry on source systems:
- "Insert row" action in the Data Hub browser opens a column-aware form;
POST /api/sql/row/insert writes to Postgres/MySQL/Mongo/Cassandra/Neo4j.
- Inserts into CDC sources (PG/MySQL/Mongo) are captured by Debezium and
streamed to Kafka in real time; UI flags this and pulses the data flow.
Build dedicated de-duplicated entity indices (atc-customers ~39k,
atc-employees ~39k, atc-products) from Trino so users can search by
customer/employee name, id and email directly instead of scanning raw
transaction rows. Raise per-table sample caps for the big source tables
(sales_orders, employee_events, supplychain events, device_metrics) and
all iceberg/Hadoop tables for broader row-level coverage.
UI: add Datasets quick-filter chips (Customers, Employees, Products,
Sales orders, HR events, Supply chain, Telemetry, Lakehouse) and default
the Search tab to business data instead of an infra-looking empty view.
When switching source engine, the sample effect fired with the previous
engine selected object (e.g. public.sales_orders against MySQL), surfacing
"Table hr.sales_orders doesn't exist". Track which engine the selected
object belongs to and only sample when it matches the active engine, plus
guard against stale responses overwriting newer ones.
Row-count for the browser used an exact count(*) which full-scanned huge
tables (~30s on 24M rows). Use planner/statistics estimates and only run a
time-bounded exact count for small (<=50k) tables.
Add a /preview endpoint that range-reads the first chunk of an object and
returns it as text. Browser gets an eye action (list + gallery) opening a
modal that pretty-prints JSON, renders CSV/TSV as a table, and shows logs/
text/yaml/xml verbatim, with a truncation note and download link.
Add a List/Gallery toggle to the bucket browser that auto-switches to a
thumbnail grid when a listing is mostly images, and a keyboard-navigable
lightbox (prev/next/esc) for full-size previews. Download endpoint serves
inline with the correct image/* content-type (ECS stores octet-stream) so
previews render instead of forcing a download.
Analytics now walks every top-level prefix with its own budget instead of a
flat alphabetical scan, so a single huge prefix (kafka/ CDC json) no longer
hides the rest — composition now correctly reflects images, logs, csv,
parquet and avro, and reports true total size.
Browser gains recursive (whole-subtree) listing, free-text name search,
file-type filter chips, type column and Load-more pagination so every object
across all folders is discoverable.
Add /api/storage/s3/analytics endpoint that scans buckets (bounded +
cached) to compute total size/objects, per-bucket distribution, file-type
and size-class breakdowns, cumulative data-growth timeline and largest /
recent objects. Track every S3 op routed through the API for a live
storage-activity timeline.
Rebuild StorageView into a tabbed view: Overview (KPI cards + SVG charts:
growth area, bucket donut, type/size bars, activity sparkline, top folders,
largest & recent objects) and the original bucket Browser.
Add Graph sub-tab in Data Sources UI for Neo4j: force-directed SVG view of
Product-Supplier nodes and SUPPLIES/RELATED_TO/PART_OF/COMPATIBLE_WITH edges.
Backend GET /api/sql/graph/neo4j with rel_type filter and edge limit.
New "Data Sources UI" tab with per-database Browser (catalog + sample data),
Query Console (SqlWorkbench) and embedded interactive Shell (DbShell via SSH).
Backend:
- Extend sql_console.py with Cassandra (CQL) + Neo4j (Cypher) engines
- Add GET /api/sql/catalog/{engine} and GET /api/sql/sample/{engine}
- ssh_terminal: optional initial_command for auto-launching DB CLIs
Frontend:
- DataSourcesView with 5-DB rail, health dots, Browser/Console/Shell sub-tabs
- DbShell embedded xterm terminal with docker exec CLI per engine
- Deep-link topology DB nodes to Data Sources UI (no SQL dock on platform)
- WorkbenchPanel restricted to agent mode only — frees dashboard space
Align nodes into pipeline columns (producers -> sources -> CDC -> lakehouse ->
Trino) with governance centred at the bottom, so lineage reads in order instead
of Iceberg/Trino floating mid-canvas.
Query postgres/mysql/mongo directly (fast, early LIMIT) instead of full Trino
table scans; Trino remains the path for the curated lakehouse + as a fallback.
- pii_catalog: persistent per-column masking policy (default masked); GET/POST
/api/pii/policy and POST /api/pii/lookup which redacts masked values server-side.
- get_pii masked flag now reflects the policy; dataflow exposes the dataset key.
- Data Flow PII inspector: per-column lock/unlock toggles + mask-all/unmask-all,
so operators control exactly which data the assistant may reveal.
- decide_approval now executes the underlying data movement when an approved
request carries an executor=movement payload (real human-gated executor).
- rag-api service gets OPENMETADATA_* (via atc.env) + COMMAND_CENTER_URL so it can
sync the catalog and call platform tools.
- Knowledge Chat gains an Agent-mode toggle (SSE tool-loop with step chips) and a
'Sync catalog' button.
lab_context now includes an in-process governance section so the LLM knows the
live CDC volume, data movements + last-run state, the PII catalog (OM+heuristic)
with masked/unmasked status, and source->masked lineage.
Data Flow graph now shows OpenMetadata as a governance node linked to all
sources and Trino (catalog edges); node inspector exposes an 'Open in
OpenMetadata' deep link and PII-columns-cataloged metric.
Add OpenMetadata node (atc-docker02 .47) to node_registry with UI links.
pii_catalog now reads OM column PII tags (Presidio auto-classification) as the
authoritative source, merged with the name heuristic; OPENMETADATA_URL wired
into the api service (token via atc.env).
- DAG hadoop_to_trino (deployed to Airflow) + worker hadoop_to_trino.py (on the
Hadoop master) move HDFS historical_sales -> iceberg.hadoop.historical_sales_hdfs.
- api/movements.py: movement registry, Airflow trigger+watch, run tracking
(state/duration/rows), endpoints /api/movements, /{id}/run, /runs.
- agent_ops.py: ETL-agent loop autonomously triggers movements on an interval
and logs each run; /api/agent-ops/etl/toggle + etl status.
Add api/cdc_consumer.py: aiokafka background consumer subscribes to the CDC
topics (postgres_sales/mysql_hr/mongodb_supplychain + cassandra/neo4j), parses
Debezium before/after envelopes, keeps a ring buffer and publishes each change
live as type=cdc_change. Endpoints /api/changes, /api/changes/stats, /status.
Deploys Apache Hive 3.1.3 (Derby metastore + external table over the HDFS
historical CSV, MapReduce exec) on the Hadoop master, so the engine comparison
shows a REAL measured Hive latency (~5.2s) next to live Trino (~0.3s); Impala
stays clearly-labelled representative. The API re-measures Hive over SSH on a
30-min TTL (cached + persisted, with a committed seed). Adds filters
(year/region/category/channel), a region×category heatmap, Trino exec stats,
and a "where is this data read from" lineage panel (Trino->S3/Iceberg/Parquet
with snapshot+files, Hive->HDFS/CSV with namenode+files). Mounts host SSH key
read-only into the api container for the live Hive benchmark.
Adds an Analytics sub-tab to the Hadoop view with live KPIs and revenue
breakdowns (by year/region/category/channel) queried from Trino over
iceberg.hadoop.historical_sales, plus a query-engine comparison panel. Trino
latency is measured live; Impala and Hive are shown as clearly-labelled
representative figures (those engines are not deployed). New cached endpoints
/api/hadoop/analytics and /api/hadoop/engines.
mo