ATC/atc-agents
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(hadoop): real Hive engine + filterable, detailed analytics with lineage

Browse Source 
Deploys Apache Hive 3.1.3 (Derby metastore + external table over the HDFS
historical CSV, MapReduce exec) on the Hadoop master, so the engine comparison
shows a REAL measured Hive latency (~5.2s) next to live Trino (~0.3s); Impala
stays clearly-labelled representative. The API re-measures Hive over SSH on a
30-min TTL (cached + persisted, with a committed seed). Adds filters
(year/region/category/channel), a region×category heatmap, Trino exec stats,
and a "where is this data read from" lineage panel (Trino->S3/Iceberg/Parquet
with snapshot+files, Hive->HDFS/CSV with namenode+files). Mounts host SSH key
read-only into the api container for the live Hive benchmark.
This commit is contained in:
mo
2026-06-26 17:08:57 +00:00
parent 71a64d5a21
commit 96d490807a
7 changed files with 570 additions and 221 deletions
Download Patch File Download Diff File Expand all files Collapse all files
api/Dockerfile
+1 -1
View File
@@ -4,7 +4,7 @@ WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY main.py lab_context.py agent_terminal.py workload.py node_registry.py node_ops.py topology_views.py supervisor.py approval_service.py db.py dockhand_envs.py presentation.py database_inventory.py presentation_upload.py presentation_static.py storage_s3.py elasticsearch_api.py sql_console.py hdfs_api.py ssh_terminal.py pipeline_ops.py hadoop_analytics.py .
COPY main.py lab_context.py agent_terminal.py workload.py node_registry.py node_ops.py topology_views.py supervisor.py approval_service.py db.py dockhand_envs.py presentation.py database_inventory.py presentation_upload.py presentation_static.py storage_s3.py elasticsearch_api.py sql_console.py hdfs_api.py ssh_terminal.py pipeline_ops.py hadoop_analytics.py hive_bench_seed.json .
RUN mkdir -p /data
ENV DATABASE_URL=sqlite:////data/atc-agents.db
EXPOSE 3201