Commit Graph

11 Commits

Author SHA1 Message Date
mo 96d490807a feat(hadoop): real Hive engine + filterable, detailed analytics with lineage
Deploys Apache Hive 3.1.3 (Derby metastore + external table over the HDFS
historical CSV, MapReduce exec) on the Hadoop master, so the engine comparison
shows a REAL measured Hive latency (~5.2s) next to live Trino (~0.3s); Impala
stays clearly-labelled representative. The API re-measures Hive over SSH on a
30-min TTL (cached + persisted, with a committed seed). Adds filters
(year/region/category/channel), a region×category heatmap, Trino exec stats,
and a "where is this data read from" lineage panel (Trino->S3/Iceberg/Parquet
with snapshot+files, Hive->HDFS/CSV with namenode+files). Mounts host SSH key
read-only into the api container for the live Hive benchmark.
2026-06-26 17:08:57 +00:00
mo 1147826dee fix(superset): disable Secure session cookie over HTTP so mo/bart can log in
Talisman forced Secure on the session cookie while Superset is served over
plain HTTP, so browsers never returned the cookie and every login failed CSRF
validation ("CSRF session token is missing"). Set session_cookie_secure=False
(+ SESSION_COOKIE_SECURE/WTF_CSRF_SSL_STRICT). Also rename S3 lakehouse prefix
iceberg-warehouse -> hadoop (Trino iceberg.hadoop.historical_sales) and repoint
the Superset dataset. Adds sanitized infra backup scripts.
2026-06-26 15:27:34 +00:00
mo 2d14eae55f feat(historical): offload historical_sales to S3 (Iceberg/Parquet on Dell ECS), repoint dashboard, drop Postgres copy 2026-06-26 11:38:30 +00:00
mo 37c03b6881 feat(historical): durable Postgres-backed historical_sales + repoint Superset dashboard (no Trino downtime) 2026-06-26 09:43:21 +00:00
mo be2bbcbd92 feat(cdc): app-level Cassandra+Neo4j change events to Kafka (->S3) on generate 2026-06-26 09:21:17 +00:00
mo f4cd33b784 infra: historical sales -> iceberg.historical (Trino) + Superset dashboard scripts 2026-06-26 09:04:33 +00:00
mo 9cbdccb403 infra: fix mongodb uuid encoding + combined DAG honors rows conf 2026-06-26 08:41:44 +00:00
mo d26698a0c7 infra: hadoop historical sales loader (HDFS partitioned by year) 2026-06-26 01:26:43 +00:00
mo 2f2cecd16f infra: neo4j generator creates indexes for fast relationship MERGE 2026-06-26 01:24:56 +00:00
mo 89b31c680a infra: fix cassandra (prepared stmt) + neo4j (drop APOC, grouped MERGE) generators 2026-06-26 01:24:06 +00:00
mo 62b416d210 infra: mirror light per-source Airflow DAGs + generators (GEN_ROWS), fix script path 2026-06-26 01:07:23 +00:00