feat: Trino federation + Hadoop external tables + LLM data catalog

Trino Federation tab (3 sub-views):
- Federated: catalog landscape + a single cross-source SQL that joins
  PostgreSQL + MySQL + MongoDB (region scorecard) — the federation proof,
  computed in the background and cached (large full scans take ~2 min).
- Hadoop Lake: all federated business data materialized as external Iceberg
  tables on HDFS (iceberg.hadoop.*_ext, ~120k rows each) with live, fast
  business analytics (revenue by region/channel, top customers, HR by
  department, supply by type, telemetry averages). Includes a one-click
  "rebuild external tables" job.
- Data Dictionary: every business table + column with masked / visible PII
  badges and categories.

Backend trino_federated.py: /catalogs, /marquee(+refresh), /lake,
/materialize(+status), /dictionary. Name-based PII detection flags raw PII
in derived/lake tables as visible vs physically-masked curated layer.

LLM context: platform_context now emits a full BUSINESS DATA CATALOG section
(tables, columns, types, source row counts, federated scorecard) with exact
per-column masked/visible status, so the assistant knows the data in detail
and what is masked vs not.
This commit is contained in:
mo
2026-06-28 18:01:25 +00:00
parent 8d28695868
commit ea3e59cf9c
8 changed files with 855 additions and 5 deletions
+3
View File
@@ -19,6 +19,7 @@ import { ChangesView } from './components/features/ChangesView'
import { DataFlowView } from './components/features/DataFlowView'
import { SearchView } from './components/features/SearchView'
import { DataExplorerView } from './components/features/DataExplorerView'
import { TrinoFederationView } from './components/features/TrinoFederationView'
import { DataHubView } from './components/features/DataHubView'
import { SshTerminal } from './components/features/SshTerminal'
import { TerminalDock } from './components/features/TerminalDock'
@@ -144,6 +145,8 @@ export default function App() {
<SearchView />
) : cc.mainView === 'dataexplorer' ? (
<DataExplorerView />
) : cc.mainView === 'trino' ? (
<TrinoFederationView />
) : (
<ApprovalInbox agents={cc.agents} livePending={cc.approvals} onDecide={cc.decide} />
)}