Add a dedicated business-data view (separate from infra/search):
- New "Data Explorer" tab: KPIs (customers, employees, products, source
row totals, sample revenue) + charts driven by Elasticsearch aggregations
(revenue over time, by region/channel/status, top customers & products,
HR by department/role/event, supply chain by type, telemetry averages),
with region + free-text filters.
- Backend /api/search/business endpoint: battery of ES aggregations with
numeric/date index template (so amount sums and date histograms work),
source totals via cheap planner estimates; biased the orders sample to
rows carrying customer names so people are searchable/visible.
Manual data entry on source systems:
- "Insert row" action in the Data Hub browser opens a column-aware form;
POST /api/sql/row/insert writes to Postgres/MySQL/Mongo/Cassandra/Neo4j.
- Inserts into CDC sources (PG/MySQL/Mongo) are captured by Debezium and
streamed to Kafka in real time; UI flags this and pulses the data flow.
Build dedicated de-duplicated entity indices (atc-customers ~39k,
atc-employees ~39k, atc-products) from Trino so users can search by
customer/employee name, id and email directly instead of scanning raw
transaction rows. Raise per-table sample caps for the big source tables
(sales_orders, employee_events, supplychain events, device_metrics) and
all iceberg/Hadoop tables for broader row-level coverage.
UI: add Datasets quick-filter chips (Customers, Employees, Products,
Sales orders, HR events, Supply chain, Telemetry, Lakehouse) and default
the Search tab to business data instead of an infra-looking empty view.
mo