- etl_offload.py: autonomous agent backfills/tails source DBs (PG/MySQL/
Mongo/Cassandra) to S3 as Parquet in small chunks, accumulates a live
federated business matrix (/api/etl/status, /api/etl/business, /run, /config).
- storage_s3.py: buffer generated CDC + masked curated rows to S3, overlay
live last-write into analytics; put_object_bytes for Parquet parts.
- trino_federated.py: capture generated rows + archive to S3; generator_active.
- dataflow.py: pulse generate + kafka/spark->S3 archive edges when active.
- StorageView: realtime ETL ingest panel; TrinoFederationView: realtime
business KPIs/charts from /api/etl/business.
- ChangesView: top KPIs/charts now overlay the live WS stream on server stats
so they update in lock-step with the bottom feed; faster 2.5s refresh.
- useCommandCenter: retain 800 live CDC changes.
Add a /preview endpoint that range-reads the first chunk of an object and
returns it as text. Browser gets an eye action (list + gallery) opening a
modal that pretty-prints JSON, renders CSV/TSV as a table, and shows logs/
text/yaml/xml verbatim, with a truncation note and download link.
Add a List/Gallery toggle to the bucket browser that auto-switches to a
thumbnail grid when a listing is mostly images, and a keyboard-navigable
lightbox (prev/next/esc) for full-size previews. Download endpoint serves
inline with the correct image/* content-type (ECS stores octet-stream) so
previews render instead of forcing a download.
Analytics now walks every top-level prefix with its own budget instead of a
flat alphabetical scan, so a single huge prefix (kafka/ CDC json) no longer
hides the rest — composition now correctly reflects images, logs, csv,
parquet and avro, and reports true total size.
Browser gains recursive (whole-subtree) listing, free-text name search,
file-type filter chips, type column and Load-more pagination so every object
across all folders is discoverable.
Add /api/storage/s3/analytics endpoint that scans buckets (bounded +
cached) to compute total size/objects, per-bucket distribution, file-type
and size-class breakdowns, cumulative data-growth timeline and largest /
recent objects. Track every S3 op routed through the API for a live
storage-activity timeline.
Rebuild StorageView into a tabbed view: Overview (KPI cards + SVG charts:
growth area, bucket donut, type/size bars, activity sparkline, top folders,
largest & recent objects) and the original bucket Browser.
mo