feat(governance): close the 6 data-disease gaps — DQ monitoring, ownership, access posture, lineage & observability

Adds native Command Center features (no new containers) integrated as sub-tabs
in the existing Data Explorer and Data Quality views:

- Continuous Data Quality (dq_monitor.py): live completeness/uniqueness/validity/
  freshness scorecards via Trino with rolling trends → DataQuality "Live Monitoring".
- Ownership & stewardship (catalog_governance.py): owner/steward/tier matrix,
  orphan detection, business glossary; local store best-effort synced to
  OpenMetadata (owner PATCH) → Data Explorer "Ownership".
- Access & policy posture: per-dataset compliance combining PII masking, ownership,
  live DQ and observability alerts vs data contracts → Data Explorer "Access & Policies".
- Lineage (lineage.py): staged source→CDC→Spark→S3→Iceberg→Trino→serving graph with
  live row counts and column-level PII/masking tracing → Data Explorer "Lineage".
- Observability (observability.py): volume/freshness/schema-drift monitoring with
  alerts → Data Explorer "Observability".
- Shared lake_meta.py dataset registry + bounded Trino client; fast native row-count
  and PK-indexed freshness so monitors stay cheap on 25-54M-row tables.
- LLM context (lab_context.py) enriched with DQ scores, ownership and active alerts.
This commit is contained in:
mo
2026-06-29 17:43:37 +00:00
parent d066def8b4
commit f36c8906bc
15 changed files with 2479 additions and 4 deletions
+8
View File
@@ -57,6 +57,10 @@ from spark_workbench import router as spark_workbench_router
from pii_catalog import router as pii_router
from trino_federated import router as federated_router
from etl_offload import router as etl_offload_router
from lineage import router as lineage_router
from dq_monitor import router as dq_router
from observability import router as observability_router
from catalog_governance import router as governance_router
from ssh_terminal import ssh_session
from node_registry import NODE_IDS, NODE_AGENT, NODE_REGISTRY, is_node_id
from node_ops import build_node_detail, probe_node, run_node_probe_task
@@ -768,6 +772,10 @@ app.include_router(spark_workbench_router)
app.include_router(pii_router)
app.include_router(federated_router)
app.include_router(etl_offload_router)
app.include_router(lineage_router)
app.include_router(dq_router)
app.include_router(observability_router)
app.include_router(governance_router)
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],