feat(governance): close the 6 data-disease gaps — DQ monitoring, ownership, access posture, lineage & observability
Adds native Command Center features (no new containers) integrated as sub-tabs in the existing Data Explorer and Data Quality views: - Continuous Data Quality (dq_monitor.py): live completeness/uniqueness/validity/ freshness scorecards via Trino with rolling trends → DataQuality "Live Monitoring". - Ownership & stewardship (catalog_governance.py): owner/steward/tier matrix, orphan detection, business glossary; local store best-effort synced to OpenMetadata (owner PATCH) → Data Explorer "Ownership". - Access & policy posture: per-dataset compliance combining PII masking, ownership, live DQ and observability alerts vs data contracts → Data Explorer "Access & Policies". - Lineage (lineage.py): staged source→CDC→Spark→S3→Iceberg→Trino→serving graph with live row counts and column-level PII/masking tracing → Data Explorer "Lineage". - Observability (observability.py): volume/freshness/schema-drift monitoring with alerts → Data Explorer "Observability". - Shared lake_meta.py dataset registry + bounded Trino client; fast native row-count and PK-indexed freshness so monitors stay cheap on 25-54M-row tables. - LLM context (lab_context.py) enriched with DQ scores, ownership and active alerts.
This commit is contained in:
@@ -57,6 +57,10 @@ from spark_workbench import router as spark_workbench_router
|
||||
from pii_catalog import router as pii_router
|
||||
from trino_federated import router as federated_router
|
||||
from etl_offload import router as etl_offload_router
|
||||
from lineage import router as lineage_router
|
||||
from dq_monitor import router as dq_router
|
||||
from observability import router as observability_router
|
||||
from catalog_governance import router as governance_router
|
||||
from ssh_terminal import ssh_session
|
||||
from node_registry import NODE_IDS, NODE_AGENT, NODE_REGISTRY, is_node_id
|
||||
from node_ops import build_node_detail, probe_node, run_node_probe_task
|
||||
@@ -768,6 +772,10 @@ app.include_router(spark_workbench_router)
|
||||
app.include_router(pii_router)
|
||||
app.include_router(federated_router)
|
||||
app.include_router(etl_offload_router)
|
||||
app.include_router(lineage_router)
|
||||
app.include_router(dq_router)
|
||||
app.include_router(observability_router)
|
||||
app.include_router(governance_router)
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
|
||||
Reference in New Issue
Block a user