feat(openmetadata): profiler + data-quality tests for data insight
Profiler: - pg/mysql: table-level metrics via DB statistics (exact-ish row counts) only; source tables hold 24-53M rows and OM's column profiler full-scans per column (no TABLESAMPLE pushdown), which would hammer the live CDC source. - trino/iceberg: full column metrics + sample data on the small (20-25k row) curated_masked + hadoop tables. Data quality: 16 test cases across 5 tables (row counts on big sources; row count + uniqueness/not-null/range on curated/masked + historical), all passing. This populates OM's Data Quality dashboard (coverage, healthy assets, dimensions, test results).
This commit is contained in:
@@ -0,0 +1,44 @@
|
||||
source:
|
||||
type: TestSuite
|
||||
serviceName: atc_data_quality
|
||||
sourceConfig:
|
||||
config:
|
||||
type: TestSuite
|
||||
entityFullyQualifiedName: "atc_trino.iceberg.curated_masked.sales_orders_masked"
|
||||
processor:
|
||||
type: orm-test-runner
|
||||
config:
|
||||
testCases:
|
||||
- name: sales_orders_masked_has_rows
|
||||
testDefinitionName: tableRowCountToBeBetween
|
||||
parameterValues:
|
||||
- name: minValue
|
||||
value: "1"
|
||||
- name: maxValue
|
||||
value: "100000000"
|
||||
- name: sales_orders_masked_order_id_unique
|
||||
columnName: order_id
|
||||
testDefinitionName: columnValuesToBeUnique
|
||||
- name: sales_orders_masked_order_id_not_null
|
||||
columnName: order_id
|
||||
testDefinitionName: columnValuesToBeNotNull
|
||||
- name: sales_orders_masked_email_not_null
|
||||
columnName: customer_email_masked
|
||||
testDefinitionName: columnValuesToBeNotNull
|
||||
- name: sales_orders_masked_amount_range
|
||||
columnName: amount
|
||||
testDefinitionName: columnValuesToBeBetween
|
||||
parameterValues:
|
||||
- name: minValue
|
||||
value: "0"
|
||||
- name: maxValue
|
||||
value: "100000000"
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: http://openmetadata-server:8585/api
|
||||
authProvider: openmetadata
|
||||
securityConfig:
|
||||
jwtToken: "__JWT__"
|
||||
Reference in New Issue
Block a user