mo
21af36a591
Profiler: - pg/mysql: table-level metrics via DB statistics (exact-ish row counts) only; source tables hold 24-53M rows and OM's column profiler full-scans per column (no TABLESAMPLE pushdown), which would hammer the live CDC source. - trino/iceberg: full column metrics + sample data on the small (20-25k row) curated_masked + hadoop tables. Data quality: 16 test cases across 5 tables (row counts on big sources; row count + uniqueness/not-null/range on curated/masked + historical), all passing. This populates OM's Data Quality dashboard (coverage, healthy assets, dimensions, test results).
41 lines
1.1 KiB
YAML
41 lines
1.1 KiB
YAML
source:
|
|
type: postgres
|
|
serviceName: atc_postgres
|
|
serviceConnection:
|
|
config:
|
|
type: Postgres
|
|
username: mo
|
|
authType:
|
|
password: Dell2026!
|
|
hostPort: 10.0.21.51:5432
|
|
database: postgres
|
|
# Hard server-side cap so no profiler query can run away on huge tables.
|
|
connectionArguments:
|
|
options: "-c statement_timeout=90000"
|
|
sourceConfig:
|
|
config:
|
|
type: Profiler
|
|
# Table-level metrics from DB statistics only. The source tables hold tens
|
|
# of millions of rows; OM's column profiler full-scans per column (it does
|
|
# not push down TABLESAMPLE), which would hammer the live CDC source. Row
|
|
# counts come from pg_class stats instantly.
|
|
computeTableMetrics: true
|
|
computeColumnMetrics: false
|
|
useStatistics: true
|
|
threadCount: 5
|
|
schemaFilterPattern:
|
|
includes:
|
|
- public
|
|
processor:
|
|
type: orm-profiler
|
|
config: {}
|
|
sink:
|
|
type: metadata-rest
|
|
config: {}
|
|
workflowConfig:
|
|
openMetadataServerConfig:
|
|
hostPort: http://openmetadata-server:8585/api
|
|
authProvider: openmetadata
|
|
securityConfig:
|
|
jwtToken: "__JWT__"
|