feat(masking): mask_to_curated DAG + OpenMetadata lineage + versioned OM artifacts
- deploy/airflow/mask_to_curated_dag.py: Trino-SQL masking (hash/redact/generalize) of PII from postgres/mysql sources into iceberg.curated_masked.* (verified: 14k+11k masked rows, all PII masked). - OM source->masked lineage edges created; curated tables cataloged. - Version OM compose + ingestion configs + om_api helper under deploy/openmetadata.
This commit is contained in:
@@ -0,0 +1,35 @@
|
||||
source:
|
||||
type: postgres
|
||||
serviceName: atc_postgres
|
||||
serviceConnection:
|
||||
config:
|
||||
type: Postgres
|
||||
username: mo
|
||||
authType:
|
||||
password: Dell2026!
|
||||
hostPort: 10.0.21.51:5432
|
||||
database: postgres
|
||||
sourceConfig:
|
||||
config:
|
||||
type: AutoClassification
|
||||
enableAutoClassification: true
|
||||
storeSampleData: true
|
||||
confidence: 80
|
||||
schemaFilterPattern:
|
||||
includes:
|
||||
- public
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- sales_orders
|
||||
processor:
|
||||
type: orm-profiler
|
||||
config: {}
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: http://openmetadata-server:8585/api
|
||||
authProvider: openmetadata
|
||||
securityConfig:
|
||||
jwtToken: "__JWT__"
|
||||
Reference in New Issue
Block a user