Data deduplication

AD
DM

Administrators and data managers can create data maintenanceClosed User directed automated jobs that improve data quality by targeting specific data quality issues like sub-object inactivation and data deduplication detection. subscriptions to identify duplicate HCP and HCO records in their Network instance.

Duplicate records can exist for the following reasons:

  • poor match rules

  • DCR processing errors

  • records are added without searching for existing records first

Use data deduplication jobs to compare specific records in your Network instance against all other records in your instance.

This feature is enabled by default in your Network instance.

Supported objects

Data deduplication subscriptions are supported for Veeva objects only (HCPs, HCOs). They do not support custom objects.

Finding duplicate records

Similar to source subscriptions, data deduplication subscriptions use the following tools:

  • Data groups - to narrow the number of records being compared
  • Match rules - to determine if records are the same or not

Match logs contain a summary of the records that were merged or suspect matched during the job. Comparisons are done at the entity level. Existing merge rules are used for sub-objects when records are merged.

All previous unmerges that have occurred in the instance are tracked and can be ignored from comparisons when you create a data deduplication job. However, rejected suspect match tasks and duplicates are tracked only from the feature implementation and onwards.

The jobs can be thoroughly tested before you commit to merging any records.

Considerations for data deduplication subscriptions

Before you create a subscription, review the following considerations:

  • Match types - It is a best practice to begin using data deduplication with ASKClosed A single incoming record matches multiple incoming or Network records without one clear "best" match. Ambiguous matches typically require human review. matches only. After you have tested the job and are confident in your match rules, you can introduce ACTClosed A high confidence match between two records. ACT matches result in a merge without any human review. matches.
  • Job size - Jobs can take some time to run, so prioritize your data and work with small, logical sets of data for each data deduplication job.

  • Applicable records - Only active and valid customer-owned HCP and HCO records from the specified Network Entity IDs (VIDs) are compared. These records are compared against all of the records in the instance, which includes Veeva OpenData records, third party mastered records and other customer-owned records.

    Valid records for a country where match rules are not defined are skipped because the data deduplication process cannot match those records.

    The following record types are excluded from the comparison: invalid, under reviewClosed A record that has not yet been validated by a data steward., externally managed (Veeva OpenData or third party), and candidate records. Active and inactive sub-objects are considered in the match comparisons to help identify duplicate entities in your Network instance.

  • Match and Download from Veeva OpenData - Customer owned-records are compared against all records that exist in your Network instance. You can enable the option to match against records in the OpenData instances. OpenData records that match will be downloaded to your Network instance during the job.

  • Job limits

    • Suspect match task - record limit - Each task that is created by the job is limited to 20 records. If the limit is exceeded, the job completes with errors.

    • Suspect match task - number of tasks - Each job can produce a maximum of 200 suspect match tasks. If the limit is exceeded, the job completes with errors.

    • Number of matches - Each job can produce a maximum of 20 million matches. If the limit is exceeded, the job fails.

  • Match and Download from OpenData limits

    • Downloaded records limit - A maximum of 5000 records can be matched and downloaded from OpenData in each job. If the job tries to download more than 5000 records, the job will fail.

      This limitation applies to customers that purchase OpenData per record for a country. It applies when records are matched and downloaded from OpenData in source subscriptions and data deduplication subscriptions.

      It does not apply to customers that purchase all records for a country.