Deduplicating source data

AD
DM

In source subscriptions, use the Source Dedup option to prevent duplicate records in the source feed from being mistakenly added to your Network MDMinstance.

It can be common for a data feed to contain duplicates in initial loads from source systems, but it can occur in subsequent loads as well.

By default, Network MDM matches incoming records against records that are currently in Network MDM. If no matches are found for an incoming record, the incoming record is inserted into Network MDM as a new, unique record. To prevent duplicate records from being created, the Source Dedup option adds a second iteration of matching.

First, all incoming records are matched against existing records in your Network MDM instance (default behavior).
Second, incoming records that didn't ACT match to existing records are matched against other records in the same source feed.

This occurs prior to final merge so that duplicates are not added to your Network MDM data.

Enable source feed deduplication

When Source Dedup is set, the subscription's defined grouping and match rules are used to deduplicate any records in the source feed that did not match or ASK matched to existing records in your Network MDM instance.

Enable it in one of the following ways:

Subscription settings

In the Settings section, select Source Dedupe and choose Entities (HCPs, HCOs) and/or the sub-objects.
Advanced Mode
- Entity source dedupe - "job.match.dedup": "true". The value can be either true or false. The value is false, by default.
- Child source dedupe - "job.merge.childDedup": "ADDRESS,PARENTHCO". The value can be any combination of ADDRESS, LICENSE, or PARENTHCO. By default, there is no value. The value must be in uppercase letters for the configuration to work.
Note: Selecting the Source Dedup options in the Settings populates these job properties.

Deduplicating sub-objects

Administrators and data managers can choose to merge any duplicate sub-objects that are loaded during a source subscription job.

This feature does not deduplicate existing sub-objects in your Network MDM instance. It applies only to data that is loaded through source subscriptions after the feature is enabled in each subscription.

Sub-object duplicates cannot be matched and merged in Veeva OpenData subscriptions or third-party source subscriptions.

Matching sub-objects

Duplicate sub-objects are matched and merged using the methods in this order:

Key matching (custom keys are the same)

Key matching is not available for Parent HCO objects.
Field matching (using duplicate detection rules).

Duplicate detection rules

Network MDM contains a set of internal sub-object comparison rules that are applied in any merge comparison.

Fields used in comparison rules

The following fields are used by default for the Veeva standard sub-objects to compare for duplicates.

Address License Parent HCO

Address	License	Parent HCO
Addresses are compared to each other only if they have the same address verification status. Fields that are compared for these address verification statuses: V - Verified, A - Ambiguous, or P - Partially Verified `thoroughfare__v` `premise_number__v` `locality__v` `country__v` `postal_code_primary__v` (except in China, use `administrative_area__v`instead) Fields that are compared for any other verification status: `address_line_1__v` `locality__v` `country__v` `postal_code__v`	`license_number__v` `type_value__v` `license_degree__v`	`parent_hco_vid__v` `relationship_type__v` (except in Japan, use `department_name__v`instead) `hierarchy_type__v`

Addresses are compared to each other only if they have the same address verification status.

Fields that are compared for these address verification statuses:

V - Verified, A - Ambiguous, or P - Partially Verified

thoroughfare__v
premise_number__v
locality__v
country__v
postal_code_primary__v
(except in China, use
administrative_area__vinstead)

Fields that are compared for any other verification status:

address_line_1__v
locality__v
country__v
postal_code__v

license_number__v
type_value__v
license_degree__v

parent_hco_vid__v
relationship_type__v
(except in Japan, use
department_name__vinstead)
hierarchy_type__v

Address comparisons

Addresses are compared to each other only if they have the same address verification status.

Address Verification Status	Behavior
Same status	If the verification status is identical, the default comparison rules are then used. If the addresses are byte-to-byte identical based on the fields, the addresses are considered a match and are merged.
Different status	Addresses could be identical, but if their address verification status differs, the incoming address will not merge into the existing one; it will be added as a new address.

This behavior applies to addresses that are added in any job in Network MDM.

Overridden addresses

Existing addresses - Addresses that have been overridden by a Data Steward will never be merged with an incoming address because the address verification statuses will not be the same.
New addresses - When new addresses are included in add or change request and they are overridden by the Data Steward that is processing the request, they are not re-cleansed.

Tie-breaker rules

For sub-object source deduplication, if duplicate sub-objects exist (there's more than one match) then Network MDM breaks the tie by matching the objects to the best sub-object based on the following criteria:

Address: status, ordinal__v, lowest Network ID
License: status, best_state_license__v, lowest Network ID
ParentHCO: status, is_primary_relationship__v, lowest Network ID

Sub-object deduplication examples

Review the following examples to understand how Network MDM deduplicates sub-objects.

Example 1 - Dedup entities and sub-objects

In this example, multiple HCPs are loaded with the same address multiple times. The HCPs have already been determined to match and merge.

The custom key uses the following configuration:

Source: SAP®
Item: ADDRESS
Value: Network ID

HCP

Network ID	First Name	Last Name	NPI Number	Custom Key
59259	Bob	Smith	9925125	SAP:HCP:59259
7513B	Bob	Smith	9925125	SAP:HCP:7513B

Address

Network ID	Address_ID	Address Line 1	City	ZIP	Country	Custom Key
59259	4292151	123 Main Street	Salem	97310	United States	SAP:Address:4292151
59259	4292152	123 Main Street	Salem	97310	United States	SAP:Address:4292152
20012	4292153	123 Main Street	Salem	97310	United States	SAP:Address:4292153
20012	4292154	123 Main Street	Salem	97310	United States	SAP:Address:4292154

Result

When the data is loaded, Network MDM checks for matches in this order:

Custom keys - The keys for the HCP and the Address do not match.
Duplicate detection rules - The addresses are the same so one address is loaded. Address with Network ID 20012 wins because it was the last row to be added.

All of the custom keys are added to the sub-object.

HCP

Network ID	First Name	Last Name	NPI Number	Custom Key
VID1	Bob	Smith	9925125	SAP:HCP:7513B SAP:HCP:59259

Address

Network ID	Entity_ID	Address_ID	Address Line 1	City	ZIP	Country	Custom Key
VID1A		4292154	123 Main Street	Salem	97310	United States	SAP:Address:4292154 SAP:Address:4292153 SAP:Address:4292152 SAP:Address:4292151

Example 2 - Duplicate addresses

A record is loaded with duplicate addresses. There are no duplicate custom keys, so Network MDM uses duplicate detection rules to deduplicate the sub-objects.