Rejecting records with duplicate custom keys


Custom keys are cross-references between an identifier from an external system and a Network ID. All custom keys should be unique throughout your Network instance. In a model map, a custom key is defined by source, item, and value identifiers and has an active status; for example: SAP:SAP ID:242. Records that contain duplicate custom keys cannot be updated by users, so any duplicates should be removed.

When this feature is enabled, during a source subscription load, records that contain duplicate custom keys are rejected. This helps to preserve data quality and to ensure that records can be updated.

During data load, one record is accepted, but any other records with the same custom key are rejected. This applies to duplicate custom keys on entities and sub- objects. The entire record is rejected if a sub-object has a duplicate custom key. Review the examples below to understand how Network handles different situations with duplicate custom keys.

Note: This feature does not apply to duplicate custom keys that have previously been loaded to your Network instance. To clean up any existing duplicate custom keys, contact Veeva Support for a list of the duplicates and recommended action.

Enabling this feature

By default, this feature is enabled for Network instances created after version 10.0. For these instances, the following options do not appear in the Network UI unless visibility is enabled by Network Support.

To enable this feature for instances created for Network v10.0 or earlier:

  1. In the Admin console, click Settings > General Settings.
  2. On the General Settings page, click Edit.
  3. Select the Reject Duplicate Keys from Data Load option.

  4. Save your changes.

Error logs

Errors occur for records that were not loaded because of duplicate custom keys.

To view the records that were not loaded, on the Job Details page, view the Job Error Log section. Each error is cataloged here so that you can review the error and update the affected custom keys so that they are not longer duplicates.

Updating existing records that have duplicate custom keys

Records in your Network instance that have a duplicate custom key cannot be updated by matching on the custom key using source subscriptions. Previously, Network would update the first record it sees if the same custom key is associated to multiple records. Now, during data load, if Network detects that there are updates for a key that is associated to multiple records, then no update will be loaded and an error occurs. This change in behavior ensures that data quality is preserved; records are not updated unless Network can determine exactly which record, containing the custom key, the update is intended for.

Examples for duplicate custom keys

Review the following examples to understand how Network handles duplicate custom keys during data load.

Example 1

A data load includes a record that has duplicate custom keys for the HCP and an address.

Source file

Source Source HCP ID Source Address ID First Name Last Name Specialty Address Line 1 City Country
SAP 2421 2421 Sam Smith Cardiology 502 Port St New York United States

HCP

Network Entity ID First Name Last Name Specialty Custom Key
- Sam Smith Cardiology SAP:SAP ID:2421

Address

Network Entity ID Entity ID Address Line 1 City Country Custom Key
- 51292 502 Port St New York United States SAP:SAP ID:2421

Custom Key

Network Entity ID

entity_vid__v

entity_type

Source Item Value
- 52192 HCP SAP SAP ID 2421
- 21482 Address SAP SAP ID 2421

Result: The entire record will be rejected during the source load because the HCP and address have the same custom key. This typically occurs from an error in the model map.

Tip: To create unique custom keys, use the entity type as the Item value to prevent duplicates. In this example, if the entity type is used, there would be two unique custom keys: SAP:HCP:2421 and SAP:Address:2421.

Example 2

A data load includes a two HCPs that have the same address (duplicate custom keys).

Source file

Source Source HCP ID Source Address ID First Name Last Name Specialty Address Line 1 City Country
SAP 9521 90281 Sam Smith Cardiology 421 Long St California United States
SAP 4981 90281 Tony Matthews Oncology 421 Long St California United States

HCP

Network Entity ID First Name Last Name Specialty Custom Key
- Sam Smith Cardiology SAP:HCP: 9521
75211 Tony Matthews Oncology SAP:HCP: 4981

Address

Network Entity ID Entity ID Address Line 1 City Country Custom Key
14076 51292 421 Long St San Jose United States SAP:Address: 90281
51659 75211 421 Long St San Jose United States SAP:Address: 90281

Result: One of the records will be accepted, subsequent records will not be imported and an error will occur. In this example, Tony Matthews is loaded into Network and Sam Smith is rejected.

Tip: To create unique custom keys, append the Value from the HCP's custom key to the front of the Value for the address's custom key. In this example, the two unique custom keys would be SAP:ADDRESS:9521-90281 and SAP:ADDRESS:4981-90281.

Example 3

A data load includes a two different entities that have the same custom key.

Source file

Source Source HCO ID Corporate Name HCO Type
SAP 9521 Sunnybrook Hospital Hospital
SAP 9521 Sunnybrook Dialysis Hospital Dialysis Hospital

HCP

Network Entity ID Corporate Name HCO Type Custom Key
52192 Sunnybrook Hospital Hospital SAP:HCO: 9521
- Sunnybrook Dialysis Hospital Dialysis Hospital SAP:HCO: 9521

Result: One of the records will be accepted, subsequent records will rejected and an error will occur. In this example, Sunnybrook Hospital is loaded into Network and Sunnybrook Dialysis Hospital is rejected.

Example 4

A data load includes an HCP record that has different addresses, but the same custom key for the addresses.

Source file

Source Source HCP ID Source Address ID First Name Last Name Specialty Address Line 1 City Country
SAP 9521 90281 Sam Smith Cardiology 421 Long St California United States
SAP 9521 90281 Sam Smith Cardiology 97 Spin Lane California United States

HCP

Network Entity ID First Name Last Name Specialty Custom Key
- Sam Smith Cardiology SAP:HCP: 9521

Address

Network Entity ID Entity ID Address Line 1 Country Custom Key
- 51292 421 Long St United States SAP:Address: 90281
- 51292 97 Spin Lane United States SAP:Address: 90281

Result: The HCP record, Sam Smith, is rejected. Network does not load multiple addresses with the same key on the same entity.

Example 5

A data load includes multiple records with the same information and the same custom keys. Source deduplicationClosed The process of merging duplicate records. is enabled in Network

Source file

Source Source HCP ID Source Address ID First Name Last Name Specialty Address Line 1 City Country
SAP 9521 90281 Sam Smith Cardiology 421 Long St San Jose United States
SAP 9521 90281 Sam Smith Cardiology 421 Long St San Jose United States
SAP 9521 90281 Sam Smith Cardiology 421 Long St San Jose United States
SAP 9521 90281 Sam Smith Cardiology 421 Long St San Jose United States

HCP

Network Entity ID First Name Last Name Specialty Custom Key
52192 Sam Smith Cardiology SAP:HCP: 9521

Address

Network Entity ID Entity ID Address Line 1 Country Custom Key
14076 51292 421 Long St United States SAP:Address: 90281

Result: Network deduplicates the records and creates one entity with one address.

Example 6

A data load includes multiple records with the same information and the same custom keys. Source deduplication is not enabled in Network.

Source file

Source Source HCP ID Source Address ID First Name Last Name Specialty Address Line 1 City Country
SAP 9521 90281 Sam Smith Cardiology 421 Long St San Jose United States
SAP 9521 90281 Sam Smith Cardiology 421 Long St San Jose United States
SAP 9521 90281 Sam Smith Cardiology 421 Long St San Jose United States
SAP 9521 90281 Sam Smith Cardiology 421 Long St San Jose United States

HCP

Network Entity ID First Name Last Name Specialty Custom Key
52192 Sam Smith Cardiology SAP:HCP: 9521
- Sam Smith Cardiology SAP:HCP: 9521
- Sam Smith Cardiology SAP:HCP: 9521
- Sam Smith Cardiology SAP:HCP: 9521

Address

Network Entity ID Entity ID Address Line 1 City Country Custom Key
14076 51292 421 Long St San Jose United States SAP:Address: 90281
- 51292 421 Long St San Jose United States SAP:Address: 90281
- 51292 421 Long St San Jose United States SAP:Address: 90281
- 51292 421 Long St San Jose United States SAP:Address: 90281

Result: Network accepts the first HCP record that is loaded and rejects the other HCP records and addresses because of they have duplicate custom keys.

Example 7

A data load includes updated information (middle name) for an existing HCP in Network. The records have duplicate custom keys.

Existing HCP

Network Entity ID First Name Middle Name Last Name Specialty Custom Key
42149 Sam Melvin Smith Cardiology SAP:Address: 9521

Source file

Source Source HCP ID First Name Middle Name Last Name Specialty
SAP 9521 Sam Evan Smith Cardiology
SAP - Sam Evan Smith Cardiology
SAP - Sam Evan Smith Cardiology
SAP - Sam Evan Smith Cardiology

HCP

Network Entity ID First Name Middle Name Last Name Specialty Custom Key
42149 Sam Evan Smith Cardiology SAP:Address: 9521

Result: Network accepts the record and updates the entity with the new middle name.

Example 8

A data load includes updated sub-object information (address) for an existing HCP in Network. The records have duplicate custom keys.

Existing HCP

Network Entity ID First Name Last Name Specialty Custom Key
42149 Sam Smith Cardiology SAP:HCP: 9521

Existing address

Network Entity ID Entity ID Address Line 1 City Country Custom Key
14076 51292 421 Long St San Jose United States SAP:Address: 90281

Source file

Source Source HCP ID Source Address ID First Name Last Name Specialty Address Line 1 City Country
SAP 9521 90281 Sam Smith Cardiology 421 Peter St San Jose United States
SAP 9521 90281 Sam Smith Cardiology 421 Peter St San Jose United States
SAP 9521 90281 Sam Smith Cardiology 421 Peter St San Jose United States
SAP 9521 90281 Sam Smith Cardiology 421 Peter St San Jose United States

Updated HCP

Network Entity ID First Name Last Name Specialty Custom Key
42149 Sam Smith Cardiology SAP:HCP: 9521

Updated address

Network Entity ID Entity ID Address Line 1 City Country Custom Key
14076 51292 421 Peter St San Jose United States SAP:Address: 90281

Result: Network accepts the updates. There will be one HCP record with one address.