Rejecting records with duplicate custom keys
DM
Custom keys are cross-references between an identifier from an external system and a Network ID. All custom keys should be unique throughout your Network instance. In a model map, a custom key is defined by source, item, and value identifiers and has an active status; for example: SAP:SAP ID:242. Records that contain duplicate custom keys cannot be updated by users, so any duplicates should be removed.
When this feature is enabled, during a source subscription load, records that contain duplicate custom keys are rejected. This helps to preserve data quality and to ensure that records can be updated.
During data load, one record is accepted, but any other records with the same custom key are rejected. This applies to duplicate custom keys on entities and sub- objects. The entire record is rejected if a sub-object has a duplicate custom key. Review the examples below to understand how Network handles different situations with duplicate custom keys.
Note: This feature does not apply to duplicate custom keys that have previously been loaded to your Network instance. To clean up any existing duplicate custom keys, contact Veeva Support for a list of the duplicates and recommended action.
Enabling this feature
By default, this feature is enabled for Network instances created after version 10.0. For these instances, the following options do not appear in the Network UI unless visibility is enabled by Network Support.
To enable this feature for instances created for Network v10.0 or earlier:
- In the Admin console, click Settings > General Settings.
- On the General Settings page, click Edit.
- Select the Reject Duplicate Keys from Data Load option.
- Save your changes.
Error logs
Errors occur for records that were not loaded because of duplicate custom keys.
To view the records that were not loaded, on the Job Details page, view the Job Error Log section. Each error is cataloged here so that you can review the error and update the affected custom keys so that they are not longer duplicates.
Updating existing records that have duplicate custom keys
Records in your Network instance that have a duplicate custom key cannot be updated by matching on the custom key using source subscriptions. Previously, Network would update the first record it sees if the same custom key is associated to multiple records. Now, during data load, if Network detects that there are updates for a key that is associated to multiple records, then no update will be loaded and an error occurs. This change in behavior ensures that data quality is preserved; records are not updated unless Network can determine exactly which record, containing the custom key, the update is intended for.
Examples for duplicate custom keys
Review the following examples to understand how Network handles duplicate custom keys during data load.
Example 1
A data load includes a record that has duplicate custom keys for the HCP and an address.
Source file
| Source | Source HCP ID | Source Address ID | First Name | Last Name | Specialty | Address Line 1 | City | Country |
|---|---|---|---|---|---|---|---|---|
| SAP | 2421 | 2421 | Sam | Smith | Cardiology | 502 Port St | New York | United States |
HCP
| Network Entity ID | First Name | Last Name | Specialty | Custom Key |
|---|---|---|---|---|
| - | Sam | Smith | Cardiology | SAP:SAP ID:2421 |
Address
| Network Entity ID | Entity ID | Address Line 1 | City | Country | Custom Key |
|---|---|---|---|---|---|
| - | 51292 | 502 Port St | New York | United States | SAP:SAP ID:2421 |
Custom Key
| Network Entity ID |
entity_vid__v |
entity_type |
Source | Item | Value |
|---|---|---|---|---|---|
| - | 52192 | HCP | SAP | SAP ID | 2421 |
| - | 21482 | Address | SAP | SAP ID | 2421 |
Result: The entire record will be rejected during the source load because the HCP and address have the same custom key. This typically occurs from an error in the model map.
Tip: To create unique custom keys, use the entity type as the Item value to prevent duplicates. In this example, if the entity type is used, there would be two unique custom keys: SAP:HCP:2421 and SAP:Address:2421.
Example 2
A data load includes a two HCPs that have the same address (duplicate custom keys).
Source file
| Source | Source HCP ID | Source Address ID | First Name | Last Name | Specialty | Address Line 1 | City | Country |
|---|---|---|---|---|---|---|---|---|
| SAP | 9521 | 90281 | Sam | Smith | Cardiology | 421 Long St | California | United States |
| SAP | 4981 | 90281 | Tony | Matthews | Oncology | 421 Long St | California | United States |
HCP
| Network Entity ID | First Name | Last Name | Specialty | Custom Key |
|---|---|---|---|---|
| - | Sam | Smith | Cardiology | SAP:HCP: 9521 |
| 75211 | Tony | Matthews | Oncology | SAP:HCP: 4981 |
Address
| Network Entity ID | Entity ID | Address Line 1 | City | Country | Custom Key |
|---|---|---|---|---|---|
| 14076 | 51292 | 421 Long St | San Jose | United States | SAP:Address: 90281 |
| 51659 | 75211 | 421 Long St | San Jose | United States | SAP:Address: 90281 |
Result: One of the records will be accepted, subsequent records will not be imported and an error will occur. In this example, Tony Matthews is loaded into Network and Sam Smith is rejected.
Tip: To create unique custom keys, append the Value from the HCP's custom key to the front of the Value for the address's custom key. In this example, the two unique custom keys would be SAP:ADDRESS:9521-90281 and SAP:ADDRESS:4981-90281.
Example 3
A data load includes a two different entities that have the same custom key.
Source file
| Source | Source HCO ID | Corporate Name | HCO Type |
|---|---|---|---|
| SAP | 9521 | Sunnybrook Hospital | Hospital |
| SAP | 9521 | Sunnybrook Dialysis Hospital | Dialysis Hospital |
HCP
| Network Entity ID | Corporate Name | HCO Type | Custom Key |
|---|---|---|---|
| 52192 | Sunnybrook Hospital | Hospital | SAP:HCO: 9521 |
| - | Sunnybrook Dialysis Hospital | Dialysis Hospital | SAP:HCO: 9521 |
Result: One of the records will be accepted, subsequent records will rejected and an error will occur. In this example, Sunnybrook Hospital is loaded into Network and Sunnybrook Dialysis Hospital is rejected.
Example 4
A data load includes an HCP record that has different addresses, but the same custom key for the addresses.
Source file
| Source | Source HCP ID | Source Address ID | First Name | Last Name | Specialty | Address Line 1 | City | Country |
|---|---|---|---|---|---|---|---|---|
| SAP | 9521 | 90281 | Sam | Smith | Cardiology | 421 Long St | California | United States |
| SAP | 9521 | 90281 | Sam | Smith | Cardiology | 97 Spin Lane | California | United States |
HCP
| Network Entity ID | First Name | Last Name | Specialty | Custom Key |
|---|---|---|---|---|
| - | Sam | Smith | Cardiology | SAP:HCP: 9521 |
Address
| Network Entity ID | Entity ID | Address Line 1 | Country | Custom Key |
|---|---|---|---|---|
| - | 51292 | 421 Long St | United States | SAP:Address: 90281 |
| - | 51292 | 97 Spin Lane | United States | SAP:Address: 90281 |
Result: The HCP record, Sam Smith, is rejected. Network does not load multiple addresses with the same key on the same entity.
Example 5
A data load includes multiple records with the same information and the same custom keys. Source deduplication
The process of merging duplicate records. is enabled in Network
Source file
| Source | Source HCP ID | Source Address ID | First Name | Last Name | Specialty | Address Line 1 | City | Country |
|---|---|---|---|---|---|---|---|---|
| SAP | 9521 | 90281 | Sam | Smith | Cardiology | 421 Long St | San Jose | United States |
| SAP | 9521 | 90281 | Sam | Smith | Cardiology | 421 Long St | San Jose | United States |
| SAP | 9521 | 90281 | Sam | Smith | Cardiology | 421 Long St | San Jose | United States |
| SAP | 9521 | 90281 | Sam | Smith | Cardiology | 421 Long St | San Jose | United States |
HCP
| Network Entity ID | First Name | Last Name | Specialty | Custom Key |
|---|---|---|---|---|
| 52192 | Sam | Smith | Cardiology | SAP:HCP: 9521 |
Address
| Network Entity ID | Entity ID | Address Line 1 | Country | Custom Key |
|---|---|---|---|---|
| 14076 | 51292 | 421 Long St | United States | SAP:Address: 90281 |
Result: Network deduplicates the records and creates one entity with one address.
Example 6
A data load includes multiple records with the same information and the same custom keys. Source deduplication is not enabled in Network.
Source file
| Source | Source HCP ID | Source Address ID | First Name | Last Name | Specialty | Address Line 1 | City | Country |
|---|---|---|---|---|---|---|---|---|
| SAP | 9521 | 90281 | Sam | Smith | Cardiology | 421 Long St | San Jose | United States |
| SAP | 9521 | 90281 | Sam | Smith | Cardiology | 421 Long St | San Jose | United States |
| SAP | 9521 | 90281 | Sam | Smith | Cardiology | 421 Long St | San Jose | United States |
| SAP | 9521 | 90281 | Sam | Smith | Cardiology | 421 Long St | San Jose | United States |
HCP
| Network Entity ID | First Name | Last Name | Specialty | Custom Key |
|---|---|---|---|---|
| 52192 | Sam | Smith | Cardiology | SAP:HCP: 9521 |
| - | Sam | Smith | Cardiology | SAP:HCP: 9521 |
| - | Sam | Smith | Cardiology | SAP:HCP: 9521 |
| - | Sam | Smith | Cardiology | SAP:HCP: 9521 |
Address
| Network Entity ID | Entity ID | Address Line 1 | City | Country | Custom Key |
|---|---|---|---|---|---|
| 14076 | 51292 | 421 Long St | San Jose | United States | SAP:Address: 90281 |
| - | 51292 | 421 Long St | San Jose | United States | SAP:Address: 90281 |
| - | 51292 | 421 Long St | San Jose | United States | SAP:Address: 90281 |
| - | 51292 | 421 Long St | San Jose | United States | SAP:Address: 90281 |
Result: Network accepts the first HCP record that is loaded and rejects the other HCP records and addresses because of they have duplicate custom keys.
Example 7
A data load includes updated information (middle name) for an existing HCP in Network. The records have duplicate custom keys.
Existing HCP
| Network Entity ID | First Name | Middle Name | Last Name | Specialty | Custom Key |
|---|---|---|---|---|---|
| 42149 | Sam | Melvin | Smith | Cardiology | SAP:Address: 9521 |
Source file
| Source | Source HCP ID | First Name | Middle Name | Last Name | Specialty |
|---|---|---|---|---|---|
| SAP | 9521 | Sam | Evan | Smith | Cardiology |
| SAP | - | Sam | Evan | Smith | Cardiology |
| SAP | - | Sam | Evan | Smith | Cardiology |
| SAP | - | Sam | Evan | Smith | Cardiology |
HCP
| Network Entity ID | First Name | Middle Name | Last Name | Specialty | Custom Key |
|---|---|---|---|---|---|
| 42149 | Sam | Evan | Smith | Cardiology | SAP:Address: 9521 |
Result: Network accepts the record and updates the entity with the new middle name.
Example 8
A data load includes updated sub-object information (address) for an existing HCP in Network. The records have duplicate custom keys.
Existing HCP
| Network Entity ID | First Name | Last Name | Specialty | Custom Key |
|---|---|---|---|---|
| 42149 | Sam | Smith | Cardiology | SAP:HCP: 9521 |
Existing address
| Network Entity ID | Entity ID | Address Line 1 | City | Country | Custom Key |
|---|---|---|---|---|---|
| 14076 | 51292 | 421 Long St | San Jose | United States | SAP:Address: 90281 |
Source file
| Source | Source HCP ID | Source Address ID | First Name | Last Name | Specialty | Address Line 1 | City | Country |
|---|---|---|---|---|---|---|---|---|
| SAP | 9521 | 90281 | Sam | Smith | Cardiology | 421 Peter St | San Jose | United States |
| SAP | 9521 | 90281 | Sam | Smith | Cardiology | 421 Peter St | San Jose | United States |
| SAP | 9521 | 90281 | Sam | Smith | Cardiology | 421 Peter St | San Jose | United States |
| SAP | 9521 | 90281 | Sam | Smith | Cardiology | 421 Peter St | San Jose | United States |
Updated HCP
| Network Entity ID | First Name | Last Name | Specialty | Custom Key |
|---|---|---|---|---|
| 42149 | Sam | Smith | Cardiology | SAP:HCP: 9521 |
Updated address
| Network Entity ID | Entity ID | Address Line 1 | City | Country | Custom Key |
|---|---|---|---|---|---|
| 14076 | 51292 | 421 Peter St | San Jose | United States | SAP:Address: 90281 |
Result: Network accepts the updates. There will be one HCP record with one address.