Merge prevention and data partitioning

AD
DM

Administrators can create rules to prevent records from merging if they contain specific values for fields. During the merge process, the records are compared to identify any fields or values that are used in the rules. If a rule fails, the records will not be merged.

You can create rules to prevent records from merging for the following scenarios:

  • Non-duplicate records - Use field values to prevent non-duplicate records from being merged. For example, prevent two records from merging if they have different NPI numbers.

  • Data partitioning - Isolate records with specific field values and prevent merges with records that don't share the same value. For example, prevent merges between records that have different reference values for the HCP Type field.

Example rule

This rule will prevent two HCP records from being merged if both records have an NPI number but the numbers are not the same.

This feature is enabled by default in your Network instance.

Supported jobs

Rules for merge prevention and data partitioning are applied to all subscriptions for existing records and incoming data:

  • source subscriptions - data loading and bulk merge jobs

  • data updater merge jobs

  • suspect match tasks (Network UI and Network API)

  • data deduplication jobs

These Network processes will use all of your enabled rules to prevent merges.

Supported objects

Merge prevention and data partitioning occurs at the entity level. The following entities are supported:

  • HCPs

  • HCOs

  • custom main objects

Example scenarios

Review the following merge scenarios that use the following example rules:

  • Rule 1 - NPI number - Do not merge if both records have a value for the NPI field, but the values are different.

  • Rule 2 - HCP Type - Do not allow HCP Type A or B to be merged with any other type but themselves (A > A and B > B).

Note: Only one rule needs to fail to prevent the merge from occurring.

Attempted merge outcomes

Record 1 Record 2 Will Records Merge? Reason
NPI: 1902849797
HCP Type: C 		NPI: <empty>
HCP Type: C 		One of the records does not have a value for NPI field, so the NPI rule is ignored.
HCP type C is not one of the specified values in the rules, so the HCP Type rules are ignored.
NPI: 1902849797
HCP Type: C 		NPI: 1366677486
HCP Type: C 		Both records have a value for NPI but they are different, so the rule fails.
It doesn't matter which record has which value. If they are different, the records will be prevented from merging.
NPI: 1902849797
HCP Type: C 		NPI: 1902849797
HCP Type: C 		Both records have a value for NPI and they are the same, so the rule is ignored.
Both records have a value for HCP Type and neither type is a value specified in the rules, so the HCP type rules are ignored.
NPI: 1902849797
HCP Type: A 		NPI: 1902849797
HCP Type: C 		Both records have the same value for NPI, so that rule is ignored.
Both records have a value for HCP Type but they are different. The rule fails because HCP Type A cannot be merged with HCP Type C.
NPI: 1902849797
HCP Type: A 		NPI: 1902849797
HCP Type: <empty> 		Both records have the same value for NPI, so that rule is ignored.
Only one record has a value for HCP Type. The HCP Type rule passes because type A isn't being merged with another HCP Type.
NPI: 1902849797
HCP Type: A 		NPI: 1902849797
HCP Type: B 		Both records have the same value for NPI, so that rule is ignored.
Both records have a value for HCP Type and they are different, but the records can merge because the rule allows HCP Type A to merge with HCP Type B.

Define rules

You can create rules to prevent merges and to partition data.

To create a rule:

  1. In the Admin console, click Data Model > Merge Prevention and Data Partitioning.

  2. Click Add Rule.

  3. On the New Rule page, type a Name and Description for the rule.

    Names can contain alphanumeric (A-Z, a-z, 0-9), underscore (_), and hyphen (-) characters only. A maximum of 100 characters is permitted.

    Descriptions can contain any characters. A maximum of 100 characters is permitted.

  4. In the Countries list, select the countries that the rule applies to. Multiple countries can be selected.

    All countries that are available from Veeva OpenData display. Countries are not restricted by your data visibility profile.

  5. Rules are enabled by default. To disable a rule, click the Status toggle.

  6. In the Entity list, select the entity. One entity can be applied to each rule.

    Only main entities display in the list. HCPs, HCOs, and main custom objects are supported.

  7. Choose the Field to identify the criteria for the rule.

    The list displays fields that apply to the selected entity and that are enabled in your Network instance. Veeva standard fields and custom fields are supported.

    Excluded fields

    • system fields (for example, vid__v and record_state__v)

    • field sets (addresses__v, custom_keys__v, licenses__v, parent_hcos__v)

    • date and date/time fields

    Reference type fields

    If you select a reference type field, you can choose to define a reference value. Veeva values and custom values are supported.

    In this example, no value is provided for the HCP Type field. When no value is provided, two records cannot be merged unless they both have the same value in the HCP Type field.

    A message displays below the field to explain the merge prevention behavior.

    When one value is defined, the merge is prevented when one of the records has that value and the other record has a value and but it is not the same.

    When multiple values are selected, the rule allows for merges only when both records have a value that is defined in the list.

    In this example, if Record1 has the HCP type value Doctor, it will be allowed to merge with Record2, which has the HCP type Resident. If Record2 has the HCP Type Dentist, the merge is prevented.

  8. Save the rule.

Rules view

When you create rules, they display on the Rules for Merge Prevention and Data Partitioning page.

You can see all of the rules that have been created in your Network instance for every country.

Available actions

  • Search - Type a keyword in the search bar to find rule names, countries, entities, and field names.

  • Sort - To quickly find a rule, you can sort the table by the Name, Entity, Field, or Status columns.

  • Delete - Click the Delete icon to remove a rule from your Network instance.

  • Enable or disable - Click the toggle in the Status column.

Source subscriptions

Merges can be prevented when the source subscription loads data and when it is configured for bulk merge.

Data loading

When a source subscription runs, the job completes but any merges that are not allowed will be prevented. A limit of 500 records can be rejected due to merge prevention rules. When that limit is reached, the job will fail.

Merge prevention rules are not applied to matches found through key match (Veeva IDs and custom keys). They are only applied to matches that are found through fuzzy matchClosed Matching that is not exact; a comparison of similar values (for example Smith/Smyth) depending on match rules set by the customer administrator. when loading data.

On the Job Details page, the Match Summary will identify all matches that were found during the job.

Example

In this job, two matches were identified.

In the Job Result Summary, two rejected matches are identified.

The Job Error Log section displays the count of records that were prevented from being merged. This also displays in the job error log that is exported to your FTP server.

Bulk merge

When source subscriptions are configured for bulk merge, the advanced property "job.merge.allowSourceMerge": "true" is defined in the Module Properties.

When bulk merges are loaded, the job will complete but the records that cannot be merged are skipped.

The Job Details page displays the results of the job.

  • Job Result Summary The Rejected column does not display a count of records that were prevented from merging for bulk merge jobs. This column shows rejected records, but only the merge was rejected in these jobs, not the records.

  • Job Error Log section shows a count of the records that we not merged.

Example error message

If the job error log file was created by the subscription, it also displays the count of VIDs that were prevented from merging.

Tip: If two records should be merged, adjust or remove the value in the identified field and try the merge again.

Data updater

When records fail a merge rule, the rows are skipped in the Data Updater Merge job.

On the File Summary tab, the Job Results section displays a count of the skipped rows.

The Job Error Log section displays the count of records that were prevented from merging.

Tip: If the records are duplicates and should be merged, adjust or remove the value in the identified field and try the merge again.

Suspect match tasks

Data Stewards processing suspect match tasks will receive immediate feedback if the two records cannot be merged based upon the configured rules.

Example

When Data Stewards click Apply to merge two records, a message displays: These records cannot be merged because HCP Type, NPI values differ.

Possible actions when merges are prevented:

  • Click No Match to reject the task.

  • Open the profiles to validate or edit the data. If the records are duplicates, update the values so they are the same on both records or remove the incorrect/less used value. Reload the suspect match task and merge the records.

Note: Merge prevention rules do not apply to suspect match tasks that originate from source deduplication in source subscription jobs. Individual fields are not considered in these types of suspect matches.

Data deduplication jobs

Data deduplication jobs will prevent merges based on all enabled rules in your Network instance.

When the subscription runs, the job completes and you can view the counts on the Job Details page to see what merges were actually completed.

Viewing the merges that were prevented is not available in this release.

Disabled fields and reference values

Merge rules support enabled fields and reference values only.

If a field is used in a rule and then it is disabled, the rule will no longer apply. When you view the rule from the Rules for Merge Prevention and Data Partitioning page, the field displays in the Field column; however, if you open the rule, the field no longer displays in the configuration; a new field must be selected for the rule.

This behavior is the same for disabled reference values.

Managing configurations

You can export the rules in a configuration package and import them to a target environment. For example, you might want to test the rules in Sandbox and then export to them to your Production instance.

The rules are located in the Merge Prevention Rules category in the Export Package configuration.

Custom reference codes

If custom codes on a custom reference type are used in a merge prevention rule, ensure that you also choose the custom codes in the export package. Custom codes are not added automatically as dependencies.

For more information about creating export packages, see Managing configurations.