Bulk merge

You can bulk merge existing HCP, HCO, and custom objects in your Network instance using a source subscription by providing the surviving Network custom keys (or the Network Entity ID) in a .csv file.

Merging existing records in bulk is helpful if you have loaded records in your Network instance and they have under matched during the job, leaving you with a number of duplicates. To resolve this, you can update your match rules to prevent future data loads from under matching and run a bulk merge job to merge the existing duplicate records.

Remember that for HCPs and HCOs, Veeva CRM only permits merging 5,000 records at a time; most other systems have similar restrictions.

Specify surviving records

In a .csv file, provide the Network custom key of the surviving record in the merge_survivor_key column; for example:

vid__v	corporate_name__v	merge_survivor_vidkey
462313821042639872	University Health Center	ERP:ACCOUNT:00OF0000006w4WY

Custom Key	corporate_name__v	merge_survivor_vidkey
ERP:ACCOUNT:00OF0000009u0D	University Health Center	ERP:ACCOUNT:00OF0000006w4WY

You can also provide the Network Entity ID of the surviving record in the merge_survivor_vid column, for example:

vid__v	corporate_name__v	merge_survivor_vid
462313821042639872	University Health Center	462313820975531008

 

Supported record states

Records can be merged using bulk merge depending on their record state. In general, VALID and INVALID records are the only records that can be merged into other records, depending on the record state of the target record.

Enable bulk merge

Create a new source subscription specifically for running a bulk merge job.

In the source subscription configuration, accept the default subscription settings, configure a model map, and ensure that the required advanced properties are added.

The bulk merge job will load the specified .csv file with the Network entity IDs and use the advanced properties to merge the records.

Model map

Add a model map to define the objects and identify how the data is provided in the incoming files.

Example model map

In this example, HCPs and HCOs files are being loaded. The model map tells the loader how to extract the columns for each file.

[
  {
    "entity": "HCP",
    "from": "HCP",
    "attributes": [
      "HCP.* AS *"
    ]
  },
  {
    "entity": "HCO",
    "from": "HCO",
    "attributes": [
      "HCO.* AS *"
    ]
  }
]

Advanced properties

Bulk merge is enabled using advanced subscription properties.

Click the Advanced button in the source subscription and set the properties in the Module Properties dialog.

Mandatory properties

The following properties must be set for bulk merge to work:

• "job.merge.allowSourceMerge": "true"

• "job.merge.mergeInstruction": "InclusiveMerge"

  When this property is enabled using InclusiveMerge, all items (keys, sub-objects, attributes) are considered for merging. The survivorship rules for the Network instance are applied to determine which values end up in each field on the winning record. Survivorship rules are defined by administrators using source rankings by system (System Interfaces > Source Rankings) or for individual fields (Data Model > Data Domains).

Additional property

If you want the job to merge Invalid records into Valid records, include this additional property:

• "job.merge.allowInvalidSourceRecordMerge": "true",

Example Advanced Properties

{
    "feed.uri": "file://inbound",
    "job.use.repo.home": "true",
    "job.match.export": "",
    "job.match.validation.rules": "",
    "job.match.networkMasterLinkage": "false",
    "job.merge.unmatchAction": "local",
    "archive.provider": "vaos",
    "map.item.trimtrailing": "0",
    "entity.model.rules": "network:nex, network:normalize_table, network:normalize_alias, network:normalize_cleanup",
    "uic.hidden": "false",
    "feed.dcrProvided": "false",
    "uic.readonly": "false",
    "feed.file.trimleading": "0",
    "job.match.blockAnalysis": "",
    "map.item.trimleading": "0",
    "job.match.skipMerge": "false",
    "job.merge.mergeInstruction": "ExclusiveMerge",
    "feed.rules": "",
    "feed.adhoc": "true",
    "job.applyJobValidationRules": "true",
    "feed.file.include": "bulk_merge",
    "job.match.dedup": "false",
    "feed.stash": "false",
    "root.directory": "/work",
    "feed.exclude": "glob:__MACOSX,.DS_Store",
    "job.pre.final.rules": "",
    "job.merge.allowSourceMerge": "true",
    "istore.path": "${feed.directory}/istore",
    "job.merge.mergeInstruction.ADDRESS": "NoKeysNoDataMerge",
    "job.merge.allowCustomerOwnedChildren": "true",
    "feed.file.trimtrailing": "0",
    "feed.unwrap": "false",
    "job.match.keyMatch": "true",
    "job.match.networkmasterlinkage": "false",
    "uic.disabled": "false",
    "job.merge.childDedup": "",
    "archive.directory": "/archive",
    "job.simulation": "false",
    "job.merge.allowInvalidSourceRecordMerge": "true",
    "archive.path": "${archive.directory}/${jobid}",
    "job.immutable": "false",
    "feed.retain.null": "true",
    "feed.image.include": "undefined",
    "script.Groovy": "/scripts",
    "feed.master": "false",
    "job.log.export": "true",
    "job.match.fuzzyMatch": "false",
    "job.match.analysisAdviceOptions": "ACT,ASK,UNMATCHED",
    "job.merge.restrictPostMergeRules": "false",
    "istore.provider": "vh2os",
    "feed.item.alias.HCP": "hcp*",
    "native.key.HCP": "vid__v",
    "parser.HCP": "csv",
    "parser.csv.quotechar.HCP": "\"",
    "parser.csv.separator.HCP": ",",
    "feed.item.alias.HCO": "hco*",
    "native.key.HCO": "vid__v",
    "parser.HCO": "csv",
    "parser.csv.quotechar.HCO": "\"",
    "parser.csv.separator.HCO": ","
}

Bulk merge behavior

By default, using bulk merge completes the following actions:

• Updates record states
• Moves custom keys

• Moves locally managed sub-objects and attributes.

  If locally managed records are merged into Veeva managed records, use the following properties to ensure that locally managed sub-objects are retained on the Veeva record:

  • "job.merge.allowCustomerOwnedChildren": "true"

  • "job.merge.allowCustomerOwnedChildren.CHILD": "true"

  For more information, see Merge instructions for sub-objects.

Merge from custom key value

In most cases the source file will only include the custom key value. You can use a Network expression A user-defined rule applied to Network data loads which can be used to apply transformations to the data during the load. to construct a full custom key from the value.

Example

Incoming file:

loser_id	corporate_name__v	survivor_id
00OF0000004d9XQ	University Health Center	00OF0000006w4WY

Using a transformation Network expression you can define the merge_survivor_vidkey:

[
  "merge_survivor_vidkey = CONCAT('ERP:ACCOUNT:', survivor_id)"
]

Considerations

A single source subscription can process additions, updates, merges, and invalidations.

You should ensure that no circular merges exist. For example, a merge of A > B > C > D will work properly, but a merge of A > B > C > A results in an error. The latter merge operations will not be processed, but all other updates and merges will be unaffected.