Source subscriptions

AD
DM

Use source subscriptions to load and merge external data into your Network instance. Source subscriptions refer to an external source of data or system you subscribe to. You can schedule subscription activity, determine what data is imported, specify whether records from a source can be unmerged, and specify whether certain data sources should be considered proprietary.

When you create a source subscription, you define properties in the subscription that manage the incoming data to create new records or to merge incoming data with existing records.

Veeva standard objects and custom objects are supported.

Source subscription process

During a source subscription job, Network picks up the raw source data, transforms it into a format consistent with the Network data model, and then runs match/merge evaluations. Preparation for data loadingClosed A record that has not yet been validated by a data steward. consists of a number of steps to ensure that data loading occurs cleanly and as expected.

The process of preparing for and configuring the data load should consist of the following activities:

  • Define a system - Add and configure a system in Network that corresponds to a data source for data loads and change requests, and which includes its own set of reference aliases. Configure parameters for proprietary or restricted data (if applicable) and define survivorshipClosed The process by which the final record is determined as duplicate records are merged, resulting in winning and losing values. trust values.
  • Define reference aliases - Create mappings between external reference codes and Network reference codes. Update mappings through a .csv file and link them to a system.

  • Create a source subscription - Define the data load schedule, incoming FTP folder, external keys, model map, data loading rules, and blockers and classifiers. Enable match testing and match log export for test purposes.

    There are two options for creating a source subscription:

    • Wizard Mode - Simplify the subscription configuration process by using a sample file to define the advanced properties, field normalization, and model map. When these settings have been configured, the classic subscription opens so you can continue defining match rules, Network expression rules, and so on.
    • Classic Mode - Create the entire subscription using the traditional configuration. Manually define all of the settings, field normalization and model map.
  • Test model map - The model map includes target entity specification, data inclusion statements, and entity cross references. Using match testing and match log export, upload a subset of data to the FTP folder, start a job from the subscription, and review the match log for errors (such as missing aliases or records).
  • Review match rules - Matching is based on features, blockers, and confidence, which use defined subsets of fields and blocks to determine confidence levels and match data. Optimize match rules by running jobs on small subsets of data and reviewing output for correctness.
  • Advanced settings - Using the Advanced button, you can set additional properties or change the default values. For example, you can set a property to determine the records that are updated according to record state (any, valid, or non-valid).
  • Load data - Disable match testing and export and run the subscription job. Addresses, licenses, and affiliations are de-duplicated and changes are committed.

Considerations

The following apply for data loaded into a Network instance:

  • Records loaded through a data load do not require mandatory fields.
  • Data added by API add requests or through the Network UI must include values for mandatory fields.
  • Mandatory fields on sub-objects are only required if the sub-object is included for a record. For example, if HCP data is added without address data, values are not required for mandatory fields under the address object for the record.
  • When you unmerge records, if a record was added by data load without an address, the resulting unmerged record will not include an address.

  • Parent HCOs that are included in subscription feeds will be dropped or rejected if the related entity is not Valid or Under_Review. This applies to candidate records and opted-out records also. If any Parent HCO records were not created, a job warning displays.

  • Address ordinals are set based on specific criteria.

  • Locally managed sub-objects are added to Veeva OpenData records during merges if the job.merge.allowCustomerOwnedChildren property value is true. For more information about this property, see Advanced subscription settings.

Note: Mandatory fields vary by country.