Add a target subscription

Use target subscriptions to export data from Network to downstream systems (such as SAP® or Veeva CRM) and to export data change requests for third party systems.

You can configure the specific data to be extracted and the schedule for the data export Writing data from a Network instance to a file. This can include HCP and HCO information as well as address, licenses and affiliations.. Target subscriptions can be used across multiple regions.

Export options

• Export records - Export records for the defined system that meet the conditions in the target subscription configuration.

• Export by VID - Include specific records by adding the Veeva IDs to a .csv file.

  The records will be exported along with any records that meet the conditions that you define in the subscription. See Exporting records by Veeva ID.

• Export DCRs - Supported for third party systems only. Export DCRs for a specific third party data provider. See Exporting DCRs for third party systems.

Prerequisite

If the source system uses code values that are different from Network's values, add reference aliases to map the codes. For more information, see Reference aliases.

Add a subscription

To add a target subscription to export data:

• In the Admin console, select System Interfaces > Target Subscriptions. Click Add Subscription.

Define basic subscription details

The Details section of the target subscription defines the type of data for export, either data or DCR, and basic information for the subscription.

Provide the following information:

Name: This value is mandatory and must be short. Special characters and spaces cannot be used in the name; use underscores in place of spaces.

Code: This does not display until the target subscription is created. The code is automatically generated from the name of the target subscription and is used for exporting Network configurations. The code does not change if the subscription name changes. For more information, see View exported configuration records.

Type: Select the type of data that will be exported.

• Data - Export data to downstream systems.
• DCR - Export change requests to third party data providers. For more information, see Exporting DCRs for third party systems.

System: Select the system to link to this subscription. All target subscriptions are linked to a system.

For DCR export jobs, only third party data sources are available in the list.

Description: Define a meaningful description for the target subscription.

Status: New subscriptions are Enabled by default.

Subscriptions can be inactivated when they are no longer used. For more information, see Inactive subscriptions.

Note: The configuration details after this point applies to Data exports.

Configure export options

In the General Export Options section, you can define the extent of the data export or DCR export and specify whether reference data is part of the export.

Targeted record options

In this section, define the options that are applicable to the type of export subscription.

Full Data Extract - Export the full data set or only the changes since the last run of the subscription.

Record State - Choose one of the following:

• All - Export records with any record state.

• Valid & Under Review - Export records only with Valid or Under Review record state.

  Note: Custom keys that are Inactive or Source Deactivated will not be exported when the Valid & Under Review option is selected.

Record Type - When candidate records are enabled select one of the following options:

• All - Export all records (including candidate records).
• Candidate - Export only candidate records.
• Non-candidate - Exclude candidate records. For more information, see Candidate records.

Unmask Customer Opt-out records - Specify if customer opted-out records should be masked when they are exported.

Network automatically masks name-related fields (field values are replaced with "Data Privacy") when records are exported; however, the record information might need to be reviewed in downstream systems even if the HCP has opted-out.

This option displays only if the data_privacy_opt_out__c field is enabled in the Network data model. It applies to full and delta exports for locally managed, Veeva OpenData, and third party records in the customer instance.

When the data_privacy_opt_out__c field is enabled, you can choose to opt out any type of record in any country to address your own privacy requirements. This is completely independent of opted-out records controlled by a Veeva OpenData country using the data_privacy_opt_out__v field. Records opted-out by Veeva OpenData are not visible in your Network instance. For more information, see Data privacy opt-out.

Apply record limit - Set a limit to the number of records that are exported if your downstream system cannot handle large data volumes.

Limits can be applied to delta exports only. If the setting is applied to a full export, it is ignored unless the target subscription is used for the Network Bridge.

The limit applies to scheduled jobs, jobs that are run through the Network API, and to exports to Veeva CRM.

Export Only Updated Sub-Objects - Choose this option if you only want to export sub-objects that have been updated.

Save Delta State - Applies to Full data set extracts only. Choose this option if you want the export to save the last delta ID A number assigned to a record that is incremented every time that a record is updated in any way. that is exported as the delta state.

Note: Records are not included in delta target exports if revisions involve changes to metadata only. For example, a record in which the HCP's first name is changed from Robert to Bob and then back to Robert again (effectively not changing the record) would result in changes to the record’s delta ID and modified date. This record would not be included in the export. This functionality compares the previously exported version of the record with the current exported version and only applies the second time a delta target subscription is run.

Include Source Data view in export files - Choose if you want to export data lineage information for the selected data source.

The exported file contains the values that were last loaded to Network for that source; it does not reflect the current values in Network, which could have been changed by another source, a DCR, or the API. Choosing this option also enables all of the entities and sub-objects in the Field & Field Selection section for export. For more information, see Including source data.

Unmapped Reference codes - Choose the action that Network should take when an unmapped reference code is found. When reference codes are added, they often require an alias to be created to map the Network code to your customer code.

The following options are available:

• Skip record - Do not export the record.
• Keep original code - Export the record with the original code.
• Replace with custom value - Specify a custom value in the Replacement Value field.

Multiple value fields delimiter - Define the delimiter for multivalued fields that are sent to downstream systems.

Change the delimiter if the downstream system expects a different delimiter.

The following delimiters are supported:

• , (comma) (default)

• : (colon)

• ; (semi-colon)

• tab

• | (pipe)

Hierarchy

The Hierarchy section indicates that one level of parent will be exported to ensure that relationships are intact. One parent is exported, even if it does not meet the criteria you set in the Export Options section for that object. For example, if you want to export only HCPs that are prescribers and HCOs that are Dialysis Centers, if a prescriber has an HCO parent that is a Hospital, the Hospital will be included in the export.

If you do not want to export the related entity to avoid unwanted records in your downstream system, select Apply "Export Options" to the target records related entities.

Note that filtering out parent HCOs from your subscription might cause stale data in your downstream system. For example, if you previously exported a parent HCO that is a Dialysis Center and then that parent HCO is changed to a Hospital, the record in your downstream system will not be updated to a Hospital and the parent/child relationship will not be updated if Apply "Export Options" to the target records related entities is now selected.

To prevent stale data in your downstream stream, use one of the following options:

• Set a flag (for example, Send to CRM) so that the parent HCO records with a Send to CRM flag are exported.
• In the Export Options, create a Source System filter (where the source is your downstream system) so that any records tagged with a custom key are exported.

Reference data

Choose whether you want to include reference data files.

Reference File Version - The reference data file version to use for this subscription.

When the layout of the file changes, a new version is introduced. All existing subscriptions retain the version selected when the subscription was created. New subscriptions use the latest available version by default, but this can be changed.

When Network adds new languages, they are added as new columns at the end of the reference file. On their own, they do not trigger a new version of this file.

For more details about the information and layout of the exported reference data files, see Reference data layout.

File format

In this section, define the exported file format.

Format - The format of the export file. This is set to CSV.

Encoding - The encoding for the export. This is set to UTF-8.

You can choose a delimiter, text qualifier, and whether to export a header row in the exported .csv files. These options help exported files to comply with a downstream system's file acceptance format. They are only available when the subscription Type is Data.

Important: Do not change these options for Veeva CRM target subscriptions. Veeva CRM expects target subscriptions to be comma delimited and to have a header row.

Delimiter- Select the character used to separate values in the exported .csv file. By default, the exported files are comma (,) delimited.

The following delimiters can be used in exported files:

• tab
• comma (,)
• colon (:)
• semi-colon (;)
• pipe (|)

Text Qualifier - Select the character to identify the beginning and end of text. On new subscriptions, double quotes (") are the default text qualifier.

The following text qualifiers can be used in exported files:

• double quotes (")
• single quotes (')

Include header row? - Choose whether to export a header row. Clear the checkbox to exclude the header row from the exported .csv file. The option is only applied to data model objects (HCP, HCO, address, license, Parent HCO, and custom keys). On new and existing subscriptions, the option is enabled by default.

Export File Format - Expand the list and select one of the following options:

• Compressed single file (default) - One compressed file is created and it contains a .csv file for each object.

  Important: Use this format for target subscriptions used by the CRM Bridge.

• Compressed individual files - A compressed file is created for each object.
• Uncompressed - An uncompressed .csv file is created for each object.

Compression Format - Choose one of the supported compression formats for the type of file you chose:

• Compressed single file - Zip (zip), Tar Gzip (tgz), or Tar Gzip (tar.gz)
• Compressed individual files - Zip (zip) or Gzip (gz)

If you chose to export an Uncompressed file, these options do not display.

Export File/ Folder Name - Choose one of the following options:

• Default - The file exported with the following naming convention: exp_########.

• Include Name and Timestamp - The file is exported with the following naming convention: <subscription name>_YYMMDDTHHMMSSSSS; for example, CRM_Target_170113T191503397.

  The UNIX timestamp uses the UTC timezone. It cannot be changed.

• Static Name - Define a name so the file will always be exported with the same name. The file and folder will be overridden each time the job runs.

If you choose to export Compressed individual files using the Default FTP path, the compressed file is placed in the outbound/test/testexport folder.

• Individual_Zip = Zip file with Static Name.
• testexport_200901T125637944 = Zip file with Include Name and Timestamp.

Export locations

In this section, choose where to export the data.

Network FTP Path - Choose the Default path or identify a Custom path.

The default path is outbound/<system_name>.

Administrators can give users specific access to FTP folders, so identifying a custom path for this subscription enables you to protect the data. For more information about identifying FTP folder access for users, see Add a user profile. Custom paths are not validated, so ensure that they are correct so your job does not fail.

For information about accessing the files in your Network file system, see File Explorer.

Export to Cloud Storage - Export the data directly to your private cloud storage. Exporting to Amazon S3 buckets are supported.

The Amazon S3 bucket must be created with write access so you can export the files. For more information, see Exporting data to cloud storage.

External Credential Location - If you chose Export to Cloud Storage, this setting displays. Select the Amazon S3 credential from the list or click the link to create the credential. Click Test Connection to ensure that the credentials are valid.

Note: The files are always exported to your FTP folder so Network FTP Path remains selected if you also choose Export to Cloud Storage.

Choose files and fields for export

You can select the level of export for each Veeva or custom object, sub-object, and relationship object, or custom key. These options apply to data exports only.

• Select Which Objects and Fields to Export - Selected by default in new target subscriptions.

  

  You can choose the following options beside each object:

  • Export All Fields - All fields are included in the export. This is selected by default for all objects.

    If you selected the Include Source Data view in export files option in the General Export Options section, choose Export All Fieldsto ensure that all of the source data is included. For more information, see Including source data.

    Note: Fields are regularly added to the data model which will impact the number of columns in the exported files. To learn about new fields, see the Veeva Network Data Governance document.

  • Export Some Fields - Choose specific fields to export.

    Click the Select Fields to Export link that displays below the entity to open the dialog and select the fields.

    • Available Fields - Fields that can be exported, but are currently not included.

    • Current Fields - Fields that are included in the export.

      Use the navigation buttons between the panes to move selected fields.

      In Current Fields pane, use the navigation buttons to move the selected fields up or down within the pane, or to the top or bottom of the pane.

    

    Note: If you are exporting a delta, only records that include selected fields that have been changed will be exported. Changes to fields that are not selected here will be filtered out of the delta export.

  • Do Not Export - No fields will be exported.

    If you inactivate data model fields in your Network instance, it will impact the fields that are exported in your target subscription.

  Considerations for inactive data model fields

  • If Export All Fields is selected, only active fields are exported each time that job runs.
  • If Export Some Fields is selected and a newly inactivated field is one of the fields selected for export, the field still displays in the Current Fields pane as a selected field; however, the export does not contain that field in any way - there is neither a column for the deactivated field nor any data, if it exists.
    • Inactive fields do not display in the Available Fields pane; they cannot be selected for export.
    • If you remove an inactive field from the Current Fields pane, it does not display in the Available Fields pane; it cannot be selected for export.

• Export All Objects and Fields - All fields are exported for all objects. The lists beside each object are inactive. Only active fields are exported each time that job runs.

• Export None - The object will not be exported from the Network main database; an export file will not be created. Choose this option if you are extracting data from the reporting database only. The lists beside each object are inactive.

  

  If you choose Export None, the Targeted Record Options and Hierarchy sections in General Export Options are immediately removed from the configuration because they no longer apply.

Filter target subscription data

For objects where you selected either Export All Fields or Export Some fields, you can specify the records to export using filters. Groups of filters are available so that you can define specific records based on entity and custom keys. These options apply to data exports only.

If you selected Export All Objects and Fields in the File & Field Selection section, the export options for all objects are set to All Records, by default. If you had previously defined export options for an object, those options are preserved. For example, if you have set the Health Care Organization Export Options to export records only when the primary country is the United States, that will not be changed.

Note: These filters do not apply to parents of entities and merge winners. As indicated in the Hierarchy section, one level of parent is exported, regardless of the applied filters, unless the Apply "Export Options" to the target records related entities is selected. Merge winners are also included in the export to ensure that the data in your downstream systems is complete. For more information, see Data exported.

Entity export options

You can export all records, or select records.

If you choose Select Records, you can specify the records using groups of filters.

Note: If you do not define filters, no records will be exported.

To start defining filters, click + Add Condition.

Field - Select the field to filter on. The following fields are available:

• Source System - The system that contributed the data through a source subscription or through the Network API. This can be used multiple times in the same group so that you can export the records for more than one system. For example, you might want to export records from Veeva CRM and a system that provides sales data.

  Source System does not mean that the first part of the custom key contains text with that value; for example, if you specify SAP as the source system, it does not extract all applicable custom keys that begin with SAP (SAP:HCP:4778349). Use the Custom Key Source filter for this requirement.

  The Source System filter will only pull records from the source that have active or inactive custom keys; it will not pull records that have custom keys that are Source Deactivated. Custom keys that are marked as source deactivated are no longer being updated by the source.

  

• All Specialties - Use to define the specific specialties for HCPs or HCOs to export. The set of specialty fields are queried for the value that you define.
• Primary Country - Use to export records based on the primary country of the data.
• Type - Use to specify the type of HCP or HCO records to export. For example, Prescriber (HCP), or Organization (HCO).
• Externally Mastered - Use to export records that are flagged with the is_externally_mastered__v field. Typically, this flag is used for contract organization data; data that is either explicitly added or excluded in a target export depending on the downstream system.
• Defined Query - Use to filter based on values in, or not in, a particular field. This field can be used multiple times in the same group.

• Custom Key Source - Records that match the source defined in the custom key are exported. For example, SAP is the source in this custom key: SAP:HCP:4778349.

  Only active and inactive custom keys are supported; custom keys that are Source Deactivated will not be exported.

  This is helpful if you use a single source subscription to load multiple custom key sources, or if you load data using the Network API. For more details, see Filtering exports by custom key source.

Note: For custom objects, the Field can be filtered by Primary Country, Source System, and Defined Query only.

Condition - choose the appropriate condition for your filter.

Value - Select or type a value and press Enter to select the value. Auto-complete information appears suggesting possible values. Depending on the field and condition that you chose, multiple values might be supported. To remove a value, click the x in the value label.

When the filter is complete you can add another condition, click Add Condition again, or click Add New Group to create another group of conditions. Multiple groups of conditions are separated by an OR condition.

Example

An administrator wants to export data to their Veeva CRM org that match the following criteria:

Primary Country = United States and (Source System = CRM (CRM custom key exists) or Send to CRM flag = Yes (Defined Query = send_to_crm_flag__c:in:Y))

Custom key export options

In this section, define the records that you want to export. For example, you can use this to export only the custom keys from your customer system. If you do not define conditions, no records will be exported.

To start defining filters, click + Add Condition.

Field - Select the field to filter on. The following fields are available:

• Source System - The system that provides the custom key data. The available values are based on the systems defined in your Network instance. Source System does not mean that the first part of the custom key contains text with that value; for example, specifying SAP as the source system will not extract all applicable custom keys that begin with SAP.
• Custom Key Sources - The custom key source that the system provides. Users can include anything in this field.
• Custom Key Status - The status of the custom key; for example, Active.
• Custom Key Item Types - The item type defined in the model map.

Condition - Select the appropriate condition from the list.

Value - Select or type a Value. Multiple values are supported for the Source System and Custom Key Sources fields.

When the filter is complete, you can add another condition or create another group. Note that the available fields can only be used once in each group.

When the subscription runs, the data that you chose will be exported in a .csv file. For more information about the data that is exported, see Exporting data.

Using defined queries

You can use defined queries to filter entities for a target subscription. Defined queries enable you to filter based on values in, or not in, a particular field, under an object. Note that these filters do not apply to sub-objects.

The syntax for specifying a defined query is either field__v:in:value or field__v:not_in:value. For example, to export objects that are active, inactive, or undetermined (not closed), you would apply the filter HCO_status__v:not_in:C in the Filter Health Care Organizations section of the target subscription.

You can specify a list of values by separating them with commas. For example, verify_record__c:in:V,K,L.

Transformation rules

Data can be transformed before it is exported to downstream systems or pushed to Veeva CRM. Create transformation rules using Network Expressions for a specific system, object, and field.

Any rules that have been applied to this target subscription display.

• To open the rule configuration, click the link in the Transformation Rule column.

• To see the NEX rule, click View Rule. This is the current rule (from the Transformation Rules configuration page).

  

For more information, see Transformation rules.

Transformation queries

This section identifies any transformation queries that are applied to the target subscription from a transformation query configuration page (System Interfaces > Transformation Queries) . The queries will transform the data within Network before exporting it in the target subscription.

For more details, see Transforming outbound data.

Create a subscription schedule

Job Schedule

You can run subscriptions manually or on a scheduled basis. If you select Manual, the subscription only runs when you click the Start Job button on the subscription page.

The Schedule section defines the schedule settings for the subscription, based on the timezone in your user profile settings. For more information, see Job schedules.

You can schedule the subscription for the following intervals:

• Hour: Runs the subscription every hour. Specify the number of minutes past the hour.

• Day: Runs the subscription every day. Specify the time of day (hour and minutes).

• Week: Runs the subscription every week. Specify the days of the week and time of day (hours and minutes).

• Month: Runs the subscription monthly. Specify the day of the month and time of day (hours and minutes). Note that if you specify a day value that doesn’t exist for all months (for example, 31), the schedule will not run for those months.

• Year: Runs the subscription every year. Specify the month, the day of the month, and time of day (hours and minutes). As with a monthly schedule, ensure that the day you choose falls within the month you choose.

To define another schedule for the job to run, click Add Schedule.

Tip: To run a job at month end, select Month and 1st as the day.

Considerations for defining multiple schedules

If jobs are scheduled to run close together, the following situations might occur:

• If there are no changes, the job does not run.
• If a job has not finished, and a second job is scheduled to begin, the second job will not run.

Job Triggers

You can choose to trigger other actions to start after a job runs.

Available triggers:

• Send email - Specify users that should be notified for successful and unsuccessful job outcomes.

• Start a job - Start a subsequent job when this job successfully completes.

  Important: Source subscriptions that are run in Test Mode or Simulation Mode cannot trigger another job. Ensure that the source subscription configuration is set to Save Changes to the Database if you want it to trigger another job to start.

For more information, see Subscription job triggers.

Save subscriptions

When you have completed the target subscription configuration, save your changes.

If you have updated an existing target subscription and have changed any of the filters or field selections, a confirmation displays so you can choose whether to export all records or export the delta of records from the last job run.

This option enables you to choose to start the export from where the previous job left off so that you don't have to unnecessarily export all records.

Network considers any changes to filters as a potential reset to the existing subscription; it is assumed that changes mean that the subscription should start over with the new defined fields and filters. Any changes to the following sections will be considered a potential reset of the subscription:

• File & Field Selection
• HCP Export Options
• HCO Export Options
• Custom Keys Export Options

Choose the appropriate export option and click Confirm. The next export job will use the selected option; it cannot be changed before the job runs.

The Confirm Next Export dialog displays any time changes are made to the filters for a target subscription. If the target subscription filters are not changed, the delta of records from the last job that ran will be exported, as usual.

Data Flow View

After you save a target subscription, a new Data Flow View is added to the Details section to visualize all of the steps of the job, including the input and output files.

Click the Data Flow View thumbnail to open the view.

These stages of the job are defined:

• Start - Indicates if the job has been triggered by another job.

• Extra Data From Main Database - An overview of the files and settings that are defined in the target subscription. If the job is configured to extract data only from the reporting database, no files display.

• Apply Transformation Queries - Details about each query that is applied to the subscription. If there are multiple queries, they are listed in the order that they run. Click the query name to open the transformation query. Click View Query to see the query. This step does not display if transformation queries are not applied.

• Generate File - The output files, their format, and exported location. This includes any files exported by the target subscription as well as any query output files.

• Export to Cloud Storage - Links to the Amazon S3 bucket that the files are exported to. This step does not display if files are not exported to cloud storage.

• End - Indicates if this job triggers email notification or another job.

To return to the job configuration, click Back to Subscription Page.

Unsaved changes

If you have made changes to the target subscription configuration, the Data Flow View does not reflect those changes until the subscription is saved. A message displays if you open the view before saving the subscription.

View job results

After a target subscription job has run, details about the records that were exported are available on the Job Details page.

On the subscription page, scroll down to the Job History section and click the link in the ID column to view information about that job.

The Job Details page displays information about each object, sub-object, and relationship object that was exported in the job.

Transformation Rules

This section displays any rules that were applied to the subscription.

• NEX Rule - The code as it existed when the job ran so you can understand the customizations that were made to the exported data if the NEX rule has since changed.

• Transformation Rule - The current name of the rule on the Transformation Rule configuration page. Click the name to open the configuration page.

  If the rule has since been deleted, the original rule name displays with a (Deleted) comment beside the name.

Job Trigger Summary

This section identifies how this subscription job started. It could have been started from a job trigger on another subscription (for example, an OpenData subscription or a source subscription).

The Job Initiation value could be any of the following:

• Manual
• Scheduled
• Job ID - The job ID link displays if this job was triggered by the successful completion of another job.

If subsequent jobs or emails were triggered by this job, the details displays beside those headings.