Network enables you to manage target subscriptions and their corresponding reference dataThe data model and related functionality in Veeva OpenData and customer data that stores the values that appear in dropdown fields (such as specialty, country, or gender) throughout Network. mappings to extract and export data to downstream systems.
Exporting records by VID
Using target subscriptions, you can export records to downstream systems using a .csv file of Network entity IDs (VIDs). This enables you to quickly reprocess a subset of records; for example, if some Network records failed to load to downstream systems. The VIDs of the failed records can be added to a .csv file and exported, instead of processing a full subscription again. The records in the .csv file are added to the export along with records that meet the conditions defined in the subscription.
When Network records are loaded to downstream systems, errors can occur because of data length, the format of the data, or other reasons. After resolving the issue in the downstream system, administrators and data managers can pull the list of failed records by Network entity ID (VID), add them to a single column .csv file and export them again.
All Veeva standard object types and custom object types are supported.
To use a file for exporting VIDs, the file must follow these requirements:
- It must be a .csv file.
- It must contain only a single column.
- The VIDs can include single (') or double (") quotation marks.
- It cannot have more than 100,000 rows.
- Comments are accepted if they are preceded by a hash sign (#).
- A header row can be included as long as it is marked with a comment.
Additional records included in the export
When the target subscription runs, the records that are listed by VID in the .csv file are exported. Network automatically includes the following records:
- Merge winners - If the file contains a record that is the losing record of a merge, the winning record of the merge is also exported so that your downstream system contains the latest data.
- Related entities - If the record has relationships to other object, for example; parent HCO affiliations, one level of parent HCO is exported. If you want related entities to be filtered based on the filter conditions that you define, enable the Apply "Export Options" to the target records related entities setting in the target subscription.
When a subscription is manually run, records that do not meet the data visibility profile of the country are not exported. For example, if the .csv file includes VIDs for China and the target subscription applies to the United States, any records for China are not uploaded.
Add the .csv file to a target subscription
Administrators and data managers can export a list of VIDs to downstream systems by adding the .csv file to an existing target subscription, or a new target subscription. A target subscription can be linked to one VID file.
To add the file to a target subscription:
- In the Admin console, click System Interfaces > Target Subscriptions.
- On the Target Subscriptions page, complete one of the following actions:
- To add the file to an existing subscription, (for example, your Veeva CRM subscription) click the name of a subscription.
- To add the file to a new subscription, click Add Subscription
- Type a Name for the subscription.
- Ensure that the Type is Data.
- Select the System for this subscription.
Click Export by VID.
In the Export by VID dialog, click Upload File. From your local computer, select the file to export.
Network automatically checks the file to ensure that it contains 100,000 rows or fewer and the size is not larger than 10 MB. If the file does not upload, remove the extra rows and upload it again.
- When the file successfully uploads, review the summary information in the message box.
The message box provides the following information:
- Rows read - The number of rows read in the .csv file. Optional - click the .csv file name to download the file to your local computer.
Rows failed - Records that did not meet the file requirements; for example, records that do not have an 18-character VID, or records that do not exist in your Network instance. Click Error Log to download a file called invalid_vids_<date>_<time> to your local computer. The error log contains the reason why the record failed.
- Records that will be exported/added in the next job - The records that will be exported to the downstream system. Optional - click Remove File to remove the .csv file from the target subscription. You can remove the file and correct any errors with the listed VIDs before uploading the file again. If errors are not corrected, you can still Save and run the target subscription.
- If this is an existing subscription, click Save. If this is a new subscription, you can click Save or optionally complete the remaining steps.
In the Schedule section, decide if you want to schedule the subscription or start the job manually.
If you select Scheduled, set the schedule and click Done.
If you run a job manually, the data visibility filters of the country are applied to records during the export. For example, if the .csv file includes VIDs for China and the target subscription applies to the United States, any records for China are not exported.
If the target subscription is scheduled, all records are exported regardless of the data visibility filters.
In the General Export Options section, the following options apply to Export by VID files:
- Records State -Export a specific record state only. For example, if Valid & Under Review is selected, only the records (including sub-objects) that have a valid or under review state are exported.
- Record Type -Export only a specific record type. If Candidate is selected, only candidate records in the list of VIDs are exported.
- Include Source Data in Export Views - Select if you want to export data lineage information for the selected data sources.
- Unmapped Reference Codes - Choose the action that Network should take when an unmapped reference code is found.
In the Hierarchy section, the following options apply to Export by VID files:
- Level of Hierarchy to Export - Set the level to 1.
- Apply "Export Options" to the target records related entities - If this option is selected, Network will filter the Parent HCO records in the export.
- In the Reference Data section, select Include Reference Data Files if you want to include that information in the export.
- In the Custom Keys Export Options section, you can choose Select Records to filter the records custom keys using the conditions you define for custom key source, type, and value.
- Save your changes.
Running a subscription
A target subscription is configured to be run manually or on a schedule. If it is scheduled, the next time the job runs it exports the .csv file of VIDs along with the regularly exported records that meet your conditions. The file is exported only one time and then it is automatically removed from the subscription; you do not have to remove the file after the subscription runs.
To run the subscription manually:
- On the Target Subscription page, click the name of the subscription.
On the Details page, a message displays with information about the uploaded .csv file of VIDs. If there are any errors in the file, you can download the error log to review the issues. The .csv file can also be downloaded or removed.
- To run the subscription, click Start Job.
- When the export is complete, to view details about the job, click the job ID link in the Job History section at the bottom of the subscription.
- On the Job Details page, review the following information:
- Export by VID - The file that was used to export the records.
- Job Result Summary - The type and count of records that were exported.
Considerations for Veeva CRM
Exporting a file containing VIDs is supported for CRM subscriptions. To export a file, log into Network and add the file to the existing CRM target subscription. You can then run the subscription job from CRM, or wait until the next scheduled job runs. After the subscription runs and the records in the file are exported, the file is automatically removed from the CRM subscription; you do not have to log into Network and remove it.
Administrators and data managers can clone target subscriptions. If a .csv file has been uploaded, the file is not cloned.
Considerations for managing configurations
Administrators can download export configurations from a Network instance to review or to import to a different Network instance. If a target subscription includes a .csv file of VIDs, the file is not included in the exported configuration.
Filtering exported data
Using the options in the target subscription, administrators can filter the data that they want to export.
Each export contains the following data:
- All data defined by the selected filters
- One level of parent
For example, if you filter the subscription to export only records for Dialysis Centers, one level of parent for the HCO records is exported. If a Dialysis Center HCO has a parent HCO that is a General Practice HCO, the General Practice HCO parent will be included in the data export.
However, if you select the Apply "Export Options" to the target records related entities option in the Hierarchy section, related entities are filtered according to the group conditions that you have defined.
- All merge winners
Merge winners are always included in exports. For example, if you filtered the data to export only HCP prescribers, but a prescriber has been merged with dentist, the dentist will be exported as well.
Including source data
You can export data lineage information for selected data sources. The exported source file; for example, hcp_source.csv, contains the values that were last loaded to Network for that particular source; it does not reflect current values, which could have been changed by another source, a data change request, or the API. Compare the Network record to the data lineage information in the file to identify any potential issues and review source survivorship.
In the target subscription General Export Options section, select Include Source Data view in export files. By default, all data sources are selected for the export.
Note: When this option is selected, all of the entities and sub-objects listed in the File & Field Selection section are enabled for export. The Export All Fields option will be selected by default even if the previous choice was set to Do Not Export. This is to ensure that the linked data is available.
To exclude data sources from the export, click the drop-down list and clear the checkbox from the data sources you want to exclude.
To refine the list of data sources for easier selection or deselection, begin typing in the search field at the top of the list.
The list updates automatically to display matching data sources.
Note: When this functionality is enabled, target aliases are not included in the export file. The target aliases have already been translated to the Network code at this point, so you can compare any of the data in the source files using the Network codes.
When you export source data, additional files are exported along with the standard six export files: hcp_source.csv, hco_source.csv, address_source.csv, parenthco_source.csv, license_source.csv, and customkey_source.csv. Remember that the exported source file contains the values that were last loaded to Network for that particular source; it does not reflect current values, so it might not match the values in the standard target subscription export files.
In the entity source files (hcp_source.csv and hco_source.csv), two additional fields are added to the export files:
- source – the source system name
- primary_ref – the entity ID of the Network record
In sub-object files (address_source.csv, parenthco_source.csv, license_source.csv, and customkey_source.csv), three additional fields are added:
- primary_ref – the entity ID of the Network sub-object record
- entity_source_vid – a system-generated value based on data lineage
- entity_source_type – the type of entity it is linked to
The exported files use the typical Network layout so they will include empty fields; specifically, Network fields (for example, Created_date) that do not have values for source data.