Exporting data

AD
DM

Network enables you to manage target subscriptions and their corresponding reference dataClosed The data model and related functionality in Veeva OpenData and customer data that stores the values that appear in dropdown fields (such as specialty, country, or gender) throughout Network. mappings to extract and export data to downstream systems.

Exporting records by VID

Using target subscriptions, you can export records to downstream systems using a .csv file of VIDs. The records in the .csv file are added to the export along with the records that meet the conditions defined in the subscription.

Example

Exporting by VID enables you to quickly reprocess a subset of records.

When Network records are loaded to downstream systems, errors can occur because of data length, the format of the data, or other reasons. After resolving the issue in the downstream system, Administrators can pull the list of failed records by VID, add them to a single column .csv file, and export them instead of processing a full subscription.

Supported objects

All Veeva standard object types and custom object types are supported.

File considerations

To use a file for exporting VIDs, the file must follow these requirements:

  • It must be a .csv file.
  • It must contain only a single column.
  • The VIDs can include single (') or double (") quotation marks.
  • It cannot have more than 10 million rows.
  • Comments are accepted if they are preceded by a hash sign (#).
  • A header row can be included as long as it is marked with a comment.

Additional records included in the export

When the target subscription runs, the records that are listed by VIDin the .csv file are exported. Network automatically includes the following records:

  • Merge winners - If the file contains a record that is the losing record of a merge, the winning record of the merge is also exported so that your downstream system contains the latest data.
  • Related entities - If the record has relationships to other object, for example; parent HCO affiliations, one level of parent HCO is exported. If you want related entities to be filtered based on the filter conditions that you define, enable the Apply "Export Options" to the target records related entities setting in the target subscription.

When a subscription is manually run, records that do not meet the data visibility profile of the country are not exported. For example, if the .csv file includes VIDs for China and the target subscription applies to the United States, any records for China are not uploaded.

Add the .csv file to a target subscription

Administrators and data managers can export a list of VIDs to downstream systems by adding the .csv file to an existing target subscription, or a new target subscription. A target subscription can be linked to one VID file.

To add the file to a target subscription:

  1. In the Admin console, click System Interfaces > Target Subscriptions.
  2. On the Target Subscriptions page, complete one of the following actions:
    1. To add the file to an existing subscription, (for example, your Veeva CRM subscription) click the name of a subscription.
    2. To add the file to a new subscription, click Add Subscription
      1. Type a Name for the subscription.
      2. Ensure that the Type is Data.
      3. Select the System for this subscription.

  3. Click Export by VID.

  4. In the Export by VID dialog, click Upload File. From your local computer, select the file to export.

    Network automatically checks the file to ensure that it contains a maximum of 10 million rows. If the file does not upload, remove the extra rows and upload it again.

  5. When the file successfully uploads, review the summary information in the message box.

    The message box provides the following information:

    • Rows read - The number of rows read in the .csv file. Optional - click the .csv file name to download the file to your local computer.

      Only the first 100,000 rows will be scanned against the database. All rows are scanned to ensure that the VIDs are 18-digit numbers.

    • Rows failed - Records that did not meet the file requirements; for example, records that do not have an 18-character VID, or records that do not exist in your Network instance. Click Error Log to download a file called invalid_vids_<date>_<time> to your local computer. The error log contains the reason why the record failed.

    • Records that will be exported/added in the next job - The records that will be exported to the downstream system. Optional - click Remove File to remove the .csv file from the target subscription. You can remove the file and correct any errors with the listed VIDs before uploading the file again. If errors are not corrected, you can still Save and run the target subscription.

  6. If this is an existing subscription, click Save. If this is a new subscription, you can click Save or optionally complete the remaining steps.

  7. In the Schedule section, decide if you want to schedule the subscription or start the job manually.

    If you select Scheduled, set the schedule and click Done.

    If you run a job manually, the data visibility filters of the country are applied to records during the export. For example, if the .csv file includes VIDs for China and the target subscription applies to the United States, any records for China are not exported.

    If the target subscription is scheduled, all records are exported regardless of the data visibility filters.

  8. In the General Export Options section, the following options apply to Export by VID files:

    • Records State - Export a specific record state only. For example, if Valid & Under Review is selected, only the records (including sub-objects) that have a valid or under review state are exported.
    • Record Type -Export only a specific record type. If Candidate is selected, only candidate records in the list of VIDs are exported.
    • Include Source Data in Export Views - Select if you want to export data lineage information for the selected data sources.
    • Unmapped Reference Codes - Choose the action that Network should take when an unmapped reference code is found.

  9. In the Hierarchy section, the following options apply to Export by VID files:

    • Level of Hierarchy to Export - Set the level to 1.
    • Apply "Export Options" to the target records related entities - If this option is selected, Network will filter the Parent HCO records in the export.
  10. In the Reference Data section, select Include Reference Data Files if you want to include that information in the export.
  11. In the Custom Keys Export Options section, you can choose Select Records to filter the records custom keys using the conditions you define for custom key source, type, and value.
  12. Save your changes.

Running a subscription

A target subscription is configured to be run manually or on a schedule. If it is scheduled, the next time the job runs it exports the .csv file of VIDs along with the regularly exported records that meet your conditions.

Note: The file is exported only one time and then it is automatically removed from the subscription; you do not have to remove the file after the subscription runs.

To run the subscription manually:

  1. On the Target Subscription page, click the name of the subscription.

    On the Details page, a message displays with information about the uploaded .csv file of VIDs. If there are any errors in the file, you can download the error log to review the issues. The .csv file can also be downloaded or removed.

  2. To run the subscription, click Start Job.
  3. When the export is complete, to view details about the job, click the job ID link in the Job History section at the bottom of the subscription.
  4. On the Job Details page, review the following information:
    • Export by VID - The file that was used to export the records.
    • Job Result Summary - The type and count of records that were exported.

Considerations for Veeva CRM

Exporting a file containing VIDs is supported for CRM subscriptions. Use the Network Bridge to export the file.

To export a file:

  • Single country bridge - Add the file to the existing CRM target subscription. You can then manually run the Network Bridge, or wait until the next scheduled job runs.

    After the Network Bridge runs and the records in the file are exported, the file is automatically removed from the CRM subscription.

  • Multi-country bridge - Add a file to a child bridge (country) within in the multi-country Bridge configuration.

    On the Details page, click Upload VID File to add the .csv for the country. When the file is added, a confirmation message displays. Click Details to review the VID details.

    After the Network Bridge runs and the records in the file are exported, the file is automatically removed from the multi-country Bridge configuration.

Cloning subscriptions

Administrators and data managers can clone target subscriptions. If a .csv file has been uploaded, the file is not cloned.

Considerations for managing configurations

Administrators can download export configurations from a Network instance to review or to import to a different Network instance. If a target subscription includes a .csv file of VIDs, the file is not included in the exported configuration.

Filtering exported data

Using the options in the target subscription, administrators can filter the data that they want to export.

Each export contains the following data:

  • All data defined by the selected filters
  • One level of parent

    For example, if you filter the subscription to export only records for Dialysis Centers, one level of parent for the HCO records is exported. If a Dialysis Center HCO has a parent HCO that is a General Practice HCO, the General Practice HCO parent will be included in the data export.

    However, if you select the Apply "Export Options" to the target records related entities option in the Hierarchy section, related entities are filtered according to the group conditions that you have defined.

  • All merge winners

    Merge winners are always included in exports. For example, if you filtered the data to export only HCP prescribers, but a prescriber has been merged with dentist, the dentist will be exported as well.

Including source data

You can export data lineage information for selected data sources. The exported source file; for example, hcp_source.csv, contains the values that were last loaded to Network for that particular source; it does not reflect current values, which could have been changed by another source, a data change request, or the API. Compare the Network record to the data lineage information in the file to identify any potential issues and review source survivorship.

In the target subscription General Export Options section, select Include Source Data view in export files. By default, all data sources are selected for the export.

Note: When this option is selected, all of the entities and sub-objects listed in the File & Field Selection section are enabled for export. The Export All Fields option will be selected by default even if the previous choice was set to Do Not Export. This is to ensure that the linked data is available.

To exclude data sources from the export, click the drop-down list and clear the checkbox from the data sources you want to exclude.

To refine the list of data sources for easier selection or deselection, begin typing in the search field at the top of the list.

The list updates automatically to display matching data sources.

Note: When this functionality is enabled, target aliases are not included in the export file. The target aliases have already been translated to the Network code at this point, so you can compare any of the data in the source files using the Network codes.

Exported files

When you export source data, additional files are exported along with the standard six export files: hcp_source.csv, hco_source.csv, address_source.csv, parenthco_source.csv, license_source.csv, and customkey_source.csv. Remember that the exported source file contains the values that were last loaded to Network for that particular source; it does not reflect current values, so it might not match the values in the standard target subscription export files.

In the entity source files (hcp_source.csv and hco_source.csv), two additional fields are added to the export files:

  • source – the source system name
  • primary_ref_vid – the entity ID of the Network record

In sub-object files (address_source.csv, parenthco_source.csv, license_source.csv, and customkey_source.csv), three additional fields are added:

  • primary_ref_vid – the entity ID of the Network sub-object record
  • entity_source_vid – a system-generated value based on data lineage
  • entity_source_type – the type of entity it is linked to

The exported files use the typical Network layout so they will include empty fields; specifically, Network fields (for example, Created_date) that do not have values for source data.