Target subscriptions
Limiting exported records
Target subscriptions can be set to limit the number of records that are exported. Some downstream systems cannot handle large data volumes, so limiting the records in each job enables you to avoid pushing too many records to these systems.
This enhancement is available by default in your Network instance. Administrators can enable it on target subscription configurations.
Supported exports
A record limit can be applied to delta exports only.
If the setting is applied to a full export, it is ignored unless the target subscription is used for the Network Bridge.
The limit applies to scheduled jobs, jobs that are run through the Network API, and to exports to Veeva CRM.
Network Bridge jobs
If a limit is applied to a Network Bridge job, the target subscription uses the delta flag defined by the bridge. After the specified records are exported to Veeva CRM, the delta flag from the target subscription is saved to the bridge.
If the Network Bridge is set to run a full export and a limit is defined on the target subscription, only the limited records will be exported. To resume full exports, the limit must be removed from the target subscription configuration.
Exported records
When a limit is applied to a delta export, the records are exported in batch jobs. The records that have the oldest update time (ascending order of the delta flag) are exported first.
The following records are also included:
-
related records to ensure the data is complete (for example, merge winners, parent HCOs, sub-object records, custom keys)
-
any records identified in the Export by VID section
This means that the record count will typically be greater than the defined limit.
Example
Your Network instance contains four million records and you specified a limit of one million records.
When the subscription runs, the following records will be exported:
-
the first one million records that meet the subscription filter criteria and that have the oldest update time
-
any additional related records (merge winners, parent HCOs, and so on)
-
any records identified in the Export by VID section
Tip: Account for approximately 20% additional records to be exported due to related records and merge winners.
The next time the subscription job runs, the delta of the next one million records are exported, and so on.
Avoiding backlogs
Use the highest number supported for your downstream system to avoid any backlogs. For example, if you regularly export one million records daily, setting the limit to ten thousand means that there will always be a backlog, and timely updates will not flow to your downstream system.
Tip: After the scheduled target subscription runs, you can click Start Job to manually start the next batch job.
Set an export limit
To define a limit:
-
Open a target subscription (System Interfaces > Target Subscriptions).
-
In the General Export Options section, select Apply Record Limit.
The option is dimmed if the subscription is not set to Delta export.
-
Type the number of records this subscriptions should export, for example, 100,000.
-
Minimum value: 10,000 (10 thousand)
-
Maximum value: 100,000,000 (100 million)
-
-
Save your changes.
When the job runs, only the specified number of records will be exported.
Job details
The Job Details page is updated to display any limit that is set for the subscription. The Record Export Limit displays in the Overview section. It does not display if a limit was not set.
Export by VID
Administrators and Data Managers can export a maximum of 10 million records using the Export by VID feature. Previously, the limit was 100,000 records.
This enhancement is enabled by default in all new and existing instances.
Exporting records
Records can be exported by adding the Veeva IDs (VIDs) to a .csv file and uploading the file to a target subscription configuration.
If more than 100,000 VIDs are included in the file, a message displays to advise the only the first 100,000 rows will be scanned against the database. All rows are scanned to ensure that the VIDs are 18-digit numbers.
If any scanned VIDs were not valid, you can download the error log for more details.
For details about this feature, see Exporting data in the Veeva Network Online Help.