Unsubscribing from Veeva OpenData records
Administrators can unsubscribe from Veeva OpenData records that are no longer needed. For example, you might want to unsubscribe to records for a particular therapeutic area that you are no longer active in or remove extraneous records that were downloaded from Veeva OpenData in error.
This feature only applies to Veeva OpenData records; it cannot be used to delete records that are managed by third party data sources or to delete locally managed records.
Unsubscribing from a Veeva OpenData record sets the record state to DELETED. The record will remain in your Network instance but will not be visible or easily accessible; for example, it cannot be found using Network Explorer. The record state of related sub-objects, including customer-owned sub-objects, is also changed to DELETED.
Enabling this feature
To enable this feature in your Network instance, contact Veeva Support. When Veeva OpenData approves the request to unsubscribe to records, the feature will be temporarily added to your Network instance so that you can run the Unsubscribe from OpenData Records data maintenance job.
When the approved records have been unsubscribed to, the feature will be turned off in your Network instance and will no longer be accessible from the Data Maintenance page. When the feature is off, previously run jobs can be viewed from the Data Maintenance Subscriptions page but they will be read-only.
Filter unsubscribed records from Search against OpenData
Unsubscribed records can display in search results if the Search against OpenData feature is enabled in your Network instance. To prevent users from unknowingly downloading and resubscribing HCP records again, you can enable the Filter Unsubscribed Records from Search against OpenData feature. Using this feature, you can prevent users from seeing HCP records in the search results based on the reason they were unsubscribed.
Important: To have the ability to filter unsubscribed records, this feature must be enabled before you run the Unsubscribe from OpenData Records job.
For more information, see Filter Unsubscribed Records from Search against OpenData.
Impact to unsubscribed records
Before unsubscribing from Veeva OpenData records, review these details to understand what happens to records and their related objects:
-
Records that are unsubscribed to will have their record state set to DELETED.
- All related customer-owned sub-objects will have their record state set to DELETED.
- All custom field values will be removed. This includes custom fields that are reference list fields.
- All custom keys will have their status changed to inactive.
- Alternate IDs will be removed. (If downstream systems require this data, this should be addressed before unsubscribing to Veeva OpenData records).
- Customer-owned records that have been merged into Veeva OpenData records are not unmerged by an unsubscribe job. These should be reviewed and unmerged, if necessary, before unsubscribing.
- Pending DCRs for unsubscribed records will give an error that it contains a deleted record. Before unsubscribing records, process any pending DCRs. Otherwise, Veeva CRM will resubscribe the records when it polls for the status of all pending DCRs.
Note: Unsubscribing to records does not change the HCP record status (and related sub-object status) to Inactive. To inactivate records, run the Anonymize Deleted HCP Records job after these records are unsubscribed (deleted).
As soon as records have been unsubscribed to in the local Network instance, run all target subscriptions immediately so the updates are pushed to downstream systems. Ensure that the proper record state selection has been made in the subscription. For more information, see the Target subscriptions section below.
Impact on Veeva CRM
For details, see the Network blog article called GDPR Compliance with Veeva Network - Part 3: Managing HCPs that opt out from your organization.
Process for unsubscribing records
Follow this process to unsubscribe from Veeva OpenData records in your Network instance.
Prepare for unsubscribe jobs
Complete the following tasks to prepare for any unsubscribe jobs:
- Confirm the list of HCP and/or HCO VIDs that should be unsubscribed.
- Confirm which downstream systems have these records.
- Run a report to identify if any custom fields have data.
- Run a report to identify if any custom keys exist on the records that will be unsubscribed.
- Manually unmerge records if there is data in custom fields and/or custom keys.
- Identify and process any outstanding DCRs for the affected records.
- Prepare target subscriptions to run (off-cycle) immediately after the unsubscribe job runs.
- Review your Veeva CRM settings for handling deleted records and advise your CRM users: Veeva CRM Record State Handling.
Unsubscribe process
When the preparation work has been completed, follow this process to unsubscribe to Veeva OpenData records:
- Contact Veeva Support to enable the Unsubscribe from OpenData records data maintenance subscription in your Network instance.
- Upon approval from Veeva OpenData, Veeva Support will enable the subscription.
- Create and run the Unsubscribe from OpenData records data maintenance job.
- Run a target subscription to update your downstream systems.
- When the records have been unsubscribed, contact Veeva Support to disable the Unsubscribe from OpenData records subscription in your Network instance.
Unsubscribe from records
When the feature is enabled in your Network instance, follow these steps to unsubscribe to Veeva OpenData records.
-
Identify the records that you want to remove from your Network instance. For example, run a report to extract all of the affected records.
-
Create a .csv file that contains a one-column list of Veeva IDs (VIDs) of the records that you want to unsubscribe to.
The file that is used must meet the following requirements:
- It must be a .csv file.
- It must contain only a single column.
- It can only contain VIDs for HCPs and HCOs; it cannot contain VIDs for sub-objects.
- The VIDs can include single (') or double (") quotation marks.
- It cannot have more than 100,000 rows.
- Comments are accepted if they are preceded by a hash sign (#).
- A header row can be included as long as it is marked with a comment.
Any rows that do not contain a valid VID are ignored.
Important: If the Filter unsubscribed HCPs from Search against OpenData feature is enabled in your Network instance, you will be asked to specify a reason for unsubscribing HCPs in the job configuration. The reason must apply to all of the HCPs that you have in your .csv file. If you are unsubscribing HCPs for other reasons, create a separate .csv file and create a different job for those records.
- In the Admin console, click System Interfaces > Data Maintenance Subscriptions.
- On the Data Maintenance page, click Add Subscription.
-
In the Add Subscription dialog, expand the list and select Unsubscribe from OpenData Records.
The configuration page opens.
- Type a Name and Description for the unsubscribe job.
-
The Reason for Unsubscribing HCPs setting displays if the Filter Unsubscribed HCPs from Search Against OpenData feature is enabled in your Network instance. This setting is mandatory if it displays.
Choose the reason why the HCPs are being unsubscribed.
Important: All of the HCPs in the job must be unsubscribed for the same reason.
- HCP requested data removal - Use for data privacy. For example, the HCP has requested to be removed from your Network instance.
- HCP is no longer targeted - Use for data storage period limitations, or if the HCP is no longer an active target.
- HCP was added unintentionally - Use if the HCP was added by mistake through the OpenData subscription's working set or downloaded by a user.
- HCP was removed for other reasons - Use for any reason other than the ones listed.
-
The Settings section contains the following settings:
- Allow File Reprocessing - This option cannot be edited. This allows a new job to start, even if the file was previously processed.
- Apply All Enabled Job Validation Rules - Apply validation rules to the job to ensure that the job isn't unintentionally unsubscribing (deleting) records. The setting is off by default. For details, see Job validation rules.
- Job Error Log - Select this option if you want to export a log to your FTP server.
- In the Select Master Records section, type the file path and name of the .csv file that contains the list of VIDs.
- Save your changes.
When the subscription is created, it displays on the Data Maintenance page.
To run the unsubscribe job, click the name of the subscription and on the subscription page, click Start Job. When the job runs, the record state for the Veeva IDs listed in the .csv file will be set to DELETED.
Tip: Subscriptions can also be copied using the Clone button. This will duplicate the subscription so you can retain all of the settings, but change the name, description, and file name.
After an unsubscribe job runs, the Job History section displays the number of HCP and HCO records that were unsubscribed to during the job. To view more information, click the job link in the ID column.
Job Details
The Job Details page for an unsubscribe subscription displays all of the entities (HCPs, HCOs) and sub-objects (addresses, licenses, and Parent HCOs) that were deleted during the job.
In the Data Load Summary section, the following details are available:
- Rows Read From File - The total number of rows in the .csv file for both HCPs and HCOs.
- Entities Parsed From File - The number of HCPs and HCOs listed by VID in the .csv file.
The Entities Updated Summary section displays the number for each object and sub-object that was deleted and custom keys that were inactivated.
The Job Error Log section displays any errors or warnings that occurred when the job ran.
Job warnings
The following warnings might occur during a job if the file contains the following types of VIDs:
- A non-valid record (merged_into, invalid, deleted).
- A third-party or customer record.
Warnings do not occur for the following situations:
- The file contains VIDs that are not complete, are not actual VIDs, or are text.
- The VID is not recognized as a VID in the Network instance.
- The VID belongs to a sub-object (address, license, Parent HCO).
- The file contains text.
- The VID was already unsubscribed.
Any warnings or issues with VIDs will not cause the job to fail; the VIDs with issues will be ignored.
Failed job validation rules
When job validation rules are applied to a job and the rule threshold is met, the data maintenance job fails and no changes are made to the data. Open the Job Details page to see an error and the rule name in the Job Error Log.
Logs
When job validation rules are applied to data maintenance subscriptions, log files are created. View the log files in the outbound > job_validation_rules directory in File Explorer. A .zip file is created for each job.
Open the .zip file to review the .csv file for the job.
The .zip file and .csv file have the following naming convention: <subscription_name>-<timestamp>-job-<job ID>.
Example job log
In the file, you can review the changes that the job tried to make and the rule that was used to detect the change.
Update linked records
After running an unsubscribe job, records that are linked to unsubscribed records might be out-of-sync. For example, if you are using Network address inheritance in your Network instance, records that are linked to addresses that have been unsubscribed to need to be updated.
To ensure that remaining records are updated to reflect changes caused by unsubscribing to other records, administrators should run the following data maintenance jobs:
- Network address inheritance refresh (Data Model > Network Address Inheritance)
- Sub-object inactivation (System Interfaces > Data Maintenance Subscriptions > Sub-Object Inactivation)
Report on unsubscribe jobs
After an unsubscribe job runs, you can report on it to see the job statistics. Use the following advanced query (Reports > SQL Query Editor):
select * from job_stats where job_id=<unsubscribe_job_id>
To start the report, click Run.
In the report results, the attributes for the Updated stats (for example, HCO Updated, Address Updated) show the counts of the records and their sub-objects that were updated by the unsubscribe job.
The results for Deleted (for example, HCO Deleted, HCP Deleted) do not show any counts for an unsubscribe job.
Considerations for Network features
Review the following sections to understand how Network features manage unsubscribed records.
Target subscriptions
Veeva OpenData records that are unsubscribed to will be sent to downstream systems one time after their record state changes to DELETED. Downstream systems can determine what happens to the record after this point. If the record is updated in your Network instance (for example, in a source subscription by VID), the update will cause the record to be exported again.
Administrators can configure the Record State setting in target subscriptions to include records that are either Valid & Under Review or All. If a target subscription exports All record states, the deleted records will always be exported.
Veeva OpenData working set
The VIDs of deleted records can remain in your Veeva OpenData working set. After a VID's record state is updated to DELETED, they will not be updated in your Network instance.
If records are mistakenly unsubscribed to, they cannot be added to the working set to resubscribe. To resubscribe to Veeva OpenData records, download the records using Ad Hoc Download or Sync with OpenData actions. The records will be downloaded to your instance again, but any custom fields that you had previously added to the record will not be restored.
Match & Download from Veeva OpenData
During data load, you can automatically match incoming records against records in Veeva OpenData and download the matched records to your Network instance. Unsubscribed records can be downloaded from OpenData again depending on how they are matched in your subscription.
If an incoming file includes Veeva IDs (VIDs) that have been have mapped in the source subscription, Network uses them as a key/ID match.
When using fuzzy match to match against OpenData, unsubscribed OpenData records can be downloaded. When the match is identified by VID, a match will not be found.
Merge
When unsubscribed records are merged in Veeva OpenData, the record is not merged in your Network instance.
- When records are merged in Veeva OpenData, but one of the records has been unsubscribed to in your Network instance, the records are not merged in your instance.
- If a record that has been unsubscribed to in your Network instance becomes a merge loser in Veeva OpenData and the merge winner is not in your instance, it is not merged into the winner.
-
If two records are merged in Veeva OpenData and both records exist in your Network instance, but the winning record has been unsubscribed to, when the losing record is updated in the customer instance through data load or ad hoc download, it will not be merged into the winning record.
The skipped merge will be logged into the Job Details of the job that attempted to perform the merge and the VIDs of both records will display so you can manually merge the records if you want to.
To manually merge the records, resubscribe to the winning record by updating that record specifically and then update the losing record. Both records will then be merged in your Network instance. If the losing record is updated before resubscribing to the winning record, the failed merge behavior will repeat because the winning record is still an unsubscribed record.
-
If a record that has been unsubscribed to in your Network instance becomes a merge winner in Veeva OpenData and the merge loser is pulled into your Network instance through a Veeva OpenData subscription or ad hoc download, the records will not be merged in your Network instance.
- For subscription jobs, the skipped merge will be logged as an error in the Job Details and the Veeva IDs of both records will be provided.
- For ad hoc downloads, a warning displays for the task in the inbox. Click the task name for more information; for example, dfb_import__v.
On the Job Warnings page, click the Job ID link to view the Job Error Log.
In the error log, note the Veeva IDs of the target (winner) and the losing record that should be merged.
If you want to merge the records (recommended so that you have complete data), resubscribe to the merge winner by updating that record specifically. Then update the losing record. Updating the losing record will merge it into the winning record.
If the merge loser is updated before the merge winner is unsubscribed, the error/warning will occur again.
Inbox tasks
If data stewards access pending tasks related to unsubscribed records, an error displays information that the task contains a deleted record. No action can be taken on the DCR.
Before unsubscribing records, any pending DCRs for any of the records must be processed and completed. Otherwise, Veeva CRM will resubscribe the records when it polls for the status of all pending DCRs.
My Requests
Completed tasks that contain a deleted record are listed on the My Requests page. If users access the link to the task, an error displays information that the task contains a deleted record. On a task, links to deleted records display but the record cannot be viewed in the Network user interface.
Profiles
Records that have been deleted cannot be found through Network Explorer. They can be viewed in the Network UI by typing the VID into the URL. This will display the record profile.
If users search for a previously unsubscribed record and the Search against Veeva OpenData feature is enabled in the Network instance, the record displays in the search results. The Download from OpenData icon displays and the record is treated as a new record to be downloaded.
Tip: To prevent users from unknowingly downloading and resubscribing HCP records again, you can enable the Filter Unsubscribed Records from Search against OpenData feature. Using this feature, you can prevent users from seeing HCP records in the search results based on the reason they were unsubscribed.
Network Reporting
When records have been unsubscribed to, the record state of those records is changed to Deleted. When the reporting database updates, users with appropriate access can report on deleted records; for example, run a report to confirm which records were deleted by a job on a specific day.
Ad hoc match
When the Match Against OpenData option is enabled in an ad hoc match job, if you include an OpenData record in your input file and it has been unsubscribed in your Network instance, it will be found as a Local match in the Match Source column in the exported match file. It will not be identified as an OpenData match because the record exists in your Network instance. For more information, see Create an ad hoc match.
Configuration management
Unsubscribe data maintenance jobs can be exported from source environments and imported to target Network environments. However, the Unsubscribe feature has to be enabled in the target Network environment to be able to run the unsubscribe subscription jobs.
Updates to unsubscribed records
Records that have been unsubscribed can receive updates from Veeva OpenData in some situations. If updates are received from the master instance, the record state is updated to the state that Veeva OpenData has and the record will be updated moving forward. This can happen in the following situations:
- The VID is entered in the Ad Hoc Download section on a Veeva OpenData subscription page.
- An add request is submitted to add a record which then merges into an existing Veeva OpenData record that has a record state of Deleted in the customer instance.
- An unsubscribed record is viewed (by typing the VID into the URL) and a user clicks Sync with OpenData on the Data Lineage page.
- Unsubscribed records are included in a Retrieve API call. These records will be re-subscribed. For more information, see the Veeva Network Developer Help.
Download Veeva OpenData records again
Network users can download records that have been unsubscribed to previously.
If an OpenData record is downloaded using any method (for example, Sync with OpenData or Download from OpenData), and that record is already in your Network instance with a record state of Deleted, the record is downloaded and the version of the record is updated. The record state changes to reflect the Veeva OpenData state and all updates are applied. If the OpenData download brings in a parent HCO and the record state of the HCO is Deleted in the Network instance, the record state remains Deleted.
Records that are downloaded again will be sent to downstream systems in applicable target subscriptions, as usual.
Supported limit
A limit of 5000 entities (HCPs or HCOs) can be resubscribed to in one download job. For example, if you are resubscribing to Veeva OpenData records using Ad Hoc Download, 5000 Veeva IDs (VIDs) can be listed in the job.
Sub-objects
Sub-objects are also downloaded with each job to ensure that they are updated properly.
A limit of 5000 sub-objects can be downloaded. If the 5000 record limit is reached, the job continues, but a warning message displays in the administrator's inbox to advise that only 5000 sub-objects were downloaded and the remaining objects will be updated during the next record refresh from Veeva OpenData.