Data maintenance

Deleting custom object records

Administrators and data managers can soft-delete custom object records using a data maintenance job. When records are deleted, the associated sub-objects and relationship objects are also deleted or inactivated. The data maintenance job soft-deletes the records, so you can update downstream systems if records were previously exported before you hard delete them in Network. The data maintenance job deletes custom object records by object type.

This feature is enabled by default in your Network instance.

Delete behavior

When you delete custom object records, the following behavior occurs:

• Record state of the main object is set to DELETED.

• Record state of associated sub-objects and relationship objects are set to DELETED.

  If the relationship object is owned by the other main object in the relationship, its record state is set to INVALID.

• All custom keys are inactivated.

• Pending tasks will be rejected and removed from the inbox.

  Pending tasks on Merged_Into records remain in the inbox until Data Stewards click the task.

Deleted objects cannot be reversed.

Supported custom object records

All main custom object records can be deleted. Sub-objects and relationship object records cannot be deleted on their own.

Exceptions

Main objects cannot be deleted if the record state is MERGE_INTO or PARENT_MERGED.

Create a data maintenance job

To create a job to delete custom object records:

1. In the Admin console, click System Interfaces > Data Maintenance Subscriptions.

2. Click Add Subscription.

3. In the Add Subscription dialog, expand the list and choose Delete Custom Object Records.

   The subscription page displays.

4. In the Details section, provide the following:

   • Name - Type a meaningful name for the subscription.

   • Action - Identifies the job that you chose. This cannot be changed.

   • Delete Records for Object - Expand the list and select a custom object. All of the enabled custom objects in your Network instance display.

     The affected sub-objects and relationship objects display below the selected object.

   • Description - Provide a meaningful description.

   • Status - New subscriptions are Enabled by default. They can be disabled when they are no longer needed.

5. In the Settings section, select the job options for the subscription.

   • Allow File Reprocessing - Process the feed again, even if the files in the feed were previously processed. When this option is set, all files in the source folder will be loaded, regardless of whether or not they have previously been loaded. This setting is enabled by default.

   • Job Error Log - Export the error log for the source subscription to your FTP server.

6. In the Record Deletion section, choose the records that you want to delete for the custom object:

   • Delete all records for the selected object - All records for the custom object will be deleted.

     If you choose this option, a table displays the objects that will be deleted by the subscription job. A count of the records that will be impacted and the update to the record_state__v field displays.

     

   • Delete records from file - Specific records that are defined in a .csv file will be deleted.

     When you select this option, create a .csv file that contains a one-column list of Network entity IDs (VIDs) that you want to delete. Network entity IDs and object type are validated during the delete job.

     Store the file on your FTP server so it can be retrieved during the job.

     In the fields, type the FTP path and .csv file name.

     

7. In the Job Trigger Configuration section, you can create a job or email trigger. For example, when the job complete, you could trigger a target subscription to update your downstream systems.
8. Save your data maintenance configuration.
9. Click Start Job to run the subscription.

Job details

When the job completes, the Job History section is updated. Click the ID column to open the Job Details page.

The Job Details displays the counts of all main object, sub-object, and relationship object records that were deleted, invalidated, and inactivated (custom keys).

Change requests

When you delete objects, any pending tasks are automatically rejected and removed from the inbox and the My requests page.

Restricting access to unsubscribed HCPs from Search against OpenData

Administrators can now prevent users from downloading HCPs that have been unsubscribed from OpenData. When records are unsubscribed, they are no longer available in your Network instance. However, if the Search against OpenData feature is enabled, the records display in the search results and include the Download from OpenData icon; users can unknowingly subscribe to HCPs again. You can now specify a reason when you unsubscribe HCPs and then those records can be filtered from OpenData search results based on that reason using data visibility profile permissions.

HCP records can be resubscribed in your local Network instance using search or if an incoming record from an add request matches a Veeva OpenData record. Records can also be resubscribed through other applications that integrate with Network using the API; for example, Veeva CRM (Network Account Search), Network search widget, and Veeva Concur Connector. Using this feature, users can be restricted from unsubscribed HCPs so they cannot be resubscribed through any of these ways.

Enable this feature

This feature is not enabled by default. Administrators must enable the feature for your Network instance.

To enable the feature:

1. In the Admin console, click Settings > General Settings.
2. At the top of the page, click Edit.
3. Select the Enable Option to Filter Unsubscribed HCPs from Search Against OpenData setting.

4. In the pop-up window, confirm that you want to enable the feature.

   Important: The feature cannot be disabled after it is enabled.

5. Save your changes.

Enabling the feature makes the following changes in your Network instance:

• Data model updates - A new data model field, data_removal_reason__v, is enabled. The field identifies the reason why the HCP is unsubscribed.

• Data maintenance job - A mandatory setting is added to the Unsubscribe from Veeva OpenData job configuration to identify the reason the HCPs are being unsubscribed.

  Note: HCP records that have been unsubscribed to before this feature was available cannot be back-filled with a reason. These records cannot be filtered because they do not include the data removal reason.

• Data visibility profiles (DVPs) - DVPs include a new permission to restrict users from searching for and downloading HCP records from OpenData that were previously unsubscribed. The permissions are based on the reason an HCP was unsubscribed.

To learn more about these changes, see the sections below.

Data model updates

New field

When you enable the feature, a new reference type field, data_removal_reason__v, is automatically enabled in your Network instance. This field is a system field that is locally managed; change requests for this field are never sent to Veeva OpenData or third party data provider.

The field can only be updated using the Unsubscribe from OpenData Records data maintenance job. It cannot be disabled after it is enabled.

New reference type

The new reference type, DataRemovalReason, is assigned to the data_removal_reason__v field. The reference codes are predefined and are used in the new data visibility setting and data maintenance job. The reference type is read-only and can only be extended by Veeva. It can be exported, but new codes cannot be added; the Import button is dimmed.

Data visibility profiles setting

A new data visibility profile permission can prevent users from searching and downloading HCPs that have been unsubscribed from Veeva OpenData. The permission enables you to choose any of the predefined reasons for unsubscribing an HCP. HCPs that have been unsubscribed for those reasons will be filtered from search for all of the users assigned to the data visibility profile.

The new permission does not display if this feature is not enabled in the Network instance.

This setting only applies to HCPs that have been unsubscribed from Veeva OpenData for the specified reasons.

Set the permission

To filter access to unsubscribed HCPs for specific user types:

1. In the Admin console, click Users & Permissions > Data Visibility Profiles.
2. Open a data visibility profile.

3. In the Search section, expand the list beside Prevent search/download for HCPs unsubscribed from OpenData Instance.

   • None (default) - The user can search for and download any HCP that has been unsubscribed from OpenData.
   • HCP requested data removal - Use for data privacy. For example, the HCP has requested to be removed from your Network instance.
   • HCP is no longer targeted - Use for data storage period limitations, or if the HCP is no longer an active target.
   • HCP was added unintentionally - Use if the HCP was added by mistake through the OpenData subscription's working set or downloaded by a user.
   • HCP was removed for other reasons - Use for any reason other than the ones listed.

   You can select multiple options; however, if None is selected, that can be the only option.

Example

On a data visibility profile for sales reps, choose the HCP requested data removal reason to restrict those users from searching or downloading HCPs that were unsubscribed for data privacy compliance. This prevents sales reps from accidentally resubscribing the HCP without their consent.

Alternatively, on data visibility profiles for Data Manager users, do not choose this reason so Data Managers have access to search and download these HCPs in case the HCP provides consent to store and use their data again.

Updates to the unsubscribe data maintenance job

The Unsubscribe from OpenData Records job now requires you to select the reason for unsubscribing HCPs. The Reason for Unsubscribing HCPs field value is applied to all of the HCP records in the job. The data visibility profile permission is then able to restrict users from searching and downloading the unsubscribed records that are flagged for that reason.

Important: The reason must apply to all of the HCPs that you have in your .csv file. If you are unsubscribing HCPs for other reasons, create a separate .csv file and create a different job for those records.

HCOs that are included in the file are not impacted; those records will be skipped when the reason is assigned to the record.

The Reason for Unsubscribing HCPs setting displays only if this feature is enabled in your Network instance.

Reusing a configured unsubscribe job

If you reuse a previous unsubscribe job configuration for a new job, the configuration cannot be saved until you select the reason for unsubscribing HCPs.

Add requests for unsubscribed HCPs

The data visibility profile permission also prevents users from adding records that have been unsubscribed from OpenData.

If an add request matches with an HCP in OpenData that was unsubscribed for a specific reason, it will be automatically rejected if it was submitted by a user who does not have permission to search and download unsubscribed HCPs for that reason.

Users can view the resolution note to understand why the add request was rejected.

Resubscribing to HCP records

When users have permission to resubscribe to an HCP record from OpenData, the data_removal_reason__v field value is updated and set to No Value/Null.

Anonymizing unsubscribed records

When unsubscribed HCP records are included in the Anonymize Deleted HCP records jobs, the data_removal_reason__v field value is retained; it is not masked or blanked during the anonymization job.

Reporting on data removal reason

Unsubscribed records can be reported on; for example, a data manager can create an overview of all of the HCPs removed from their Network instance, including the data removal reason. The data removal reason can be accessed from reporting as long as the record has not been anonymized; anonymized records are removed from reporting.

Exporting data removal reasons

You can export a list of all HCPs removed from your Network instance, including the data removal reason for each HCP. Anonymized HCPs are included in target subscriptions.

Network API

This feature also supports the Network API. Unsubscribed HCPs are excluded from the Search API if a user's DVP prevents them from searching for the records. Similarly, the Retrieve API will prevent a user from downloading an unsubscribed record based on the user's DVP.

More information

For detailed information about the data maintenance job, see Unsubscribing from Veeva OpenData records in the Veeva Network Online Help.