Deleting locally managed records

AD
DM

Administrators and data managers can delete locally managed records in Network using a data maintenance job. Deleting records is important for complying to GDPR requirements around data retention periods and the ability to remove personal data for HCPs, if requested. It is also helpful for removing locally managed records from your Network instance that are no longer being updated or are no longer useful; for example, candidate records.

When records are deleted, the record state (record_state__v) for the HCP or HCO and its associated sub-objects is set to Deleted. Users can no longer access the record using Network search.

Note: Deleting locally managed records cannot be reversed.

This feature is enabled by default in your Network instance.

Applicable records

The data maintenance job supports HCP and HCO records that meet the following requirements:

  • Records are locally managed. Records that are managed by third party data providers or Veeva OpenData are not supported.

  • The record state (record_state__v) is Valid, Deleted, or Invalid. The following record states cannot be updated to Deleted:

    • Under_review - The record is pending and must be processed by a data steward.
    • Merged_into - The record was the loser of a merge. These records cannot be deleted during this job, but HCPs with this record state will be anonymized in your Network instance if the merge winner is included in an Anonymizing Records job.

Delete job impact

When the data maintenance job runs, it has the following impact in Network:

  • The record state (record_state__v) updates to Deleted. Associated sub-objects are also deleted.
  • Associated custom keys are set to Inactive (custom_key_status__v).
  • The record state of relationships pointing to a deleted HCO is updated to Invalid.
  • The records can no longer be found using Network search.
  • Pending tasks (add and change requests and suspect matches) are automatically closed after the job runs. For more details, see the section below called "DCR considerations."

Note: The delete action is considered a "soft delete." Deleted records are restricted in the Network user interface, but they are still accessible through some features. To comply with GDPR requirements, personal data on HCP records can be completely removed from Network using the Anonymize Records feature.

Impact on Veeva CRM

For details, see the Network blog article called GDPR Compliance with Veeva Network - Part 3: Managing HCPs that opt out from your organization.

Delete records

To delete locally managed records, administrators and data managers can configure a data maintenance job. The job requires a .csv file containing the Network entity IDs (VIDs) of the applicable records.

To create the data maintenance job:

  1. In the Admin console, click System Interfaces > Data Maintenance Subscriptions.
  2. Click Add Subscription.

  3. In the Add Subscription dialog, select Delete Locally Managed HCP/HCO.

    The New Deletion Job page displays.

  4. In the Details section, type a Name and Description for the job.

    New subscriptions are Enabled by default. For more information, see Inactive subscriptions.

  5. In the Settings section, select any of the following options:

    • Allow File Reprocessing - Select if you want a permit a new job to start even if the file has been previously processed.

    • Apply All Enabled Job Validation Rules - Apply validation rules to the job to ensure that the job isn't unintentionally deleting records. The setting is off by default.

      For data maintenance subscriptions, the applicable default job validation rules are the rules that check for records being deleted. For example, if job validation rules are applied, the HCPIsDeleted rule will fail a Delete Locally Managed HCP/HCO job if the job tries to delete 100 or more HCP records.

      For details, see Job validation rules.

    • Job Error Log - Select if you want an error log to be exported to your FTP server.

  6. In the Select Records section, define the file path and name of the .csv file.

    The .csv file must have one column that contains the list of VIDs for the records that you want to delete.

    Note: A limit of 60,000 VIDs can be included in the .csv file. The job will fail if the limit is exceeded.

  7. In the Job Schedule & Trigger section, define any subsequent actions that will start when this job finishes. For example, trigger a target subscription to export the deleted records to a downstream system.

    Available triggers:

    • Send email - Specify users that should be notified for successful and unsuccessful job outcomes.
    • Start a job - Start a subsequent job when this job successfully completes.

    For more information, see Subscription job triggers.

  8. Save your changes.

Run the delete job

When the data maintenance job is configured, run it to delete the required records.

To start the data maintenance job:

  1. In the data maintenance job that you just configured, click Start Job.

  2. When the job completes, a Job History section displays at the bottom of the page with some summary information.

  3. Click the job ID to open the Job Details page.

  4. On the Job Details page, review the number and type of records that were deleted during the job.

    The Rows Read From File section indicates the number of rows in the .csv file. The Entities Parsed From File section displays how many records for each entity were loaded and used. Unrecognized and incorrect VIDs, or extraneous data, are ignored and not included in the Rows Parsed column.

    The Job Trigger Summary section identifies subsequent jobs or emails were triggered by this job.

    If errors occur during the job, a Job Error Log section displays at the bottom of the page and identifies any records that could not be deleted.

    For the records in the .csv file that are locally managed and have the record state Valid, Deleted, and Invalid, the job updates the record state (record_state__v) to Deleted; any other records are ignored during the job. Associated sub-objects are also set to Deleted and custom keys are set to inactive.

    Errors will occur for the following situations:

    • The record state of the VID is Under_review or Merged_into.
    • The record is not locally managed. Records that are managed by Veeva OpenData or third party data providers cannot be deleted using this feature.
    • Job validation rule thresholds were met and the job failed.

Failed job validation rules

When job validation rules are applied to a job and the rule threshold is met, the data maintenance job fails and no changes are made to the data. Open the Job Details page to see an error and the rule name in the Job Error Log.

Logs

When job validation rules are applied to data maintenance subscriptions, log files are created. View the log files in the outbound > job_validation_rules directory in File Explorer. A .zip file is created for each job.

Open the .zip file to review the .csv file for the job.

The .zip file and .csv file have the following naming convention: <subscription_name>-<timestamp>-job-<job ID>.

Example job log

In the file, you can review the changes that the job tried to make and the rule that was used to detect the change.

Deleted records

The delete action is considered a "soft delete." Deleted records are restricted in the Network user interface, but they are still accessible in the following ways:

  • Users can continue to access the deleted record if they enter the VID into the Network URL.
  • Tasks associated with the records are still available to view in the Inbox and from the My Requests page.
  • Data is not purged from the reporting database, so it continues to be accessible in reports, data lineage, and revision history.
  • The record and all of its data can be exported to downstream systems using target subscriptions and the API.

DCR considerations

When records are deleted, any pending tasks for the records (including queued DCRs) are automatically rejected and closed after the job runs. Resolution notes are automatically added to the rejected DCRs and the tasks remain visible in the inbox and from the My Requests page.

Tasks cannot be closed in the following situations:

Suspect matches

  • If there are more than two VIDs in the suspect match task, and the deleted VID is not the source record, the deleted VID is removed from the task and the task remains open.

Add requests

  • An HCP or HCO is added and the Create Unverified option is enabled in the Network instance. This option automatically assigns a VID to a record as soon as the task is submitted. In this case, the record state will be Under_Review, so the record cannot be deleted. The task must be processed by a data steward.
  • A parent HCO is added and the VID of the parent record is deleted. The task cannot be closed because it is associated with the sub-object (parent HCO). The task must be processed by a data steward. If the add request is approved, then the relationship is invalidated and is not visible in the Network UI.

Additionally, for DCR processing, if you are using Network address inheritance in your Network instance, if data stewards try to link to a parent HCO that has been deleted, a DCR validation error occurs. The link to the parent HCO must be deleted before the DCR can be processed.

Next steps

  • Update your downstream systems - Use target subscriptions or the Network API to export the deleted records.

  • Anonymize records - To comply with the GDPR requirements to remove all personally identifiable data at an HCP's request, administrators can now anonymize any of these deleted HCP records and export them to their downstream systems.

    Note: In Veeva CRM, records in Deleted state cannot be updated so exporting records from Network after anonymizing them will not update the records further. To ensure that personally identifiable information is removed, delete the records directly from Salesforce™.