Job validation rules


Administrators can create and manage data validation rules for source subscription and data updater jobs. Job validation rules will fail the data loading job if critical value changes try to occur on a large number of records.

Examples of critical value changes

  • updating mandatory name fields to NULL

  • deleting, invalidating, and inactivating records

  • opting out records

If a validation rule is set and a defined number of records violate the rule, the job will fail before the data is loaded and these changes take effect.

Note: Job validation rules reject certain changes for existing records; they do not ensure data quality for newly loaded records. Validation for newly added records is not supported.

Job validation rules apply to source subscriptions and Data Updater jobs.

A set of predefined rules are provided for Veeva standard objects. These rules are enabled by default but they do not impact existing data loading jobs until Administrators enable the rule settings for these jobs.

Enable rules

Job validation rules can be applied to source subscriptions and Data Updater update jobs.

The job validation rules setting is enabled on new source subscriptions by default. The setting is not enabled on existing subscriptions to ensure the rules have no impact until Administrators choose to enable it.

Job validation rules are supported for source subscriptions that load data from all source systems, including third party data providers.

To set the option for an existing source subscription:

  1. In the Admin console, click System Interfaces > Source Subscription.

  2. Select an existing subscription.

  3. In the General Settings section, select the following settings:

    • Enable All Job Validation Rules

      To review the rules, click the Job Validation Rules link to navigate to that page.

    • Apply Updates & Merge

       Job validation rules run during the merge stage of the source subscription job, so the rules are only applied if this setting is also selected.

Job validation rules are applied if the source feed contains fields that are part of a rule.

Administrators can enable job validation rules to run on data updater jobs. By default, this option is not enabled so it does not impact data updater jobs.

  1. In the Admin console, click Settings > General Settings.

  2. Select Edit.

  3. In the Application Settings section, select Apply all Enabled Job Validation Rules to Data Updater.

  4. Save your changes.

The job validation rules are applied if fields defined in the rules are part of the data feed.

Job validation rules page

Network provides a set of predefined job validation rules for each Veeva standard object.

To view the rules in the Admin console, click System Interfaces > Job Validation Rules.

The page contains a section for each enabled object in your Network instance. The objects are listed alphabetically by main object and then sub-objects and relationship objects.

The predefined rules are enabled by default but they do not impact data loading jobs until they are enabled to run for a source subscription or they are enabled for data updater jobs.

Creating validation rules

Create job validation rules to prevent critical field values changes.

Rules apply to all countries. If you create a rule using a field that is not available in a country and you run a job for that country, the job validation rule is not applied.

To create a rule:

  1. In the Admin console, click System Interfaces > Job Validation Rules.

  2. On the Job Validation Rules page, all objects that are enabled in your Network instance display. Expand an object and click Add Rule.

    Example

    Create a validation rule to ensure that a large number of HCP types are not changed from Prescriber.

    The rule will check that the hcp_type__v field value does not change from Prescriber to any value that is not Prescriber.

  3. In the New Rule section, define the following settings:
    • Rule Name - Type the name of the rule. The name must be unique.
    • Error Message - Type a message that Administrators and Data Managers will see if the rule triggers the job to fail.
    • Description - Type a description of the rule that displays on the Job Validation Rules page.
    • Threshold - Define the number of records to meet the rule criteria for the job to fail. The number must be between 1 and 20,000.
    • Reject Records Below Threshold - Select this option to reject any number of records below the threshold that meet the rule criteria. If this option is not set, records that meet the rule conditions below the threshold will be updated during the job.

      Tip: Select this option when the change will result in bad data quality; for example, if the update changes the HCP name to Null.

    • Conditions - Identify the field and values as the criteria for this rule. For most rules, this is a comparison between the old (existing value) and the new (incoming) value.
      • Field - Expand the list and choose the field. Only the fields for the object display.
      • Old Value - The existing field value on the record. Expand the list to choose the operator (Equals, Not Equals, Find, and so on) and then type the field value.

        The available operators depend on the selected field.

        Not all operators required a value. For example, the Is Changed, Is Null and Is Not Null operators do not require values.

      • New Value - The incoming value of the field that will update the record. Expand the list to choose the condition and then type the field value.

      Click Add Condition to define another field. Each rule can link multiple conditions using the AND or the OR operator. All conditions must be joined through the same operator; you cannot mix AND and OR within the same rule.

  4. Save the rule.

The rule displays in the object section. It is enabled by default.

Edit rules

Predefined and custom rules can be edited; for example, you can raise or lower the threshold. All of the rule properties can be changed except the object and the Code. The Code is used for exporting configurations to other Network environments.

Copy rules

Copy a validation rule to create a similar rule.

To copy a rule:

  1. Expand a rule and click Copy.

    You can also copy a rule from the row in the list view. Click Options > Copy

  2. In the Copy Rule pop-up, click Yes, Copy.

    The copied rule opens so you can edit it. The Rule Name is appended with _Copy.

  3. When you have finished editing the rule, select Save.

    The rule is enabled by default.

Delete rules

All predefined and custom rules can be deleted.

To delete a rule:

  1. Expand a rule and click Delete.

    You can also delete a rule from the row in the list view. Click Options > Delete.

  2. In the Delete Job Validation Rule pop-up, click Yes, Delete.

    The rule is removed from your Network instance.

Triggered validation rules

Validation rules run on source subscription and data updater jobs if the source feed or file contains the fields defined in the rule conditions.

The job outcome depends on the configuration of any triggered rules.

Failed jobs

A rule is violated and the job fails if the number of records that meet that rule condition is equal or higher than the defined threshold.

Example

If the rule threshold is 100 and the job tries to change a critical value for 100 or more records, the job fails and no records are updated.

Rejected records

If a rule threshold is not met and the job completes, individual records that violate the rule might be rejected depending on the Reject Records Below Threshold rule setting. Each rule contains this setting so you can determine the number of acceptable records to be updated.

Example

A rule threshold is 100 records and 3 records met the rule conditions.

  • Reject Records Below Threshold option is not set - The 3 records are updated by the completed job.

  • Reject Records Below Threshold option is set - The 3 records are rejected by the completed job.

Job errors

Errors are logged when a job fails because of a validation rule and when records are rejected because of a validation rule.

Source subscriptions

The source subscription error log contains entries for the following issues:

  • Job fails because a validation rule threshold was met. The log entry contains the rule name and error message defined in the rule.

  • Job completes but records were rejected because the validation rule was configured to reject records below the rule threshold. The error log contains the rule and the record that the job tried to change so you can investigate the update.

Data updater jobs

    For data updater jobs, if the job completes but a number of records below the threshold were rejected, they will not be updated and will display as Skipped Records on the Job Details page. The Job Error Log will display the job validation rule that failed.

Job validation error log

All records that met any rule conditions are logged in a separate log file so you can investigate the records that caused the job rule violations. This includes records that violated rules without hitting the rule thresholds (records that were updated). The logged records stop near the defined threshold; for example, if the threshold is 100, the log might contain 100 or more records.

The log file is located in the outbound/job_validation_rules folder. All job validation rule logs for source subscriptions and data updater jobs are contained in this folder.

Log file naming convention:

  • Source subscription jobs - <subscription_name>-<timestamp>-job-<job_ID>.zip

  • Data updater jobs - data_updater_update_records__v-<timestamp>-job-<job_ID>.zip

Download the file to review information about the records and rules that were violated.

Example log file

This job failed because a large number of HCPs were inactivated. The log file displays all of the critical changes that the job made based on the rules that were triggered.

This log displays the records that violated the HCPIsInactivated rule; this is the rule that caused the job to fail. It also displays some records that violated the HCPTypeIsChanged validation rule. These records did not trigger the job to fail because it was below the threshold, but they are still logged so you can see that a critical value would be changed.

The log files contain the following columns:

  • VID - The Veeva ID.

  • Type - The object type.

  • Parent VID - Veeva ID of the parent object.

  • Parent Type - Type of parent object.

  • Native Key - If the source file contains a native key, it displays so you can identify which incoming record experienced the error.

  • Rule Name - The job validation rule name.

  • Rule ID - The rule code (automatically defined by Network)

  • Old Value - The existing field value on the record.

  • New Value - The new incoming value from the source feed or file.

Predefined rules

These following validation rules are available in your Network instance. They are enabled by default. You can edit these rules to customize them for your requirements.

Bulk updates

If you are intentionally updating a large number of records, for example, you are inactivating more than 1000 addresses, you will not want the job validation rule to run.

There are three options to perform the bulk update:

  • Increase the threshold of the job validation rule. For example, if there are 1000 addresses to be activated, temporarily increase the rule threshold to 2000. This will impact any jobs started by other users.

  • Temporarily disable the individual job validation rule on the Job Validation Rule page. This will impact all other source subscription and data updater jobs that are running while the rule is disabled.

  • Disable all job validation rules in the source subscription. This means that all other job validation rules that can prevent other critical value changes during the job are disabled as well.

All of these actions are tracked in the System Audit Log.

Logging

The System Audit Log contains any changes that are made to job validation rules.

The log tracks the following changes:

  • Enable or disable the setting to apply job validation rules to Data Updater jobs.

    Search by the GeneralSettings Object Types to filter the log records.

  • Change job validation rule settings for a subscription.

    Search by the Subscription Object Types to filter the log records. Open the log to view the change.

  • Create, edit, or delete job validation rules.

    Search for the JobValidationRule Object Types to filter the log records. Open the record to review the JSON so you can see the details of the change.

Exporting configurations

Administrators can export the job validation rules to a target environment using configuration packages.

All of the job data validation rules and the setting to enable job validation rules for the data updater can be exported.

  • Job validation rules - Export one rule, all rules for an object, or the Job Validation Rules section to move all of the rules.

    If you include job validation rules for custom objects, the custom fields and objects for those rules are automatically added to the export package.

  • Data updater setting - Select the General Settings configuration to export the setting.

For more information about exporting configurations, see the Managing configurations.