Job validation rules

Administrators can create and manage data validation rules for source subscription and data updater jobs. Job validation rules will fail the data loading job if critical value changes try to occur on a large number of records.

Examples of critical value changes

• updating mandatory name fields to NULL

• deleting, invalidating, and inactivating records

• opting out records

If a validation rule is set and a defined number of records violate the rule, the job will fail before the data is loaded and these changes take effect.

Note: Job validation rules reject certain changes for existing records; they do not ensure data quality for newly loaded records. Validation for newly added records is not supported.

Supported features

Job validation rules can be applied to the following features:

• Source subscriptions

• Data Updater jobs

• Data maintenance jobs

  The following jobs are supported:

  • Delete Locally Managed HCP/HCO

  • Delete Custom Object Records

  • Unsubscribe OpenData Records

  Tip: Unsubscribe from Third Party Records jobs already support job validation rules because these jobs are run using NEX rules in source subscriptions.

Supported objects

Job validation rules are supported for Veeva standard objects and custom objects.

A set of predefined rules are provided for Veeva standard objects. These rules are enabled by default but they do not impact existing data loading jobs until Administrators enable the rule settings for these jobs.

Enable rules

Job validation rules can be applied to source subscriptions, Data Updater update jobs, and some data maintenance subscriptions.

Click each tab to learn how to apply rules to each feature.

• Source Subscriptions
• Data Updater Jobs
• Data Maintenance Subscriptions

Job validation rules page

Network provides a set of predefined job validation rules for each Veeva standard object.

To view the rules in the Admin console, click System Interfaces > Job Validation Rules.

The page contains a section for each enabled object in your Network instance. The objects are listed alphabetically by main object and then sub-objects and relationship objects.

The predefined rules are enabled by default but they do not impact data loading jobs until they are enabled to run for a source subscription or they are enabled for data updater jobs.

Create rules

Create job validation rules to prevent critical field values changes.

Rules apply to all countries. If you create a rule using a field that is not available in a country and you run a job for that country, the job validation rule is not applied.

To create a rule:

1. In the Admin console, click System Interfaces > Job Validation Rules.

2. On the Job Validation Rules page, all objects that are enabled in your Network instance display. Expand an object and click Add Rule.

   Example

   Create a validation rule to ensure that a large number of HCP types are not changed from Prescriber.

   The rule will check that the hcp_type__v field value does not change from Prescriber to any value that is not Prescriber.

   

3. In the New Rule section, define the following settings:
   • Rule Name - Type the name of the rule. The name must be unique.
   • Error Message - Type a message that Administrators and Data Managers will see if the rule triggers the job to fail.
   • Description - Type a description of the rule that displays on the Job Validation Rules page.
   • Threshold - Define the number of records to meet the rule criteria for the job to fail. The number must be between 1 and 20,000.

   • Reject Records Below Threshold - Select this option to reject any number of records below the threshold that meet the rule criteria. If this option is not set, records that meet the rule conditions below the threshold will be updated during the job.

     Tip: Select this option when the change will result in bad data quality; for example, if the update changes the HCP name to Null.

   • Conditions - Identify the field and values as the criteria for this rule. For most rules, this is a comparison between the old (existing value) and the new (incoming) value.
     • Field - Expand the list and choose the field. Only the fields for the object display.

     • Old Value - The existing field value on the record. Expand the list to choose the operator (Equals, Not Equals, Find, and so on) and then type the field value.

       The available operators depend on the selected field.

       Not all operators required a value. For example, the Is Changed, Is Null and Is Not Null operators do not require values.

     • New Value - The incoming value of the field that will update the record. Expand the list to choose the condition and then type the field value.
     List of supported operators

     The following operators are available, depending on the selected field type:

     • Equals / Not Equals - Values are not case-sensitive.

     • Find / Not Find - Use this operator for substring matching. Think of it as "contains / not contains". For example, if you specify the FIND operator value as "abc" then the rule is violated for all fields that contain the substring "abc" ("abcxyz", "xyzabcxyz", "xyzabc", and so on). Values are not case-sensitive.
     • In / Not In - Used for string and reference fields. It specifically looks for values in fields and it ignores blanks or nulls. Values are not case-sensitive.
     • Is Null / Is Not Null - Use to check when there is no value or when existing values are removed and are not replaced. This is helpful to ensure that records contain critical values (for example, HCP name).
     • Is Changed - Used to determine if a field value is changed. If this operator is used, the New Value is not required.
     • Match Regular Expression / Not Match Regular Expression - Can be used to check if the old or new value match a certain expression. (Network uses Java Regular Expressions). Values are case sensitive.
     • Less Than / Greater Than - Can be used for fields containing numeric values; for example, fields with rankings or dates.

     • Between - Can be used for fields containing numeric values; for example, fields with rankings or dates.

       This operator looks at the values between what's specified, not including what's specified. For example, if you provide the values of 1 and 3, the rule will only fire/fail for a value of 2. If you provide values of 1 and 2, the rule will never fire because there is no value between 1 and 2.

     Click Add Condition to define another field. Each rule can link multiple conditions using the AND or the OR operator. All conditions must be joined through the same operator; you cannot mix AND and OR within the same rule.

4. Save the rule.

The rule displays in the object section. It is enabled by default.

Edit rules

Predefined and custom rules can be edited; for example, you can raise or lower the threshold. All of the rule properties can be changed except the object and the Code. The Code is used for exporting configurations to other Network environments.

Copy rules

Copy a validation rule to create a similar rule on the same object.

To copy a rule:

1. Expand a rule and click Copy.

   You can also copy a rule from the row in the list view. Click Options > Copy

2. In the Copy Rule pop-up, click Yes, Copy.

   The copied rule opens so you can edit it. The Rule Name is appended with _Copy.

   

3. When you have finished editing the rule, select Save.

   The rule is enabled by default.

Delete rules

All predefined and custom rules can be deleted.

To delete a rule:

1. Expand a rule and click Delete.

   You can also delete a rule from the row in the list view. Click Options > Delete.

2. In the Delete Job Validation Rule pop-up, click Yes, Delete.

   The rule is removed from your Network instance.

Triggered validation rules

Validation rules run on source subscription and data updater jobs if the source feed or file contains the fields defined in the rule conditions.

The job outcome depends on the configuration of any triggered rules.

Failed jobs

A rule is violated and the job fails if the number of records that meet that rule condition is equal or higher than the defined threshold.

Example

If the rule threshold is 100 and the job tries to change a critical value for 100 or more records, the job fails and no records are updated.

Rejected records

If a rule threshold is not met and the job completes, individual records that violate the rule might be rejected depending on the Reject Records Below Threshold rule setting. Each rule contains this setting so you can determine the number of acceptable records to be updated.

Example

A rule threshold is 100 records and 3 records met the rule conditions.

• Reject Records Below Threshold option is not set - The 3 records are updated by the completed job.

• Reject Records Below Threshold option is set - The 3 records are rejected by the completed job.

Job errors

Errors are logged when a job fails because of a validation rule and when records are rejected because of a validation rule.

Source subscriptions

The source subscription error log contains entries for the following issues:

• Job fails because a validation rule threshold was met. The log entry contains the rule name and error message defined in the rule.

  

• Job completes but records were rejected because the validation rule was configured to reject records below the rule threshold. The error log contains the rule and the record that the job tried to change so you can investigate the update.

  

Data updater jobs

For data updater jobs, if the job completes but a number of records below the threshold were rejected, they will not be updated and will display as Skipped Records on the Job Details page. The Job Error Log will display the job validation rule that failed.

Data maintenance subscriptions

If a job validation rule threshold is met, the data maintenance job fails and no changes are made to the data. Open the Job Details page to see an error and the rule name in the Job Error Log.

Job validation rule error log

All records that met any rule conditions are logged in a separate log file so you can investigate the records that caused the job rule violations. This includes records that violated rules without hitting the rule thresholds (records that were updated). The logged records stop near the defined threshold; for example, if the threshold is 100, the log might contain 100 or more records.

The log file is located in the outbound/job_validation_rules folder. All job validation rule logs are contained in this folder.

Log file naming convention:

• Source subscription jobs - <subscription_name>-<timestamp>-job-<job_ID>.zip

• Data updater jobs - data_updater_update_records__v-<timestamp>-job-<job_ID>.zip

• Data maintenance subscriptions - <subscription_name>-<timestamp>-job-<job ID>.zip

Download the file to review information about the records and rules that were violated.

Example log file

This job failed because a large number of HCPs were inactivated. The log file displays all of the critical changes that the job made based on the rules that were triggered.

This log displays the records that violated the HCPIsInactivated rule; this is the rule that caused the job to fail. It also displays some records that violated the HCPTypeIsChanged validation rule. These records did not trigger the job to fail because it was below the threshold, but they are still logged so you can see that a critical value would be changed.

The log files contain the following columns:

• VID - The Veeva ID.

• Type - The object type.

• Parent VID - Veeva ID of the parent object.

• Parent Type - Type of parent object.

• Native Key - If the source file contains a native key, it displays so you can identify which incoming record experienced the error.

• Rule Name - The job validation rule name.

• Rule ID - The rule code (automatically defined by Network)

• Old Value - The existing field value on the record.

• New Value - The new incoming value from the source feed or file.

Predefined rules

These following validation rules are available in your Network instance. They are enabled by default. You can edit these rules to customize them for your requirements.

Bulk updates

If you are intentionally updating a large number of records, for example, you are inactivating more than 1000 addresses, you will not want the job validation rule to run.

There are three options to perform the bulk update:

• Increase the threshold of the job validation rule. For example, if there are 1000 addresses to be activated, temporarily increase the rule threshold to 2000. This will impact any jobs started by other users.

• Temporarily disable the individual job validation rule on the Job Validation Rule page. This will impact all other source subscription and data updater jobs that are running while the rule is disabled.

• Disable all job validation rules in the source subscription. This means that all other job validation rules that can prevent other critical value changes during the job are disabled as well.

All of these actions are tracked in the System Audit Log.

Logging

The System Audit Log contains any changes that are made to job validation rules.

The log tracks the following changes:

• Enable or disable the setting to apply job validation rules to Data Updater jobs.

  Search by the GeneralSettings Object Types to filter the log records.

  

• Change job validation rule settings for a subscription.

  Search by the Subscription Object Types to filter the log records. Open the log to view the change.

  

• Create, edit, or delete job validation rules.

  Search for the JobValidationRule Object Types to filter the log records. Open the record to review the JSON so you can see the details of the change.

  

Exporting configurations

Administrators can export the job validation rules to a target environment using configuration packages.

All of the job data validation rules and the setting to enable job validation rules for the data updater can be exported.

• Job validation rules - Export one rule, all rules for an object, or the Job Validation Rules section to move all of the rules.

  If you include job validation rules for custom objects, the custom fields and objects for those rules are automatically added to the export package.

• Data updater setting - Select the General Settings configuration to export the setting.

For more information about exporting configurations, see the Managing configurations.