Managing inbound data
DM
From a source subscription, you can define the way in which entities are provided in incoming data and map fields from incoming sources to attributes in the Network data model. You do this by configuring a model map and normalization rules in the source subscription.
Create a model map
The model map defines the objects and sub-objects and how the data is provided in the incoming files, along with instructions on how to use them. The following samples illustrate model maps for two different sources.
Note: Unless they are anchored from an address file, licenses should be modeled against the HCP or HCO.
Examples
Sample model map: NPI
NPI source is flat source that contains HCP and HCO information in a single file. HCPs can have up to two addresses (business and practice) distributed in columns, including columns for other information in the addresses. The following samples show declarations for each entity, followed by descriptions of each sample.
For the following source, the alias "NPI" is configured in the job properties.
HCP entity:
[ { "entity": "HCP", "from": "NPI WHERE NPI.ENTITY_TYPE_CODE = '1'", "attributes": [ "NPI.*", "NPI.VDM_* AS *" ], "customkeys": [ { "source": "NPI", "item": "HCP", "value": "NPI.NPI" } ] },
In this sample, a possible HCP custom key stored in Network would be NPI:HCP:1750384707
.
Property | Value | Description |
---|---|---|
"entity" | "external_data_model_entity" | This is the Network external data model entity that is being loaded (for example, "HCP" or "HCO"). It extracts them based on the "ENTITY_TYPE_CODE" column from the file. |
"from" | "SQL_statement" | This is similar to a MySQL statement, containing a WHERE clause that extracts entity data based on the column you specify in the statement. |
"attributes" |
"NPI.*", "NPI.VDM_* AS *" |
This is an array for each entity being modeled. The array contains important information that the loader uses to extract the proper columns for each file. |
"customkeys" | "value", "item", or "source" |
This is an array for the required keys, which the loader stores in the "value" represents the column within the flat file that contains the required value. "item" represents the object that the custom key belongs to. If you do not specify an item, the loader uses the "entity" value. "source" represents the source that the custom key belongs to. If you do not specify the source, the loader uses the "system" value assigned to the subscription. |
HCO entity:
{ "entity": "HCO", "from": "NPI WHERE NPI.ENTITY_TYPE_CODE = '2'", "attributes": [ "NPI.*", "NPI.VDM_* AS *" ], "customkeys": [ { "source": "NPI", "item": "HCO", "value": "NPI.NPI" } ] },
In this sample, a possible HCO custom key stored in Network would be NPI:HCO:1750384709
.
Property | Value | Description |
---|---|---|
"entity" | "external_data_model_entity" | This is the Network external data model entity that is being loaded (for example, "HCP" or "HCO"). It extracts the HCOs based on the "ENTITY_TYPE_CODE" column from the file. |
"from" | "SQL_statement" | This is similar to a MySQL statement, containing a WHERE clause that extracts entity data based on the column you specify in the statement. |
"attributes" |
"NPI.*", "NPI.VDM_* AS *" |
This is an array for each entity being modeled. The array contains important information that the loader uses to extract the proper columns for each file. |
"customkeys" | "value", "item", or "source" |
This is an array for the required keys, which the loader stores in the "value" represents the column within the flat file that contains the required value. "item" represents the object that the custom key belongs to. If you do not specify an item, the loader uses the "entity" value. "source" represents the source that the custom key belongs to. If you do not specify the source, the loader uses the "system" value assigned to the subscription. |
Address entity
{ "entity": "ADDRESS", "from": "NPI", "anchors": [ { "anchor": "NPI.PROVIDER_FIRST_LINE_BUSINESS_PRACTICE_LOCATION_ADDRESS", "attributes": [ "NPI.PROVIDER_FIRST_LINE_BUSINESS_PRACTICE_LOCATION_ADDRESS", "NPI.PROVIDER_SECOND_LINE_BUSINESS_PRACTICE_LOCATION_ADDRESS", "NPI.PROVIDER_BUSINESS_PRACTICE_LOCATION_ADDRESS_CITY_NAME", "NPI.PROVIDER_BUSINESS_PRACTICE_LOCATION_ADDRESS_STATE_NAME", "NPI.PROVIDER_BUSINESS_PRACTICE_LOCATION_ADDRESS_POSTAL_CODE", "NPI.PROVIDER_BUSINESS_PRACTICE_LOCATION_ADDRESS_COUNTRY_CODE__IF_OUTSIDE_U_S__" ], "customkeys": [ { "source": "NPI", "item": "ADDRESS", "value": "NPI.NPI" } ] } ], "attributes": [ "NPI.VDM_* AS *", "NPI.VDM_ENTITY_ID AS VDM_FKENTITY_ID", "VDM_ENTITY_ID=@NEW_ENTITY_ID" ] },
In this sample, a possible address custom key stored in Network would be NPI:NPI_ADDRESS:1750384709
.
Property | Value | Description |
---|---|---|
"entity" | "external_data_model_entity" | This is the Network external data model entity that is being loaded. |
"from" | "SQL_statement" |
For this entity declaration, this represents the file (or alias) name, since address information is in the same file and addresses are extracted using anchors. As a result, a |
"anchors" | "first_address_column" |
The anchor represents a column in the file (preferably non-empty) used for creation of the sub-object. |
"attributes" |
"entity_array_column" |
This is an array for each entity being modeled. The array contains important information that the loader uses to extract the proper columns for each file. |
"customkeys" | "value", "item", or "source" |
This is an array for the required keys, which the loader stores in the "value" represents the column within the flat file that contains the required value. "item" represents the object that the custom key belongs to. If you do not specify an item, the loader uses the "entity" value. "source" represents the source that the custom key belongs to. If you do not specify the source, the loader uses the "system" value assigned to the subscription. |
"attributes" |
"NPI.VDM_* AS *", "NPI.VDM_ENTITY_ID AS VDM_FKENTITY_ID", "VDM_ENTITY_ID=@NEW_ENTITY_ID" |
These attributes store the address of the ID of the extracted HCO or HCP as a foreign key, and create a new entity for the extracted address information. These are internal attributes used by the loader and should not be changed. |
License entity
{ "entity": "LICENSE", "from": "NPI", "anchors": [ { "anchor": "NPI.PROVIDER_LICENSE_NUMBER_%s", "index": [ "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15" ], "attributes": [ "LIC_LICENSE_NUMBER=@ANCHOR_VALUE", "NPI.PROVIDER_LICENSE_NUMBER_STATE_CODE_%s AS LIC_LICENSE_NUMBER_STATE_CODE", "LIC_TYPE='STATE'" ] } ], "attributes": [ "NPI.VDM_* AS *", "NPI.VDM_ENTITY_ID AS VDM_FKENTITY_ID", "VDM_ENTITY_ID=@NEW_ENTITY_ID" ] } ]
Property | Value | Description |
---|---|---|
"entity" | "external_data_model_entity" | This is the Network external data model entity that is being loaded. |
"from" | "SQL_statement" |
For this entity declaration, this represents the file (or alias) name, since license information is in the same file and licenses are extracted using anchors. As a result, a |
"anchors" | "anchor_array" |
This represents the relevant columns from which to extract license records from the file used for extraction. |
"anchor" | "first_anchor_column" |
The anchor represents a column in the file (preferably non-empty) used for creation of the sub-object. |
"index" | "index_number" |
This represents multiple anchor declaration for the license. |
"attributes" |
"entity_array_column" |
The attributes array contains all of the license columns required for extraction. The Network external model requires the loader to use variables such as |
"attributes" |
"NPI.VDM_* AS *", "NPI.VDM_ENTITY_ID AS VDM_FKENTITY_ID", "VDM_ENTITY_ID=@NEW_ENTITY_ID" |
These attributes store the license of the ID of the extracted HCO or HCP as a foreign key, and create a new entity for the extracted license information. These are internal attributes used by the loader and should not be changed. |
Veeva CRM source is more normalized than NPI source. It provides a single file for both HCPs and HCOs, and separate files for addresses (including licenses) and relationships.
For the following source, the alias "ACCOUNT" is configured in the job properties.
HCP entity:
[ { "entity": "HCP", "from": "ACCOUNT WHERE ACCOUNT.ISPERSONACCOUNT = 'TRUE'", "attributes": [ "ACCOUNT.*", "ACCOUNT.VDM_* AS *" ], "customkeys": [ { "source": "CRM", "item": "HCP", "value": "ACCOUNT.ID" }, { "source": "CRM", "item": "EXTERNAL" "value": "ACCOUNT.EXTERNAL_ID_VOD__C" } ] },
In this sample, a possible HCP custom key stored in Network would be NPI:HCP:1750384707
.
Property | Value | Description |
---|---|---|
"entity" | "external_data_model_entity" | This is the Network external data model entity that is being loaded (for example, "HCP" or "HCO"). It extracts the HCPs based on the "ENTITY_TYPE_CODE" column from the file. |
"from" | "SQL_statement" | This is similar to a MySQL statement, containing a WHERE clause that extracts entity data based on the column you specify in the statement. ISPERSONACCOUNT is set to TRUE because HCPs are being extracted. |
"attributes" |
"ACCOUNT.*", "ACCOUNT.VDM_* AS *" |
This is an array for each entity being modeled. The array contains important information that the loader uses to extract the proper columns for each file. These are internal attributes that the loader uses to store the attributes in the file and the internal VDM (internal loader) attributes in the intermediate store. These are internal attributes used by the loader and should not be changed. |
"customkeys" | "value", "item", or "source" |
This is an array for the required keys, which the loader stores in the "value" represents the column within the flat file that contains the required value. "item" represents the object that the custom key belongs to. If you do not specify an item, the loader uses the "entity" value. "source" represents the source that the custom key belongs to. If you do not specify the source, the loader uses the "system" value assigned to the subscription. |
HCO entity
{ "entity": "HCO", "from": "ACCOUNT WHERE ACCOUNT.ISPERSONACCOUNT = 'FALSE'", "attributes": [ "ACCOUNT.*", "ACCOUNT.VDM_* AS *" ], "customkeys": [ { "source": "CRM", "item": "HCO", "value": "ACCOUNT.ID" }, { "source": "CRM", "item": "EXTERNAL", "value": "ACCOUNT.EXTERNAL_ID_VOD__C" } ] },
In this sample, a possible HCP custom key stored in Network would be VCRM:HCO:1750384758
.
Property | Value | Description |
---|---|---|
"entity" | "external_data_model_entity" | This is the Network external data model entity that is being loaded (for example, "HCP" or "HCO"). It extracts the HCOs based on the "ENTITY_TYPE_CODE" column from the file. |
"from" | "SQL_statement" | This is similar to a MySQL statement, containing a WHERE clause that extracts entity data based on the column you specify in the statement. ISPERSONACCOUNT is set to FALSE because HCOs are being extracted. |
"attributes" |
"ACCOUNT.*", "ACCOUNT.VDM_* AS *" |
This is an array for each entity being modeled. The array contains important information that the loader uses to extract the proper columns for each file. These are internal attributes that the loader uses to store the attributes in the file and the internal VDM (internal loader) attributes in the intermediate store. These are internal attributes used by the loader and should not be changed. |
"customkeys" | "value", "item", or "source" |
This is an array for the required keys, which the loader stores in the "value" represents the column within the flat file that contains the required value. "item" represents the object that the custom key belongs to. If you do not specify an item, the loader uses the "entity" value. "source" represents the source that the custom key belongs to. If you do not specify the source, the loader uses the "system" value assigned to the subscription. |
Address entity
{ "entity": "ADDRESS", "from": "ADDRESS JOIN ACCOUNT ON ACCOUNT.ID = ADDRESS.ACCOUNT_VOD__C", "attributes": [ "ADDRESS.*", "ADDRESS.VDM_* AS *", "ACCOUNT.VDM_ENTITY_ID AS VDM_FKENTITY_ID" ], "customkeys": [ { "source": "CRM", "item": "ADRRESS", "value": "ADDRESS.ID" } ] },
Property | Value | Description |
---|---|---|
"entity" | "external_data_model_entity" | This is the Network external data model entity that is being loaded. |
"from" | "SQL_statement" |
This is similar to a MySQL statement, containing a |
"anchors" | "first_address_column" |
The anchor represents a column in the file (preferably non-empty) used for creation of the child entity. |
"customkeys" | "value", "item", or "source" |
This is an array for the required keys, which the loader stores in the "value" represents the column within the flat file that contains the required value. "item" represents the object that the custom key belongs to. If you do not specify an item, the loader uses the "entity" value. "source" represents the source that the custom key belongs to. If you do not specify the source, the loader uses the "system" value assigned to the subscription. |
"attributes" |
"ADDRESS.*", "ADDRESS.VDM_* AS *", "ACCOUNT.VDM_ENTITY_ID AS VDM_FKENTITY_ID" |
This is an array for each entity being modeled. The array contains important information that the loader uses to extract the proper columns for each file. These are internal attributes that the loader uses to store the attributes in the file and the internal VDM (internal loader) attributes in the intermediate store. They also store the ID of the account extracted as a foreign key in the address. These are internal attributes used by the loader and should not be changed. |
License entity
{ "entity": "LICENSE", "from": "ADDRESS JOIN ACCOUNT ON ACCOUNT.ID = ADDRESS.ACCOUNT_VOD__C", "anchors": [ { "anchor": "ADDRESS.DEA_VOD__C", "attributes": [ "LIC_TYPE='ADDRESS'", "LIC_VALUE=@ANCHOR_VALUE", "LIC_BODY='DEA'", "LIC_TYPE_VALUE='DEA'", "ADDRESS.DEA_EXPIRATION_DATE_VOD__C AS LIC_EXPIRATION", "ADDRESS.DEA_STATUS_VOD__C AS LIC_STATUS" ], "addresskey": { "source": "CRM", "item": "ADRRESS", "value": "ADDRESS.ID" } }, { "anchor": "ADDRESS.LICENSE_VOD__C", "attributes": [ "LIC_TYPE='STATE'", "LIC_VALUE=@ANCHOR_VALUE", "ADDRESS.STATE_VOD__C AS LIC_TYPE_VALUE", "ADDRESS.LICENSE_EXPIRATION_DATE_VOD__C AS LIC_EXPIRATION", "ADDRESS.LICENSE_STATUS_VOD__C AS LIC_STATUS" ] } ], "attributes": [ "ADDRESS.VDM_* AS *", "ACCOUNT.VDM_ENTITY_ID AS VDM_FKENTITY_ID", "VDM_ENTITY_ID=@NEW_ENTITY_ID" ] },
Property | Value | Description |
---|---|---|
"entity" | "external_data_model_entity" | This is the Network external data model entity that is being loaded. |
"from" | "SQL_statement" |
This is similar to a MySQL statement, containing a With Veeva CRM, the license information is in the address file, but joining against the account creates a link between the license record and the HCP or HCO account that holds the license. |
"anchors" | "anchor_array" |
This represents the relevant columns from which to extract license records from the file used for extraction. |
"anchor" | "first_anchor_column" |
The anchor represents a column in the file (preferably non-empty) used for creation of the child entity. |
"attributes" |
"entity_array_column" |
The attributes array contains all of the license columns required for extraction. The Network external model requires the loader to use variables such as |
"addresskey" | "value", "item", or "source" |
This is an array for the required keys, which the loader stores in the "value" represents the column within the flat file that contains the required value. "item" represents the object that the custom key belongs to. If you do not specify an item, the loader uses the "entity" value. "source" represents the source that the custom key belongs to. If you do not specify the source, the loader uses the "system" value assigned to the subscription. |
"anchor" | first_anchor_column |
This represents the first column of the state license within the file used for extraction. |
"attributes" |
"LIC_TYPE='STATE'", "LIC_VALUE=@ANCHOR_VALUE", "ADDRESS.STATE_VOD__C AS LIC_TYPE_VALUE", "ADDRESS.LICENSE_EXPIRATION_DATE_VOD__C AS LIC_EXPIRATION", "ADDRESS.LICENSE_STATUS_VOD__C AS LIC_STATUS" |
These attributes assign each column to a corresponding variable. |
"attributes" |
"ADDRESS.VDM_* AS *", "ACCOUNT.VDM_ENTITY_ID AS VDM_FKENTITY_ID", "VDM_ENTITY_ID=@NEW_ENTITY_ID" |
These attributes store the parent and child account IDs of the extracted PARENTHCO as a foreign key, store the internal VDM (internal loader) attributes in the intermediate store, and store all attributes in the file in the intermediate storage. These are internal attributes used by the loader and should not be changed. |
Parent HCO entity
{ "entity": "PARENTHCO", "from": "CHILDACCOUNT JOIN ACCOUNT CHILD ON CHILDACCOUNT.CHILD_ACCOUNT_VOD__C = CHILD.ID LEFT JOIN ACCOUNT PARENT ON CHILDACCOUNT.PARENT_ACCOUNT_VOD__C = PARENT.ID", "attributes": [ "CHILDACCOUNT.*", "CHILDACCOUNT.VDM_* AS *", "CHILD.VDM_ENTITY_ID AS VDM_FKENTITY_ID", "PARENT.VDM_ENTITY_ID AS VDM_FKENTITY2_ID" ], "customkeys": [ { "source": "CRM", "item": "CHILDACCOUNT", "value": "CHILDACCOUNT.ID" } ], "parentkey": { "source": "CRM", "item": "ACCOUNT", "value": "CHILDACCOUNT.PARENT_ACCOUNT_VOD__C" } } ]
In this sample, a possible parent HCO custom key stored in Network would be VCRM:CHILDACCOUNT:1750384707
.
Property | Value | Description |
---|---|---|
"entity" | "external_data_model_entity" | This is the Network external data model entity that is being loaded. |
"from" | "SQL_statement" |
This is similar to a MySQL statement, containing a |
"attributes" |
"CHILDACCOUNT.*", "CHILDACCOUNT.VDM_* AS *", "CHILD.VDM_ENTITY_ID AS VDM_FKENTITY_ID", "PARENT.VDM_ENTITY_ID AS VDM_FKENTITY2_ID" |
These attributes store the parent and child account IDs of the extracted PARENTHCO as a foreign key, store the internal VDM (internal loader) attributes in the intermediate store, and store all attributes in the file in the intermediate storage. These are internal attributes used by the loader and should not be changed. |
"customkeys" | "value", "item", or "source" |
This is an array for the required keys, which the loader stores in the "value" represents the column within the flat file that contains the required value. "item" represents the object that the custom key belongs to. If you do not specify an item, the loader uses the "entity" value. "source" represents the source that the custom key belongs to. If you do not specify the source, the loader uses the "system" value assigned to the subscription. |
Define normalization rules
The normalization rules define field mappings from columns in an incoming source file to the attributes in the Network data model.
These mappings are divided into arrays, which correspond to entities in the Network external data model (HCP, HCO, ADDRESS, LICENSE, PARENTHCO, and CUSTOMKEYS).
Considerations
- Field normalizations are case-sensitive. For example, if you configure the field normalization "HCP.city": "locality__v"," city is case sensitive. The column name in the UI must match exactly.
- If aliases are used in the model map (* AS *), do not prefix the entity in the field normalization. For example, if "HCP.* AS *" is defined in the model map, the field normalization for HCP should be "LASTNAME" instead of "HCP.LASTNAME". This does not apply to the VDM attributes (for example, "HCP.VDM_* AS *").
Examples
The following are examples of properties for two sources, in JSON format.
NPI
{ "HCP": { "NPI.NPI": "npi_num__v", "NPI.PROVIDER_LAST_NAME__LEGAL_NAME_": "last_name__v", "NPI.PROVIDER_FIRST_NAME": "first_name__v", "NPI.PROVIDER_MIDDLE_NAME": "middle_name__v", "NPI.PROVIDER_NAME_PREFIX_TEXT": "prefix__v", "NPI.PROVIDER_NAME_SUFFIX_TEXT": "suffix__v", "NPI.PROVIDER_CREDENTIAL_TEXT": "medical_degree_1__v", "NPI.PROVIDER_GENDER_CODE": "gender__v" }, "HCO": { "NPI.NPI": "npi_num__v", "NPI.PROVIDER_ORGANIZATION_NAME__LEGAL_BUSINESS_NAME_": "corporate_name__v" }, "ADDRESS": { "NPI.PROVIDER_FIRST_LINE_BUSINESS_MAILING_ADDRESS": "address_line_1__v", "NPI.PROVIDER_SECOND_LINE_BUSINESS_MAILING_ADDRESS": "address_line_2__v", "NPI.PROVIDER_BUSINESS_MAILING_ADDRESS_CITY_NAME": "locality__v", "NPI.PROVIDER_BUSINESS_MAILING_ADDRESS_STATE_NAME": "administrative_area__v", "NPI.PROVIDER_BUSINESS_MAILING_ADDRESS_POSTAL_CODE": "postal_code__v", "NPI.PROVIDER_BUSINESS_MAILING_ADDRESS_COUNTRY_CODE": "country__v", "NPI.PROVIDER_FIRST_LINE_BUSINESS_PRACTICE_LOCATION_ADDRESS": "thoroughfare__v", "NPI.PROVIDER_SECOND_LINE_BUSINESS_PRACTICE_LOCATION_ADDRESS": "thoroughfare_post_direction__v", "NPI.PROVIDER_BUSINESS_PRACTICE_LOCATION_ADDRESS_CITY_NAME": "locality__v", "NPI.PROVIDER_BUSINESS_PRACTICE_LOCATION_ADDRESS_STATE_NAME": "administrative_area__v", "NPI.PROVIDER_BUSINESS_PRACTICE_LOCATION_ADDRESS_POSTAL_CODE": "postal_code__v", "NPI.PROVIDER_BUSINESS_PRACTICE_LOCATION_ADDRESS_COUNTRY_CODE": "country__v" }, "LICENSE": { "LIC_TYPE": "type__v",s "LIC_LICENSE_NUMBER_STATE_CODE": "type_value__v", "LIC_LICENSE_NUMBER": "license_number__v" } }
{ "HCO": { "ACCOUNT.NAME": "corporate_name__v", "ACCOUNT.Type": "hco_type__v", "ACCOUNT.PHONE": "phone_1__v", "ACCOUNT.FAX": "fax_1__v", "ACCOUNT.WEBSITE": "URL_1__v", "ACCOUNT.NUMBEROFEMPLOYEES": "count_medstaff__v", "ACCOUNT.GROUP_SPECIALTY_1_VOD__C": "specialty_1__v", "ACCOUNT.GROUP_SPECIALTY_2_VOD__C": "specialty_2__v", "ACCOUNT.DO_NOT_CALL_VOD__C": "do_not_contact__v", "ACCOUNT.STATUS__C": "hco_status__v", "ACCOUNT.TEACHING_HOSPITAL__C": "training_facility__v" }, "HCP": { "HCP.LASTNAME": "last_name__v", "HCP.FIRSTNAME": "first_name__v", "HCP.PHONE": "phone_1__v", "HCP.FAX": "fax_1__v", "HCP.WEBSITE": "URL_1__v", "HCP.PERSONOTHERPHONE": "phone_2__v", "HCP.PERSONEMAIL": "email_1__v", "HCP.credentials_vod__c": "medical_degree_1__v", "HCP.SPECIALTY_1_VOD__C": "specialty_1__v", "HCP.SPECIALTY_2_VOD__C": "verteo_specialty__c", "HCP.GENDER_VOD__C": "gender__v", "HCP.PDRP_OPT_OUT_VOD__C": "pdrp_optout__v", "HCP.PDRP_OPT_OUT_DATE_VOD__C": "pdrp_optout_date__v", "HCP.SUFFIX_VOD__C": "suffix__v", "HCP.NPI_VOD__C": "npi_num__v", "HCP.ME__C": "me_id__v" }, "LICENSE": { "LICENSE.STATE_VOD__C": "type_value__v", "LICENSE.LICENSE_VOD__C": "license_number__v", "LICENSE.LICENSE_EXPIRATION_DATE_VOD__C": "expiration_date__v", "LICENSE.LICENSE_STATUS_VOD__C": "license_status__v", "LICENSE.type__v": "type__v", "LICENSE.drug_schedule__v": "drug_schedule__v" }, "ADDRESS": { "ADDRESS.NAME": "address_line_1__v", "ADDRESS.ADDRESS_LINE_2_VOD__C": "address_line_2__v", "ADDRESS.CITY_VOD__C": "locality__v", "ADDRESS.STATE_VOD__C": "administrative_area__v", "ADDRESS.ZIP_VOD__C": "postal_code__v", "ADDRESS.country__v": "country__v" }, "PARENTHCO": { "RELATIONSHIP.Primary": "is_primary_relationship__v" } }