ServiceNow Data Flow: Duplicate Entries created in ServiceNow

Tips
1

Under the default ServiceNow Data Flow configuration, you may find duplicate computers are being created in ServiceNow after enabling the BigFix ServiceNow Data Flow.

In the default configuration, computers are matched between BigFix and ServiceNow based on ServiceNow’s “correlation_id” value matching a BigFix Computer ID. In most cases the correlation_id for existing ServiceNow computers will not have a value, so BigFix creates new computers instead of updating existing ServiceNow computer records - there is no default matching based on Computer Name or other attributes.

This behavior should be changed by updating the “IntegrationServicesConfig.xml” file, located by default at “C:\Program Files\HCL\Integration Services\src”. See the reference at https://help.hcltechsw.com/bigfix/10.0/integrations/Ecosystem/Install_Config/c_refs_cfile.html

The default configuration file contents are as follows:

<?xml version="1.0" ?><integrationconfig>
	<datasources>
		<datasource datasourcename="BigFixRestAPI" connectionstring="https://[BigFixRestAPIURL]]:52311/api" 		verifycert="false"/>
		<datasource datasourcename="ServiceNowAPI" connectionstring="https://[ServiceNowRestAPIURL]/api/now/table" 	verifycert="false"/>
	</datasources>
	
	<dataflows>
		<dataflow displayname="Endpoint data from BigFix To ServiceNow" minimumcorrelationconfidencelevel="75" executionintervalinminutes="360">

			<sourceadapter displayname="BigFix Source Adapter" adapterclass="bigfixrest" datasourcename="BigFixRestAPI">
				<!-- Filter which endpoints to send to ServiceNow. The sample query below filters to only the Root Server(s). 
				<filterquery><![CDATA[root server flag of it]]></filterquery>-->
				
				<properties>
					<!-- Identify Source Key and Target Key -->
					<property displayname="BigFix Computer ID" 	columnname="BigFix ServiceNow Data Flow-Hardware Attributes-1" 			datatype="string" pk="true"/>
					<property displayname="ServiceNow SYS_ID" 	columnname="ServiceNow CMDB Attributes-ServiceNow Custom Properties-2" 	datatype="int"/>

					<!-- List Other Properties To Be Syncronized -->
					<property displayname="Computer Name" 		columnname="BigFix ServiceNow Data Flow-Hardware Attributes-2" 	datatype="string"/>
					<property displayname="Discovery Source" 	columnname="BigFix ServiceNow Data Flow-Hardware Attributes-3" 	datatype="string"/>
					<property displayname="Last Discovered" 	columnname="BigFix ServiceNow Data Flow-Hardware Attributes-4" 	datatype="string"/>
					<property displayname="Chassis Type"	 	columnname="BigFix ServiceNow Data Flow-Hardware Attributes-5" 	datatype="string"/>
					<property displayname="Virtual" 			columnname="BigFix ServiceNow Data Flow-Hardware Attributes-6" 	datatype="string"/>
					<property displayname="Manufacturer" 		columnname="BigFix ServiceNow Data Flow-Hardware Attributes-7" 	datatype="string"/>
					<property displayname="Model number" 		columnname="BigFix ServiceNow Data Flow-Hardware Attributes-8" 	datatype="string"/>
					<property displayname="OS" 					columnname="BigFix ServiceNow Data Flow-Hardware Attributes-9" 	datatype="string"/>
					<property displayname="OS Version" 			columnname="BigFix ServiceNow Data Flow-Hardware Attributes-10" datatype="string"/>
					<property displayname="OS Domain" 			columnname="BigFix ServiceNow Data Flow-Hardware Attributes-11" datatype="string"/>
					<property displayname="CPU Manufacturer" 	columnname="BigFix ServiceNow Data Flow-Hardware Attributes-12" datatype="string"/>
					<property displayname="CPU Name" 			columnname="BigFix ServiceNow Data Flow-Hardware Attributes-13" datatype="string"/>
					<property displayname="CPU Speed" 			columnname="BigFix ServiceNow Data Flow-Hardware Attributes-14" datatype="string"/>
					<property displayname="CPU Count" 			columnname="BigFix ServiceNow Data Flow-Hardware Attributes-15" datatype="string"/>
					<property displayname="CPU Core Count" 		columnname="BigFix ServiceNow Data Flow-Hardware Attributes-16" datatype="string"/>
					<property displayname="RAM" 				columnname="BigFix ServiceNow Data Flow-Hardware Attributes-17" datatype="string"/>
					<property displayname="Disk space" 			columnname="BigFix ServiceNow Data Flow-Hardware Attributes-18" datatype="string"/>
					<property displayname="IP Address" 			columnname="BigFix ServiceNow Data Flow-Hardware Attributes-19" datatype="string"/>
					<property displayname="MAC Address" 		columnname="BigFix ServiceNow Data Flow-Hardware Attributes-20" datatype="string"/>
				</properties>
			</sourceadapter>	

			<targetadapter displayname="ServiceNow Source Adapter" adapterclass="servicenowrest" datasourcename="ServiceNowAPI">
				<properties>
					<!-- Identify Source Key and Target Key -->
					<property displayname="BigFix Computer ID" 	columnname="correlation_id" 	datatype="int"/>
					<property displayname="ServiceNow SYS_ID" 	columnname="sys_id" 			datatype="int" pk="true"/>

					<!-- List Other Properties To Be Syncronized -->
					<property displayname="Name" 				columnname="name" 				datatype="string"/>
					<property displayname="Discovery Source"	columnname="discovery_source" 	datatype="string"/>
					<property displayname="Last Discovered"		columnname="sys_updated_on" 	datatype="string"/>	
					<property displayname="Chassis Type" 		columnname="chassis_type" 		datatype="string"/>
					<property displayname="Virtual" 			columnname="virtual" 			datatype="string"/>
					<property displayname="Manufacturer" 		columnname="manufacturer" 		datatype="string"/>
					<property displayname="Model number"		columnname="model_number" 		datatype="string"/>
					<property displayname="OS" 					columnname="os" 				datatype="string"/>
					<property displayname="OS Version" 			columnname="os_version" 		datatype="string"/>
					<property displayname="OS Domain" 			columnname="os_domain" 			datatype="string"/>
					<property displayname="CPU manufacturer"	columnname="cpu_manufacturer" 	datatype="string"/>					
					<property displayname="CPU Name" 			columnname="cpu_name" 			datatype="string"/>				
					<property displayname="CPU Speed" 			columnname="cpu_speed" 			datatype="string"/>					
					<property displayname="CPU count" 			columnname="cpu_count" 			datatype="string"/>
					<property displayname="CPU Core Count" 		columnname="cpu_core_count" 	datatype="string"/>
					<property displayname="RAM" 				columnname="ram" 				datatype="string"/>
					<property displayname="Disk space" 			columnname="disk_space" 		datatype="string"/>
					<property displayname="IP Address" 			columnname="ip_address" 		datatype="string"/>
					<property displayname="MAC Address" 		columnname="mac_address" 		datatype="string"/>
				</properties>
			</targetadapter>
		
		</dataflow>		

		<dataflow displayname="Endpoint data from ServiceNow to BigFix" minimumcorrelationconfidencelevel="75" executionintervalinminutes="360">
			<sourceadapter displayname="ServiceNow Source Adapter" adapterclass="servicenowrest" datasourcename="ServiceNowAPI">
				<!-- Filter which endpoint data to send to BigFix. The sample query below filters to only those endpoints assigned to a user whose name contains 'bigfix'. 
				<filterquery><![CDATA[assigned_to.name contains 'bigfix']]></filterquery>-->
				<properties>
					<!-- Identify Source Key and Target Key -->
					<property displayname="BigFix Computer ID" 	columnname="correlation_id" 	datatype="int"/>
					<property displayname="ServiceNow SYS_ID" 	columnname="sys_id" 			datatype="string" pk="true"/>

					<!-- List Other Properties To Be Syncronized -->
					<property displayname="assigned_to" 		columnname="assigned_to.name" 	datatype="string"/>
					<property displayname="Department" 			columnname="department.name" 	datatype="string"/>
				</properties>
			</sourceadapter>
			<targetadapter displayname="BigFix Target Adapter" adapterclass="bigfixrest" datasourcename="BigFixRestAPI">
				<properties>
					<!-- Identify Source Key and Target Key -->
					<property displayname="BigFix Computer ID"	columnname="BigFix ServiceNow Data Flow-Hardware Attributes-1" 			datatype="int" pk="true"/>
					<property displayname="ServiceNow SYS_ID" 	columnname="ServiceNow CMDB Attributes-ServiceNow Custom Properties-2" 	datatype="string"/>

					<!-- List Other Properties To Be Syncronized -->
					<property displayname="assigned_to" 		columnname="ServiceNow CMDB Attributes-ServiceNow Custom Properties-3" 	datatype="string"/>
					<property displayname="Department" 			columnname="ServiceNow CMDB Attributes-ServiceNow Custom Properties-4" 	datatype="string"/>
				</properties>
			</targetadapter>			
		</dataflow>
	</dataflows>

	<settings>
		<setting key="NumberOfConcurrentDataflows" 			value="2"/>
		<setting key="LogLevel" 							value="DEBUG"/>
		<setting key="PreviewOnly" 							value="True"/>

		<!-- Default Update Throttling Configured To 2500 Updates/hr -->
		<setting key="servicenowrest.QueueRefreshInterval" 	value="120"/>
		<setting key="servicenowrest.UpdateBatchSize" 		value="85"/>
		<setting key="bigfixrest.QueueRefreshInterval" 		value="120"/>
		<setting key="bigfixrest.UpdateBatchSize" 			value="85"/>

		<setting key="bigfixrest.ContentSiteName" 			value="ServiceNow CMDB Attributes"/>
		
		<setting key="bigfixrest.MaxComputerAge" 			value="12"/>
	</settings>
</integrationconfig>

In order to prevent duplicating existing ServiceNow computers and instead update existing ones, ensure the “Endpoint data from BigFix to ServiceNow” flow contains the ‘minimumcorrelationconfidencelevel’ value, and that at least one of the properties have ‘weight’ defined. ‘weight’ should be defined on both the source (BigFix) and target (ServiceNow) definition of the property, and if multiple properties are used their weights should add up to at least the ‘minimumcorrelationconfidencelevel’ value.
The following snippet shows using the Computer Name property to match. Extra properties are snipped and comments are inline:

<dataflows>
  <!-- Match existing computers were we are 75% confident of a match, and check for updates from BigFix every 6 hours -->
  <dataflow displayname="Endpoint data from BigFix To ServiceNow" minimumcorrelationconfidencelevel="75" executionintervalinminutes="360">

    <sourceadapter displayname="BigFix Source Adapter" adapterclass="bigfixrest" datasourcename="BigFixRestAPI">
<!-- Filter to include only 'real' BigFix agents, excluding Proxy Agents for OS Deployment or Virtual Machine templates -->
      <filterquery><![CDATA[ agent type of it = "Native" ]]></filterquery>
      <properties>
        <!-- Identify Source Key and Target Key -->
        <property displayname="BigFix Computer ID" columnname="BigFix ServiceNow Data Flow-Hardware Attributes-1" datatype="string" pk="true"/>
        <property displayname="ServiceNow SYS_ID" columnname="ServiceNow CMDB Attributes-ServiceNow Custom Properties-2" datatype="int"/>

        <!-- List Other Properties To Be Syncronized -->
        <!-- where Computer Names match between BigFix and ServiceNow, we are 80% confident these are the same machine -->
<property displayname="Computer Name" columnname="BigFix ServiceNow Data Flow-Hardware Attributes-2" datatype="string" weight="80"/>
        <!-- additional properties trimmed for example -->
      </properties>
    </sourceadapter>  

    <targetadapter displayname="ServiceNow Source Adapter" adapterclass="servicenowrest" datasourcename="ServiceNowAPI">
      <properties>
        <!-- Identify Source Key and Target Key -->
        <!--<property displayname="BigFix Computer ID" columnname="u_bigfix_computer_id" datatype="int"/> -->
        <property displayname="BigFix Computer ID" columnname="correlation_id" datatype="int"/>
        <property displayname="ServiceNow SYS_ID" columnname="sys_id" datatype="int" pk="true"/>

        <!-- List Other Properties To Be Syncronized -->
        <!-- where Computer Names match between BigFix and ServiceNow, we are 80% confident these are the same machine -->
        <property displayname="Name" columnname="name" datatype="string" weight="80"/>
        <!-- additional properties trimmed for example -->
      </properties>
    </targetadapter>
  </dataflow>    
</dataflows>

After modifying the XML file, in some cases it is necessary to delete all of the .dat files from “C:\Program Files\HCL\Integration Services" and restart the service.

3 Likes
2

Any word on when the next update will be published?

3

@JasonWalker Thank you for this example. It helps with adding weight to the properties we choose to get a better match. However, we would like to add custom properties such as serial number. We created a custom propety to gather the serial number for various platforms. If this is possible, how is it added to the XML? Looking at the default XML, I would assume the following based on “Computer Name”:

<property displayname="Computer Name" 		columnname="BigFix ServiceNow Data Flow-Hardware Attributes-2" 	datatype="string"/>

“BigFix ServiceNow Data Flow” is the name of the site
"Hardware Attributes" is the name of the analyses
"2" is the order of the property in the analyses

Is this how we would add other analyses to the XML? The next question would be how this works for custom properties which are not part of an analyses such as out custom “serial number” property.