I’m attempting to setup ServiceNow Data Flow v1.2 and facing some challenges. I’m not seeing any CSV files (PreviewOnly=True)
This is a lab environment, and there are no CIs presently in ServiceNow, and only 5 computers in BigFix, including the root server.
The BigFix to ServiceNow dataflow log has these entries:
2022-10-13 22:55:10.122861 12732 filter_duplicates LogLevels.DEBUG ServicenowManager : Droping Duplicates
2022-10-13 22:55:10.138495 12732 df LogLevels.INFO DataFrame is not initialized. DynamicDevice : 10a23ae18cad20e1a393aff7aa7f7844 cache is empty
2022-10-13 22:55:10.138495 12732 filter_duplicates LogLevels.ERROR ServicenowManager Dataloader-filter_duplicates failed. Error: 'NoneType' object has no attribute 'drop_duplicates'
2022-10-13 22:55:10.138495 12732 df LogLevels.INFO DataFrame is not initialized. DynamicDevice : 10a23ae18cad20e1a393aff7aa7f7844 cache is empty
2022-10-13 22:55:10.138495 12732 init LogLevels.ERROR ServiceNow Dataloader-Init failed. Error: object of type 'NoneType' has no len()
2022-10-13 22:55:10.138495 12732 DataFlowRunner LogLevels.ERROR DataFlowRunner: Exception occurred Discontinuing Execution of DataFlow.
You may be better of opening a support case. That’s exactly where I am too - tried to install/configure but not exactly working (although, not sure that the issues are the same), so Support are reviewing logs.
I assume these commands were run during the setup: validateconfiguration, initializeschemas? Are you using out of the box configuration created during installation?
I changed the weightages for these 2 and added weightage to BigFix Computer Name to create records in CMDB.
Enable debug, update configuration, run -reset and try again and check the logs.
@ssakunala I used the Fixlet to Deploy the data flow service. I also used OOB configuration. At this point, I am merely looking for the CSV files to be generated, not to place any data into ServiceNow.
I have enabled Debug in the logs. No change in the results of running the dataflow after performing the reset.
Hi @itsmpro92 Here is my configuration that works and had generated CSV file during testing. I have couple of custom attributes but the default configuration from template is still there. I also have filters for both adapters. Please compare it with your configuration.
<dataflows>
<dataflow displayname="Transfers Asset Data from Bigfix to ServiceNow" datatype="asset" schedule="now">
<dataflowdescription/>
<sourceadapter displayname="Bigfix Adapter" adapterclass="bigfix" datasourcename="BigFixRestAPI">
<filterquery><![CDATA[device type of it = "Server"]]></filterquery>
<device_properties>
<sourcekey displayname="BigFix Computer ID" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-1" datatype="string"/>
<targetkey displayname="ServiceNow SYS_ID" propertyname="ServiceNow CMDB Attributes-ServiceNow Custom Properties-2" datatype="int"/>
<property displayname="Computer Name" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-2" datatype="string" weight="40"/>
<property displayname="Discovery Source" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-3" datatype="string"/>
<property displayname="Last Discovered" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-4" datatype="string"/>
<property displayname="Chassis Type" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-5" datatype="string"/>
<property displayname="Virtual" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-6" datatype="boolean"/>
<property displayname="Manufacturer" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-7" datatype="string"/>
<property displayname="Model number" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-8" datatype="string"/>
<property displayname="OS" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-9" datatype="string"/>
<property displayname="OS Version" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-10" datatype="string"/>
<property displayname="OS Domain" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-11" datatype="string"/>
<property displayname="CPU Manufacturer" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-12" datatype="string"/>
<property displayname="CPU Name" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-13" datatype="string"/>
<property displayname="CPU Speed" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-14" datatype="string"/>
<property displayname="CPU Count" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-15" datatype="string"/>
<property displayname="CPU Core Count" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-16" datatype="string"/>
<property displayname="RAM" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-17" datatype="string"/>
<property displayname="Disk space" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-18" datatype="string"/>
<identityproperty displayname="IP Address" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-19" datatype="string" weight="20"/>
<identityproperty displayname="MAC Address" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-20" datatype="string" weight="40"/>
<property displayname="Serial Number" propertyname="ServiceNow-BigFix Properties CMDB-1" datatype="string"/>
</device_properties>
</sourceadapter>
<targetadapter displayname="ServiceNow Adapter" adapterclass="servicenow" datasourcename="ServiceNowAPI">
<device_properties>
<sourcekey displayname="ServiceNow SYS_ID" propertyname="sys_id" datatype="string"/>
<targetkey displayname="BigFix Computer ID" propertyname="correlation_id" datatype="string"/>
<property displayname="Name" propertyname="name" datatype="string" weight="40"/>
<property displayname="Discovery Source" propertyname="discovery_source" datatype="string"/>
<property displayname="Last Discovered" propertyname="sys_updated_on" datatype="string"/>
<property displayname="Chassis Type" propertyname="chassis_type" datatype="string"/>
<property displayname="Virtual" propertyname="virtual" datatype="boolean"/>
<property displayname="Manufacturer" propertyname="manufacturer" datatype="string"/>
<property displayname="Model number" propertyname="model_number" datatype="string"/>
<property displayname="OS" propertyname="os" datatype="string"/>
<property displayname="OS Version" propertyname="os_version" datatype="string"/>
<property displayname="OS Domain" propertyname="os_domain" datatype="string"/>
<property displayname="CPU manufacturer" propertyname="cpu_manufacturer" datatype="string"/>
<property displayname="CPU Name" propertyname="cpu_name" datatype="string"/>
<property displayname="CPU Speed" propertyname="cpu_speed" datatype="string"/>
<property displayname="CPU count" propertyname="cpu_count" datatype="string"/>
<property displayname="CPU Core Count" propertyname="cpu_core_count" datatype="string"/>
<property displayname="RAM" propertyname="ram" datatype="string"/>
<property displayname="Disk space" propertyname="disk_space" datatype="string"/>
<identityproperty displayname="IP Address" propertyname="ip_address" datatype="string" weight="20"/>
<identityproperty displayname="MAC Address" propertyname="mac_address" datatype="string" weight="40"/>
<property displayname="Serial Number" propertyname="serial_number" datatype="string"/>
</device_properties>
</targetadapter>
</dataflow>
<dataflow displayname="Transfer Asset Data from ServiceNow to Bigfix" datatype="asset" schedule="*/2 * * * *">
<dataflowdescription/>
<sourceadapter displayname="ServiceNow Adapter" adapterclass="servicenow" datasourcename="ServiceNowAPI">
<filterquery><![CDATA[ip_addressSTARTSWITH192.168]]></filterquery>
<device_properties>
<sourcekey displayname="ServiceNow SYS_ID" propertyname="sys_id" datatype="string"/>
<targetkey displayname="BigFix Computer ID" propertyname="correlation_id" datatype="string"/>
<property displayname="assigned_to" propertyname="assigned_to.name" datatype="string"/>
<property displayname="Department" propertyname="department.name" datatype="string"/>
<property displayname="Asset Tag" propertyname="asset_tag" datatype="string"/>
<property displayname="OS Domain" propertyname="os_domain" datatype="string"/>
<identityproperty displayname="IP Address" propertyname="ip_address" datatype="string" weight="20"/>
<identityproperty displayname="MAC Address" propertyname="mac_address" datatype="string" weight="80"/>
</device_properties>
</sourceadapter>
<targetadapter displayname="Bigfix Adapter" adapterclass="bigfix" datasourcename="BigFixRestAPI">
<device_properties>
<sourcekey displayname="BigFix Computer ID" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-1" datatype="string"/>
<targetkey displayname="ServiceNow SYS_ID" propertyname="ServiceNow CMDB Attributes-ServiceNow Custom Properties-2" datatype="int"/>
<property displayname="assigned_to" propertyname="ServiceNow CMDB Attributes-ServiceNow Custom Properties-3" datatype="string"/>
<property displayname="Department" propertyname="ServiceNow CMDB Attributes-ServiceNow Custom Properties-4" datatype="string"/>
<property displayname="Asset Tag" propertyname="ServiceNow CMDB Attributes-ServiceNow Custom Properties-5" datatype="string"/>
<property displayname="OS Domain" propertyname="ServiceNow CMDB Attributes-ServiceNow Custom Properties-6" datatype="string"/>
<identityproperty displayname="IP Address" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-19" datatype="string" weight="80"/>
<identityproperty displayname="MAC Address" propertyname="BigFix ServiceNow Data Flow-Hardware Attributes-20" datatype="string" weight="20"/>
</device_properties>
</targetadapter>
</dataflow>
</dataflows>
<settings>
<setting key="MinimumConfidenceLevel" value="80"/>
<setting key="NumberOfConcurrentDataflows" value="1"/>
<setting key="LogLevel" value="INFO"/>
<setting key="CacheRefreshLimit" value="1"/>
<setting key="PreviewOnly" value="False"/>
<setting key="servicenowrest.QueueRefreshInterval" value="120"/>
<setting key="servicenowrest.UpdateBatchSize" value="85"/>
<setting key="servicenowrest.batch_size" value="9999"/>
<setting key="bigfixrest.QueueRefreshInterval" value="120"/>
<setting key="bigfixrest.UpdateBatchSize" value="85"/>
<setting key="bigfixrest.ContentSiteName" value="ServiceNow CMDB Attributes"/>
<setting key="bigfix.MaxComputerAge" value="12"/>
</settings>
</dataflowconfig>
My DataflowsConfig.xml looks like yours, for the most part. There are a few non-consequential differences, like LogLevel=DEBUG and PreviewOnly=True. I don’t have any filters at this time.
Interesting - I think I’ve uncovered an edge case.
There were no Computer records in my ServiceNow Instance, since this is just a lab environment. The Deduplication logic fails when there are no records in cmbd-ci-computer in ServiceNow. This causes the dataflow process to fail.
I added one Computer record, with just a name (Dummy01), not matching any of the devices in my BigFix managed environment, and it worked.
I now have RecordCache files, and the Preview-[0]-[servicenow]-yyymmddhhmmss.csv file with the expected 5 records in it.
Side note: even though I am running just the Transfers Asset Data from BigFix to ServiceNow dataflow, it appears that some data is collected from ServiceNow, and this is portion of the process fails with the error:
2022-10-14 18:09:09.299262 14840 SaveMailbox LogLevels.ERROR BigFix DataUpdater-LoadMailbox failed. Error Saving Binary Cache File: Mailboxes-ad1236f05aec162916c83010d05a79b9.dat
I think it is looking for the Dummy01 device in the BigFix managed environment.
2 Likes
Final update: I ran both dataflows, with PreviewOnly=False, and encountered no errors. Not all the properties showed up in ServiceNow, but that is a different topic for a future conversation.
Aram
October 14, 2022, 9:10pm
9
I don’t know that I’d describe the scenario above as an edge case necessarily. I’ll be following-up internally, but it sounds to me like we should better handle cases where there are no computer records in CMDB (while admittedly rare, it can happen).
1 Like