Relay - Failed to Synchronize 404

csimonsen · June 16, 2022, 1:31pm

We are seeing a failure to synchronize fixlet content from the official fixlet sites across our relays. I’ve narrowed the issue down to possibly our top relay.

A snippet from the client at the top relay.

What is of interest here is that clients directly connected to the BigFix server experience no problem retrieving their content. For example, a fresh install of the client without relay will immediately connect to the BigFix server and synchronize properly with all sites. But as soon as we install the Relay via the console, the client begins to fail to synchronize with the BigFix server because the connection is now running through the relay instead of directly with the BigFix server.

We see the same problem at lower level Relays, but I imagine if we manage to fix the top relay, the others will follow suit.

Already taken remediation tasks:

Fresh installation of Client and Relay
Initiated a BESAdmin -repair command
Attempted a Gather Reset on Relay
Attempted a Gather Reset on Relay with Client __Besdata folder content removed
Removed computers from affected sites
Entirely removed Sites and enabled them again
Change of Internet Proxy at Bigfix Server

As of last note, if you take the address from the log and put it into a browser, you get

“Error providing site directory.
The requested Fix site cannot be serviced by this server.
Site not yet available: initiating first gather of site.(19)”

However, if you remove the home address and input the address of the BigFix server, it responds just fine.

I’m suspecting there might be some issue with how the Relay handles 127.0.0.1 addresses, but I have not been able to find information on the topic to assist me. If you have any extra ideas beyond the remediation tasks, let me know

JasonWalker · June 16, 2022, 1:37pm

Anything in the Relay’s logfile.txt?

csimonsen · June 16, 2022, 2:02pm

Hey Jason

(I’ve had to remove the http part as I am not allowed to post links as a new user).

Yes, we are getting some errors which I’ve tried to research, but not getting a solution which we could apply (a lot of the old links direct you to IBM which no longer host the content).

For example,

Thu, 16 Jun 2022 15:52:25 +0200 - 15896 - Beginning gather of site ://sync.bigfix.com/cgi-bin/bfgather/bessecurity
Thu, 16 Jun 2022 15:52:25 +0200 - 15896 - Entering GET ://bigfixserver:52311/cgi-bin/bfenterprise/BESGatherMirror.exe?url=://sync.bigfix.com/cgi-bin/bfgather/bessecurity&ManyVersionSHA1=da39a3ee5e6b4b0d3255bfef95601890afd80709&Expect404Flag=true&Time=1655387544
Thu, 16 Jun 2022 15:52:25 +0200 - 15896 - Exiting GET ://bigfixserver:52311/cgi-bin/bfenterprise/BESGatherMirror.exe?url=://sync.bigfix.com/cgi-bin/bfgather/bessecurity&ManyVersionSHA1=da39a3ee5e6b4b0d3255bfef95601890afd80709&Expect404Flag=true&Time=1655387544 (7 ms)
Thu, 16 Jun 2022 15:52:25 +0200 - 15896 - 3: class NoMastheadMatchesURL

NoMastHeadMatchesURL would indicate that there is a mismatch between Actionsite of the Client and the Server. I would have expected a clean install of both Client and Relay with fresh Actionsite from the BigFix server would clear it out, but it doesn’t seem to have done anything.

Another example,

Thu, 16 Jun 2022 15:52:22 +0200 - 15196 - Beginning gather of site ://bigfixserver.:52311/cgi-bin/bfgather.exe/actionsite
Thu, 16 Jun 2022 15:52:22 +0200 - 15196 - Entering GET ://bigfixserver:52311/cgi-bin/bfenterprise/BESGatherMirror.exe?url=://bigfixserver.:52311/cgi-bin/bfgather.exe/actionsite&ManyVersionSHA1=da39a3ee5e6b4b0d3255bfef95601890afd80709&Expect404Flag=true&Time=1655387541
Thu, 16 Jun 2022 15:52:22 +0200 - 15196 - Exiting GET ://bigfixserver:52311/cgi-bin/bfenterprise/BESGatherMirror.exe?url=://bigfixserver.:52311/cgi-bin/bfgather.exe/actionsite&ManyVersionSHA1=da39a3ee5e6b4b0d3255bfef95601890afd80709&Expect404Flag=true&Time=1655387541 (11 ms)
Thu, 16 Jun 2022 15:52:22 +0200 - 15196 - 8: class NotASignedMessage

Here’s another error which would again indicate a mismatch between Actionsite and BigFix Server.

For now, I’m only aiming at fixing the top relay as it used to work, but suddenly stopped. I would have assumed a fresh install of client and relay would do it, but apparently not.

Our current predicament is that this issue limits our visibility into what fixlets are relevant and which are not as none of the agents can report back.

JasonWalker · June 16, 2022, 2:28pm

If you haven’t yet, I suggest you should open a support incident for live troubleshooting.

I notice the two source sites in your log messages here are different - did you anonymize one of them to ‘bigfixserver:52311’ but not the other? If these two are literally different in the log, it looks like at least one client is requesting a site that doesn’t exist because it was masthead switched between different BES deployments. The first error “NoMastheadMatchesURL” indicates a site that doesn’t exist, likely because it’s requested from the wrong BigFix deployment. The second, “NotASignedMessage” usually indicates an error message (which isn’t signed and will be rejected by the relay client). You could try that URL from a browser on the relay, and see what the error is - possibly a message from a proxy server between the relay and the root, which will require some proxy config on the relay?

csimonsen · June 16, 2022, 2:44pm

Hey Jason

They are part of the same BigFix environment. I just merely forgot to sanitize both outputs from the same log properly

I’ll try your suggestion and see how it goes.

csimonsen · June 16, 2022, 6:04pm

Another observation I wanted to add:

It seems the 404 problem is separate from Masthead and NotSigned. I remembered that we had an isolated Relay next to the BigFix server with only 1 client reporting. That client and relay are 100% healthy and operating properly.

So I thought “why not see what happens if I connect the current Top Relay with this isolated one?”.

The action didn’t solve the problem, but it did highlight something: The “current” top Relay was still not pulling down the sites despite now having access to a Relay which did pull them down correctly.

JasonWalker · June 16, 2022, 6:40pm

I still think we could help better with some live troubleshooting, but I wonder whether this top-level relay has some Proxy settings in place that could be sending traffic through a proxy server? I’d check for each of the settings at https://help.hcltechsw.com/bigfix/10.0/platform/Platform/Installation/c_proxy_on_relay.html#configuringaproxyconnectiononparentrelay and see whether any are set.

Also some ‘netstat’ or packet captures could be useful.
Is there any antivirus / endpoint security software running on the Relay that could be interfering with the traffic or with writing the files to disk?

csimonsen · June 17, 2022, 9:42am

Some internal company mix-up is preventing us from opening a support case, so I’ll have to make due with forum suggestions and own research .

Another interesting observation:

I think the problem may be coming from a bad / corrupted version of the actionsite masthead. We have another test relay in a different vlan which was also operating perfectly fine. However, upon uninstalling the relay (to mimic the behavior seen in Top relay), the agent had the opposite reaction: it began failing to synchronize with the BigFix server despite just having worked moments prior. Aka, we are seeing the opposite behavior.

Top Relay: Agent works without Relay, but not with.

Test Relay: Agent works with Relay, but not without.

Upon reinstalling the Relay, synchronizations resumed within the agent just fine.

This led me to digging deeper into what exactly is being requested by the agents at the Relay / main server. Looking into the RelayLog at the BigFix server, I did a curl against this address:

http://bigfixserver:52311/cgi-bin/bfenterprise/BESGatherMirror.exe?url=http://sync.bigfix.com/cgi-bin/bfgather/patchingsupport&ManyVersionSHA1=da39a3ee5e6b4b0d3255bfef95601890afd80709&Expect404Flag=true&Time=1655369448

It responded as previously, but what if I removed the server address itself and just went straight to the source, http://sync.bigfix.com/cgi-bin/bfgather/patchingsupport&ManyVersionSHA1=da39a3ee5e6b4b0d3255bfef95601890afd80709&Expect404Flag=true&Time=1655369448 or merely http://sync.bigfix.com/cgi-bin/bfgather/patchingsupport&ManyVersionSHA1=da39a3ee5e6b4b0d3255bfef95601890afd80709 ?

This provided me a bit more of info cause if you try to input that address, you get:

Error providing site directory.
The requested Fixsite cannot be serviced by this server.
Site configuration file does not exist.

Which now makes the rest of the Relay behavior make sense. “da39a3ee5e6b4b0d3255bfef95601890afd80709” as far as I gather is an empty string hashified, so in reality the server is requesting Version = Empty and all the sites are reporting back saying: “I don’t know any ‘Empty’ version” hence the agents failing synchronization.

I’ll have to continue looking into our test relay where the opposite is happening (Relay works, but not without). Still befuddled that a full reinstallation of client and relay do not reset the gathering state.

JasonWalker · June 17, 2022, 12:40pm

Try the BESRemove tool, which will remove all the components, reg entries, and configuration data.

csimonsen · June 17, 2022, 12:57pm

Hey Jason

Do you have an updated version of that tool? The latest we have is 9.5 and we have used it quite a bit in this process, but unfortunately doesn’t change the end result.

JasonWalker · June 17, 2022, 2:34pm

I wouldn’t expect those to work when hitting sync.bigfix.com directly. The URL parameters (ManyVersionSHA1, Time, etc.) are for a root server to provide delta differences between site versions - so the relay/client can just download the differences since the last sync, but sync.bigfix.com isn’t a BES Server, it’s just a website, and I don’t think it can provide delta versions.

(I haven’t actually tried though)

JasonWalker · June 17, 2022, 2:41pm

I’m still inclined to look at firewall or proxy. It’s worth noting that a client, but not a relay service, will try using automatic proxy config based on what’s defined in Internet Explorer’s proxy settings.

https://help.hcltechsw.com/bigfix/10.0/platform/Platform/Installation/c_proxy_on_client.html#configuringaproxyconnectiononclient

csimonsen · June 20, 2022, 1:10pm

Hey @JasonWalker

This is just an update to let you know that the problem has been fixed .

In the end, it turned out to be a corrupted registry for the BigFix Relay installation. I didn’t mention this in earlier posts, but when we attempted to remove the Relay installation via the removal tool, we were always met with errors. Same thing happened if we attempted to uninstall the Relay via BigFix (error message: couldn’t find Registry key). This explains why the Client could function, but not with the Relay installed.

We opted for a full re-initialization of the underlying VM instead of combing through the registry and performed a clean install. As soon as both Client and Relay were installed on the new VM, everything started to work again from top to bottom.

Thank you for your assistance and suggestions.

TimRice · June 20, 2022, 1:44pm

TIP: You can always find the latest downloads for BigFix by visiting https://support.bigfix.com/bes/release/