Downloading slowness in Nated environment for All OS platforms

Hi Bigfix Masters,

We are having issues after deploying the patches onto all OS platforms(Windows,centos,rhel,suse,Ubuntu)

Issues we are facing when we are deploying the patches onto the systems of the OS, it’s taking a lot of time to download onto Server/relays/clients.

Issues onto the Windows OS servers:-

Earlier we were using the WSUS for windows patching and now we have moved to bigfix but while deploying patches through bigfix its taking 6-7 hours to download patches and execute,

AS we are onto the testing phase we have tested patching activity onto the 10 different servers but we are getting the same result every time.

similarly, when we are deploying the same amount of the patches through WSUS, the deployment is getting completed within 20-30 minutes. and we are deploying the same numbers of patches through WSUS what we are deploying through BigFix.

Issues onto Non-Windows server(Rhel,Suse,Centos,Ubuntu):

We are using the Plugins method to deploy to patches onto non-windows servers. Similarly, we get the same result the patches are taking a lot of time to download and execute.

But whenever we are using Redhat satellite we able to deploy all applicable patches within 30-45 minutes and when we deployed 5 -8 patches through Bigfix its taken almost 2-3 hours to download and execute.

Also, we have opened the support case with HCL support, but we have not got any proper resolution.

Bellows are some brief information about our environment.

The environment is nated where UDP protocol is blocked and alls clients are communicating over the Internet with nated public IP with relays /Severs.

Kindly help us to resolve these issues.

Thanks In Advance!!
Arjit

1 Like

The first thing to try is enabling Persistent Connections, described at https://help.hcltechsw.com/bigfix/9.5/platform/Platform/Config/c_persistenconn.html

That can help quite a bit in the NAT situation you describe. Prior to Persistent Connections, we would have recommended Command Polling, which is also still an option.

hi @JasonWalker COmmand polling is enabled onto our environment, but the result was the same. We will initiate the Persistent settings onto relay and clients as described into the link and update you.

What had you set for the Command Poll Interval?

Command polling interval we have set to 15 minutes.

Well since you’re doing a PoC, i want to assure you that what you’re seeing is not normal and I’m confident you’ll be able to find the problem (maybe with a little help from your friends here). There’s really no scale that should be that slow.

If you’d like to get some hands-on help we could possibly engage our Tech Advisors to evaluate what’s going on (paging @DanPaquette )

To troubleshoot, I’d start with the top-level and work your way down. How long does it take to send a patch to

  • the root server itself
  • a top-level relay (reporting directly to the root)
  • a second-level relay (reporting to a too-level relay)
  • a “normal” endpoint (connected to a second-level relay).

You could always be suffering from “normal” network configuration issues - like a link speed/duplex mismatch between one of your servers and switch, or on any of the switchport uplinks, or MTU or routing issues, but I’ll try to stick to the bigfix part for now.

During each of these tests above, check for network and disk utilization (easiest way is via Resource Monitor on Windows) to see whether the actual on-wire speed is what you’d expect from the link.

An excerpt from the client and relay logs would be helpful. One thing to check would be large gaps between you taking the action, amd the client receiving the action; or large gaps between reporting the action Relevant and requesting the downloads.

1 Like

I have sent PM for contact info.

Bigfix root server is taking a long time to download the patches. 400MB of patches taken around 1.5 hours to download onto Bigfix server.

and when we are downloading the patches directly onto the root server it’s downloading very quickly and But during the action deployment, we are observing very slowness in downloading through Bigfix.

We have only 2 relays in our environment and both relays are directly reporting to Bigfix root server we don’t have any top-level relays and all the clients are authenticating and communicating with Both relays.

There is no issue with network speed as we have high speed of internet connection. as we are able to download the same patch manually onto the Bigfix root server and relays with very little time, but during the actions deployment phase its taking long time to download through Bigfix.

Also, we did not find any high resource utilization during the deployment of action Bigfix root server and both relays in Resource monitor.

We will initiate the Persistent settings onto relay and clients as described into the link and update you.

Thanks
Arjit

It’s interesting that even the patch download from Internet to the root server is taking a while. Do you have some kind of proxy configuration in place, that maybe getting configured automatically with the browser, but would need to be configured manually for the BES Root Server service (via BESAdminTool)?

Persistent Connections would help with clients learning that new actions were sent, and informing them when the Relay has cached the download and the client can retrieve the patch from the Relay. Let us know whether that helps too.

Hi Jason,

Patches download from internet is very fast on Bigfix server and relays, 400MB of patches downloaded in 2-3 minutes,

but during the action Deployment 400 MB patches taken 1.5 hours to get cached onto the Bigfix Server/relays.

We don’t have proxy environment, so i don’t think so that we need to do configuration for proxy.

We will enable to Persistent connection onto relays and clients and will update you outcome.

Thanks
Arjit

@JasonWalker we tested by enabling the persistent connection settings onto the relay and client machines but no luck, 26MB of patches taken around 40 Mintus to download and transfer to the client machine.

I’d try to narrow down whether the problem is downloading files, or getting responses from the clients.

Try taking an empty action, with no file downloads, and see how long it takes the clients to report back statuses.

Some client logs showing when they receive, process, and report the action would be helpful.

@JasonWalker
Whenever we initiate the blank action its gets completed very quickly.also we are not observing any delay in execution in any deployment.but issue with downloading.

Thanks
Arjit

Ok that’s a very good sign, often the most difficult part is getting notifications down to the clients so it’s good you don’t seem to have a problem there.

Let’s stick to Windows patches first, since Windows and RHEL downloads are very different.

How are you measuring when the download gets cached on the root server? You may be watching the download status in the Console (where it should show “cached on server” for the downloads very quickly. Or you may be looking at the Relay Diagnostics page for the root server. Or watching the server’s wwwrootbes\bfmirror\downloads\sha1 folder directly to see the new download file getting cached.

I’m not sure I was clear on your earlier response - if you send an action to just the root server to download & apply a patch, is that also very slow?

There is a lot we can do to make downloads more efficient, in terms of increasing the cache sizes on server, relay, and client, and configuring clients to download directly from Internet instead of the Relay/root server, if there is a slow WAN pipe, but I’d want to start by just sending download actions to the root server and see how long those take. For downloads as small as you describe no tuning should be needed yet.

We are measuring from the Relay diagnostic page and console in action summary “Cached on server”.

we will deploy the patches onto the Bigfix server and update the status.

Thanks
Arjit

1 Like