10.0.9 upgrade - no fun

[vk.khurava]

Just reporting my experience upgrading BigFix to version 10.0.9-

Today I upgraded our BigFix infrastructure from version 10.0.4 to version 10.0.9, and it was rife with issues for the master, console, relays, and clients.

I upgraded our other BigFix infrastructure from 10.0.4 to 10.0.8 last month, and it went without a hitch, save for a problem with download certificates and a problem with WebUI NMO’s ability to take action.

Error -1: MS Visual C++ failure, it was auto fixed when I initiated another run with my NT ID.

Error -2: MS ODBC Driver 17 Failure - I was able to fix it by upgrading our SSMS from 18.X to 19.X

Error -3: Console & Client installer was not able to copy masthead during upgrade - Once I click OK it just proceesed further but failed to start the client so had to manually copy the masthead for both console & client.

Error -4: BESClient installer was failed to update - to resolve this had to uninstall client & relay completely.

Error -5: BESRelay failed to install, had to remove BESClient & BESClient helper to fix the issue.

Error -6: Auto BESRelay upgrade ended with weird exit code on multiple relays - still under investigation.

[vk.khurava]

Error -7: WebUI NMO cant take action, anyone got hotfix from HCL? PFYI - this is known bug with v10.0.8 as well, for that its pending with HCL product support team to fix.

Related KBs from HCL side -
https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0104196
https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0102980

But given that there was a recognized issue with version 10.0.8 (still not fixed even with provided hotfix for v10.0.8), it is odd that it was left out of version 10.0.9. I assume that the crew is not working well with the introduction of new content.

[DanieleColi]

Hi @vk.khurava, sorry for your inconvenience, but let me try to clarify:

• KB0102980 affects the WebUI component, so it cannot be fixed with the v10.0.9 as it upgrades the Platform components only; it will be fixed with the next release of the WebUI (you can reach the support for an hotfix)

• KB0104196 was discovered after the release of v10.0.9, so it was impossible to fix it on v10.0.9: you can reach the support team for an hotfix as well

Regards.

[vk.khurava]

https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0102980

Thanks @DanieleColi ! I already raised case # CS0385094 with support team & its still pending for both of the infra with v10.0.8 & v10.0.9

We have infra of 200K+ endpoints, also with 250+ operators & most of them are on WebUI !!

[DanieleColi]

You are one of our priority: stay tuned!

[FatScottishGuy]

Eeeeek! This is where my upgrade plan was going - 10.0.4 to 10.0.9

10.0.8 was a mess for me, I had to abandon it in dev and revert my snapshot due to the cert issues etc…

[JasonWalker]

I suspect all of the first six issues could be related. For the Visual C++ and ODBC installations, we simply execute the command-line installers with silent switches, their success or failure is outside the control of the BES installers.

If these were failing on only one server I’d suspect an OS corruption, particularly in the MSI installation database/registry/ or component-based servicing, but having the failure on multiple Relays leads me to wonder whether you have some fine-grained security policy or endpoint protection that are blocking the installers? I’d look to anything that has an explicit Deny permission on the filesystem for LocalSystem, Administrators, or any groups to which your accounts are members; as well as any endpoint protection or privileged-access modules that are monitoring the BigFix directories.

I’d like to hear more about your configuration and troubleshooting. I believe you’re correct to open the support incident and the team may be able to help, especially given debug logs from the installers, but given problems installing both VC++ and ODBC I think this will be more troubleshooting your Windows installations than BigFix itself.

[vk.khurava]

I suspect all of the first six issues could be related. For the Visual C++ and ODBC installations, we simply execute the command-line installers with silent switches, their success or failure is outside the control of the BES installers.

Yeah ! I attempted to use the automated component upgrade fixlet in my Dev/DR environment, but it somehow lost track of it and I was left with no choice except to launch the manual installers, which ultimately saved me. However, there must be some code connected to the ODBC Driver v17 that checks the SQL Server Management Studio version.

I used some online SQL support that directed me to an outdated version of SSMS and instructions for upgrading it.

If these were failing on only one server I’d suspect an OS corruption, particularly in the MSI installation database/registry/ or component-based servicing, but having the failure on multiple Relays leads me to wonder whether you have some fine-grained security policy or endpoint protection that are blocking the installers? I’d look to anything that has an explicit Deny permission on the filesystem for LocalSystem, Administrators, or any groups to which your accounts are members; as well as any endpoint protection or privileged-access modules that are monitoring the BigFix directories.

My ID is local administrator on the box, and before starting the upgrade, I also put them in local approval mode within our endpoint security controls, so I seriously doubt that. Otherwise, I might have received the message from Carbon Black that “Your Exe is not authorized to run,” which is common with recent releases of executable from the HCL side due to improper signing.
I had this issue with 10.0.9 as well; I’ve brought it up to Roberto (HCL Support), who may contact the internal HCL Development team.

Additionally, it is not specific to one computer for the relay or client. When I attempted to manually upgrade my Top level relays, I encountered numerous issues & had to install both component from scratch.

Currently, we’re only focusing on saving the days due to the numerous teams chasing after our WebUI issues.

[vk.khurava]

10.0.8 was a mess for me, I had to abandon it in dev and revert my snapshot due to the cert issues etc…

Lucky you I deployed it in our another BigFix infra & we are still struggling with Download certs & WebUI NMO take action issues.

[fermt]

We are facing the same issue after upgrading to 10.0.9. Have you got any solution from support?

[vk.khurava]

@fermt if you are talking about WebUi issue, HCL provided hotfix to resolve the issue, please raise the case.