We have a new installation of BigFix (9.5.2) which we’ve spent a lot of time installing and configuring. Prior to the install, we asked IBM if UNC installs were supported to which they said they were (we’re using a VM (ICS with NetApp all SSD)).
We performed the install successfully a month ago and all has been well. The only BigFix application components that are on the UNC (SMB3.0) is the BES Server\BESReportsServer and BES Server\wwrootbes (everything else is on C). The BES Installers directory is there as well but I believe that is inconsequential.
When we performed a manual upgrade to 9.5.3, meaning running the Server setup.exe file from within the BES Installers directory, it immediately failed; but not before it hosed the whole system (and we had to restore).
We deal with a VAR and they were able to reproduce the problem stating that it is an issue in the MSI that has an issue with a UNC.
That all being said, I’ve elected to open a PMR to find out if IBM unequivocally supports UNC paths and if they don’t, I was given poor information from them originally. If they do, which I am hoping very much is the case, that they can fix the script.
Rebuilding is not an option I want to consider do to all the work, but perhaps moving the BES Server folder as mentioned above to a local drive would be “easier”? I’m unsure on what all within the application would have to change to reference the new location.
I can’t contribute a ton on this topic (IBM Support is definitely the right route) but out of curiosity does the (elevated?) account you’re using to run the server installer have access to that UNC path?
It does. We have a service account that we used to perform the initial install (though the BigFix services are using the standard Local System). The account as well as the computer itself has access to the Shares. The install worked as expected. SQL, which is locally installed on the server is using the service account and its database is pointing to a UNC share; but as far as the BigFix app, the database location in that regards should not be relevant.
Logged in with the service account (the same and only account used for the original install), I downloaded the update from the support.bigfix.com website. I ran that,which generated the installers. I then ran the Server setup.exe file.
How naive am I to think that I could stop all the BES Services, copy the contents from the UNC to a lettered drive, go into [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\BigFix] and replace any \server\share instances with E:, then start the services and everything works?
If you move items that an installer places, the uninstaller will fail as it only knows the location that it moves things too. This is standard for an installer.
Additionally a UNC path may require credentials to reach and thus the services need the credentials (so run as a specific user) to be able to reach the drives.
Also I’m presuming these are network drives, that will drop the performance of the servers as they hit these drives a lot.
Not sure how a dedicated 10GB network using SMB3.0 to a NetApp with SSD storage is slow. Works well for our 1000+ VMs. I don’t think these two areas are part of the installer; its almost like moving a relay cache. I’m also using the same account I used for the initial install. Anyway, I’m working with the VAR to talk about options.
Obviously if I could just reinstall and point to the same database I would go that route (so I can keep all my custom content and other customizations).
So…at this point, are you trying to keep the installation on the UNC path, or move it to local storage?
Just a guess, the MSI installation may be triggering the Windows Installer service to elevate portions. In that case, the installer would be running under the root server’s LOCALSYSTEM account, not as your logged-in user account or the BigFix Services “log on as” account. Check whether the UNC path has Full Control permissions for the computer account - either by adding DOMAIN\COMPUTERNAME$ to both the Share and NTFS permissions, or by adding a group in which the computer account is a member.
If the UNC were hosted by a Windows machine, I’d also say to check the User Rights assignments for “Access this computer from the network” and ensure that the BigFix Server’s computer account is included. There may be an equivalent for your NetApp device (that’s using Samba under the hood, is it not?)
At this point I’m going to have iSCSI LUNs connected to the VM like follows:
D Drive 10GB SQL Logs
F Drive 200GB AppCache
G Drive 40GB SQL Database
The database/logs are easy enough, just unmount the database and do a file copy.
The part that I have to think about, and is explained earlier in this post, is moving the two directories within the BES Server directory to the F drive.
Its not like I can just re-install BigFix and point it to my existing database and be good to go.
The moral of this story is not to use UNC paths. IBM has now identified this as a bug (aka I identified it). Simple test? Install to a UNC path and then immediately try to uninstall; it fails.
I had to blow away the whole VM and literally install from scratch; even using the BF Remove Tool caused issues.
A three week lesson learned… two take aways:
Don’t use UNC’s for BES install. If using a VM/NetApp like me, use LUNs (iSCSI).
If you have to rebuild the BES, install a fresh OS and don’t trust the remove tool (even if you totally cleared every regkey and file you can think of)
it should have been possible to treat it like a migration. Set up a new VM and then move everything into place from the old VM but into the new locations. I’m not sure what this would have helped with.
Network storage will be much slower in terms of latency and IOPS as compared to PCI Express NVMe storage.
Performance matters a lot for some of the storage, and much less for the rest. The DB stuff and FillDB matter the most.
Absolutely no way to migrate from a UNC path install. As mentioned, you can’t even uninstall BigFix when succesfully installed to a UNC path. With regards to the storage I’m using (NetApp Cluster SSD with dedicated 10GB), the 250K IOPs I’m getting seems to work fine.
You misunderstand. You could have treated it like a server migration. Set up the new VM, then migrate the files and DB from the old server to the new one. As long as you can access the files, it should be possible. The BigFix installer / uninstaller is irrelevant.
If you haven’t done much with the old server with the UNC install, then it wouldn’t have saved you much work and in that case it was better to just start fresh.
That should definitely work fine and you probably don’t need any more than that, I’m just pointing out that there are relatively inexpensive SSDs with ~20Gbit/s throughput and over 400k IOPS with latency far lower than would ever be possible over Ethernet (iSCSI / SMB).
If you had multiple of these SSDs in a server and split the workload up among them, then you would have even more maximum throughput.
This kind of performance is really only concern in extreme scenarios.
