In planning for DR, I am considering the utility of having a pre-configured “recovery” server at the ready.
The various BigFix services would be disabled, of course, but the “recovery” server would be online with SQL Server already installed. All the registry settings would be in place. It would also house synchronized copies of the files in the various BigFix Server folders, such as UploadManagerData and wwwrootbes, among others.
Thoughts, comments, or suggestions?
virtual or physical?
Physical is plan A, but virtual is not off the table.
so there might be an issue around decrypting these files on the original machine:
EncryptedServerSigningKey,
EncryptedClientCAKey,
EncryptedAPIServerKey
EncryptedPlatKey,
EncryptedWebUICAKey
it wouldn’t be hard to script and should be included in your backup.
If it were me, I wouldn’t invest in a physical machine doing nothing… I’d back up (plus decrypt) all necessary items daily and in a DS situation, spin up a VM. You could have it up and runnning in hours.
Thanks - I’ve noted that I’ll need the Server Signing Key Tool from the WIki …
There’s definitely a cost consideration with a stand-by physical server, but it wouldn’t be all that different from running a DSA server.
We went through this exercise (had DSA for years and then considered stand-by) but the delay of especially DB replication, cost of the server + actual maintenance was just not efficient. What we ended up doing is install ESXi directly on the primary root server as designed/spec’ed out and configure a single VM using all the physical’s resources and pass-through disks. That allowed us to essentially use standard VMware replication methodology for DR (we use Zerto but there plenty others around), so it mirrors the root server VM “as is” to another shared host in our situation (it’s not as powerful but DR is not meant to be), so if there is anything a flip of the button and fails over to another host the same exact VM with the same exact data/db. Just removes the hassle completely and very cost-effective too!
There’s a new official DR method on its way out which is way better than the HA one available just now but we can’t use it as it needs additional (expensive) SQL licences.
Our DR method is:
DNS entry for prod that we switch the IP address on if DR needs to be invoked
DR Server being completely blank (no BigFix installed)
SQL server for it on another server where there is SQL Log Shipping from the Prod
We then RoboCopy the required files (including custom site exports for Analysis and Fixlets) over to the DR server every 6 hours
When DR hits we stop log shipping, change the SQL server from Read Only to production, modify the IP address on the DNS entry and then fresh install BigFix. As all the relays are configured to use a hostname, everything falls over nicely (though not always the quickly).
If you can afford the SQL Always On then this would be the best way to go but it’s super expensive and you’ll need a few servers with it too.
We do have an internal solution paper on this. We can discuss, along with product management.
Boyd, if you want to contact me directly we can review and see what makes sense. HA/DR solutions tend to be very business specific, and there can be good reasons for either physical or virtual deployments. The cost implications are much different.
Thx.