If you can take a few hours of downtime, it’s far easier IMO to perform a disaster-recovery scenario.
Build the new server with a temp name & IP address, and install SQL. Then, on your old server, disable BigFix, swap names & IP addresses between your new and old servers, restore the Bigfix database & files onto the new server, reinstall Bigfix Server on the new server (the installer will treat it as an upgrade/reinstall) and run the SQL stored procedure for renaming a server.
You can practice the restore-and-upgrade scenario to get an idea of the downtime required and to practice the procedure. Until you are ready to swap the names and addresses, the old and new server can coexist, you just won’t have any clients reporting on the new server.