I’ll defer to Support on these, since they’ve been able to do some analysis from your logs…
… I’m not sure that we document the behavior for “a system that was once DSA, but has had the DSA functionality removed”. I think you’re correct though, and a system that has ever been DSA will probably need to consider DSA ramifications ever after.
I’ve done it both ways. If there’s already a clean BigFix install, restoring the data over it (including re-encrypting the keys for the new host) overwrites the default data and works fine; if you do the restore first and then the new install, when the new install runs it reuses the existing data (in fact each server upgrade performs an uninstall of the old version followed by an install of the new version, which is kind of scary to watch when you do it manually).
I always like to do dry-runs first, testing that we can get the database connected and the new server reporting to itself, then have a separate outage for a “final” restore of the files and database to the new machine (skipping the encrypted keys on that final copy, as they should have already been restored & fixed up during the dry-runs).
Edit:
If you do the “restore first, then install the software”, I don’t recall whether you’re prompted for masthead options on the new install; if so, be sure to specify install using an existing masthead.