This sounds like a user privelages issue. I would recommend validating that:
the user you are logged into the server has local administrator privileges
the user has access to the remote database and has sysadmin rights in SQL
You are running the installation as admin (right click-launch as admin)
Alternatively, you can create a dedicated service account for this, and apply the above logic accordingly.
I also highly recommend that you run the BESRemove tool if you have attempted multiple installations, and also re-boot the machine before starting again. Ensure that you have backed up your .pvk, .crt and masthead files and select the masthead instead of a license authorisation file when you run the installer again.
Link to latest BESRemove tool: https://software.bigfix.com/download/bes/100/util/BESRemove-10.0.1.41.exe
Also make sure there is no firewall blocking communication between your BigFix server, and the SQL server (you should be able to telnet on 1433)
You can also use ODBC to validate/test connection.
Related to the SQL sysadmin rights, this can be reduced to dbo for the BFEnterprise and BESReporting databases after the install. You may need to elevate back to sysadmin when you are running a future upgrade though.