There is a private key that is the root of trust for all bigfix content in your environment. Only the root server has a copy of it. Access to this private key and BESAdmin is secured with a password.
The matching public key is in the masthead file that is required for clients to join and trust the bigfix root server. With this public key they can validate all content the bigfix root server provides. Since all content is signed with the private key and validated with this public key, it isn’t possible for someone to modify bigfix content and actions and get downstream clients to accept it, even if they had compromised a bigfix relay and directly modified files. This means even without any encryption, you can’t get BigFix clients to accept or run something arbitrary.
In the default setup, any client with the masthead can join a bigfix environment, but this is no longer the case if you enable “Client Authentication” on the relays. At this point, a client can only join bigfix if they have a prior connection and have attained a client certificate. This is the “right” way to have BigFix relays that allow connections from outside the corp network. (DMZ relays) You could also enable this for internal relays, but then you must provide an extra mechanism for all clients to join bigfix, which could be a password or a one time token.
I believe all content is gathered from relays over SSL, as well as any client reports sent up to relays is over SSL. But there is actually an additional protection available called Message Encryption. If enabled in BESAdmin, then client reports are additionally encrypted at rest such that only the root server or top level relays can decrypt them. This added protection is such that if a BigFix relay were to be compromised, then client reports being passed to that relay could not be decrypted unless it happened to be a “decrypting relay” which is generally only the root server or top level relays.
Relays do send UDP notifications to clients. These notifications themselves are not encrypted or secured, but they also do not contain any content. They only serve to let clients know that there is something new for it to gather, at which point the client reaches out over a secure TCP connection to get the content from the relay. This allows BigFix clients to be responsive without having to wait around for them to check for new content on an interval and instead check nearly real time.
There is some good related discussion about BigFix DMZ / Cloud DMZ Relays and how to secure them starting here: Relay Selection Over VPN with Split Tunneling
Someone from the BigFix platform team could give more details on how it works, but there should also be some documentation about it. CC: @AlanM