I didn’t go down that path, but yes if you can retrieve the DPAPI key that would be useful to me, thanks!
We have opened a case with Citrix. The initial contact was not hopeful. We are waiting for an escalated answer. I can’t believe that HCL cannot setup a Citrix PVS environment.
I am not sure Bigfix Management realizes what a big problem this is. We have 300 of these Citrix PVS servers. We are going to have to remove the Bigfix agent from these systems if this problem cannot be resolved. We cannot continue to create 300 duplicate instances every day. This means that the viability of Bigfix is in question if it cannot manage all of our servers.
I still think there needs to be a solution to be looked in - BigFix should be able to accommodate the support of such machines! This is where infrastructure technology is most-likely going to go with the “Infrastructure as code” concept making strides where machines will be immutable (cattle-vs-pets) and machines will just be “rebuilt” on every release, rather than “patched”/“maintained”…
The above discussion is great in the sense of explaining WHY the workaround is not working with current design of the agent but that doesn’t take away from the fact that we need permanent solution. Today’s Citrix PVS, tomorrow will be ALL machines provisioned to CI/CD pipelines running on all virtualizations and so on…
@ageorgiev - you mentioned a workaround. What workaround are you referring to? Do you mean the persistent partition with saving the Bigfix registry before restart and the reapplying it at startup?
I meant what you guys are trying by restoring reg keys/certs/files/folder/etc. I don’t consider this a “solution” but more of a “workaround” under current agent/rootserver design, even if you get it working. Solution would be a native out-of-box support for this kind of technologies where we configure what are the minimum requirements to be matched on registration requests and the root server is smart enough to not reset client IDs but reuse them and will require a redesign of the processes.
HCL should allocate resources to build the lab environment to simulate and test different virtualization technologies in today and moving forward to tomorrow world. This helps to deal with different issues and how the BigFix agent behaves especially dealing with duplicate computer matter which is a big issue in virtualization environment. My technical and logical sense is that regardless how the same virtual server is restored with preinstalled agent from snapshot or system restore method, it is still the same virtual server (guy) with the same UUID, hostname, IP and MAC. Why the bigfix client needs to reset itself to create a new ID? That is the part not making sense. Think of the bigger picture, a person moved out of the house and moved back to the same house later or different house located in different city or country. Isn’t this person carrying the same identify information (security ID, same driver license #, etc…) Why would you issue a new identify information to this person? Same logic apply here when dealing with duplicate computer matter.
The workaround provided to manually take a backup of the registry key and other information to a backup location and then manually restored them before restarting the besclient service in order to avoid duplication issue, is completely not a good workaround per say. Adding extract workload for someone to do these steps are not doable especially dealing with hundreds of duplicate computers daily. Not a happy workaround.
HCL can’t really rely on its customers to try testing out this workaround and hoping it will work without conducting a fully tested in HCL lab environment. After it is fully tested and verified it is 100% working in the logical sense, please document it the step-by-step how to get it working.
The main difficulty there is that the UUID, hostname, IP, and MAC are all reported by the client, and a malicious client could impersonate any other machine and report whatever values it wants up to the Relay/Root Server. Our protection against that is the client authentication certificate that is used to sign that report when it is made, and removing the certificate authentication reduces how much the server should trust the given client report.
We’re still considering options, but must take care as there’s a security implication to any changes we make in this space.
As far as a tested/proven procedure, the ClientIdentityMatch option is a tested & proven procedure - but it’s a solution that solves a different problem, the VM Snapshot/Rollback or Disk Write Filter/DeepFreeze scenario. That doesn’t apply in the Citrix PVS scenario because the machine Citrix provisions really is a new operating system base every time, without preserving the system’s keys that make the endpoint unique.
I can’t commit to whether or when a workaround could be provided, but if there is one it would be very specific to this scenario, would in some way reduce the security of the server infrastructure, and care would need to be taken to avoid applying such workaround generally, one would only perform it on the specific systems on an as-needed basis.
2 Likes
When we migrate a root server to new hardware, we use the ServerKeyTool to decrypt and save the otherwise encrypted configuration keys. I wonder if a similar approach could be developed for managing the client’s private key.
Well, can I suggest a potential security measure? Maybe additional advanced system settings that not only enables/configures this level of ClientIdentityMatch functionality but also allows you to configure the approved subnets from which to accept such registration requests. For example:
- ClientIdentityMatch=200
- ClientIdentityMatchCriteria=UUID,hostname,IP Address,MAC Address
- ClientIdentityMatchCIDR=1.2.3.0/24,2.3.4.0/24,3.4.5.0/24
This way, it only allows the level 200 on requests coming from those subnets and for all others it treats it as ClientIdentityMatch=100
@JasonWalker - I used a tool called CredentialsFileView to view the credentials on my test PVS server. The credentials did not change after a restart. If you would like to work with me on my test environment, that would be great.
I performed several restarts on the PVS server yesterday and the Bigfix client ID did not change. I have no idea why I am not getting a new ID now.
As we are facing similar issue it would interesting to know if there is any progress on this topic?
Yes, on the endpoint there’s a Citrix service that actually does restore the machine identity correctly, but BESClient was starting before that restore was complete. Adding a startup delay to the BES Client of several minutes (I think they used 15 minutes in that case) appears to resolve it.
3 Likes
Thanks, Sounds good. We will give it a try.
We are still working to get it tested (just internal resource prioritization thing) but same was officially posted as KB article.
1 Like
I am still working with support on this issue. The workaround which is to save the Bigfix registry to a persistent drive and then restore it on restart of the PVS worker server, works until there is a promotion on a master server. Then all of the workers generate a new ID. However, if you re-install the Bigfix agent on the master after the promotion and before restarting the worker servers, the ID is not recreated. Of course, the master servers get a new ID. At least, we avoid 150+ worker servers getting a new ID. Additionally, once the workers start getting a new ID, it does not stop until the Bigfix agent is re-installed on the master.
It all makes absolutely no sense to me. Especially, when Crowdstrike has such a simple answer. You install the Crowdstrike agent with a parameter that says to use the host name as the ID. There is no overhead of a persistent partition, shutdown scripts, and startup scripts.
1 Like
Why do I keep hearing from people on the HCL Bigfix side, that no one else in the world is having the problem I am having with the regeneration of client IDs in the Citrix PVS environment? It sounds like from this thread that others are having the same problem.
I have also heard that no one uses Bigfix in the Citrix PVS environment. I do not use it for delivering updates, but I do use Bigfix Inventory to capture the hardware and software inventory. Bigfix Inventory is dependent on the Bigfix agent working correctly.
Not true, we have the same exact problem but we are just moving very slow towards resolution (in fact, we had done some calculations and because of it the size of our BFEnterprise database is 4 times bigger than what would be expected for environment of our size, so it is not just impact on managing these devices which is bad enough but clear implication on performance & stability of the environment). We have a positive testing on a few “clusters” but haven’t deployed it yet globally. Our Citrix team are migrating from PVS to MCS which as far as the problem is concerned is not making any difference and changing some of the other tooling around managing the devices, which is slowing things down significantly but it is certainly a problem and I am keeping an eye on your posts (much appreciated by the way!). That said, we haven’t seen the issue after promotion of the master server but it is possible that we just haven’t rolled out to large enough group to notice it.
The issue with promotion of the master only seems to happen in our production environment where there are two groups and therefore two masters. In our DR environment, we are not seeing the issue.
Yes, can confirm. We have the same problem; We had a case with HCL that was going on for months. In the end, we just gave up and now we do a manual cleaning everyday, but we have hundreds of duplications each day, all of them from our Citrix Environment.
Until we have a working solution or any documentation, we’ll be doing this manually.