Any official doc to address duplicate computer caused by Citrix "Citrix Provisioning Services"?

[yorkly1]

Good day folks, I am aware that this topic has been discussed many times in the pass on how to resolve duplicate computer generated by “Citrix Provisioning Services”. Then there is a new setting called “clientIdentityMatch” available in latest version of BigFix. However, there are customers reported back that this setting does not work on Citrix environment.

Is the a real youtube video demonstrated or going through the whole procedure to prove this setting will work on Citrix environment?

Is it just setting up “clientIdentityMatch=100” on BigFix console (1 single step) and it will fix the issue?

Then you have steps described in this doc link below are so confused as it seems that there are too many steps involved just to enable this setting. Can someone re-define the steps more clearly to address this duplicate computer issue in Citrix environment? Thanks and appreciate it.

https://help.hcltechsw.com/bigfix/9.5/platform/Platform/Installation/t_preserving_bundling_when_clientreinstalled.html

==============

clientIdentityMatch
This advanced option can help you to avoid having duplicate computer entries when the endpoints are detected as possible clones by the BigFix Server. Starting from BigFix Version 9.5.7, the BigFix Server can use the existing computer information to try to match the identity of a Client and reassign the same ComputerID to computers that might have been rolled back or restored. To guarantee the correct applicability of this option, it is necessary that the following components are at least at 9.5.7 level:

    The BigFix Server.
    All Clients that will apply the option.
    All Relays that are in the configuration tree between the Clients and the Server.

If clientIdentityMatch=0, the BigFix Server performs strict clone detection. This means that, if the BigFix Server receives a registration request from a Client that was rolled back or restored, the Server invalidates the old ComputerID, resets the old Client definition, and assigns a new ComputerID to the registering Client. This is the default behavior and is the same way the BigFix Servers earlier than V9.5.7 operate.

If clientIdentityMatch=100, the BigFix Server performs an additional check before assigning a new ComputerID to a registering Client to avoid creating cloned computer entries. This means that the BigFix Server tries to determine if the information about the rolled-back Client sufficiently matches the data held for that ComputerID. If the identity of the Client is matched, the Client keeps using the old ComputerID and its identity is not reset.

[JasonWalker]

As I understand it, when provisioning cloned VMs each VM is a new computer. New MAC address, potentially new hostname, etc.
The Client identity match is more useful when rolling back to a VM snapshot, where the computer ID and most importantly the client identification certificate is present in the snapshot.

I don’t actually have a Citrix machine to check, but if I’m mistaken and Citrix is just rolling back to a machine snapshot repeatedly, you might have some luck with installing the BESClient, registering to the deployment, setting Client identity match, and then snapshotting the VM as a reference at that point.
You’ll need to keep unique snapshots.of each VM though - if you are cloning a single reference VM to multiple target VMs, each of those targets would register a new computer object every time.

[yorkly1]

Thanks Jason. HCL is a big company who owns BigFix product. I am hoping that it should also have a Citrix environment to test this setting below in the lab to ensure it also works as well. Then write up more clearly steps and perhaps a youtube video showing step-by-step to demonstrate that it works or how it works.

clientIdentityMatch=100

Are you able to engage the development team to provide how this setting would work or not work for Citrix environment to prevent duplicate computer? Then create a youtube video on how to implement it.

Thanks.

[JasonWalker]

If you engage with your TA we may be able to help set it up on your environment. I’ve access to VMWare, Azure, AWS, and GCP but afraid I don’t have easy access to Citrix. There are also a lot of variations based on however you have your VDI provisioning set up.

[mbartosh]

Hi yorkly1,

I am also having an issue with the Citrix PVS environment where every night when the PVS servers are restarted, a new computer ID is created. Although, oddly I think I have seen the ID preserved in some odd cases. We have setup a persistent W: drive and save the Bigfix registry during shutdown and then restore it back at startup. We also have ClientIdentityMatch=100.

Currently, what I am seeing is the following message in the BES Client log: Scheduling client reset; Computer id changed to 551744474 The odd thing is that this message does not appear in the BES client log until about 6 hours after the Citrix PVS server is restarted. I think it is going to be important to understand what is triggering this message.

The biggest problem for me with the creation of a new computer ID is that Bigfix Inventory sees the computer as a new computer and I have to go through all of the software classification all over again. Also, since BFI never removes computers, I am adding about 300 servers per day to BFI. This is a lot of overhead.

[yorkly1]

Hi Mike, this feature to resolve duplicate computer is NOT doable as described in this doc link below.

https://help.hcltechsw.com/bigfix/9.5/platform/Platform/Installation/t_preserving_bundling_when_clientreinstalled.html

Staring for step 5 to step 8 are completely adding a lot more overhead as you are seeing this as well to those affect Citrix PVS virtual servers. If you think of it that those steps are required to do through for each servers manually for every time that a new Citrix PVS virtual server is created. There are approx. up to 300+ servers in my environment created each day. I can’t be doing those steps for each 300+ servers. I need another full-time body to do this job for me.

I just gave up to implement this solution as it is completely NOT reasonable. It has to be a better way to handle duplicate computer.

I ended up choosing a better option to remove the BigFix Client installation on the golden image where these PVS servers are spawning from since there is no IBM product installed anyway and each PVS servers is shutdown at the end of the day anyway. No point to have a BigFix Client installed on them to track IBM software usage. That is my side of the story NOT to use such feature.

[JasonWalker]

Not being entirely familiar with PVS, I have to ask - when one of the managed servers is restarted, is it reprovisioned with all the same identity info (MAC addresses, serial number, UUID)?

When you run the tool that exports the current BESClient registry keys, are you also preserving the __BESData and KeyStorage folders? Are you stopping the BESClient service before saving these, and restoring them before starting the BESClient service?

Can you post a BESClient log where the server is requesting the client reset?

[yorkly1]

Hi Jason, it’s good to hear from you. To answer your question “is it reprovisioned with all the same identity info (MAC addresses, serial number, UUID)”, I would have to dig more into it if I still see these PVS virtual servers are still coming in since I have implemented a solution to remove the BigFix Client last week.

I will give you more information if I found some duplicate servers.

Technically and logically speaking and thinking, why would the BigFix Client needs to re-create a new ID if someone restore the snapshot? That part is really not making sense. If a virtual server with BigFix Client installed and snapshot was taken as usual in virtual environment, it should and must remain the same BigFix Client ID after a snapshot restored. Think of it, it is coming from the same server image. It can’t be recreating a new client ID after each snapshot restored. In fact, I can understand if BigFix Client was reinstalled and a new client ID is created. That makes a lot more sense.

On top of that, I would expect to see the same information such as VMWare UUID, MAC address, IP and hostname after the snapshot restored. So why does it need to create a new client ID?

It is just those steps described to deal with duplicate computers are completely NOT making sense to do extra work manually (preserving and restoring…).

HCL is a big company as IBM. Personally speaking when this issue is reported by a customer, I would expect it to conduct a full test to see how the Citrix PVS works and behaves.

HCL needs to come up a better logical way to deal with duplicate computer issue as it is also affecting other virtualization vendors such as VMWare and Hyper-V.

[mbartosh]

20220723.bes (512.2 KB)

My understanding of the PVS environment is that it boots off a master image. At some point I believe there is a rename of the computer name back to the original name. I am trying to prove that the MAC address, serial number and UUID remain the same. There was not a property for UUID so I created one. Is this correct?

UUID of hardware

We have local GPOs that run a shutdown script and start up script. We are using a persistent partition so the __Besdata and and KeyStorage folders do not change. Although, the KeyStorage folder changes when the computer ID is reset.

Shutdown Script
Net stop besclient

sc config besclient start=demand

reg export “HKLM\SOFTWARE\WOW6432Node\Bigfix” “w:\Bigfix.reg” /Y

Startup Script

reg import “w:\Bigfix.reg”

sc config besclient start=auto

net start besclient

I posted a copy of the log. I had to rename the extension to .BES.

Interestingly, I just restarted the server I am testing with and did not get a duplicate ID. I thought the issue was reproducible every time.

[mbartosh]

Hi Jason,

I have proven that the UUID, MAC address, and serial number do not change, but the computer ID does.

[JasonWalker]

I think you’ll need a support incident escalated to development in that case. If everything’s being preserved as described, the client shouldn’t be resetting. I think they’ll need some extended debug logging to figure out what’s happening with your clients.

[JasonWalker]

One more thing - can you block the startup script on one of the clients and verify the BESClient service is actually stopped and set to manual?

One oddity on the ‘sc config’ command line is that there must be a space after ‘start=’, i.e.

sc config BESClient start= demand

And

sc config BESClient start= auto

It might also be handy to have the shutdown and startup scripts echo the service states to an output file for logging, with sc query BESClient and sc qconfig BESClient. The statuses before & after disabling/enabling might be interesting.

[mbartosh]

I am unable to change the startup script because it is in the master image. I am sure that problem is not with the service starting prematurely because I have to manually change the path of besclient.exe for the service to start. This is because on most servers the BES Client path is W:\Bigfix Enterprise. As part of testing, I changed the path to W:\Bigfix Enterprise\BES Client. This shows that the registry is coming from the master image.

Does renaming a computer trigger a new computer ID?

I opened a case with support Friday afternoon.

[JasonWalker]

Renaming does not trigger a new client ID. The conditions I know of that should generate a new ID are

• Client certificate revoked on the server
• ‘client reset’ ActionScript command is sent
• Computer sends an older-versioned report and server has seen a newer report (this is what the ClientIdentityMatch logic is supposed to avoid on client snapshot rollback)

Hostname isn’t involved, it’s common to have multiple active computers with the same hostnames. I’m not even sure changing Serial or UUID resets the computer.

[mbartosh]

I am seeing a number of Event ID 36871 in the system event log at startup. A fatal error occurred while creating an SSL client credential. The internal error state is 10013. I don’t think this is referring to the BES Client, but would it have anything to do with this issue?

I neglected to mention that all the servers in the Citrix PVS environment are Windows Server 2012R2.

[ageorgiev]

I did have a support on both Inventory & Platform side that went on for more than 6 months and never went anywhere. Eventually it went to PMs because “BigFix client is not design to support this use case of devices” from Platform side and Inventory the answer was “This should be fixed on Platform side and all of the issues on Inventory will just work” and now I periodically ask for updates but nothing is forthcoming as far as I know…

The above is exactly what Support had me tried and we were never successful. I would be very interested if you do find a workaround but at the end of the day it is just that. We really need a solution…

Ideal solution is another level of ClientIdentityMatch (let’s say 500) where you can configure UUID/SerialNumber/etc to be the only criterias for the root server to match to establish that a registration request is coming from the same exact machine irrespective of the fact that reg keys/files/folders are not there! The root server is obviously issuing client certificates, so surely it should be able to “re-issue” a cert and obviously the ClientIdentityMatch is designed to overcome certain client-side configs/files missing and recognize that it doesn’t need to re-register, it just doesn’t go in-depth enough.

Just FYI, we are investigating another repercussion of this same issue - the fact that our BFEnterprise is extremely bloated to the magnitude of performance/stability impacting. We keep all computer data of deleted machines for 180 days for audit purposes and just a quick math of an example 1000 of those citrix instances (600 reset weekly and 400 reset daily), where we do retain the BES Client folder post reset.

• 600 machines reset weekly, means we would be keeping in the DB roughly 25-26 duplicates for each (180 days / 7 days a week = 25.7) = we retain the equivalent of 15600 additional endpoints worth of data in the BFEnterprise data accumulated over 180 days

• 400 reset daily, means we would be keeping in the DB roughly 180 duplicates for each (one per each day) = we retain the equivalent of 72000 additional endpoints worth of data in the BFEnterprise data accumulated over 180 days

So those 1000 instances would equate to data in BFEnterprise equal to = 87600 endpoints

[yorkly1]

What customers really need is an automation process to fix this matter behind the scene without adding extra manual procedures such as backing up registry keys first to preserve the information and then restore the old information back after the VMs are either restored from the snapshot or creating by the Citrix PVS model.

This matter is consuming more database table usage just to store those duplicate computer data in the ILMT or BFI databases daily.

[JasonWalker]

Yes, I understand the inconvenience, but it’s a much more complex problem than just handling the legitimate duplicate computer use-case.

When the client registers, it generates a client auth certificate unique to that endpoint. The private key never leaves the client (as it shouldn’t), and uniquely identifies it to the server.

If we allowed another client, without that certificate/key pair, to overwrite the existing clients’ object on the server, that opens up possibilities that a malicious client could overwrite existing computers with fake data. There might be difficulty distinguishing real clients from imposters in that case.

To be clear, I’m not on the platform dev team and there are probably other complexities here, I’m just pointing out that it’s not as simple as “always overwrite the existing machine record”.

At least in part, not accepting an older report with an old Report Sequence Number and forcing a client reset in that case also helps protect against replay attack.

[mbartosh]

I am still trying to understand why the client thinks it needs to be reset. The keys in the keystorage folder have not changed since they are on a persistent partition. L2 found the following in the debug log on the client.

Failed automatic client authentication key exchange with server message: There’s already a public key associated with that id
** Resetting computer ID**

Why wouldn’t there already be a public key associated with that ID? It was an existing client. This message does not make sense to me.

[gus]

Hi @mbartosh, I’ve also done some extensive work in this space. My recommendation is to review the before and after of the same endpoint for these 3 client relevance queries. Do all of these 3 queries persist and are exactly the same after the endpoint has been restarted?

Q: virtual of hardware

Q: info of client

Q: concatenation "_" of elements of set of mac addresses of ipv4or6 interfaces whose ( exists mac address of it and length of mac address of it = 17 ) of adapters of network

-Gus