I wouldn’t completely rule out AV. It definitely could be a contributing factor, and if you are seeing a trend micro av having a handle on the files, then that could be part of the problem, especially if this seems rather sporadic.
We weren’t being serious about that. Obviously if it is installed then it should be patched.
You keep mentioning the global folder location, but you never answer my question about what is or isn’t here:
The location i’m interested in above corresponds to the site the baseline is run from, which shouldn’t be patches for windows / updates for windows applications.
I’d also like to know more about what you are seeing in the global location. Are there any files in the directory? ( BES Client\__BESData\__Global\__Download\Updates for Windows Applications or similar )
is there anything common running on the systems with the issue?
You still haven’t answered many of my questions.
If this is really happening with many different patches, then give me a list of the patches you know this is happening with and the baseline component that run right before the one that is having the issue. Even if you don’t find a pattern, you could still provide a list of them, which you still haven’t.
If you want to reliably replicate this issue so you can investigate what happens in general:
open a CMD window
browse to a folder like C:\Program Files (x86)\BigFix Enterprise\BES Client\__BESData\CustomSite_SITENAME\__Download
run an action with a download from that site
You will get the same error you are currently getting in the agent logs
I would have to mess around with some baselines to see if I can figure out how to reproduce this on purpose using the location here: C:\Program Files (x86)\BigFix Enterprise\BES Client\__BESData\__Global\__Download\CustomSite_SITE\FOLDER but as far as I know, that isn’t the issue. I’m fairly certain you are looking in the wrong place.
This will tell you what actions have this problem, though not which action may be causing the problem:
(id of it, name of it) of bes actions whose(exists detailed statuses whose(it contains "The client is waiting for another process to release files in the action download folder.") of results of it)
The action that is causing the problem should be the one before what this session relevance identifies, but it would be nice to know which ones have this status as well to help track things down.
It seems that once the client gives up on the download, the detailed status switches to A required download failed.
The values for #### matched that location that the named folder was in.
This wasn’t with a baseline, which could have slightly different behavior, but this error message is not the same that was provided above by @RaphaelWahl which supports my theory that you are looking in the wrong folder for the issue.
In my experience, the error Cannot empty _Download directory refers to the location that is NOT under the __Global folder.
This will show where all the different __Download folders are:
pathnames of (folders of folders of folders "__Download" of folders "__Global" of it ; folders "__Download" of folders whose(name of it does not start with "__") of it) of data folder of client
This will show which of the __Download folders are not empty:
pathnames of it whose(exists descendants of it) of (folders "__Download" of folders whose(name of it does not start with "__") of it) of data folder of client
This is normal right after an action has been run. The client does not try to empty these folders right away. Generally not until another action tries to run in the same site. It is at that time that you get issues if the folder doesn’t empty.
I simply couldn’t reproduce the issue anymore today whatever I did.
Tried to push the same baseline to same kind of device and lock the folder using cmd to browse to the folder.
In the mean time, parallel to this thread, two things happened:
We managed to reproduce the issue last week and sent a procmon trace file to IBM. We haven’t gotten any further than to determine that the folder C:\Program Files (x86)\BigFix Enterprise\BES Client\__BESData\__Global\__Download\CustomSite_xxxxx is locked by the SYSTEM process and that the Antivirus is not the cause seeing that all the BESData subdirs are excluded from the antivirus scanning.
2)We found a workaround to keep our devices secure in the mean time, that is, only to deploy the critical updates this month and not the important ones.
The strange thing is, that this issue is occurring only with one kind of device, not an entire site (1 of 2 kind of mobile devices within a custom site) so we’re trying to determine now what is unique in that specific environment that isn’t present in the other.
But I’m pretty sure it’s C:\Program Files (x86)\BigFix Enterprise\BES Client\__BESData\__Global\__Download\CustomSite_xxxxx that gets locked because it’s its content that I cannot manually delete when the isssue occurs. I can only manually delete it when I stop the BESClient service.
I still think you are incorrect about this. These are the files that the BES Client is trying to copy from, not to.
If stopping the BESClient is sufficient to allow you to delete the files, then that means that only the BES client has a lock on those files, which means those files are NOT the problem. The problem you are having is occuring because something OTHER than the BESClient has a lock on a file.
You are still looking in the wrong place.
This is definitely interesting and a good thing to look into. I would recommend looking at what software is common between them that is not common on other systems.
I still highly suspect that there is a small set of patches that are at fault here.
The BESClient is a SYSTEM process, so you should expect that the files are locked by SYSTEM/BES Client. This is completely normal.
The longer this thread goes, the more strongly I feel that the issue is not in the folder that you are looking into and you are not providing me any evidence to make me feel differently. I’m more and more certain over time that the issue is somewhere like this: C:\Program Files (x86)\BigFix Enterprise\BES Client\__BESData\CustomSite_SITENAME\__Download
It is very likely that there is a patch or something creating a file in a download folder and not releasing it’s lock on it right away.
The following relevance MIGHT give you a list of files that are locked:
files whose(not exists lines of it) of (folders of folders of folders "__Download" of folders "__Global" of it ; folders "__Download" of folders whose(name of it does not start with "__") of it) of data folder of client
Did you provide a link to this thread in the PMR so that support can look over it?
Also, you might want to make sure that the systems with the problem have enough free hard drive space at the time of the error. The BigFix client should give a different error if the hard drive is out of space, but it is worth checking if that is the case.
This says that the office installer finished immediately. This most likely means the office installer spawned a NEW process and ended the original and thus while the agent thinks its done, the installer is still working and still locking the file.
Good catch. Normally I’d expect to see some delay between Command started and Command succeeded with some timing info and possibly other events in between.
As far as I know, they are using IBM patch content, not custom content. This should mean that it is the IBM content that needs fixed to use Completion=job for cases like this, though if they identify the problematic patches then a custom copy could be created using this override to test… but they haven’t identified a set of problematic patches.