This question may be ILMT specific but it happens on the BigFix side of things so I figure I’d ask it here.
The ILMT scans (hardware & software, and maybe even the VM Manager Tool) all write to directories under UploadManagerData ( C:\Program Files (x86)\BigFix Enterprise\BES Server\UploadManagerData to be precise).
On a customer’s BigFix installation, I saw a warning in BigFix Warnings (under BigFix Management) that this directory on the BigFix server was full. The description of this warning, and associated fixlet, said that I needed to increase this directory’s maximum size and that the server would appear to stall until this was done. The default for this setting was 20MB. I checked the properties for this directory tree and there were 170MB of files in it. I changed the setting to 1GB. The fixlet ran and the warning went away. I didn’t restart any BigFix services after I did this.
My questions:
Exactly what does it mean that “the server will stall”? Could someone describe what I would see when the server “stalls”?
How is this directory managed? Does it keep growing or is there a periodic cleanup task that I am supposed to do? (I never saw a fixlet to clean up this directory tree).
Should I have restarted any of the BigFix services after I modified this setting?
I saw a corresponding setting for the maximum number of files under the UploadManagerData directory tree but the number of files present (21K) did not exceed this setting (30K), so I left it alone.
Are there recommended values for these two settings?
Just noticed this thread and I was having some problems with the uploadmanagerdata directory.
All the SUA scan data from endpoints is being uploaded to the UploadManagerData\BufferDir\sha1 folder.
I have set the following settings.
Now my sha1 directory is 1.50gb already. I have noticed that it contains files from 3rd Feb. My assumption was upload manager will start cleaning up the files but that is clearly not happening. Am i missing something? I even restarted besclient and server after changing these settings.
Also i notice there are some files in BufferDir/Temp folder from 5th Feb. I did not get why those are not processed yet by upload manager?
It is on you to keep the upload manager folder structure clean.
We run a daily script to move data out of upload manager to several different destinations, then clear those files to make way for the next set the following night.
Be careful if you are using upload manager for multiple types of data that they all get to their end destination successfully. Splitting it out can be a bit tricky. We add some descriptive tags/metadata to some of the stuff we upload to facilitate easier splitting out on the back end.
I think the cleanupHours setting only gets rid of partially uploaded files, not the ones that were successful. I believe they sit inside the SHA1 directory itself. I normally have a handful of these after a week or so hanging out. I just started purging all of the SHA1 directory/file contents recently, mine tends to accumulate about 10MB a day.
Ok this is something that I did not know. I will proceed to manually cleanup that folder. The dbo.uploads table should give an idea which ones are successfully uploaded. And then once the SUA import is successful, cleanup can be done from the sha1 folder based on that.
I looked at those settings. They really don’t answer my questions. I understand how things are getting in there (results of uploading ILMT scans and information from the VM Manager Tool). I don’t understand when things get out of UploadManagerData, if they do so at all.
Does BigFix manage the UploadManagerData folder somehow or is it up to applications that use this folder to clean up after themselves? In my specific case of ILMT (and @rohit for SUA), we see this directory growing, which could eventually cause the BigFix warning about this directory being full to occur, but haven’t seen anything about how to manage this. Do I simply set the max size and max file count values for UploadManagerData to an arbitrarily high number and forget about it, or do I have to do some cleanup work myself? There’s no guidance for this that I have seen. And for ILMT, what are the implications of cleaning this folder up myself (for any ILMT IBMers who follow this forum)?
I did read somewhere that the iem filldb process monitors the sha1 folder of uploadmanager data and it then updates the dbo.uploads which have information about these files. SUA and probably ILMT as well reads this table and gets information about the files.
But it seems that the sha1 folder is not cleaned up by IEM itself.
My guess is that BESRelayUploadManager_BufferDirectoryMaxCount and BESRelayUploadManager_BufferDirectoryMaxSize settings would affect the relay which is sending the data to server. If the values specified are hit, there will be no more uploads coming in. What I am unsure of is whether filldb itself will stop updating dbo.uploads if the max values are reached.
I did some checking internally and the expected behavior of the relays in regards to its UploadManager folder is:
On the relay’s the UploadManagerData\BufferDir\sha1 directory should generally be empty as its contents are forwarded to the relay’s parent. If the ‘UploadManagerData\BufferDir\sha1’ folder contains many files that do not appear to be moving, it is possible that the parent is not accepting new uploads. Check the following 3 UploadManager settings on the parent relay (or Root) to make sure they are adjusted accordingly:
_BESRelay_UploadManager_BufferDirectoryMaxSize
Default (8.2+): 1GB (1024 * 1024 * 1024)
Default (prior to 8.2): 20MB (20 * 1024 * 1024)
Max value (8.0+): 2^64-1 bytes.
Max value (prior to 8.0): 2^31-1 bytes (2147483647) bug#18939.
_BESRelay_UploadManager_CompressedFileMaxSize
Default: 20 * 1024 * 1024
Max value (pre-8.0) is 2^31-1 (2147483647) bug#18939.
Max value (post-8.0) is 2^64-1.
_BESRelay_UploadManager_BufferDirectoryMaxCount
Default: 10,000
Max value (pre-8.0) is 2^31-1 (2147483647) bug#18939.
Max value (post-8.0) is 2^64-1.
What about the behavior of the main server when the uploads reach it’s final destination in the SHA1 directory under the UploadsManagerData folder? Do they every purge or anything to that effect? SAlso, how does SUA interact with that, do those purge as well, or do we need to manually purge this? If so, how do we ensure that SUA uploads have been imported so we don’t miss the scanned data? Thank you
They do not purge. I believe for the SUA scan uploads, they get replaced/overwritten by subsequent uploads. So the folder’s will always contain the the latest SUA scan results received.
One potential problem that could happen, is if computers reset themselves overtime, or new computers join and leave, nothing clears out the data to those obsolete computer instances.
I think there may be some custom solutions out there that might be implemented to purge out old Upload Manager data. Maybe on BigFix.me.
There may be 3rd party scripts/programs that were also developed that could help to clear out these directories of old files as well.
@BigFixNinja, thanks for checking into this for us.
What I’m concluding is, if BigFix is being used for ILMT/BFI, absent any IBM recommendations, we really should consider increasing these settings to an arbitrarily large number to prevent the UploadManagerData folder warning from occurring:
Your posting seems to answer the question about how the UploadManagerData directory is managed.
What about this question: When the “UploadManagerData directory full warning” occurs, its description says that the server will stall until this limit is increased (or data in UploadManagerData is deleted manually). Exactly what does it mean that “the server will stall”? Could someone describe what I would see when the server “stalls”?
I upvoted this. Also I have noticed that if you go ahead and delete all the contents of the uploadmanagerdata/sha1 folder, and do an import, SUA shows missing scan results for all the computers. So it expects the scan files to be sitting in that directory for existing computers. These files are referenced via dbo.uploads table in BFEnterprise.
Thanks @rohit, I’ll have to check this table for myself and see what’s there.
I have an open PMR regarding a larger than normal number of computers that ILMT reports “missing (software) scan results” and maybe this has something to do with it.
Are you actually missing results from computers for SUA, or do you believe that the error message is there only because the scan files are no longer under the UploadManagerData folder? I don’t manage the SUA piece personally but I have been clearing out that directory for a few weeks now, and have not heard back from the SUA guys. Just curious if this is just cosmetic or clearing out the directory actually causes a data issue.
The issue with my PMR is this … For some endpoints, ILMT’s scan results widget reports missing software scan results even though when I log onto these endpoints, I can see that the software scan has run and I can see the software scan data files in the UploadManagerData folder. I don’t know how ILMT determines that a software scan has run but its results are missing. Maybe it’s a caching issue, where new results aren’t getting put in UploadManagerData, but I don’t know.
I’m not deleting anything from UploadManagerData unless I get guidance from IBM that it is OK to do so. Nothing definitive out of the PMR yet.
(ILMT is SUA “lite” where it only sees IBM software; SUA can see other vendors’ software),
If the scans are periodic , it would replace the files you clean out. I have observed that after cleaning out the uploadmanagerdata folder and running a data import, I get lot of missing scan results.