Automation of Deleting Orphaned Custom Fixlet / Task Content BESServer

(imported topic written by mbp911)

As I am the Owner / Primary Admin of the TEM Solution for my Enterprise, I am looking for a tool to automate removing depreciated custom content. Currently after identifying and deleting custom Fixlets / Tasks, I have to manually navigate through the wwwrootbes folder, locate the unique folder and delete it. This can be time consuming and should be part of the deletion process within the console. I understand that this is by design, however, very inefficient.

I was hopeful that the new Manage Software Distribution Packages may have this built in, but unfortunately, no.

(imported comment written by SystemAdmin)

If i understand what you are looking for, its that when you delete a fixlet that has a reference to a file, then that file should be removed from the cache on the server?

The new software distributions site (which the “Manage Software Distribution Packages” dashboard is a part of) is actually designed around this idea. No matter how many fixlets reference a file, there should be only one copy on the server, and it should be automatically removed when all the fixlets have been removed.

Are you deleting the fixlet through the dashboard, or by hand in teh console? Also, are you sure there are no other fixlets that reference that file? There is often a bit of a delay between when you delete the fixlet and when the server gets the update, and the server plug in (which you deploy from teh setup node in the SWD site) detects the change and deletes the file.

-Zak

(imported comment written by mbp911)

I have worked a little bit with the “Manage Software Distribution Packages” dashboard, but found that it was a bit disappointing as it wasn’t what I considered a solid package management solution. I did play around with deleting files and did not see the files get purged immediately. Perhaps, as you stated, there is a delay. Having said all of that, I would still like something to assist with task of cleaning up orphaned files in the uploads folder.

(imported comment written by MattBoyd)

Coincidentally, I’ve been investigating the same thing – I would like to clean out old, unused files from the Uploads folder. I’m almost finished with a VBScript that should be able to do this. Here’s the strategy I’m using:

-Using relevance, retrieve every unique sha1 hash from every Fixlet, Task, and Action. Load them into an array.

-Loop through each folder in the Uploads folder

-If the name of the folder is not in the array of sha1 hashes AND its modified time is greater than 365 days ago, delete it.

Here’s the relevance I’m using to retrieve the list of all unique sha1 hashes (I hope it’s right):

elements of union of ( set of (unique values of (following text of first 
"/Uploads/" of preceding text of last 
"/" of it) of (substrings separated by 
"%0a" of it as trimmed string) whose ((it starts with 
"download" or it starts with 
"prefetch" or it starts with 
"add prefetch item") and it contains 
"http:%2f%2f<BESSERVER>:52311/Uploads/") of action scripts of bes actions); set of (unique values of (following text of first 
"/Uploads/" of preceding text of last 
"/" of it) of (substrings separated by 
"%0a" of it as trimmed string) whose ((it starts with 
"download" or it starts with 
"prefetch" or it starts with 
"add prefetch item") and it contains 
"http:%2f%2<BESSERVER>:52311/Uploads/") of scripts of actions of bes fixlets) )

(imported comment written by mbp911)

This is very encouraging. I will check periodically to see if you have a working script for the masses.

(imported comment written by SystemAdmin)

mbp911

I have worked a little bit with the “Manage Software Distribution Packages” dashboard, but found that it was a bit disappointing as it wasn’t what I considered a solid package management solution.

Are there some specific limitations that make you disappointed (so we can work to correct them)?

mbp911

Having said all of that, I would still like something to assist with task of cleaning up orphaned files in the uploads folder.

We actually have a prototype cache management dashboard that i intend to add to our new BigFix Labs domain, sometime shortly after its released (check the Labs forum for details). The dashboard you what files are on your relay/server caches (using technique very similar to what boyd is using), and points out files that are currently in your cache, but that dont have a fixlet or action that actually uses them.

It also lets you create actions to clear your caches of specific files, or based on some criteria (such as files being older than a certain time, or over a certain size)

-Zak

(imported comment written by MattBoyd)

Thanks for the info Zak, I’m looking forward to trying that.

mbp911, below is a messy VBScript that I have started. I’m a little gun shy, so the script is only supposed to move the downloads to another location for archiving purposes, not delete them. Also, if you don’t specify an archive folder, it will just report the unused downloads. Please carefully read the comments at the top before using it.

Option Explicit 
'===================================================================== 
'     TEM Unused Uploads Cleaner 
'  Author:         Matthew Boyd (iboyd.net) 
'    Created:        09/23/2011 
'  Modified:        
'    Version: 1.0 
' 
'    Cleans up TEM file uploads that are not used in any fixlet or action  
'       and are older than 365 days. 
' 
'    Usage:  cscript.exe TEMUnusedUploads.vbs <username> <password> <url> <uploads folder> [<Archive Folder>] 
'      username = Web Reports username (Must be able to access ALL BES data!) 
'      password = Web Reports password 
'     url = Web Reports URL 
'       uploads folder = The folder where uploads are stored 
'        archive folder = The folder where unused uploads will be moved to for archiving purposes. If not specified, 
' then this script will report on the unused folders, but not move or delete them. 
'    Note: The script was tested and worked with Web Reports version  
'    8.1.617.0. 
'===================================================================== Dim ShaList Set ShaList = CreateObject (
"System.Collections.ArrayList") Dim username, password, relevance, url, moveTo, NodeList, path, totalSize, totalFolders, maxDays, fso, i, f, f2, rootfolder totalSize = 0 totalFolders = 0 maxDays = 365 If Wscript.Arguments.Length = 5 Then moveTo = Wscript.Arguments.Item(4) Else moveTo = 
"" path = Wscript.Arguments.Item(3) url = Wscript.Arguments.Item(2) username = Wscript.Arguments.Item(0) password = Wscript.Arguments.Item(1) Set fso = CreateObject(
"Scripting.FileSystemObject")   If Not fso.GetFolder(path).ParentFolder.Name = 
"wwwrootbes" Then Wscript.echo 
"Uhh... I don't think " & path & 
" is the uploads folder. The parent folder is " &  fso.GetFolder(path).ParentFolder.Name & 
", not wwwrootbes." Wscript.Quit 1 End If   
''
' This is where the relevance string is set. You can set it to any relevance you'd like. Remember, quotes in the relevance clause must be escaped by using two quotes (see below near: name of it starts with 
"CB") Function GetRelevanceResult (wrurl, rel, user, pass) Dim nl, result, objXML, objHTTP, SoapRequest 
''
'This  section builds the Soap Request Body  that will be sent to the Web Reports web server SoapRequest = 
"<?xml version="
"1.0"
" encoding="
"utf-8"
"?>" SoapRequest = SoapRequest & 
"<env:Envelope xmlns:xsi="
"http://www.w3.org/2001/XMLSchema-instance"
" xmlns:xsd="
"http://www.w3.org/2001/XMLSchema"
" xmlns:env="
"http://schemas.xmlsoap.org/soap/envelope/"
"> " SoapRequest = SoapRequest & 
"<env:Body>"   
''
'This section contains the actual call to the GetRelevanceResult method. SoapRequest = SoapRequest & 
"<GetRelevanceResult xmlns="
"https://schemas.bigfix.com/Relevance"
">" SoapRequest = SoapRequest & 
"<relevanceExpr><![CDATA["& rel & 
"]]></relevanceExpr>" 
'This is the relevance parameter. The relevance expression must be enclosed in <relevanceExpr></relevanceExpr> brackets. SoapRequest = SoapRequest & 
"<username>"& user & 
"</username>" 
'This is the username parameter in the Soap Request. Notice the credentials are part of the SOAP request. This is why all Web Reports SOAP request (or any authentication to Web Reports) should be done over HTTPS! Otherwise, they are passed in clear text. SoapRequest = SoapRequest & 
"<password>"& pass & 
"</password>" 
'This is the password parameter in the Soap Request. SoapRequest = SoapRequest & 
"</GetRelevanceResult>"   
''
'This section cloases the Soap Request Body SoapRequest = SoapRequest & 
"</env:Body>" SoapRequest = SoapRequest & 
"</env:Envelope>"   
''
'Now that we've built the Soap request, we need to send it over HTTP. This section handles that. Set objHTTP = CreateObject(
"Msxml2.XMLHTTP") 
'Create an HTTP object to send the SOAP Request to the BigFix Server objHTTP.open 
"POST", wrurl, 

false 
'Configure the HTTP object to use POST, point it to the Web Reports URL. objHTTP.setRequestHeader 
"Content-Type", 
"text/xml" objHTTP.setRequestHeader 
"SOAPAction", url & 
"GetRelevanceResultResponse" objHTTP.send SoapRequest 
'Sendthe Soap Request to the Web Reports Server, collect the result.   result = objHTTP.responseText  
'At this point we have the result of the relevance query. However, it's in XML format, with each result record between <a></a> tags.  We can extract 

this information using a DOMDocument object (see below)   
''
'This section extracts the text data from the result, which is in XML format. We do this by loading the result string into an MSxml2.DOMDocument object and then using SelectNodes with an XPATH statement to select all nodes enclodsed in <a></a> tags. Set objXML = CreateObject (
"Msxml2.DOMDocument") objXML.LoadXml result 
'Load the result from the SOAP Request     If not objXML.SelectSinglenode (
"//faultstring") is nothing then 
'If a faultstring node exists in the XML, GetRelevanceResult returned an error. Raise an exception with the source and description. Err.Raise 1,  
"GetRelevanceResult:" & objXML.SelectSingleNode(
"//faultcode").text, Trim(objXML.SelectSingleNode(
"//faultstring").text) End If   Set nl = objXML.SelectNodes(
"//a") 
'Select the nodes enclosed in <a></a> from the XML. Load into NodeList object. Set GetRelevanceResult =  nl   End Function     relevance = 
"elements of union of ( set of (unique values of (following text of first "
"/Uploads/"
" of preceding text of last "
"/"
" of it) of (substrings separated by "
"%0a"
" of it as trimmed string) whose ((it starts with "
"download"
" or it starts with "
"prefetch"
" or it starts with "
"add prefetch item"
") and it contains "
"http:%2f%2fbes.win.psu.edu:52311/Uploads/"
") of action scripts of bes actions);  set of (unique values of (following text of first "
"/Uploads/"
" of preceding text of last "
"/"
" of it) of (substrings separated by "
"%0a"
" of it as trimmed string) whose ((it starts with "
"download"
" or it starts with "
"prefetch"
" or it starts with "
"add prefetch item"
") and it contains "
"http:%2f%2fbes.win.psu.edu:52311/Uploads/"
") of scripts of actions of bes fixlets))" wscript.echo 
"Retrieving list of upload sha1 hashes that are in use" Set NodeList = GetRelevanceResult(url, relevance, username, password) wscript.echo 
"Count: " & NodeList.length For i = 0 to NodeList.length - 1 
'Iterate through each node in the NodeList object, write the text contents to the console. shaList.Add(NodeList.item(i).text) Next       Set rootfolder = fso.GetFolder(path) For Each f in rootfolder.subfolders If Not shaList.Contains(f.Name) and DateDiff(
"d", f.DateLastModified, Now()) > maxDays Then wscript.Echo 
"UNUSED: " & f.Name & vbNewLine & vbTab & 
" Last Modified: " & f.DateLastModified & vbNewLine & vbTab & 
" Size: " & Round(f.Size/(1024*1024), 2) & 
" Megabytes" For Each f2 in f.Files wscript.echo vbTab & f2.Name Next totalFolders = totalFolders + 1 totalSize = totalSize + f.Size If Not moveTo = 
"" Then wscript.echo vbTab & 
"Moving... " f.Move moveTo End If End If Next   wscript.echo 
"Total unused uploads: " & totalFolders wscript.echo 
"Total size of unused uploads: " & Round(totalSize/(1024*1024), 2) & 
" Megabytes"

Hi there, this article looks very interesting… I’m using TEM Version: 9.1.1117.0 but I get errors trying to use both the relevance and the .VBS
The relevance returns that the properties ‘bes actions’ and ‘bes fixlets’ are not defined
The VBS returns 'compilation error: expected statement’
Do you have any idea if this should run on Webreports 9.1 ?
Is there any other way to get orphnaed Uploads?

thanks a lot in advance
frank