DownloadWhitelist.txt (Again)

[D.Dean]

I have found a ton of contradictory information in the forums about this file and the format. I found entries as new as 6 years ago that say they do not need to have the Perl script syntax and KBs that say we do

Download error: The requested URL does not pass this deployment's download...

Sidd in 2016 said he found it did not have to have the escape characters.

DownloadWhitelist.txt issue

Hi, I am using external repository to store some tools which I prefetch in actionscript. But action is not able to cache tool exe from repository on IEM server. Error : The requested URL does not pass this deployment's download whitelist URL : http://bigfix-repo.mydomain.net/utilities/unzip.exe Contents in DownloadWhitelist.txt http.*bigfix-repo\.mydomain\.net/.* Still it is not white listing my prefetch URL.

Other resources in my company have said they do not use escape characters.

So, as of this week we did not patch Java newer than 8u201. We now have a customer that has enterprise licensing for Java and I am setting up patching for them.

I setup the manual caching. Setup the folder, deployed the _BESClient_AllowCustomRepoDownloads settings. I can copy the link in the error and put it in the browser, yes, it downloads the file So the only issue is the whitelist even though I have our server in the whitelist.

what gives?

[trn]

Whitelist is all http, download is https???

[D.Dean]

I bet that is the issue…

The _BESClient_AllowCustomRepoDownloads probably has https in it. Let me check and get back to you.

[trn]

Long time since I had to adjust ours, but I do have a suspicion that it is escaped

[JasonWalker]

It definitely uses regular expressions, but where I think some are confused is that the regex “.” for “match any character” also matches the literal “.” symbol in a URL.

Best practice would be to use the regex escape ’\.’ because the intended server “www.mydomain.com” could also be matched against a fake server “wwwXmydomain.com” if the dots are not escaped in the regex.

[D.Dean]

It was the “_BESClient_AllowCustomRepoDownloads” setting that contained the https

Removed the S and it worked

As you can see, the whitelist file does not have the regular expressions.

@JasonWalker see where the confusion comes from?

I will test it with regex next week when I return. I really want to know if it still works. As for now, it is working

[JasonWalker]

It’s…a regex already.
In the DownloadWhiteList.txt, the value you have

http://myserver.mydomain.com:52311/Repository/.*

is already a regular expression. The ‘.*’ at the end means ‘match any character (.), any number of times (*)’. That’s why you don’t have to specify the jre-8u391-windows-i586.exe in the whitelist, that whole part of the string is matched by the regex ‘.*’

In the 'myserver.mydomain.com' portion, each ‘.’ means to match ‘any character, one time’ - and in expected usage, the one character it actually matches happens to be the literal ‘.’. It could also match myserverXmydomain-com, or any other single character that fall in the positions of the dots…but as it is, it’s a regular expression that does work, it just allows more than intended.

Where your download is failing to match the whitelist, I agree it’s the ‘s’ on ‘https’

q: exists matches(regex("http://myserver.mydomain.com:52311/Repository/.*")) of "http://myserver.mydomain.com:52311/Repository/jre-8u391-windows-i586.exe"
A: True
T: 0.407 ms
I: singular boolean

q: exists matches(regex("http://myserver.mydomain.com:52311/Repository/.*")) of "https://myserver.mydomain.com:52311/Repository/jre-8u391-windows-i586.exe"
A: False
T: 0.296 ms
I: singular boolean

q: exists matches(regex("http://myserver.mydomain.com:52311/Repository/.*")) of "http://myserver-mydomain-com:52311/Repository/jre-8u391-windows-i586.exe"
A: True
T: 0.101 ms
I: singular boolean

For the flexibility of both ‘http’ and ‘https’, and the stricter checking of the hostname, the best value to use in the DownloadWhiteList.txt would be

https?://myserver\.mydomain\.com:52311/Repository/.*

The ? on https? makes the last character optional - it matches ‘http’ or 'https’
Normall a ‘.’ in a regex will ‘match any single character’, so we escape them… '\.' will only match the literal dot symbol.
/ and : are not special symbols to regex, so we don’t have to escape them.
At the end of the url, the ‘.*’ pattern means ‘match any characters, any number of times’ so as long as the first part of the pattern matches, your URLs can end with anything.

Also, I did recently encounter an issue that these are case-sensitive regular expressions. These examples won’t match HTTP://MyServer.MyDomain.Com:52311/repository/java.exe

[D.Dean]

Not dismissing your advice, I am just saying there is a lot of confusion out there and that caused me a lot of grief today.

The KB I posted did say it should have the escape characters. I did modify my whitelist but that didnt help because I broke it with the https in the “_BESClient_AllowCustomRepoDownloads” setting.

May I suggest that you take your last response and put it in Tips and Tricks? Include the part about putting https in the _BESClient_AllowCustomRepoDownloads setting.