External Prefetch

Content Authoring
1

So I need some help intepreting what exactly is happening in this action script. A majority if not all of the Windows Updates are executed with an external prefetch:

Example: Office 2010 SP2 KB2687455

prefetch officesp2010-kb2687455-fullfile-x86-en-us.exe sha1:faa6e4d481e8ae534cbf47afc46c22dc4f43ee3c size:669173640 http://download.microsoft.com/download/4/9/9/499A1B0D-6FF3-466E-888E-DC7D7C788A61/officesp2010-kb2687455-fullfile-x86-en-us.exe sha256:156a3cb15148549f6ddc6ca4e3c7e9e1784b40c1b61f1dd21e80719deed8aceb

waithidden __Download\officesp2010-kb2687455-fullfile-x86-en-us.exe /quiet /norestart

action may require restart "faa6e4d481e8ae534cbf47afc46c22dc4f43ee3c"

So, with this, it seems to download directly from MS and not from the internal relays. Is that correct? I’m confused because then when we run the action, in the downloads header it says it’s cached:

So, what is exactly happening here?

Thanks!

2

The prefetch is sent to the relays for execution from the client. If the relay doesn’t have it, it requests it from either the core server or, if the settings are set, the internet to fulfill the request.

3

details of the client settings that change the download behaviour are here
Search for _BESClient_Download_Direct

1 Like
4

The prefetch statement has the URL to the file location on Microsoft’s server.

This is the default behavior:

The client processes the prefetch, and looks for a file with a matching SHA1 in it’s cache.

If the client has a matching file with the same SHA1 in it’s cache, then it checks the file size to see if that matches as well. The client may also verify that the SHA256 matches.

If the client does not have a matching SHA1 in it’s cache, then it requests it from it’s parent relay.

The parent relay does the same process, checking it’s cache, and then requesting it from it’s parent relay.

This process continues until the root server is reached. If the root server does not have the download in it’s cache, then it reaches out to Microsoft and downloads it, placing it in it’s cache, and verifying the file size, SHA1, SHA256, etc…

Once the root server has this cached, it cascades down the chain by notifying the child relay that it has the download cached by sending it a UDP packet. The child relay downloads it, verifies it, and then notifies it’s children that requested the file using a UDP packet. If any children don’t receive the UDP packet, then they will wait until their next download polling interval, and ask their parent if they have the download available again. The children will poll their parents, backing off each time by doubling the polling interval until they are told that the download is available through the polling or through UDP and download it, unless they are told that the download is not available too many times.

Eventually the client that originally requested the file will download it from it’s parent relay, and then it will run the action once all prefetches have been downloaded and verified.

If a 2nd client requests the same file from the same relay, the relay will serve it from it’s cache if the relay’s cache has not rolled over and deleted the file due to lack of space.

In almost all cases, the download should only ever happen once from Microsoft to the Root Server.

If you do not want your Root Server to be involved with downloading files from Microsoft / Others, you can tell a Top Level Relay that it should handle internet downloads instead of asking it’s parent for them. This would reduce the storage needs of the Root Server.

The “cached on server” message you see in the console means that the root server has the download already cached from Microsoft successfully.

You may need to increase the cache sizes of your top level relays and the webcache of your root server to keep them from rolling over and deleting downloads due to lack of space.

From the perspective of your relays and clients, there is no such thing as an “internal prefetch”. They are all external, they just sometimes happen to be hosted on the root server.

You could host all of your “internal” installer files on a webserver that is only accessible by your root server and/or top level relays, and then point all of your prefetches at that location instead of uploading them to the root server upload cache. This would cause these downloads to go into the root server’s webcache which is managed automatically by the root server, which reduces the burden of cleaning up the root server’s manual upload cache.