9.5.2 problem with detecting restart?

[JasonWalker]

So on one of these clients, about 90 minutes ago I removed all of the values at HKLM\software\wow6432Node\BigFix\EnterpriseClient\BESPendingRestart and restarted the BES Client. It did not recreate any of the entries in that key, but has also not updated any of the action statuses to Complete or Failed; they all still show Pending Restart.

How does the BES Client detect that a reboot has actually occurred and the action statuses need to be re-evaluated? I thought that the absence of the values under BESPendingRestart would have done that, but so far no joy.

[JasonWalker]

I’ve also cleared the values from __BESData\Global__PendingRestartList and restarted the BES Client service, no change 30 minutes later.

[AlanM]

There are many places in the system the client looks to determine if the OS has a pending restart. The values you removed were just indicating that specific actions had a pending restart and only used by the BigFix agent, not by the OS.

The OS keeps track of things all over the place to determine what it needs to fix up on a restart to complete installs. And it may take multiple restarts to clear it up depending on the work being done.

[JasonWalker]

But how does the BES Client detect that a restart has occurred and it needs to re-evaluate the pending actions? Does it compare the system boot time to the age of the _PendingRestartList XML file or something else?

[strawgate]

Have you checked pending file operations? http://www-01.ibm.com/support/docview.wss?uid=swg21506002

The TEM Client also looks in the registry under the key “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager”. If the value “PendingFileRenameOperations” exists, then this indicates that Windows will do some file operations during the next restart.

[JasonWalker]

I’m not too worried about the BES Client flagging that the system has a Pending Restart. There may have been a PendingFileRenameOperations value. I can go look at that to get more data, but that’s not really what I’m looking for. I’m looking for the post-reboot evaluation of the individual pending restarts.

For instance, one of my actions put a restart flag out there - action requires restart PatchWindowReboot. After the system rebooted, even that simple flag was not cleared and the Action that triggered it was still in the Pending Restart state. Even if something else in the system like PendingFIleRenameOperations is flagging a reboot is needed, I’d still expect at least that one Action to change its result to Complete after a reboot occurs.

I haven’t had enough time to know whether it’s caused because this is the first reboot after upgrading to 9.5.2, or whether I have a bigger problem with reboot detection.

[strawgate]

My understanding is that actions and those keys will remain in pending restart until the bes client confirms a lack of pending restart. Only then are the bes pending restart keys cleared.

I have seen HP printer drivers among other things populate that key and all actions stay at pending restart until you fix the issue or add an exception to the client ( _BESClient_ActionManager_PendingRestartExclusions) for a specific pending rename value

[JasonWalker]

Good to know! So if I understand it correctly, any Pending Restart holds up all Pending Restarts…

[AlanM]

There’s more than just that location

    HKLM\System\CurrentControlSet\Control\Session Manager  value: PendingFileRenameOperations
    HKLM\Software\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired
    HKLM\Software\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending
    HKLM\Software\Microsoft\Updates  value: UpdateExeVolatile

[JasonWalker]

I do see a number of entries in the PendingFileRenameOperations registry key.

So I may have a Chicken-and-Egg problem. During my Patch Window, I have a Policy Action running to trigger the actual reboots. The logic on this Policy Action checks whether any of my baselines are still trying to execute, and when they are finished it executes to trigger a reboot. The Reboot policy action has a Relevance that includes
not pending restart "PatchWindowRestartPoller"

The Action Script is simple -
action requires restart "PatchWindowRestartPoller"

So the system runs its patch baselines, and when they are finished this restart polling fixlet executes. It sets the pending restart, and its post-action triggers the reboot (with a delay for the user). I’m depending on the reboot to clear the pending restart for PatchWindowRestartPoller. But if another external check (like PendingFileRenameOperations) prevents this action’s pending restart from getting cleared, this restart polling fixlet won’t execute again to trigger the second reboot.

I can’t say I’ve encountered this as a problem before, so before I start changing my polling logic, I just want to be clear that I’m understanding this correctly. Is it true that after rebooting, the BES Client will not clear the BESPendingRestart registry keys and re-evaluate any of the actions that were previously Pending Restart, if there are any other items that are still Pending Restart?

I understand the client should be relevant for the “Pending Restart - Not Triggered by a BES Action” fixlet, but I think it still should have moved ahead with clearing the older BESPendingRestart values.

[strawgate]

The issue here is something is continuously setting the pending file rename operations. After a reboot this key should be clear. If it’s continuously populated you run into the issue you’re seeing. You’ll want to figure out why it’s not cleared after a reboot. These pending operations are performed by Windows on boot – i believe this key is always cleared on boot, something else just happens to be setting it again before the bigfix client starts.

My understanding is the client has to see the state as needing a restart, and then see the client as not needing a restart (all areas that could indicate the need for a pending restart need to be clear) before considering a reboot to have occurred.

Maybe Alan will swing in and correct me

[Entaille]

in case it’s helpful, here’s a good powershell script that will tell you the source of the pending restart.

https://gallery.technet.microsoft.com/scriptcenter/Get-PendingReboot-Query-bdb79542

[AlanM]

There are some annoying programs that will populate things immediately after a restart and we have a setting which you can ignore them if you have such a program.

If you “name” a pending restart then the settings mentioned earlier were the ones that would have that “named” type of restart, but they will NOT be cleared on a restart unless all the “restarts” are cleared out on a true restart as we don’t know which restart flag applies to which thing.

[JasonWalker]

Thanks @strawgate and @AlanM. Now that I understand the behavior I think I can accommodate it. This isn’t happening constantly for me, I think it’s a particular patch that has required two reboots rather than one; all of the Pending Rename operations look like they deal with RDP Printing (mxdwdrv.dll, tspring.dll, xpssvcs.dll, unidrvui.dll, unidrv.dll, unires.dll, fxsdrv.dll, fxsui.dll, fxswzrd.dll, fxstiff.dll, fxsres.dll, and fxsapi.dll).

Knowing now that my original “action requires restart” will not be cleared until the client detects that no restarts are pending at all, I can modify the logic on my reboot poller action to take that into account and trigger a second restart when necessary.

[fermt]

I can modify the logic on my reboot poller action to take that into account and trigger a second restart when necessary

What did you mean by “trigger a second restart when necessary”?

Did you get a solution for this? I am facing the same with some devices.

[JasonWalker]

I just have an action that I run during my patch windows to reboot the systems after the patches are installed; I trigger a second action on it when systems still report “Pending Restart” after the first reboot.

[cjwolford]

Hi @JasonWalker We run into this from time to time while pushing updates to servers. Before we have just had the Baseline action trigger a reboot after applying the updates but occasionally they hang on “pending restart”

It seems to happen when we’re also pushing software/symantec updates which falls in line with the PendingFileRenameOperations being populated.

I’ve tried setting the _BESClient_ActionManager_PendingRestartExclusions setting to exclude the symantec values to see if that helps.

How did you finally accomplish triggering a second reboot? I’m interested in a smarter way of triggering the post patch reboots.

I’ve had issues letting it reapply the “Restart Needed” task during it’s maintenance window because it either reboots too many times or still constrains the task somehow.

[strawgate]

BigFix doesn’t normally switch between actions while running, it normally completes one full action before running the second one. For this reason if you schedule the reboot as a second action i do not believe it will reboot between each patch of your earlier baseline.

After the reboot occurs the original baseline action will start re-evaluation and any patches which required a reboot before installing will now install.

You can schedule the reboot action to wait xx minutes between retries or reapplications if you want to allow it to reboot more than once during the window and you can rely on pending restart for the relevance to only reapply the reboot if it’s required.

[jbmaz]

This is a really bad design decision on the BigFix developers part.This is a constant complaint by our server ops team. BigFix should track restart status independently of other apps that may litter the registry with values. Either that or they should add a BigFix-only tracked restart status to the action view AND a “PendingRestart-Other than BigFix” status into two columns. Otherwise, our guys don’t know if the restart even happened without manually checking total up-time.

[JasonWalker]

I’m not defending it and the behavior can be annoying, but I hope this is helpful…

There are other detections available (such as the fixlets for “Pending Restart - not triggered by a BES Action” and “Pending Restart - triggered by a BES Action”, at least I thino those are the names).

Bear in mind, BigFix isn’t the product that needs a restart to complete, it is the patch installers that require a reboot, and BigFix can’t really tell whether they’re finished until the installers themselves clear the reboot flags.

A number of products will fail to install at all if there is a pending restart from something else, and I believe they use the same detection logic.

As for checking uptime individually, there is a ‘System Boot Time’ property you might want to add to your column view in the console, that might help.