When creating your baseline, uncheck the box “Baseline will be relevant on applicable computers where this component is relevant” for your restart service task.
That way the baseline won’t become relevant unless at least one of your Fixlets is relevant.
The checkbox is visible when you expand a component on the components tab of the Edit Baseline dialog.