This depends how you have set the action in general…
If you used computer groups then you can select Dynamically Target by Property > All Computers > By Group and select the groups you want making sure to hold down the Ctrl key to select multiple.
If, instead, you are doing it by relevance of a specific service running (which I think you are) then I would create a custom computer group with the servers you want to exclude from the job and update the relevance of the action to exclude those servers
AND (not (exists true whose (if true then (exists setting "__Group_0_Group_Name_Here" whose (value of it is "True") of client) else false)))