TIP: multi-tenancy Part 1

[D.Dean]

We are an MSP with a 130+ customers and tens of thousands of systems. Multi-tenancy is a beast with every customer having unique needs and being managed by hundreds of Desktop and Server System Admins. Automation is key and we are currently managing all daily patching activities with just two full time employees.

Each of our customers have a site for just their systems and unique customer policies. Customer systems are Identified by an Entity ID and added to the site with that ID. Each customer has unique policies which covers their unique needs.
Outside of patching, we are also managing security tool installs as well as configuration settings for vulnerability management and hardening.

This tip is how we automate security tool installs with BigFix to replace GPO software installs. Many customers are moving away from installing tools via GPO for several reasons, including many systems not being on the domain or are full time, remote, off the internal network.

We have a global policy for the tool deployment with relevance that a certain registry key must exist with a certain value, in the registry. The policy is always running dynamically. To make a system applicable we simply add the key to the system.

When we onboard a customer, we deploy the key to a few systems at first, as a test. Then we add the key to the customer custom policy so all new systems that call in get the key, and therefore the tool.

Most BigFix client installs are automated, through a GPO and remote install, using a script. The GPO is unavoidable but with BigFix installing other tools, this may be the only GPO install in the environment.

I am often asked we don’t use the unmanaged assets and client deployment tool. With 130+ customers, and 150+ domains, each having unique user names and passwords, the necessary access to the environment to install anything is untenable. We do setup a GPO for BigFix client install that runs from the DFS. The script has some AI in it to build the clientsettings.cfg on the fly, to find known good relays and add the customers Entity ID. When we have a new agent version of a new version of the script, we update it with BigFix. No need to log into the customers environment.

I also have a policy in place to copy the ActionSite.afxm file from the local client folder (On DCs) to the DFS when the date is older, to keep it up to date.

The customer specific dat file contains the customer’s relays, domain and entity ID. The install script uses the relays from the list to check to see if they are responding and have the port open. It also checks the domain the system is on compared to the domain listed in the dat file. This is a failsafe to prevent a customer dat file from one customer being used at another customer. If the domain is the same, the entity ID is added as a client property so when it calls in, it will be added to the correct customer site.

I have a lot of “multi-tenant” oriented processes and configurations. It is critical we do not allow a system from one customer to show in reports for another customer. As I said, multi-tenancy is a beast.

[Mike]

Thanks for sharing @D.Dean! It’s great to hear how folks are using BigFix in the more complex configurations and seeing just how flexible it really can be.

Might I suggest a little tip here: don’t use UNC references unless there is absolutely no other alternative, especially within applicability relevance statements. Since the client runs as local system, it does not have a user context when reaching out over SMB to access shares so the path must be accessible via a NULL user share. In this case, netlogon is one of those special shares so this should work (unless locked down) but that is not always the case. Additionally, every client loop is causing the share to be accessed which might not be the best thing for a network impact case… but you do lessen the impact here to reduce first the number of clients who potentially will try and access the share as well as reference only the local share. What I would do here instead is to change the relevance to reference the local path instead of the network share path like:

if (exists network share "netlogon") then (exists files "CustomerSpecificMSSPM.dat" of folders "BESAgent" of folders (path of network share "netlogon")) else false

With this approach, you eliminate the whole UNC path part, ensure the local server does in fact have a netlogon share without trying to establish a connection to it first, bypass any permissions that might otherwise be present on the share that would prevent writing to this file (just relying upon the filesystem permissions alone), and should afford you a little more control over any error correction you might wish to add upon.

[D.Dean]

Thanks for the suggestion.

How would I used the suggested process to search the line of text containing “ScriptVer=5”?

[SLB]

Maybe something like

exists network shares whose ((name of it as uppercase = "NETLOGON") and (exists file "InstallationPatch.bat" whose (exists lines whose (it contains "ScriptVer=5") of it) of folder "BESAgent" of folder (path of it)))

[Mike]

You can do it like this:

if (exists network share "netlogon") then ((not exists files "InstallPatch.bat" of folders "BESAgent" of folders (path of network share "netlogon")) or (exists files "InstallPatch.bat" whose (exists lines whose (it contains "ScriptVer=5") of it) of folders "BESAgent" of folders (path of network share "netlogon"))) else false

I would optimize this fixlet a bit if using this in production however:

1. Check for Windows OS before trying to use the WMI or NETWORK SHARE inspectors.

name of operating system starts with "Win"

2. Modify the WMI call relevance to prevent duplicate WMI queries for the same value:

exists unique values whose (it = 4 or it = 5) of integer values of selects "DomainRole FROM Win32_ComputerSystem" of wmis 

3. Check for the network share (just once)

exists network share "netlogon"

4. Check for the CustomerSpecificMSSPM.dat file (and the “BESAgent” folder as a side-effect)

exists files "CustomerSpecificMSSPM.dat" of folders "BESAgent" of folders (path of network share "netlogon")

5. Finally check for the script version

((not exists files "InstallPatch.bat" of it or exists files "InstallPatch.bat" whose (exists lines whose (it contains "ScriptVer=5") of it) of it) of folder "BESAgent" of folder (path of network share "netlogon")) | false

[JasonWalker]

Great notes! I think we can skip the WMI query though in favor of a built-in inspector (but need to check this against read-only domain controllers, which I don’t have available)

product type of operating system = nt domain controller product type

[Mike]

Why yes, yes it does. Good catch @JasonWalker!

            WMI Query                    Native Relevance Inspector
DomainRole   DomainRole.ToString()       product type of operating system
----------   -------------------------   ---------------------------------
    0        Standalone Workstation      nt workstation product type
    1        Member Workstation          nt workstation product type
    2        Standalone Server           nt server product type
    3        Member Server               nt server product type
    4        Backup Domain Controller    nt domain controller product type
    5        Primary Domain Controller   nt domain controller product type

[D.Dean]

Unfortunately we would need to know which is the PDC and the BDC. Because we are updating the netlogon share, we only push to the PDC and let replication do the rest but sometimes the PDC is missing the BigFix client and all we see is the BDC or the PDC is locked at the customers request so we will push it to the BDC.

[jgstew]

@Mike

4 Backup Domain Controller nt domain controller product type
5 Primary Domain Controller nt domain controller product type

This should work on either domain controller type: (I haven’t tested it)

(it = operating system product type 4 OR it = operating system product type 5) of product types of operating systems

It is a tad awkward though, but if you can avoid a call to WMI, it is often a good idea.

I’m annoyed that this does NOT work: (it as integer) of product types of operating systems

It might be possible that the underlying inspector considers the value of 4 and 5 to be the same nt domain controller product type and does not differentiate, but I would guess that is not the case and it is just a missing option.

References:

• https://developer.bigfix.com/relevance/reference/operating-system-product-type.html

[D.Dean]

@JasonWalker

product type of operating system = nt domain controller product type

Does not seem to be working for the integer. This is on a PDC in my lab

[gus]

Here’s another possible approach but will not run the WMI query against your non-Domain Controller endpoints because it will “FALSE” out the first part of the relevance and not continue with the “AND”:

Backup Domain Controller
(product type of operating system = nt domain controller product type) AND (exists unique values whose(4 = it) of integer values of selects "DomainRole from Win32_ComputerSystem" of wmis)

Primary Domain Controller
(product type of operating system = nt domain controller product type) AND (exists unique values whose(5 = it) of integer values of selects "DomainRole from Win32_ComputerSystem" of wmis)

[jgstew]

@D.Dean

This is on a PDC in my lab

Thanks for testing that, I didn’t have one handy. That is odd that it doesn’t work, seems like the relevance inspector is hiding some complexity and not actually exposing all types individually.