GatherDB import duration

[IanDM]

Has anyone else observed imports are taking a much longer time in the last year? Enterprise Security import has gone from ~1 hour in Jan of 2021, ~2 hours Jan of 2022 and as of today, ~4.5 hours. Just wanted to ask for input especially from those with a Linux/DB2 based environment, before opening a case to investigate.

It seems to be the same in both our production and test environments:

Production Env:

Mon, 04 Jan 2021 11:32:40 -0500 -- Beginning DB import of version 3684 of site Enterprise Security
Mon, 04 Jan 2021 12:33:42 -0500 -- Import of version 3684 of site Enterprise Security completed successfully

Tue, 04 Jan 2022 13:57:28 -0500 -- Beginning DB import of version 3900 of site Enterprise Security
Tue, 04 Jan 2022 15:59:40 -0500 -- Import of version 3900 of site Enterprise Security completed successfully

Fri, 04 Nov 2022 14:36:52 -0400 -- Beginning DB import of version 4090 of site Enterprise Security
Fri, 04 Nov 2022 19:04:02 -0400 -- Import of version 4090 of site Enterprise Security completed successfully

Test Env:

Mon, 04 Jan 2021 11:20:27 -0500 -- Beginning DB import of version 3684 of site Enterprise Security
Mon, 04 Jan 2021 12:21:56 -0500 -- Import of version 3684 of site Enterprise Security completed successfully

Tue, 04 Jan 2022 14:26:15 -0500 -- Beginning DB import of version 3900 of site Enterprise Security
Tue, 04 Jan 2022 16:25:43 -0500 -- Import of version 3900 of site Enterprise Security completed successfully

Fri, 04 Nov 2022 14:49:55 -0400 -- Beginning DB import of version 4090 of site Enterprise Security
Fri, 04 Nov 2022 18:17:24 -0400 -- Import of version 4090 of site Enterprise Security completed successfully

[Aram]

I would recommend opening a Support case here. For what it’s worth, in my (Windows Root Server) lab environment, the average gather duration for Enterprise Security is 12.5 minutes, and if anything, it’s decreased in duration recently:

[Mike]

Same here. Over the past year my median import time for Enterprise Security is 12 minutes 3 seconds.

[Mike]

If anyone is interested in generating the same metrics for their environments, here is the PowerShell function for processing the GatherDB Log for site import times:

function Measure-SiteImportTimes {
	[CmdletBinding()]
	param (
		[Parameter(Mandatory = $true, Position = 1, HelpMessage = 'Path to the GatherDB log file')]
		[System.IO.FileInfo]$Path
	)
		
	process {
		try {
			$stream = [System.IO.FileStream]::New($Path, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read, [System.IO.FileShare]::Read)
			$reader = [System.IO.StreamReader]::New($stream)
			while ($line = $reader.ReadLine()) {
				$fields = $line -split ' -- '
				if (2 -eq $fields.Length) {
					$timestamp = [DateTime]$fields[0]
					if ($fields[1] -match '^Beginning DB import of version (\d+) of site (.+)$') {
						$start = $timestamp
						$site = $Matches[2]
						$version = $Matches[1]
					} elseif ($fields[1] -match '^Import of version (\d+) of site (.+) completed successfully$') {
						if ($Matches[2] -eq $site -and $Matches[1] -eq $version) {
							[PSCustomObject]@{Date=$start; Site=$site; Version=$version; Duration=$($timestamp - $start).TotalSeconds}
						}
					}
				}
			}
		}
		finally {
			if ($logstream) {
				$logstream.Close()
			}
		}
	}
}

Measure-SiteImportTimes "C:\Program Files (x86)\BigFix Enterprise\BES Server\GatherDBData\GatherDB.log"

[IanDM]

I would recommend opening a Support case here.

Thank you @Aram. Will do.

Thank you @Mike for sharing that script.

[IanDM]

In the end, I did not open a support case. When I discovered this, we were in the final planning to perform a migration from RHEL7 to 8 and application upgrade 10.0.3 to 10.0.7 and I elected to wait until after to reassess (if the issue persisted).

The migration (Dec 15) did not have any impact to the import duration as one additional import ran that night on the new OS and VMs and still took ~6 hours.

After the upgrade, the imports dropped to the ~15 minute range. Sometimes under 12 minutes.

Wed, 14 Dec 2022 11:33:56 -0500 -- Beginning DB import of version 4108 of site Enterprise Security
Wed, 14 Dec 2022 17:45:32 -0500 -- Import of version 4108 of site Enterprise Security completed successfully
Wed, 14 Dec 2022 18:46:10 -0500 -- Beginning DB import of version 4109 of site Enterprise Security
Thu, 15 Dec 2022 00:58:28 -0500 -- Import of version 4109 of site Enterprise Security completed successfully

**Migration RHEL7 - RHEL8**

Thu, 15 Dec 2022 18:25:44 -0500 -- Beginning DB import of version 4111 of site Enterprise Security
Fri, 16 Dec 2022 00:25:05 -0500 -- Import of version 4111 of site Enterprise Security completed successfully

**Upgrade 10.0.3 - 10.0.7**

Mon, 19 Dec 2022 13:17:07 -0500 -- Beginning DB import of version 4112 of site Enterprise Security
Mon, 19 Dec 2022 13:32:16 -0500 -- Import of version 4112 of site Enterprise Security completed successfully
Tue, 20 Dec 2022 16:08:11 -0500 -- Beginning DB import of version 4113 of site Enterprise Security
Tue, 20 Dec 2022 16:22:02 -0500 -- Import of version 4113 of site Enterprise Security completed successfully

Not exactly clear what the cause was, but happy to say it seems to have been resolved by upgrading and grateful for one less case to wrangle at the start of the year!