Installing Splunk on endpoints - status running but its not

I have tested the following command locally on machine which ran succesful
I don’t see in windows event log . any ideas?
url to commands.
https://docs.splunk.com/Documentation/Splunk/9.3.2/Installation/InstallonWindowsviathecommandline

`type or paste code here`

msiexec.exe /I splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi /l*v C:\TEMP\SplunkInstall.log /qn /norestart WINEVENTLOG_SYS_ENABLE=1 SPLUNKPASSWORD=LearnSplunk AGREETOLICENSE=yes

I have tested multiple attempts to run using BigFix - the action shows running but there is no activity . I attempted to run from the download folder and then I attempted to copy to directory and run from there same issue every time. there is no error in relevance command . I turned on verbose logging and don’t see the issue.
log file
Search “splunk” (28 hits in 1 file of 1 searched) [RegEx]
C:\Users\johngib\OneDrive - CDW\Documents\besclientdebug.log (28 hits)
Line 233: Fri, 29 Nov 2024 12:15:48 -0600 DebugMessage Relevant - Software Distribution - Deploy: Splunk (fixlet:103961)
Line 242: Fri, 29 Nov 2024 12:15:49 -0600 DebugMessage ActionLogMessage: (action:103961) Download url: ‘http://SHSEGVBFXPVW002.forsythehosting.com:52311/Uploads/44b1998a0b527aa868bfc0e3f18bd186b4092601/splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi.tmp
Line 347: Fri, 29 Nov 2024 12:17:39 -0600 DebugMessage Command succeeded (Prefetch download manager collected file) prefetch 44b1998a0b527aa868bfc0e3f18bd186b4092601 sha1:44b1998a0b527aa868bfc0e3f18bd186b4092601 size:129245467 http://SHSEGVBFXPVW002.forsythehosting.com:52311/Uploads/44b1998a0b527aa868bfc0e3f18bd186b4092601/splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi.tmp sha256:5c8ac79ae89447a8e6e9c147ac0bc6016cb36041c773c73fa651170b0d427f59 (action:103961)
Line 357: Fri, 29 Nov 2024 12:17:42 -0600 VerboseMessage RelevanceSubstitution::SubstituteStrings() entering, input = “{pathname of system folder & “\msiexec.exe”}” /i “c:\Temp\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi /lv C:\TEMP\SplunkInstall.log" /qn /norestart / WINEVENTLOG_SYS_ENABLE=1 SPLUNKPASSWORD=LearnSplunk AGREETOLICENSE=yes
Line 362: Fri, 29 Nov 2024 12:17:42 -0600 DebugMessage Command started - waithidden “C:\Windows\system32\msiexec.exe” /i "c:\Temp\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi /l
v C:\TEMP\SplunkInstall.log” /qn /norestart / WINEVENTLOG_SYS_ENABLE=1 SPLUNKPASSWORD=LearnSplunk AGREETOLICENSE=yes (action:103961)
Line 672: Fri, 29 Nov 2024 12:27:15 -0600 DebugMessage Command failed (Action ended while waiting for another process to complete) waithidden “{pathname of system folder & “\msiexec.exe”}” /i “c:\Temp\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi /lv C:\TEMP\SplunkInstall.log" /qn /norestart / WINEVENTLOG_SYS_ENABLE=1 SPLUNKPASSWORD=LearnSplunk AGREETOLICENSE=yes (action:103961)
Line 841: Fri, 29 Nov 2024 12:29:39 -0600 DebugMessage Relevant - Software Distribution - Deploy: Splunk (fixlet:103962)
Line 962: Fri, 29 Nov 2024 12:30:23 -0600 DebugMessage ActionLogMessage: (action:103962) Download url: ‘http://SHSEGVBFXPVW002.forsythehosting.com:52311/Uploads/44b1998a0b527aa868bfc0e3f18bd186b4092601/splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi.tmp
Line 969: Fri, 29 Nov 2024 12:30:30 -0600 DebugMessage Command succeeded (Prefetch download manager collected file) prefetch 44b1998a0b527aa868bfc0e3f18bd186b4092601 sha1:44b1998a0b527aa868bfc0e3f18bd186b4092601 size:129245467 http://SHSEGVBFXPVW002.forsythehosting.com:52311/Uploads/44b1998a0b527aa868bfc0e3f18bd186b4092601/splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi.tmp sha256:5c8ac79ae89447a8e6e9c147ac0bc6016cb36041c773c73fa651170b0d427f59 (action:103962)
Line 977: Fri, 29 Nov 2024 12:30:36 -0600 VerboseMessage RelevanceSubstitution::SubstituteStrings() entering, input = “{pathname of system folder & “\msiexec.exe”}” /i "{(pathname of client folder of current site) & "__Download\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi /l
v C:\TEMP\SplunkInstall.log”}" /qn /norestart / WINEVENTLOG_SYS_ENABLE=1 SPLUNKPASSWORD=LearnSplunk AGREETOLICENSE=yes
Line 981: Fri, 29 Nov 2024 12:30:36 -0600 VerboseMessage RelevanceSubstitution::EvaluateRelevanceAsString() entering, input = (pathname of client folder of current site) & “__Download\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi /lv C:\TEMP\SplunkInstall.log"
Line 985: Fri, 29 Nov 2024 12:30:36 -0600 DebugMessage Command started - waithidden “C:\Windows\system32\msiexec.exe” /i "C:\Program Files (x86)\BigFix Enterprise\BES Client__BESData\CustomSite_Gibson_Custom_Content__Download\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi /l
v C:\TEMP\SplunkInstall.log” /qn /norestart / WINEVENTLOG_SYS_ENABLE=1 SPLUNKPASSWORD=LearnSplunk AGREETOLICENSE=yes (action:103962)

I just saw this issue.

Line 672: Fri, 29 Nov 2024 12:27:15 -0600 DebugMessage Command failed (Action ended while waiting for another process to complete) waithidden “{pathname of system folder & “\msiexec.exe”}” /i “c:\Temp\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi /lv C:\TEMP\SplunkInstall.log" /qn /norestart / WINEVENTLOG_SYS_ENABLE=1 SPLUNKPASSWORD=LearnSplunk AGREETOLICENSE=yes (action:103961)

That message indicates the client was restarted while the ‘waithidden’ command was still running.

I see you downloading a .msi.tmp file, but not where you extract or rename it to take the .tmp off the filename; and then you’re running msiexec against a file in c:\Temp but I don’t see where you would have moved the MSI package to the temp directory. Normally you would reference that with the relative path __Download\filename.msi

Jason below are the two options I have used to run from the client download foler or copy the installer to c:temp directory and run from there. Interesting you say the client restarted during the process . where did you see that as that seems like it would casuse the process issue

prefetch 44b1998a0b527aa868bfc0e3f18bd186b4092601 sha1:44b1998a0b527aa868bfc0e3f18bd186b4092601 size:129245467 http://SHSEGVBFXPVW002.forsythehosting.com:52311/Uploads/44b1998a0b527aa868bfc0e3f18bd186b4092601/splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi.tmp sha256:5c8ac79ae89447a8e6e9c147ac0bc6016cb36041c773c73fa651170b0d427f59
extract 44b1998a0b527aa868bfc0e3f18bd186b4092601

Copying the file to the C:temp directory .
//copy “{(client folder of current site as string) & “__Download\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi”}” “c:\Temp\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi”

This is running from the client directory

//waithidden “{pathname of system folder & “\msiexec.exe”}” /i “c:\Temp\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi /l*v C:\TEMP\SplunkInstall.log” /qn /norestart

//waithidden “{pathname of system folder & “\msiexec.exe”}” /i “{(pathname of client folder of current site) & “__Download\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi /l*v C:\TEMP\SplunkInstall.log”}” /qn /norestart / WINEVENTLOG_SYS_ENABLE=1 SPLUNKPASSWORD=LearnSplunk AGREETOLICENSE=yes

This is running from the C:Temp directory
//waithidden “{pathname of system folder & “\msiexec.exe”}” /i “c:\Temp\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi /l*v C:\TEMP\SplunkInstall.log” /qn /norestart / WINEVENTLOG_SYS_ENABLE=1 SPLUNKPASSWORD=LearnSplunk AGREETOLICENSE=yes

That copy command won’t be correct - ‘client folder of current site as string’ will not contain the trailing backslash, so the string will be some thing like “c:\BES\client__BESData\CustomSite_MySiteName__Download\splunkforwarder-9.2.3-something.msi”
Missing the "" between sitename and __Download.

Please grab an excerpt of the client log while this action is running so we can see how all of the relevance substitutions are evaluated.

will do so but the copy command did copy the file to the directory.

image

Here are the action lines. Running from the client download folder

prefetch 44b1998a0b527aa868bfc0e3f18bd186b4092601 sha1:44b1998a0b527aa868bfc0e3f18bd186b4092601 size:129245467 http://SHSEGVBFXPVW002.forsythehosting.com:52311/Uploads/44b1998a0b527aa868bfc0e3f18bd186b4092601/splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi.tmp sha256:5c8ac79ae89447a8e6e9c147ac0bc6016cb36041c773c73fa651170b0d427f59
extract 44b1998a0b527aa868bfc0e3f18bd186b4092601

waithidden “{pathname of system folder & “\msiexec.exe”}” /i “{(pathname of client folder of current site) & “__Download\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi /l*v C:\TEMP\SplunkInstall.log”}” /qn /norestart / WINEVENTLOG_SYS_ENABLE=1 SPLUNKPASSWORD=LearnSplunk AGREETOLICENSE=yes

This is our installation command and it works just fine.

msiexec.exe /i splunkforwarder-9.3.1-0b8d769cb912-x64-release.msi INSTALLDIR=“c:\Program Files\SplunkUniversalForwarder” DEPLOYMENT_SERVER=“xxx.xxx.xxx.xxx:8089” AGREETOLICENSE=YES /quiet

Try including a line

action uses wow64 redirection false

before the waithidden. Since BigFix by default would launch the 32-bit version of msiexec that could be an issue.

so I attempted to use the custom site to install but now getting ActionLogMessage: (action:104314) Action signature verified for Execution
ActionLogMessage: (action:104314) Cannot empty _Download directory

So I have deleted the folder from the custom site 104314 and have used task to clear the cached and stop and started client. none changes this issue. So I then changed to the master action site and the process is stuck just running which is what always happens.

ActionLogMessage: (action:104316) Action signature verified for Downloads
DownloadsAvailable: checking for ‘http://SHSEGVBFXPVW002.forsythehosting.com:52311/bfmirror/downloads/104316/0
DownloadsAvailable: true (action id 104316)
ActionLogMessage: (action:104316) Non-Distributed - DownloadsAvailable
ActionLogMessage: (action:104316) Submitting download request
ActionLogMessage: (action:104316) Download url: ‘http://SHSEGVBFXPVW002.forsythehosting.com:52311/Uploads/44b1998a0b527aa868bfc0e3f18bd186b4092601/splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi.tmp
At 12:17:55 -0600 -
Report posted successfully
At 12:18:24 -0600 -
ActionLogMessage: (action:104316) Action signature verified for Execution
ActionLogMessage: (action:104316) starting action
At 12:18:24 -0600 - actionsite (http://SHSEGVBFXPVW002.forsythehosting.com:52311/cgi-bin/bfgather.exe/actionsite)
Command succeeded (Prefetch download manager collected file) prefetch 44b1998a0b527aa868bfc0e3f18bd186b4092601 sha1:44b1998a0b527aa868bfc0e3f18bd186b4092601 size:129245467 http://SHSEGVBFXPVW002.forsythehosting.com:52311/Uploads/44b1998a0b527aa868bfc0e3f18bd186b4092601/splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi.tmp sha256:5c8ac79ae89447a8e6e9c147ac0bc6016cb36041c773c73fa651170b0d427f59 (action:104316)
At 12:18:31 -0600 - actionsite (http://SHSEGVBFXPVW002.forsythehosting.com:52311/cgi-bin/bfgather.exe/actionsite)
Command succeeded extract 44b1998a0b527aa868bfc0e3f18bd186b4092601 (action:104316)
Command started - waithidden “C:\Windows\system32\msiexec.exe” /i “C:\Program Files (x86)\BigFix Enterprise\BES Client__BESData\actionsite__Download\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi /l*v C:\TEMP\SplunkInstall.log” /qn /norestart / WINEVENTLOG_SYS_ENABLE=1 SPLUNKPASSWORD=LearnSplunk AGREETOLICENSE=yes (action:104316)
At 12:19:04 -0600 -
Report posted successfully
At 12:22:11 -0600 -
Report posted successfully
At 12:22:26 -0600 -
PollForCommands: Requesting commands
PollForCommands: commands to process: 0

i will add the wow64 to the fixlet

adding action uses wow64 redirection false had no impact. the action stays in state of running , never ends.

DownloadPing command received (ID=104329)
At 18:21:28 -0600 - mailboxsite (http://SHSEGVBFXPVW002.forsythehosting.com:52311/cgi-bin/bfgather.exe/mailboxsite544284006)
Relevant - Software Distribution - Deploy: Splunk (fixlet:104329)
At 18:21:28 -0600 -
ActionLogMessage: (action:104329) Action signature verified for Downloads
At 18:21:52 -0600 -
ActionLogMessage: (action:104329) Non-Distributed - DownloadsAvailable
ActionLogMessage: (action:104329) Action signature verified for Execution
ActionLogMessage: (action:104329) starting action
At 18:21:52 -0600 - actionsite (http://SHSEGVBFXPVW002.forsythehosting.com:52311/cgi-bin/bfgather.exe/actionsite)
Command succeeded (Prefetch download manager collected file) prefetch 44b1998a0b527aa868bfc0e3f18bd186b4092601 sha1:44b1998a0b527aa868bfc0e3f18bd186b4092601 size:129245467 http://SHSEGVBFXPVW002.forsythehosting.com:52311/Uploads/44b1998a0b527aa868bfc0e3f18bd186b4092601/splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi.tmp sha256:5c8ac79ae89447a8e6e9c147ac0bc6016cb36041c773c73fa651170b0d427f59 (action:104329)
At 18:22:00 -0600 - actionsite (http://SHSEGVBFXPVW002.forsythehosting.com:52311/cgi-bin/bfgather.exe/actionsite)
Command succeeded extract 44b1998a0b527aa868bfc0e3f18bd186b4092601 (action:104329)
Wow64 redirection disabled. action uses wow64 redirection false (action:104329)
Command started - waithidden “C:\Windows\system32\msiexec.exe” /i "C:\Program Files (x86)\BigFix Enterprise\BES Client__BESData\actionsite__Download\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi INSTALLDIR=“C\Program Files\SplunkUniversalForwarder” /lv C:\TEMP\SplunkInstall.logINSTALLDIR=“:\Program Files\SplunkUniversalForwarder” /qn /norestart / WINEVENTLOG_SYS_ENABLE=1 SPLUNKPASSWORD=LearnSplunk AGREETOLICENSE=yes (action:104329)
At 18:22:03 -0600 -
Report posted successfully
At 18:22:03 -0600 - actionsite (http://SHSEGVBFXPVW002.forsythehosting.com:52311/cgi-bin/bfgather.exe/actionsite)
Command succeeded (Exit Code=1603) waithidden “C:\Windows\system32\msiexec.exe” /i "C:\Program Files (x86)\BigFix Enterprise\BES Client__BESData\actionsite__Download\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi INSTALLDIR=“C\Program Files\SplunkUniversalForwarder” /l
v C:\TEMP\SplunkInstall.logINSTALLDIR=“:\Program Files\SplunkUniversalForwarder” /qn /norestart / WINEVENTLOG_SYS_ENABLE=1 SPLUNKPASSWORD=LearnSplunk AGREETOLICENSE=yes (action:104329)
At 18:22:03 -0600 -
ActionLogMessage: (action:104329) ending action

So now I tried running

Command started - waithidden “C:\Windows\system32\msiexec.exe” /i “c:\Temp\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi INSTALLDIR=“C\Program Files\SplunkUniversalForwarder” /l*v C:\TEMP\SplunkInstall.log” /qn /norestart / WINEVENTLOG_SYS_ENABLE=1 SPLUNKPASSWORD=LearnSplunk AGREETOLICENSE=yes (action:104330)

It’s still missing the backslash between the site name and the __Download folder that I suggested could be a problem earlier - and now it looks like you have several Unicode extended versions of “smart quotes”, look for those in the log file to show where the characters are incorrect

With a bad path to the MSI package, the msiexec command is likely just promoting you an error message; but since the interface is hidden, you cannot see or dismiss the message, so the msiexec is stuck running and has the folders locked.

Jason can you show me what you mean with this exampl

waithidden “{pathname of system folder & “\msiexec.exe”}” /i “{(pathname of client folder of current site) & “__Download\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi /l*v C:\TEMP\SplunkInstall.log”}” /qn /norestart / WINEVENTLOG_SYS_ENABLE=1 SPLUNKPASSWORD=LearnSplunk AGREETOLICENSE=yes

You may need to retype all the double quotes, I’m typing on my phone now but what I mean is this…note the backslash before __Download. I also changed some of the quoting. I can’t test this but try

waithidden "{pathname of system folder}\msiexec.exe" /i "{(pathname of client folder of current site)}\__Download\splunkforwarder-9.2.3-282efff6aa8b-x64-release.msi" /l*v C:\TEMP\SplunkInstall.log /qn /norestart / WINEVENTLOG_SYS_ENABLE=1 SPLUNKPASSWORD=LearnSplunk AGREETOLICENSE=yes