BIGFIX 10.0.8 - Linux master - prefetch download error when link states https:// - works with http:// - " HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate"

Usage and Config
1

BIGFIX 10.0.8 - Linux master - prefetch download error when link states https:// - but works with http:// - " HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate"

on a case by case basis …so if i copy off the fixlet , edit the prefetch line to use http: instead of https: then the files are downloaded.

BIGFIX master is linux … we also have a BIGFIX master that is Windows and the same " https prefetch fixlets " that fail on the LINUX master work no problem on the Windows master.

we use a PROXY to get out to the internet - sounds like some type of certificate or http / https redirect issue but not sure what to modify to correct the overall issue… since it works with our windows based bigfix master, there should be something we can modify on our LINUX based bigfix master to get this working

2

If your proxy is rewriting the TLS connection, every HTTPS connect will appear to have a certificate from your proxy – which BigFix doesn’t trust (yet).

You need to update the root server’s ca-bundle.crt to include the root and issuing certificates uses by your proxy. There’s a bit of description here listing the file paths

https://help.hcltechsw.com/bigfix/10.0/platform/Platform/Config/c_https_gathering.html

I think I’ve talked in more detail here before, will try to find & link older posts.

1 Like
3

This should help, I think

1 Like
4

One fairly simple way to retrieve the proxy’s certificate chain with OpenSSL:

 echo| openssl s_client -connect google.com:443 -prexit -showcerts --proxy proxyserver:3128

(substituting your proxy server hostname and port)

The real Google certificate chain should look like

Certificate chain
 0 s:CN = *.google.com
   i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
-----BEGIN CERTIFICATE-----
MIIOHDCCDQSgAwIBAgIRAORLNyJztwuHEsXHBNxhF0swDQYJKoZIhvcNAQELBQAw
    <snip>
U/wPetR3nxUVOjEkBpZo9g==
-----END CERTIFICATE-----
 1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
   i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
-----BEGIN CERTIFICATE-----
MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQsw
    <snip>
1IXNDw9bg1kWRxYtnCQ6yICmJhSFm/Y3m6xv+cXDBlHz4n/FsRC6UfTd
-----END CERTIFICATE-----
 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
-----BEGIN CERTIFICATE-----
MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBX
<snip>
d0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8=
-----END CERTIFICATE-----
---
Server certificate
subject=CN = *.google.com

issuer=C = US, O = Google Trust Services LLC, CN = GTS CA 1C3

The certificates rewritten by your proxy will be different, they’ll be your company’s certificates. You need to copy those certificates into your ca-bundle.crt file.
The chain may or may not include the CA Root Certificate, depending on your proxy and how it’s configured. You may need to get in touch with the proxy owner to find your company’s root certificate if you use an internal CA.

1 Like
5

Jason,

I work with Big fixer who made this post and I have been troubleshooting this issue on our side with your suggestions and some googling

Issue: Prefetch Jobs fail with “Reason: HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate”

Things we know/have tried…

  • We know that our proxy does SSL intercption
  • We are only having this SSL intercept issue with Prefretch fixlets that use https://
  • I can confirm that when our proxy admin add’s affected sites to the proxy’s SSL Decryption Exclusion list the prefetch https:// fixlet’s work; however this is not feasable to do everytime
  • Adding the Cert(s) to the /opt/BESServer/Reference/ca-bundle.crt has no affect on the https:// prefretch fixlets running…they still fail with the same error
  • Copying the fixlet and changeing the https:// to http:// allows it to work
  • The guy on this post On an Initial Install, Untrusted Certificate to https://gatherer.bigfix.com Prevented Loading of BigFix Management Fixlets interestingly says
    "Can go into many details for security reasons, however the problem is certainly related to the proxy. To avoid “man in the middle attacks”, the https gather includes also a peer verification, and your proxy could not be configured to work with the above peer verification."
  • Purposely breaking the entire ca-bundle.crt file has no affect on the error for the prefetch https:// however it does break other traffic with an error such as this:
    HTTPS connection to {https://sync.bigfix.com/cgi-bin/bfgather/cyberfocus?Time=1674576987} was unsuccessful due to {HTTP Error 77: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations: CAfile: /opt/BESServer/Reference/ca-bundle.crt CApath: none}; retried using HTTP
  • The above behavior leads me to believe that the prefetch fixlets are not using the specified ca-bundle.crt file to verify certificates.

Verbose log snippit of example failure from BESRelay.log…
Thu, 26 Jan 2023 10:46:03 -0500 - GatherMain (3571107584) - Queueing download: {aid=16396,index=1,sha1=bd8ea599f044e3834b779bd99e8732a92ae869a8,size=null,url=https%3a%2f%2fcatalog.s.download.windowsupdate.com%2fc%2fmsdownload%2fupdate%2fsoftware%2fupdt%2f2022%2f09%2fsqlserver2019-kb5017593-x64_bd8ea599f044e3834b779bd99e8732a92ae869a8.exe}
Thu, 26 Jan 2023 10:46:03 -0500 - GatherMain (3571107584) - DownloadState: download {aid=16396,index=1,sha1=bd8ea599f044e3834b779bd99e8732a92ae869a8,size=null,url=https%3a%2f%2fcatalog.s.download.windowsupdate.com%2fc%2fmsdownload%2fupdate%2fsoftware%2fupdt%2f2022%2f09%2fsqlserver2019-kb5017593-x64_bd8ea599f044e3834b779bd99e8732a92ae869a8.exe} changed from FAILED to QUEUED
Thu, 26 Jan 2023 10:46:03 -0500 - GatherMain (3571107584) - Starting download: {aid=16396,index=1,sha1=bd8ea599f044e3834b779bd99e8732a92ae869a8,size=null,url=https%3a%2f%2fcatalog.s.download.windowsupdate.com%2fc%2fmsdownload%2fupdate%2fsoftware%2fupdt%2f2022%2f09%2fsqlserver2019-kb5017593-x64_bd8ea599f044e3834b779bd99e8732a92ae869a8.exe}
Thu, 26 Jan 2023 10:46:03 -0500 - 2470385408 - Entering GET https://catalog.s.download.windowsupdate.com/c/msdownload/update/software/updt/2022/09/sqlserver2019-kb5017593-x64_bd8ea599f044e3834b779bd99e8732a92ae869a8.exe
Thu, 26 Jan 2023 10:46:03 -0500 - 2470385408 - The currently configured proxy server is: lsproxy.palmbeach.k12.fl.us:3128
Thu, 26 Jan 2023 10:46:03 -0500 - 2470385408 - Actually used proxy server depends on the proxy exception list which is: localhost,palmbeach.k12.fl.us,fhbigfix
Thu, 26 Jan 2023 10:46:03 -0500 - 2470385408 - Proxy authentication methods allowed are: all
Thu, 26 Jan 2023 10:46:03 -0500 - 2470385408 - You can’t use the proxy for downstream
Thu, 26 Jan 2023 10:46:03 -0500 - 2470385408 - The proxy doesn’t use secure channel
Thu, 26 Jan 2023 10:46:03 -0500 - 2470385408 - Enabling Host verification for connection with url https://catalog.s.download.windowsupdate.com/c/msdownload/update/software/updt/2022/09/sqlserver2019-kb5017593-x64_bd8ea599f044e3834b779bd99e8732a92ae869a8.exe
Thu, 26 Jan 2023 10:46:03 -0500 - GatherMain (3571107584) - DownloadState: download {aid=16396,index=1,sha1=bd8ea599f044e3834b779bd99e8732a92ae869a8,size=null,url=https%3a%2f%2fcatalog.s.download.windowsupdate.com%2fc%2fmsdownload%2fupdate%2fsoftware%2fupdt%2f2022%2f09%2fsqlserver2019-kb5017593-x64_bd8ea599f044e3834b779bd99e8732a92ae869a8.exe} changed from QUEUED to DOWNLOADING
Thu, 26 Jan 2023 10:46:03 -0500 - Main Thread (1409366080) - /data/site-readers - 10.254.16.16
Thu, 26 Jan 2023 10:46:03 -0500 - /data/site-readers (2716587776) - Running plugin /data/site-readers with client 10.254.16.16
Thu, 26 Jan 2023 10:46:03 -0500 - /data/site-readers (2716587776) - select ServerID from DBINFO
Thu, 26 Jan 2023 10:46:03 -0500 - /data/site-readers (2716587776) - select FieldContents from ADMINFIELDS where FieldName = ? and IsDeleted = 0
Thu, 26 Jan 2023 10:46:03 -0500 - /data/site-readers (2716587776) - select Certificate
from CERTIFICATES
Thu, 26 Jan 2023 10:46:03 -0500 - 2470385408 - Exiting GET https://catalog.s.download.windowsupdate.com/c/msdownload/update/software/updt/2022/09/sqlserver2019-kb5017593-x64_bd8ea599f044e3834b779bd99e8732a92ae869a8.exe (14 ms)
Thu, 26 Jan 2023 10:46:03 -0500 - /data/site-readers (2716587776) - select U.UserID, MastheadUsername, Username, IsMaster, CustomContent, UserCreationTime, ShowOtherUsersActions, StopOtherUsersActions, UnmanagedAssetPrivilege, ApproverRoleID, LdapID, LdapDN, GUID, CanCreateActions, PostActionBehavior, ActionScriptCommands, CanLock, CanSendMultipleRefresh, CanSubmitQueries, ConsoleLogin, APILogin, WebUILogin, IsDeleted, LoginPermission, UL.LastLoginTime, SignedData from USERINFO U join USER_LOGIN UL on UL.UserLoginID = U.UserID where UserID = ? and IsDeleted = 0
Thu, 26 Jan 2023 10:46:03 -0500 - /data/site-readers (2716587776) - select R.RoleID, R.Name, R.ModificationTime, R.IsDeleted, R.IsMaster, R.CustomContent, R.ShowOtherUsersActions, R.StopOtherUsersActions, R.CanCreateActions, R.CanLock, R.CanSendMultipleRefresh, R.CanSubmitQueries, R.ConsoleLogin, R.APILogin, R.WebUILogin, R.PostActionBehavior, R.ActionScriptCommands, R.UnmanagedAssetPrivilege, R.Description, R.Computers, R.SubscriptionSMIME, R.SignedData from ROLE_USER_ASSIGNMENTS RA join ROLES R on R.RoleID = RA.RoleID where R.IsDeleted = 0 AND RA.UserID = ? AND ( RA.Explicit = 1 or RA.Inherited = 1 )
Thu, 26 Jan 2023 10:46:03 -0500 - /data/site-readers (2716587776) - select SiteID, Privilege from ROLE_EXTERNAL_SITE_ASSIGNMENTS where RoleID = ? and IsDeleted = 0
Thu, 26 Jan 2023 10:46:03 -0500 - /data/site-readers (2716587776) - select Sitename, Privilege from ROLE_CUSTOM_SITE_ASSIGNMENTS where RoleID = ? and IsDeleted = 0
Thu, 26 Jan 2023 10:46:03 -0500 - /data/site-readers (2716587776) - select RA.UserID, RA.Explicit, RA.Inherited, RA.InheritedFrom from ROLE_USER_ASSIGNMENTS RA join USERINFO U on U.UserID = RA.UserID and U.IsDeleted = 0 where RA.RoleID = ? and ( RA.Explicit = 1 or RA.Inherited = 1 )
Thu, 26 Jan 2023 10:46:03 -0500 - /data/site-readers (2716587776) - select LdapID, GroupDN, GroupName from ROLE_GROUP_ASSIGNMENTS where RoleID = ? and IsDeleted = 0
Thu, 26 Jan 2023 10:46:04 -0500 - GatherMain (3571107584) - DownloadState: download {aid=16396,index=1,sha1=bd8ea599f044e3834b779bd99e8732a92ae869a8,size=null,url=https%3a%2f%2fcatalog.s.download.windowsupdate.com%2fc%2fmsdownload%2fupdate%2fsoftware%2fupdt%2f2022%2f09%2fsqlserver2019-kb5017593-x64_bd8ea599f044e3834b779bd99e8732a92ae869a8.exe} changed from DOWNLOADING to FAILED
Thu, 26 Jan 2023 10:46:04 -0500 - GatherMain (3571107584) - Download failed: aid=16396,index=1,sha1=bd8ea599f044e3834b779bd99e8732a92ae869a8,size=null,url=https%3a%2f%2fcatalog.s.download.windowsupdate.com%2fc%2fmsdownload%2fupdate%2fsoftware%2fupdt%2f2022%2f09%2fsqlserver2019-kb5017593-x64_bd8ea599f044e3834b779bd99e8732a92ae869a8.exe
Reason: HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate

1 Like
6

Also, the verbose log above looks like it’s pulling certificates from the Database with a select statement like this.

select Certificate
from CERTIFICATES

Furthermore this leads me to believe that there is another separate certificate store in the database.

1 Like
7

I just posted about this in the related BFI thread.

I ran into this issue today, following our upgrade to v10.0.8.

Our issue is two fold. We use a proxy with an internal cert… BUT some sites are whitelisted. So when BES downloads a file, some sites would need to validate against the internal cert and others to the real world cert.

Initially setting the property _BESClient_Download_CACertPath solved my issue for downloads until I tried a site that was whitelisted. I then set _BESRelay_Download_UntrustedSites = 1. That solved my whitelist issue. I am trying removing the CACertPath and rechecking. Also opened a case with HCL to get best practice here. I like the idea of cert validation but I may just have to leave it disabled.

These are new properties as part of v8.
https://help.hcltechsw.com/bigfix/10.0/platform/Platform/Config/c_managing_downloads.html

8

Thanks, @DerrickD and @mjudson and yes it looks like there are significant changes with downloads in 10.0.8.

As I read that article, it looks like Reference\ca-bundle.crt is not used anymore, and instead we now distribute an updated ca-bundle.crt in the BES Support site.
For a Proxy that does interception, the way I read it is

  1. On your server (or Relay performing Internet downloads), you need to generate a ca-bundle.crt containing all of the root and intermediate certificate issuers you want to trust. You may obtain a current bundle of publicly-trusted CA authorities from Mozilla, or curl, or use the one we update in the BES Support site (by copying it from BES Client\__BESData\BES Support\ca-bundle.crt from a client to a path you can maintain)

  2. Update the ca-bundle.crt with your internal CA Root / CA Issuing certificates (the same as we would previously update the “BES Server\Reference\ca-bundle.crt”). This is literally pasting your Intermediate and Root authorities’ PEM-encoded text certificates onto the end of the ca-bundle.crt file.

  3. Apply client setting _BESRelay_Download_CACertPath on your Root (and any Relay performing direct Internet downloads) to reference the path of your custom copy of ca-bundle.crt.

  4. Restart BESClient / BESRelay to apply the new setting.

If you have any Clients performing Direct Downloads, you’d need to do the same setup to create a custom ca-bundle.crt, but apply the client setting _BESClient_Download_CACertPath instead of the _BESRelay version of the setting.

I’m not in a position to test this out at the moment, but if you can give it a try and report back I’d love to hear some real-world results.

1 Like
9

@DerrickD I will give this a try…you situation sound just like ours…proxy with internal cert where some are whitelisted

I had previously tried setting this paramerter but the one referenced there is different.

ARGH… I hope it works. Thanks

[Software\BigFix\EnterpriseClient\Settings\Client_BESGather_CACert]
value = /opt/BESServer/Reference/ca-bundle.crt

2 Likes
10

@mjudson

Yes, that is our setup as well.

You will likely still need _BESGather_CACert for the BES Gather service

You ALSO need the _BESClient_Download_CACertPath and/or _BESRelay_Download_UntrustedSites properties set for fixlet/other downloads.

For me I have
_BESGather_CACert = C:\path\to\cert.crt
_BESRelay_Download_UntrustedSites = 1

I have a case open with HCL. They said they have a lot of cases right now from customers. I am waiting to hear back and asked for best practice here, given our situation where some sites have an internal cert and others are whitelisted for the public cert.

1 Like
11

@DerrickD

YESS It worked…

Adding these 2 to the besserver.config did the trick…

[Software\BigFix\EnterpriseClient\Settings\Client\_BESRelay_Download_UntrustedSites]
value = 1

[Software\BigFix\EnterpriseClient\Settings\Client\_BESRelay_Download_CaCertPath]
value = /opt/BESServer/Reference/ca-bundle.crt

OMG thank you so much… This has been driving me up the wall…I also just realized google was showing me all the 9.x parameters instead of the 10.0 paramaters…I should have been paying more attention and I would have found this sooner. ARGH!

vs.

and

vs

Successfull Download now
Thu, 26 Jan 2023 17:00:35 -0500 - GatherMain (3571107584) - Processing successful download: {aid=16396,index=1,sha1=bd8ea599f044e3834b779bd99e8732a92ae869a8,size=null,url=https%3a%2f%2fcatalog.s.download.windowsupdate.com%2fc%2fmsdownload%2fupdate%2fsoftware%2fupdt%2f2022%2f09%2fsqlserver2019-kb5017593-x64_bd8ea599f044e3834b779bd99e8732a92ae869a8.exe}
Thu, 26 Jan 2023 17:00:35 -0500 - Main Thread (1409366080) - /data/deleted-objects?sequence=(1369831961,1369832107) - 10.254.62.6
Thu, 26 Jan 2023 17:00:35 -0500 - /data/deleted-objects (2716587776) - Running plugin /data/deleted-objects?sequence=(1369831961,1369832107) with client 10.254.62.6
Thu, 26 Jan 2023 17:00:35 -0500 - /data/deleted-objects (2716587776) - select ServerID from DBINFO
Thu, 26 Jan 2023 17:00:35 -0500 - /data/deleted-objects (2716587776) - select FieldContents from ADMINFIELDS where FieldName = ? and IsDeleted = 0
Thu, 26 Jan 2023 17:00:35 -0500 - /data/deleted-objects (2716587776) - select Certificate
from CERTIFICATES
Thu, 26 Jan 2023 17:00:39 -0500 - GatherMain (3571107584) - DownloadState: download {aid=16396,index=1,sha1=bd8ea599f044e3834b779bd99e8732a92ae869a8,size=null,url=https%3a%2f%2fcatalog.s.download.windowsupdate.com%2fc%2fmsdownload%2fupdate%2fsoftware%2fupdt%2f2022%2f09%2fsqlserver2019-kb5017593-x64_bd8ea599f044e3834b779bd99e8732a92ae869a8.exe} changed from DOWNLOADING to COMPLETED

image
image1185×400 14.9 KB

2 Likes
12

Ideally, if you have ca-bundle.crt set up to include both the public trusted root authorities, and your internal trusted authorities, you shouldn’t need to set _BESRelay_Download_UntrustedSites. All of the HTTPS connections should be chained up to an authority trusted in ca-bundle.crt (once you’ve added your own to it).

Setting _BESRelay_Download_UntrustedSites = 1 is setting the server to ignore all certificate errors, so if a web server presents an invalid, untrusted, or expired certificate, this setting will allow the BigFix server to download the file anyway. In most BigFix cases that really isn’t much of an exposure (since we perform hash-checks against the downloaded file anyway), but a strict reading of the security standards would have us reject the connection.

2 Likes
13

@JasonWalker ok i do have the ca bundle setup with our internal and the public certs so i will try to back out that setting

1 Like