Splunk Openssl

The eNcore add-on for Splunk requires Python 3.6+ and openSSL, in the latest Splunk 8.0 release Python3 is included, however the python mods for openssl are not included, which require additional configuration steps outlined in this update.

  • Splunk Web supports 2048 bit keys, but you can specify larger keys if they are supported by your browser. $SPLUNKHOME/bin/splunk cmd openssl genrsa -aes256 -out myCAPrivateKey.key 2048 Note that in Windows you may need to append the location of the openssl.cnf file: $SPLUNKHOME bin splunk cmd openssl genrsa -aes256 -out myCAPrivateKey.key 2048.
  • Generate a Self Signed OpenSSL certificate for SplunkWeb Copy Splunk inputs.conf, web.conf, authentication.conf, and alertactions.conf for system local Create default admin user seed file Accept Splunk license and set Splunk to start at boot using splunk user.
  • Aug 03, 2020 openssl req -out serverconf.csr -new -newkey rsa:2048 -keyout serverconf.key Windows REM SPLUNKHOME is the root of your Splunk Enterprise installation set SPLUNKHOME='C: Program Files Splunk'.

From Splunk Wiki

Jump to: navigation, search

Configuring Splunk forwarding to use the default SSL server certificate

This procedure will explain how to configure Splunk to send data from your forwarders to your indexer(s) using the default SSL server certificate.

This configuration will ensure that your data will be encrypted when in transit on your network, but it will not be securely encrypted since we will be using the default SSL server certificate that ships with every Splunk package : $SPLUNK_HOME/etc/auth/server.pem

If you want to ensure that no one can easily snoop on your Splunk-to-Splunk traffic or wrongfully send data to your indexers, we recommend that you use new SSL certificates signed by your own certificate authority.

Instructions to create your own root certificate and use it to sign new server certificates for Splunk-to-Splunk traffic can be found [[1]].

Instructions to use your own certificate authority to sign new server certificates for Splunk-to-Splunk traffic can be found [[2]].


1 - Set up the indexer(s) to use the default server certificate that ships with Splunk and listen for Splunk to Splunk traffic on a given port :

We will be using port 9997 to receive data from forwarders.

  • In $SPLUNK_HOME/etc/system/local/inputs.conf (or in the appropriate directory of any app you are using to distribute your forwarding configuration), set up the following stanzas :

[SSL]
rootCA = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem
password = password
[splunktcp-ssl:9997]
compressed = true
We will not be using 'requireClientCert = true' since it would be pointless to check the validity of the default server certificate that ships with Splunk and that the forwarders will present to the indexer.

Despite that fact, it is still necessary to indicate the path to the certificate authority public key with 'rootCA = $SPLUNK_HOME/etc/auth/cacert.pem'.

  • Restart splunkd after making these changes.

# $SPLUNK_HOME/bin/splunk restart splunkd
Note that the server certificate pass phrase will be hashed and stored in $SPLUNK_HOME/etc/system/local/inputs.conf, overwriting the clear-text value of 'password' if it was defined there. If 'password' was defined in clear-text in an inputs.conf located in an app, it *will not* be hashed there and will still be present in clear text in that location. This doesn't matter too much in this case since the pass phrase for the default server certificate is well known.


2 - Set up the forwarder(s) to use the default server certificate that ships with Splunk and to send Splunk to Splunk traffic to the indexer(s) receiving port :

In our example, the indexer's IP address is 10.1.12.112.

  • Define the following stanzas in $SPLUNK_HOME/etc/system/local/outputs.conf (or in the appropriate directory of any app you are using to distribute your forwarding configuration) :

[tcpout]
defaultGroup = splunkssl
[tcpout:splunkssl]
server = 10.1.12.112:9997
compressed = true
sslVerifyServerCert = false
sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = password
Just as we did for the indexer, we will not be using 'sslVerifyServerCert = true' since it would also be pointless to ask the forwarders to check the validity of the default server certificate that ships with Splunk and that the indexer(s) will be presenting.

Here too, it is still necessary to indicate the path to the certificate authority public key in with 'sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem'.

If you are distributing data to several indexers, you can simply add their HOST:PORT address as a comma-separated list in the 'server' configuration parameter of the target group definition stanza.

  • Restart splunkd after making these changes.

# $SPLUNK_HOME/bin/splunk restart splunkd

Note that the server certificate pass phrase will be hashed and stored in $SPLUNK_HOME/etc/system/local/outputs.conf, overwriting the clear-text value of 'sslPassword' if it was defined there. If 'sslPassword' was defined in clear-text in an outputs.conf located in an app, it *will not* be hashed there and will still be present in clear text in that location. This doesn't matter too much in this case since the pass phrase for the default server certificate is well known.

3 - Check for a successful connection in splunkd.log :

  • This is what you should see during the indexer start-up sequence in $SPLUNK_HOME/var/log/splunkd.log :

02-06-2011 19:19:01.552 INFO TcpInputProc - using queueSize 1000
02-06-2011 19:19:01.552 INFO TcpInputProc - SSL cipherSuite=ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
02-06-2011 19:19:01.552 INFO TcpInputProc - supporting SSL v2/v3
02-06-2011 19:19:01.555 INFO TcpInputProc - port 9997 is reserved for splunk 2 splunk (SSL)
02-06-2011 19:19:01.555 INFO TcpInputProc - Port 9997 is compressed
02-06-2011 19:19:01.556 INFO TcpInputProc - Registering metrics callback for: tcpin_connections

Openssl
  • This is what you should see during the forwarder start-up sequence in $SPLUNK_HOME/var/log/splunkd.log :

02-06-2011 19:06:10.844 INFO TcpOutputProc - Retrieving configuration from properties
02-06-2011 19:06:10.848 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
02-06-2011 19:06:10.848 INFO TcpOutputProc - found Whitelist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
02-06-2011 19:06:10.848 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
02-06-2011 19:06:10.850 INFO TcpOutputProc - Will retry at max backoff sleep forever
02-06-2011 19:06:10.850 INFO TcpOutputProc - Using SSL for server 10.1.12.112:9997, sslCertPath=/opt/splunk/etc/aut/server.pem
02-06-2011 19:06:10.854 INFO TcpOutputProc - ALL Connections will use SSL with sslCipher=
02-06-2011 19:06:10.859 INFO TcpOutputProc - initializing single connection with retry strategy for 10.1.12.112:9997

  • And this is what a successful connection attempt will look like indexer-side :

02-06-2011 19:19:09.848 INFO TcpInputProc - Connection in cooked mode from 10.1.12.111
02-06-2011 19:19:09.854 INFO TcpInputProc - Valid signature found
02-06-2011 19:19:09.854 INFO TcpInputProc - Connection accepted from 10.1.12.111

  • ...and forwarder-side :

02-06-2011 19:19:09.927 INFO TcpOutputProc - attempting to connect to 10.1.12.112:9997...
02-06-2011 19:19:09.936 INFO TcpOutputProc - Connected to 10.1.12.112:9997


4 - Troubleshooting :

  • First, check in $SPLUNK_HOME/var/log/splunk/splunkd.log on both ends for errors. On the indexer, check for the messages from the TCP input processor 'TcpInputProc', and on the forwarder, check the messages from the TCP output processor 'TcpOutputProc'.
  • In general, it is a good idea to increase the logging level of the appropriate processors on the indexer and the forwarder in $SPLUNK_HOME/etc/log.cfg.
    On the forwarder, set 'category.TcpOutputProc=DEBUG', on the indexer set 'category.TcpInputProc=DEBUG'. Restart Splunk for these to take effect and observe the start-up sequence for the pertinent component. Most configuration issues are explicitly revealed by this method.
  • Check the SSL configuration as it is seen by Splunk using btool.
    • On the indexer :
      $SPLUNK_HOME/bin/splunk cmd btool inputs list --debug
    • On the forwarder :
      $SPLUNK_HOME/bin/splunk cmd btool outputs list --debug
  • Make sure that the certificates are readable by the user that Splunk runs as. Indexer-side, two common problems are :
    • The path to the server certificate file set as the value of 'serverCert' in inputs.conf is wrong, or the file cannot be read. This will generate the following error :
      12-16-2010 16:07:30.965 ERROR SSLCommon - Can't read certificate file /opt/splunk/etc/auth/server.pem errno=33558530 error:02001002:system library:fopen:No such file or directory
    • The password to the RSA private key contained in the server certificate file is wrong. This password is set as the value of 'password' in inputs.conf. This will generate the following error :
      12-07-2010 07:56:45.663 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem

On *nix, you can manually test the password of the RSA key contained in the file by running the following openssl command :
# openssl rsa -in /opt/splunk/etc/auth/server.pem -text

The same can be done on Windows with the openssl binary that ships with Splunk :
C:Program FilesSplunkbin>openssl.exe rsa -in 'c:Program FilesSplunketcauthserver.pem' -text

  • More information regarding the configuration of splunk2splunk SSL connections can be found here in the online documentation:

The appropriate sections of the spec files for inputs.conf and outputs.conf are also a very good resource :

These files can be found in $SPLUNK_HOME/etc/system/README.

Retrieved from 'https://wiki.splunk.com/index.php?title=Community:Splunk2Splunk_SSL_DefaultCerts&oldid=57402'

Overview

Openssl

Lately, Splunk has made the secured connections (https protocol) between the Splunk instance and the client a requirement for the Splunk Add-on certification. Hence the PureStorage Unified Add-on for Splunk adheres to this requirement and has eliminated the check-box option like 'Verify SSL Certificate' when configuring the array using the technology add-on (TA). This means, it is not only required to have an SSL certificate at the array level but the certificate can be verified.

The Challenge?

Splunk Openssl Version

Configuring an array using the new TA is not an issue if you are using commercial SSL certificates for the FlashArray or FlashBlade as it is issued by a trusted Certificate Authority (CA) and Splunk includes various trusted CA root certificates as part of the Add-on builder.

What if you are not using commercial SSL certificates for your FlashArrays or FlashBlades but relying on the self-signed certificates created by the array itself as part of the installation?

You will run into an error message as follows when trying to configure an array on the PureStorage Unified Add-on because the certificate cannot be verified against the list of trusted CA root certificates given this was self-signed.

Solution

This can be overcome by following the two-step process that will allow the self-signed certificates to work without any errors.

  1. Create a new self-signed certificate with the 'Common Name' matching the Server Address that will be entered in the 'Add Account' dialog box.
  2. Add the certificate into the cacert.pem file under the following directories.
    1. $SPLUNK_HOME/etc/apps/TA-purestorage-unified/bin/ta_purestorage_unified/aob_py3/certifi/cacert.pem
    2. $SPLUNK_HOME/etc/apps/TA-purestorage-unified/bin/ta_purestorage_unified/aob_py2/certifi/cacert.pem

Creating a new self-signed certificate

The self-signed certificate created by Pure Storage generally has the Common Name field as 'Pure Storage' and hence if you export this certificate and add it to the cacert.pem file you will still encounter the following error message.

Here are the steps to generate a new self-signed certificate with the 'Common Name' matching the Server Address.

The process is different between FlashArray and FlashBlade as FlashArray offers the option of generating a new self-signed certificate from the GUI but FlashBlade is yet to implement this functionality.

FlashArray

1. Select Create Self-Signed Certificate option from the SSL Certificate frame within the Settings=>System tab of the FlashArray GUI.

2. Enter all the fields that are blank and make sure the Common Name field matches the management IP address or FQDN of the array.

The entry you are providing against Common Name should resolve to the Array either through DNS or the hosts file on the Splunk server.

3. After providing all the details, click Create and select Yes to override the current certificate.

4. After five seconds, the FlashArray GUI is reloaded prompting you with the warning message about the certificate. Accept it to establish a new secured connection.

5. Export the certificate that was just created from the SSL Certificate frame under the Settings=>System tab.

6. Copy the certificate or download the certificate.

7. Open the $SPLUNK_HOME/etc/apps/TA-purestorage-unified/bin/ta_purestorage_unified/aob_py3/certifi/cacert.pemfile in an editor and paste the certificate at the end of the file.

Alternatively, if you have downloaded the certificate, you can concatenate the certificate file to the cacert.pem file. Make sure the certificate file is transferred to the Splunk instance where the PureStorage Unified Add-on is installed.

FlashBlade
Splunk openssl training

1. Generate a new self-signed certificate from a Unix host using the openssl command.

Provide the details when prompted and ensure the Common Name matches the FQDN or the management IP address of the FlashBlade.

Transfer the selfsigned-cert.pem and private_key_nocrypt.pem file to your laptop/server where you are accessing the FlashBlade GUI.

2. Import this new certificate along with the private key to the FlashBlade by choosing Import Certificate option under SSL Certificate under Settings=>System tab (Purity//FB <= 3.1.x) or Certificates under Settings=>Certificates=>Array Certificates (Purity//FB >= 3.2.x)

Purity//FB >= 3.2.x

3. Provide the certificate files that were generated from step 1. Select the file selfsigned-cert.pem for Certificate field and private_key_nocrypt.pem for the Private Key field and click Import.

4. Refresh the FlashBlade GUI due to the change in the certificate. The SSL Certificate frame should show the certificate details.

5. Export the certificate that was just created from the SSL Certificate frame under the Settings=>System tab (for Purity//FB <= 3.1.x) or Certificates under Settings=>Certificates=>Array Certificates (Purity//FB >= 3.2.x).

6. Copy the certificate or download the certificate.

7. Open the $SPLUNK_HOME/etc/apps/TA-purestorage-unified/bin/ta_purestorage_unified/aob_py3/certifi/cacert.pemfile in an editor and paste the certificate at the end of the file.

Alternatively, if you have downloaded the certificate, you can concatenate the certificate file to the cacert.pem file. Make sure the certificate file is transferred to the Splunk instance where the PureStorage Unified Add-on is installed.

Workaround

Splunk Openssl Vs

X509

Using the commercial certificate or self-signed certificate as described above are the suggested approaches to have a secured connection between the array and the Splunk instance. But in exceptional cases, if you would like to disable the SSL verification and do not go through generating self-signed certificates and updating the cacert.pem file, you can use the following workaround.

Please use the workaround at your own discretion as you have to edit a python script in the backend to disable the SSL verification. This is not Pure or Splunk's suggested approach. Never use this workaround in the production environment.

1. Navigate to the path $SPLUNK_HOME/etc/apps/TA-purestorage-unified/bin directory on the Splunk instance where the Add-on is installed.

Splunk Openssl Certification

2. Edit the file purestorage_unified_utils.py file and look for the following code immediately under the import statements.

Splunk Openssl

Openssl

Splunk Openssl Command

3. Change the value from True to False to disable the SSL verification.