Setting up SplunkWeb with Third Party Certificates End-to-End on a Built Environment (Part 1)
Author: Laurence Everitt
Release Date: 20/03/2024
Introduction
Recently, I was working on a project which included the need to promote the communications in Splunk from using regular unencrypted HTTP to encrypted HTTPS from browser to SplunkWeb and also replace the Splunk-provided certificates for Splunk to Splunk communication, so that all communication with Splunk is secure. The documentation from Splunk which says “generate a CSR, send it to the Certification Provider and then apply the certificates to Splunk” or “how to set up a self-signed certificate for Splunk”, which doesn’t provide much insight if you don’t know how to generate a CSR in the first place.
So, to help others (and maybe myself in a few years, here is the whole process, end-to-end (truly), including various gotchas in the process, so that you can learn from my experience and include as much background as necessary to get the job done (but no, this is not going to be the extremely complex openssl manual regurgitated).
This is the first part of a 3 part explanation of how to uplift SplunkWeb from presenting a regular HTTP website to using a full-blown SSL/TLS HTTPS website, end-to-end. Please see the other parts, The Details and Troubleshooting.
The Environment
For this guide, I have set up the following demo Splunk Enterprise system. It is not exhaustive, but does demonstrate the items that you will need to be aware of in most cases:
Role(s) | Server FQDN | Machine Name |
|---|---|---|
Licence Master Cluster Master Search Head Deployer Deployment Server | manager.example.com | sa-vm-pool-05
|
Search Head Cluster
| shcluster.example.com
| sa-vm-pool-06
|
Indexer Cluster | idxcluster.example.com | sa-vm-pool-07
|
Universal Forwarder | uf.example.com
| sa-vm-pool-08 |
NOTE: This small setup is PURELY for demonstration purposes. This Splunk system was created EXPRESSLY for this blog to provide the screenshots and to demonstrate a functional system in the smallest footprint possible. NEVER try to create such a system for production purposes - it is not in line with Splunk good practice; Splunk provides documentation on the instance colocation here. You have been warned - don’t try this at home.
This system is running Splunk 9.1.1 on 64-bit CentOS 7 and the encryption provider is an enterprise CA. I have chosen this combination of setups as:
• It is free to set up (including the 30-day Full Splunk Enterprise Splunk License) and
Many organisations have an enterprise CA provider.
• The system is set up with out-of-the box Splunk-to-Splunk encryption (using default Splunk HTTPS certificates) and SplunkWeb is running on HTTP (port 8000) and it is fully functioning.
The system is set up with out-of-the box Splunk-to-Splunk encryption (using default Splunk HTTPS certificates) and SplunkWeb is running on HTTP (port 8000) and it is fully functioning.
Splunk is installed in (and SPLUNK_HOME is set to) /opt/splunk or /opt/splunkforwarder on Enterprise or UF devices, respectively.
NOTE: Although I use the term “SSL” in the document (like many public documents), as the Secure Sockets Layer protocol has been deprecated, this document is actually referring to the subsequent standard of Transport Layer Security (TLS) protocol instead.
The Prerequisites
The following are prerequisites for setting up Splunk to encrypt communications with certificates:
• Ensure that DNS for the servers is set up and working between each server.
• Ensure that there is no time variance between the servers (especially the date!) Configure NTPd and/or W32Time. This is for two reasons:
1) Server time consistency is critical for Splunk - the whole system works on time and
2) if time/date is enough out on a machine, the certificates might be invalid.
• Alternate ports (port 443 or 8443) for encrypted communication are opened on operating system and infrastructure firewalls on the Splunk servers that run SplunkWeb, rather than using Splunk’s default 8000.
• GOTCHA: Many versions of Linux stop not-root users from listening on ports below 1024, so if you want to use port 443 on the server, you will need Splunk to listen on a high port (eg 8443) and then use the OS firewall to redirect inbound traffic from port 443 to Splunk on 8443.
The High Level Process
Overall, the process for elevating Splunk to using third-party certificates is:
• Create server private keys and Certificate Signing Requests (CSR) for each server
• Submit the CSRs to the Certification Authority (CA) (may be a third party CA)
• Receive the Certificate Bundle(s)
• Extract the bundle(s)
• Convert the certificate bundle to Splunk’s PEM format
• If the CA Root Certificate is not trusted by the client browser, add this certificate as a trusted root certificate on the client machines that will be used to connect to Splunk. As this is an extremely rare situation and is performed by Active Directory/Group Policy or another external product, this operation is not included here.
• Create a web certificate app
• Copy the private key files and associated certificates to the Splunk servers.
• Distribute the web certificate app to all of the Splunk servers that run SplunkWeb, and test connectivity, such as:
• Licence Master
• Cluster Manager (if included)
• Search Head Deployer (if included)
• Search Head Cluster (if included)
• Deployment Server (if included)
• Heavy Forwarder(s)
• Test browser connectivity to SplunkWeb on each machine.
As mentioned previously, this blog post is part of a trilogy series. If you would like to be notified when parts 2 and 3 are released, please sign up here.
More Resources Like This One:
is a free of charge, value added service from Somerford.
