Getting the best from your Splunk Implementation
17/07/2018 – Author: Andy Gibbs – Security Consultant
Somerford Associates has assisted hundreds of organisations across the UK to achieve a successful implementation of Splunk, helping to integrate their tooling into the heart of our customers’ businesses to provide effective management of their IT services. Sadly, we’ve also been called upon to help out some organisations who were struggling to realise the benefits they were anticipating from Splunk. So what are the keys to a successful implementation and what are some of the pitfalls? I’d like to share some tips based upon our experiences (good and bad) from working with our customer base.
The ‘holy trinity’ – people, process and technology
A three-legged stool is inherently stable but remove one leg and it falls over. Successful implementation of any new system into operations needs a careful blend of people, process and technology. This is particularly true when implementing Splunk, and we’ve found that where our customers have struggled, this has often been due to lack of attention in one or more of these key aspects.
- People – Splunk is a feature-rich and sophisticated tool, so it’s important to ensure that employees are adequately trained in its use to the level at which they will be expected to operate it. Putting a novice driver at the wheel of a Formula 1 car is likely to have catastrophic results. Skills need to be developed over time and built upon as experience, skill-sets and aptitude to operate the tolling effectively. Include Splunk skills in regular reviews of relevant employee development and appraisal. if possible, establish internal user forums and take advantage of sector specific user groups. Attend Splunk events and webinars. These are great ways of further developing skills, exchanging knowledge and getting the best from the tools.
- Process – Splunk is a disruptive technology and its implementation will often require a reappraisal of supporting processes to obtain optimal effectiveness. There’s little value in alerting on some new operational or security even if there is no supporting process to respond to the issue and ensure it is properly resolved. Implementation of Splunk works well in conjunction with a review of operational and security practices. A process review using a standards framework such as ITIL, ISO 27001, Cyber Essentials or the CIS Top 20 Controls is a great way to organise this, and a number of white-papers are available explaining how Splunk can help support such initiatives (for example see Splunk and the CIS Critical Security Controls).
- Technology – Great. So now we’ve got a well trained, highly competent team, using Splunk to support a robust set of operational processes, but how is the technology performing for you? Good preparation, based upon a clear understanding of the expected business outcomes will help enormously here. Conduct a data source Assessment to be clear about what data you need to gather, its source location and how Splunk will be used to access or gather this. Ensure that your Splunk architecture is well configured for the data sources and volumes you’ll be handling. Performance test new reports and dashboards where these are due to become part of your regular operational monitoring patterns. Poorly devised searches and reports can have significant impact on system resources so make sure you have good control over who is able to create and launch ad-hoc reports and ensure they are well versed in the basic rules for creating efficient search processing language (SPL) and other query types. if performance remains an issue and you don’t understand why, Somerford can work with you to perform a data/architectural health check to help address the issue.
Just Starting Out?
If you are in the earlier stages of implementation, take care to understand your business, service and IT priorities and focus on addressing these first. It is easy to get swept away with the functionally rich features Splunk has to offer, especially for the technically-minded. That’s not to say you shouldn’t explore the extensive capabilities of Splunk – but there’s a time and place for experimentation. Assess your security and operational risks and opportunities, and use these to prioritise activities that will yield the biggest benefits early on. Keep your initial reports and dashboards simple and make any refinements once you’ve proven they work. Always avoid using your live production environment for trying out new features or testing new reports – build a sandpit for the ‘techies’ and actively encourage them to try out new features and functions there. Keep your stakeholders informed of progress regularly and demonstrate successes as they happen.