A more dynamic Splunk environment

Dynamic Splunk Deployment

In our previous blog we reviewed Technical add-ons (TA’s) that can be used in your Splunk environment.

TA’s are used to gather data from your cloud and input into Splunk. As we identified, these are not always the best solution, so we went on to review the benefits of using HTTP Events Collectors (HECs).

HECs are the more preferred option in your Splunk environment as they provide a robust and scalable method of data collection. Having established the use of HECs as the longer-term solution for data collection, in this blog we explore the different requirements of Splunk architecture you’ll need to ensure a seamless flow of data into Splunk. We go on to discover how these can be configured for maximum success and effectiveness. 

Firstly, let’s take a look at how a classic Splunk deployment stacks up against desired cloud-based practices and why Splunk’s standard approach may fail to meet your requirements.  

So, the question we now ask is: can we overcome these aspects of Splunk architecture and still meet best practice with a simple and elegant design? Most definitely!

High Availability

The first aspect to consider is high availability. The conventional solution is to use TA’s, but as we have already identified these have several issues. The alternative is to use HECs (HTTP Event Collectors). These achieve high availability, which as we have stated is best practice, because HECs can exist on multiple Splunk instances with the same token with no fear of data duplication. 

Scalability

Now we move onto scalability. The biggest question here is: how do we scale heavy forwarders up and down without worrying about maintenance of configuration? Answer: by using a cloud-based solution with a simple autoscaling group and load balancer. You can use this alongside the HEC configuration to achieve not only scalability based on load but also a single load balanced HEC endpoint that can be used without fear of changing any configuration. 

With this solution we keep the configuration as simple as possible for these Splunk instances. Therefore, we only really require four pieces of configuration:

1. the HEC
2. the index tier endpoint for forwarding
3. the license master endpoint
4. possibly the index time knowledge of the data being sent and received

This configuration is easy to compile and requires almost no maintenance. It can either be placed on the server at installation time of received from a deployment server. 

Moving to a cloud-based solution

So far, we have achieved an easy to configure, highly available and scalable solution that required almost no Splunk knowledge beyond installation. To take this to the next level we need to make our solution cloud managed and serverless thereby removing the need for installation and maintenance. What if we told you this could be achieved with a single line command? Well it can!

This is where the architecture becomes really beautiful. Splunk Docker is perfect for creating these discardable instances in 30 seconds or less. The only notable configuration here is the definition of the HEC and the pointer to the deployment server. 

Upon creation, it uses Splunk: latest – the latest version of Splunk – so the system will keep up to date for you. By running an autoscaling group of these docker containers in a cloud managed service such as ECS or EKS we have now achieved our final two goals. 

In Summary

Let’s run through what we’ve achieved with the solution we have outlined above:

• A cloud managed service with 99.99% availability and 30 second or less up time on new containers.
• Scalability to almost any load. Being able to cope with even the largest data feeds such as VPC Flow Logs. By configuring an AWS Firehose or Azure Event Hub we can send practically any cloud data we’d like straight to Splunk. The lack of static architecture also means you can reduce your cloud costs when necessary.
• A very configuration light solution that keeps the setup out of the Splunk side and in the cloud.
• A completely serverless architecture.
• Automatic updates, keeping the system up to date with the latest Splunk versions. 

Now we’ve created a dynamic Splunk environment with an elegant solution, can we achieve the same with other tiers of Splunk? Besides the index tier, the answer is most definitely yes.

If you have any questions about your Splunk environment, please do get in touch on +44(0) 845 226 3351 or email is at enquiries@aptosolutions.co.uk

Or find out about our other Splunk solutions