13 June 2019

A more dynamic Splunk environment


Dynamic Splunk Deployment

In the last blog we touched on how Splunk HECs can be used to create a more dynamic Splunk environment. In this blog we will dig a little deeper to think about which tiers of the Splunk architecture can be built in such a way.

Firstly let’s looks look at desirable cloud best practices and how a classic Splunk deployment may fail to meet it.

Best PracticeSplunk
High availability Achieved with Indexer Clusters and Search head clusters. However many single points of failure remain such as heavy forwarders with TAs (Technical Addons)
ScalabilityAgain achievable with Indexer Clusters and Search head clusters. However heavy forwarders lack horizontal scalability when using TAs which means the forwarding tier struggles when confronted with a huge data source, such as, cloud traffic logs which can total millions per hour
Keep configuration as simple as possibleCompanies often lack Splunk proficiency but have plenty of cloud resources. How can we take the work away from technical Splunk knowledge? 
Use cloud managed services where possibleUnless using Splunk Cloud it is unlikely your solution is cloud managed
Go serverlessUp until now Splunk deployments have almost always used heavy duty virtual machines, can we go serverless?

Do tiers of the Splunk architecture exist in which we can we instil these best practices with a simple and elegant design? Most definitely!

The previous blog explains why TAs cause issues for High availability and scalability. So the first solution to making this tier dynamic is to replace all TAs with a HEC (HTTP Event Collector solution). We now have high availability because HECs can exist on multiple Splunk instances with the same token and no fear of data duplication.


Now we move onto scalability, How do we scale Heavy Forwarders up and down without worrying about configuration to maintain? In the cloud a simple autoscaling group and load balancer can be used alongside the HEC configuration to achieve not only scalability based on load but a single load balanced HEC endpoint that we can use without fear of changing any configuration.

Already we have kept the configuration as simple as possible for these Splunk instances, we only really require four pieces of configuration – The HEC, the index tier endpoint for forwarding, the license master endpoint and possibly the index time knowledge of the data being sent and received. This configuration is easy to compile, and requires almost no maintenance. It can either be placed on the server at installation time or received from a deployment server.

So we have achieved an easy to configure, highly available and scalable solution that requires almost no Splunk knowledge beyond installation. To take this to the next level we need to make our solution cloud managed and serverless thereby removing the need for installation and maintenance. What if I told you this could be achieved a single line command? Well it can!

This is where the architecture becomes really beautiful. Splunk Docker is perfect for creating these discardable instances in 30 seconds or less. The only notable configuration here is the definition of the HEC and the pointer to the deployment server. Upon creation, it is using Splunk:latest – the latest version of Splunk so the system will keep up to date for you. By running an autoscaling group of these docker containers in a cloud managed service such as ECS or EKS we have now achieved our final two goals.

In Summary

In summary lets run through what we’ve achieved with this solution:

  • A cloud managed service with 99.99% availability and 30 second or less up time on new containers
  • Scalability to almost any load that can cope with even the largest data feeds such as VPC Flow Logs. By configuring an AWS Firehose or Azure Event Hub we can send practically any cloud data we’d like straight to Splunk. The lack of static architecture also means you can reduce your cloud costs when necessary.
  • A very config light solution that keeps the setup out of the Splunk side and in the cloud
  • A completely serverless architecture tier
  • The system will keep up to date with the latest Splunk versions

Now we’ve turned one Splunk tier into an elegant solution can we achieve the same with any more? Besides the index tier, the answer is most definitely yes.

Find out more about our Splunk Consultants

Liked our article?

Subscribe and get the latest from us every month or so :)

We'll never send spam or sell your data, see our privacy policy

See how we can build your digital capability,
call us on +44(0)845 226 3351 or send us an email…