Licence Growth and Role Management in Splunk Environment

Overview

A prominent company approached us with a pressing issue regarding the size and growth rate of their Splunk licence. As a data-driven organisation, they were ingesting large amounts of data but were not managing or storing it efficiently. The lack of a standardised approach to managing their indexes and roles in Splunk was driving their licence growth at an unsustainable pace, leading to spiraling costs. Our goal was to help them reduce their licence size and optimise their data management processes.

 

The Challenge

The company’s primary concern was the rapid and erratic growth of their Splunk licence. Their users were ingesting data without proper oversight from system admins, which led to inconsistent licence increases. This uncontrolled ingestion not only inflated their costs but also complicated their Splunk environment.

Moreover, we uncovered a broader issue during our investigation: their role-based access control (RBAC) system within Splunk was spread across 20 different roles, resulting in confusion and inefficiencies. Users were granted access to ingest data without following a proper approval process, and there was a lack of accountability across the departments using the platform. This uncontrolled access was costing the company more money and complicating the management of their Splunk environment.

 

Why They Chose Us

The company sought our help because of our deep knowledge of Splunk’s data feeds and our expertise in networking and licence management. We were able to jump straight into the problem-solving process without requiring much guidance from the client, which was crucial given the complexity of their environment. Our ability to propose immediate solutions to work around the growing licence costs made us an ideal partner for this engagement.

 

Approach and Solution

To tackle the licence growth, we began by creating a model to analyse their current licence usage. We developed a detailed spreadsheet showing which indexes and roles were contributing the most to the data being ingested into Splunk. This allowed the client to identify the key areas where they needed to reduce data ingestion. One critical aspect we focused on was the retention rates of their data, which had a direct impact on the size of their licence. Many users weren’t aware of how their retention settings affected data storage, so we presented the findings in a way that allowed them to interact with the data and simulate different retention settings.

Given that they didn’t have a separate testing environment, we modeled these changes outside of their live environment to mitigate any risks of losing important data. This model helped them understand the effects of reducing retention times without affecting their actual system.

Additionally, we created a technical note outlining how they could offload the data causing excessive storage costs to AWS. By doing this, they could store less critical data externally and bring it back into Splunk only when needed, which reduced costs significantly.

We also discovered inefficiencies in their RBAC system. As a result, we worked on streamlining the roles in their Splunk environment by implementing a more structured role-based access control system. This involved reviewing the roles, speaking with stakeholders, and understanding how different users interacted with the platform. Ultimately, we consolidated roles and implemented a more efficient access control system to prevent unchecked data ingestion and improve system management.

 

Outcome

As a result of our engagement, the company now has a better understanding of their licence growth and the underlying causes. They are now spending significantly less on their Splunk licence, and the systems team is empowered to manage data ingestion more effectively. Additionally, with a restructured RBAC system, the company has greater control over who is using Splunk, what data they are ingesting, and how it’s being stored.

 

Business Impact

From a business perspective, the company can now avoid unnecessary licence increases and has a clear understanding of where data ingestion is coming from. This has led to a more cost-efficient operation, with the systems team better equipped to communicate with other departments about their Splunk usage. Furthermore, this process has prompted them to take a more critical look at how Splunk is used across the organisation, ensuring that data ingestion is intentional and managed properly.

Recommendations for Other Splunk Users

For other Splunk users facing similar challenges, we recommend the following:

• Understand Your User Base: Ensure there is a clear understanding of who is using Splunk and why. Overloading the platform with users without proper oversight can lead to significant inefficiencies.
• Manage Data Retention: Regularly review and adjust your retention policies. Data retention has a massive impact on your licence size and costs.
• Educate Users and Admins: Provide training for both users and system admins on how to use Splunk efficiently. This can prevent costly mistakes, such as excessive storage costs due to improper use of safe searches and lookups.
• Centralise Role Management: Ensure that access control and data ingestion responsibilities are well defined and that roles are consolidated and structured to avoid confusion and inefficiencies.

This case study demonstrates how proactive management and optimisation of Splunk environments can lead to significant cost savings and improved operational efficiency.