Data Pipelining and How it can benefit the Public Sector

In a recent blog we sat down with Berwyn Jones, the public sector lead at Cribl, to talk about telemetry data pipelining and how it can help the public sector. Telemetry data includes logs, metrics, and traces from IT systems and applications, which are crucial for maintaining operational efficiency and security in healthcare environments. Data pipelining involves collecting, processing, and routing data from various sources to different destinations. This process can significantly improve how organisations like Universities manage their IT and security information.

How long have you been involved in public sector?

I’ve been involved in technology into the public sector for over 20 years. During that time, I’ve worked with all the big central governments, departments and agencies e.g. HMRC, DWP, Home Office, DVLA, DHSC etc; but I’ve also worked across universities and NHS Trusts as well.

What trends have you seen over the last few years in particular, that persist today?

In those 20+ years there have been plenty of trends, but I’d say the biggest I’ve seen happening in the last few years are around the digitisation of services and providing more online services to the citizen. More organisations have also been moving data and systems into the cloud, rather than managing them on premise within their own data centers. And more recently, we’re seeing a rise in the use of generative AI which creates even more data for IT and security teams to manage.

As you’ve alluded to above there has been a lot of change in the public sector, along with adoption of technology. What ongoing challenges do you see for public sector organisations as they look to continue to deliver on these trends?

When it comes to delivering on these trends, one major challenge that I’m seeing is the massive growth in data. According to IDC, there’s a 28% compound annual growth rate in terms of the rise in telemetry data. The second challenge faced is vendor lock-in. Once they’ve got all this data and they place this data within their systems, that data then becomes proprietary to those systems, so there’s an element of vendor lock-in there. Lastly, organisations are struggling with the skills gap. It’s pretty well documented across the UK that there is an IT skill shortage, and that those problems even worse across the public sector. As a result there aren’t enough technical resources to drive through the required transformation at the pace that people want to drive it. It’s holding people back.

Read the full blog here

Additional Reading for Universities

A recent published Government paper detailed the significant impact of Cyber attacks on education institutions, highlighting higher education….

“Higher education institutions are more likely to be affected by cyber-attacks – 97% identified a breach or attack in the past year. Just under six in ten of the higher education institutions identified that they had been negatively impacted by a breach.”

Universities and schools also became prime targets for data breaches and ransomware attacks. With vast amounts of personal and financial data stored on their systems, educational institutions presented a treasure trove for cybercriminals. A notable example includes the 2024 ransomware attack on a major university in Canada, which disrupted research and student records. 

Conclusion

With the ferocious rate of growth in data and digital services students demand, the attack surface is growing rapidly.  A robust threat detection posture and data pipelining strategy is paramount.  We welcome you getting in touch for further advice.