The Telemetry Pipeline Playbook 

Introduction 

Every organisation we work with has the same problem. They are drowning in data. Log volumes are growing 20–40% year-on-year. SIEM and observability platform costs are spiralling. Security and operations teams are overwhelmed by noise, and the data they actually need is buried in a flood of verbose, duplicated, low-value telemetry. 

The root cause is architectural. Most organisations send 100% of their raw data directly to expensive analytics platforms. There is no processing layer in between, no place to filter, aggregate, enrich, or route data intelligently before it hits a licensed backend. The result is runaway costs, degraded platform performance, and analysts spending more time managing data than analysing threats. 

This playbook introduces the concept of the telemetry pipeline — the “missing middle layer” between your data sources and your analytics platforms. It explains why pipelines are the single most impactful architectural decision you can make for cost control and data quality, and provides a practical framework for implementation using Cribl Stream and OpenTelemetry. 

The Telemetry Pipeline Architecture

A telemetry pipeline is a dedicated processing layer that sits between your data sources (applications, infrastructure, security tools) and your analytics destinations (SIEM, observability platform, data lake). Its job is to receive all telemetry, process it intelligently, and route it to the right destination at the right cost. 

The Two-Tier Model 

A well-designed pipeline follows a two-tier architecture: 

Tier 1 — Edge Collection: Lightweight agents run on your hosts, containers, or pods. Their only job is to collect local telemetry (logs, metrics, traces) and forward it to the central pipeline as quickly as possible, with minimal local processing. This tier uses OpenTelemetry Collectors, Cribl Edge agents, or vendor-specific forwarders. 

Tier 2 — Central Processing: A dedicated, horizontally scalable fleet of pipeline processors (Cribl Stream or OpenTelemetry Collector gateways) receives all data from Tier 1. This tier performs the heavy processing: filtering, aggregation, enrichment, transformation, and routing. It is the control plane for your entire data architecture. 

This separation of concerns is critical. Edge agents stay lightweight, ensuring minimal impact on application performance. Central processing is independently scalable and provides a single point of control for data policies. 

The Core Pipeline Functions 

Filtering: Dropping the Noise 

The most immediate value from a pipeline is filtering — removing known low-value data before it reaches your expensive platforms. Common examples include: 

• Health check and heartbeat logs (often 30–50% of web server log volume) 

• Debug and verbose-level logs from production systems 

• Successful authentication events in high-volume environments 

• Kubernetes orchestration events that repeat every few seconds 

• Duplicate events from overlapping collection agents

Aggregation: Summarising at Scale 

Aggregation converts high-volume, low-value raw data into low-volume, high-value summaries. Instead of ingesting one million individual 200 OK web server logs (each carrying full request details), the pipeline can aggregate them into a single metric: “1,000,000 successful requests to /api/checkout in the last 5 minutes with a p99 latency of 245ms.” 

The raw logs are gone, but the operational intelligence is preserved — at a fraction of the storage and licence cost. For compliance, the raw data can simultaneously be routed to cheap object storage. 

Enrichment: Adding Context 

Raw telemetry often lacks the context that analysts need. A log event might contain an IP address, but without knowing that IP maps to a specific user, department, or geographic location, the analyst must manually cross-reference multiple systems. Pipeline enrichment adds this context in-stream: 

• GeoIP lookups on source and destination IP addresses 

• User identity enrichment from Active Directory or CMDB 

• Asset classification (production vs development, critical vs non-critical) 

• Threat intelligence lookups against known indicators of compromise 

Enriched data reduces Mean Time to Resolution by eliminating manual cross-referencing during incident investigation. 

Routing: Right Data, Right Destination, Right Cost 

The pipeline’s routing capability is what transforms your data architecture from a single-destination firehose into an intelligent, cost-optimised distribution network. A single data stream can be routed simultaneously to: 

• Your SIEM (high-value security events only, enriched and normalised) 
• Your observability platform (metrics and sampled traces for performance monitoring) 
• Cheap object storage like S3 (100% raw data for compliance and retroactive analysis) 
• A security data lake (OCSF-normalised events for advanced threat hunting) 

This “full-fidelity-to-lake, high-signal-to-tool” model is the most cost-effective architecture available. You keep everything for compliance, but you only pay premium licence costs for the data that delivers real-time operational value. 

Sampling Strategies for Trace Data 

Distributed traces are invaluable for observability but generate enormous data volumes. Effective sampling is essential for cost management without sacrificing diagnostic fidelity. 

Head-Based Sampling 

Makes a keep/drop decision at the start of a trace (e.g., “keep 10% of all traces”). This is simple to implement but has a critical flaw: it will inevitably drop the error traces and high-latency traces that you most need to diagnose problems. At 10% sampling, you have a 90% chance of missing any given incident. 

Tail-Based Sampling 

Buffers 100% of trace spans, waits for the trace to complete, then makes a keep/drop decision based on the complete trace. You can configure rules like “always keep traces with errors”, “always keep traces above 2 seconds latency”, and “sample 5% of healthy traces.” This preserves 100% of your diagnostically valuable traces while dramatically reducing volume. 

The Data-Lake-First Model 

The most sophisticated approach inverts the traditional cost model entirely: 

1. Collect 100% of all telemetry at the edge 
2. Route 100% of raw data to cheap object storage (S3, Azure Blob, Cribl Lake) for compliance and retroactive analysis 
3. In the same pipeline, apply filtering, tail-based sampling, and aggregation 
4. Route only the processed, high-signal data (typically 1–10% of the original volume) to your expensive real-time platforms 

This gives you full-fidelity data for compliance and forensics, with dramatically lower real-time platform costs. When an incident occurs, you can retroactively search the full data set in the lake, then promote relevant data to your SIEM for detailed analysis. 

Next Steps 

Ready to take action? Apto Solutions offers a range of entry-point engagements designed to give you clarity before commitment: 

• Free Assessment: A no-obligation conversation with one of our platform specialists to understand your current state and identify quick wins. 

• SIEM Health Check: A structured review of your existing SIEM deployment covering architecture, detection coverage, data quality, and operational efficiency. 

• Observability Maturity Assessment: A framework-driven evaluation of your monitoring and observability capabilities against industry best practice. 

• Data Mapping and Discovery: An analysis of your telemetry data flows, identifying redundancy, gaps, and optimisation opportunities. 

Book your free assessment

Or email enquiries@aptosolutions.co.uk to start the conversation.

Assessed. Architected. Operated.