Why intelligence needs to move upstream in the modern tech stack

Modern systems generate more telemetry and data than ever, but more telemetry and data haven’t made teams faster or more productive. In fact, it’s doing the opposite. Fragmented telemetry data signals, rising storage costs, and delayed insights are making it harder to understand and act when it matters most. As systems become more dynamic and ephemeral, teams often find themselves dealing with fragmented telemetry and unclear signals. It takes longer to understand what’s gone wrong, and even longer to fix it. At the same time, the cost of storing and processing all this telemetry and data continues to rise.

 

There’s a growing sense that something needs to change. Instead of trying to analyse everything after the event, organisations are starting to look at how they can build better intelligence into their systems from the start, so the telemetry they rely on is more relevant and easier to act upon.

 

 

The real cost of poor telemetry and data quality

The challenge isn’t that organisations are short of telemetry and data; it’s that they’re short of good quality signals. As AI-driven capabilities become deeply embedded in software delivery and operations, the quality of observability telemetry becomes even more important. When the telemetry is collected and unnecessary telemetry is filtered out, employees end up spending more time managing the telemetry than learning from it.

 

Part of this issue is fragmentation: when telemetry is stored across several tools and agents, each of which generates signals with somewhat varied forms. During an outage, teams are forced to stitch together inconsistent signals across tools, losing valuable time while the business impact grows.

 

The financial part comes next. Large-scale telemetry and data processing and storage are expensive, and the costs are only rising. Organisations are spending a significant amount of their resources on processing and storing data that only adds noise, not insights. In the traditional sense, this isn’t a technology problem. It’s a telemetry and data quality concern, and the longer it’s ignored, the worse it usually gets.

 

 

Moving intelligence upstream

Moving intelligence upstream means filtering, enriching and routing meaningful telemetry before it reaches downstream systems. Increasingly, organisations are starting to think about how to address these fragmentation and financial challenges. Teams are beginning to collect, process, and route relevant observability telemetry in real-time before it is ingested into downstream systems. As a result, there is less volume to handle, reduced expenses and cleaner telemetry to deal with since only the relevant signals are sent downstream.

 

This is becoming increasingly feasible at scale thanks to open standards such as OpenTelemetry. For example, teams can gather telemetry consistently across multiple environments without being dependent on a particular vendor or set of tools. When working with intricate, distributed systems that span several clouds, security and storage platforms, that type of flexibility is crucial. Furthermore, as environments continue to evolve, it makes adapting to them that much easier.

 

 

What this means for AI-powered operations

AI systems are only as robust as the telemetry and data that powers them. They depend on accurate, contextualised telemetry to generate insights and automate decisions. Teams are not only slowed down by incomplete or cluttered telemetry, but it’s this simple: poor-quality telemetry leads to poor-quality insights. When those insights are applied to large-scale automated decision-making, the margin for error drastically lowers.

 

Another aspect that is frequently disregarded is trust. The teams in charge of these automated systems must be able to explain what happened and why the systems are making decisions about performance or security. If the information supporting those choices is inconsistent, it becomes extremely challenging. This matters far beyond just the technical teams. Senior management and board members are increasingly being asked to support AI-driven choices, and this becomes a much simpler discussion when the telemetry and data behind the insights are credible and trustworthy.

 

The operational advantages are also evident. Teams can work quicker if they aren’t continuously battling erratic telemetry and data. They spend less time on manual research, detect issues early, and automate with greater confidence. A strong telemetry and data foundation and a unified, context-driven platform not only enhance daily operations but also increase the successes of the AI-powered capabilities the organisation invests in over time.

 

 

The next steps start at the source

For organisations to innovate faster and deliver greater value to their customers, they must have the right telemetry, in the right signal, at the right time. The good news is that it doesn’t require starting from scratch; it’s about making choices early in the telemetry lifecycle.

 

By bringing intelligence earlier into the lifecycle, teams can ensure that the telemetry flowing into their observability platforms is scalable from the start. The organisations that win won’t be the ones with the most telemetry; they’ll be the ones with the best telemetry, shaped at the source and ready to drive action.

 

---

 

Mala Pillutla is WW VP - Observability & Security at Dynatrace

 

Main image courtesy of iStockPhoto.com and gremlin