Infrastructure teams learn a pattern that rarely makes it into budget presentations: observability typically consumes 10-30% of total infrastructure spend. For systems running at scale, that percentage often lands at the high end. Some exceed it entirely.
Most executives don't scrutinize this line item until it's already embedded in operations. By the time observability becomes visible in the budget, questioning whether the economics make sense feels academic. The infrastructure is already dependent on it.
The cost trajectory reveals itself through a pattern most organizations discover too late. Start with $50,000 annually for a mid-sized engineering team running a monolithic application—manageable, predictable. Then the architecture evolves. One organization's spend grew from $50,000 to over $14 million across fifteen years, including 40% year-over-year growth for five consecutive years. That's not exponential growth from business expansion. That's what happens when a monolith becomes twenty microservices, each generating its own telemetry.
| Architecture Stage | Annual Observability Cost | Growth Driver |
|---|---|---|
| Monolithic application | $50,000 | Baseline |
| 20 microservices | $200,000-$300,000 | 100x telemetry multiplication |
| Mature distributed system | $14,000,000+ | Compounding complexity |
Here's how the multiplication works: microservices architectures generate approximately 100x more telemetry data than monolithic applications from a decade ago. Each service boundary creates new instrumentation points. Distributed tracing across dozens of services produces data volumes that weren't part of the original budget model. That $50,000 baseline doesn't scale linearly—it compounds. Move to twenty microservices and observability costs might reach $200,000-$300,000 annually before anyone realizes the budget model broke.
The pricing mechanisms accelerate this. Datadog bills based on 99th percentile usage—if infrastructure autoscales from 100 hosts to 150 for a few hours during a traffic spike, billing reflects approximately 149 hosts for the entire month. That temporary scale event becomes a permanent cost increase. Splunk's data volume model charges based on daily ingestion, where a single Java microservice with debug logging enabled can generate 5-10GB daily on its own. Twenty microservices, each logging aggressively, can push daily ingestion past 100GB—turning a $50,000 annual commitment into $200,000+ before the architecture stabilizes.
Logs alone consume over 50% of observability spend. They're the noisiest, highest-volume signal—and the hardest to optimize. Organizations collect everything "just in case," paying premium prices for data they rarely query. The fear of missing critical information during an incident drives over-collection. The economics penalize caution.
The visible costs are only part of the structure. Infrastructure teams discover categories that don't appear on vendor invoices:
Data retention economics create deferred costs. Longer retention periods increase storage expenses, forcing organizations to balance regulatory requirements against escalating bills. Keeping logs searchable for 90 days instead of 15 can triple storage costs. Most teams don't discover this until compliance audits force retention policy decisions.
Maintaining observability infrastructure itself consumes engineering capacity. Someone has to manage dashboards, optimize queries, tune alert thresholds, and investigate why last month's bill spiked 30%. That's headcount that could be building features. Organizations rarely budget for the operational overhead of operating their observability platform.
Ephemeral infrastructure creates permanent costs. In cloud environments where containers spin up and down constantly, each short-lived instance generates telemetry. The infrastructure might be temporary, but the observability costs are permanent. That asymmetry creates cost structures that don't map cleanly to business value.
Below 10,000 hosts or 100GB daily log ingestion, observability is a manageable line item. Above it, costs require dedicated optimization effort—the economics fundamentally shift from "we need visibility" to "we need a strategy for what visibility is worth."
For organizations building reliable web automation infrastructure at scale—like we do at TinyFish orchestrating enterprise web agents—these patterns become visceral. When you're running thousands of concurrent browser sessions across fragmented web surfaces, observability depth determines reliability. Each session generates logs. Each authentication attempt produces telemetry. Each bot defense encounter creates data. But the real observability challenge isn't volume—it's correlation. Understanding why a workflow succeeded on 9,999 sites but failed on one specific property requires infrastructure that can trace session state across retries, correlate authentication patterns with regional variations, and surface the exact anti-bot strategy that triggered the failure. That observability depth doesn't come from collecting more data. It comes from structuring data so failures become debuggable, not just visible.
Infrastructure planning faces a particular challenge: observability costs scale with system complexity, not business metrics. Your revenue might double while your observability costs triple, and that asymmetry catches organizations off guard. A 2x increase in traffic might require 3-4x more observability data to maintain the same visibility. Some organizations report observability spend exceeding their cloud compute budget—where monitoring costs more than what you're monitoring.
The 10-30% benchmark exists because observability has become infrastructure's unavoidable tax. Understanding what drives those costs—and how they compound—helps organizations make better decisions about where to invest and what trade-offs actually matter. The costs are real. What they buy is worth examining.