OpenTelemetry at Scale: Preventing High-Cardinality Traces from Inflating APM Bills by 10x

Adopting OpenTelemetry is one of the most effective ways to establish vendor-neutral observability across distributed microservices. However, without strict cardinality controls and intelligent trace sampling, standard production deployments routinely trigger catastrophic APM bills, inflating monthly observability invoices from $800 to $12,000 within weeks of rollout.

The surge in cloud spend does not happen because OpenTelemetry code is inefficient. It happens because commercial observability platforms (including Datadog, Honeycomb, Dynatrace, and New Relic) price their ingestion tiers on two core metrics: unique metric time-series streams and raw gigabytes of trace spans ingested.

When application developers attach dynamic metadata like user_id, request_id, or shopping cart UUIDs to metric labels, or sample 100% of high-volume HTTP health checks, the math of distributed telemetry turns against you.

Here is an operational guide to cardinality math, trace sampling mechanics, and the exact OpenTelemetry Collector configurations required to cut observability costs by 70% while improving error diagnostic accuracy.

The Napkin Math of a Cardinality Explosion

Metric databases such as Prometheus, Datadog Custom Metrics, and AWS CloudWatch allocate an independent time-series stream for every unique permutation of label key-value pairs. Adding a single high-cardinality dimension causes an exponential multiplication of metric streams.

Let’s do the napkin math on a standard API endpoint metric: http_requests_total.

Consider an application with conservative dimensions:

• HTTP Method: 4 values (GET, POST, PUT, DELETE)
• Route: 25 endpoints (/api/v1/orders, /api/v1/checkout, etc.)
• Status Code Class: 5 values (200, 300, 400, 500, 503)
• Deployment Region: 2 values (us-east-1, eu-west-1)

The baseline time-series count is:

4 (methods) * 25 (routes) * 5 (status codes) * 2 (regions) = 1,000 active time-series

At $0.05 per custom metric per month, 1,000 custom metrics cost $50 per month. This is completely sustainable.

Now, consider what happens when a software engineer adds one seemingly innocent label for user identification:

// DANGEROUS: Injecting high-cardinality UUID into metric labels
meter.createCounter('http_requests_total').add(1, {
  method: req.method,
  route: req.route,
  status: res.statusCode.toString(),
  region: process.env.AWS_REGION,
  userId: req.user.id, // <-- High-cardinality attribute!
});

If your platform has 50,000 active monthly users, the permutation calculation becomes:

1,000 baseline * 50,000 unique user IDs = 50,000,000 active time-series

At commercial APM pricing tiers ($0.05 to $0.10 per custom metric stream), 50 million custom metrics result in an estimated monthly invoice exceeding $250,000. Even if your vendor caps ingestion, their automated overage penalties or rate-limiting will drop telemetry data and trigger unexpected enterprise billing disputes.

High-Cardinality Rule: Separate Metrics from Traces and Logs

Metrics, distributed traces, and structured logs serve fundamentally different architectural roles and have completely different cost structures. High-cardinality data belongs exclusively in traces and logs, never in metrics.

+---------------------------------------------------------------------------------+
|                       Observability Telemetry Routing Matrix                    |
+---------------------------------------------------------------------------------+
| Data Type      | Target Signal  | Engine Storage Model | Cost per Item          |
|----------------|----------------|----------------------|------------------------|
| Aggregate Rates| Metrics        | Time-Series Array    | Expensive per Label    |
| User IDs, UUIDs| Traces / Spans | Key-Value Event Log  | Pennies per GB         |
| Payload Blobs  | Cold S3 Storage| Raw Compressed Blob  | Fractions of a cent    |
+---------------------------------------------------------------------------------+

• Metrics: Designed for aggregate rates, gauges, and counters (e.g. system.cpu.utilization, http.server.duration). Dimensions must be bounded to low-cardinality enums (under 100 possible values).
• Traces: Designed for execution paths and context propagation. Spans can comfortably accommodate high-cardinality attributes like user.id, tenant.id, order.uuid, and db.query.statement because traces are indexed as document events rather than real-time time-series matrices.
• Logs: Designed for high-volume contextual detail. Like traces, modern log engines (Loki, Elasticsearch, Datadog Logs) index strings without creating geometric dimension matrices.

If an engineering team needs to answer “Which specific user received an error?”, they should query distributed traces or logs by user.id. If they need to answer “What is our overall 5xx error rate across regions?”, they query metric counters.

Trace Volume Math: The Cost of 100% Ingestion

Capturing 100% of distributed trace spans in high-throughput production environments generates overwhelming data volume that offers almost zero engineering value.

Let’s do the ingestion math for a modest microservices architecture:

• Peak Traffic: 2,500 requests per second
• Average Spans per Request: 8 spans (API gateway -> auth lambda -> order service -> database query -> Redis cache -> external webhook)
• Total Spans Generated: 2,500 * 8 = 20,000 spans per second
• Monthly Span Count: 20,000 * 86,400 seconds * 30 days = 51.8 billion spans per month

At typical cloud ingestion rates of $0.10 per million spans, 51.8 billion spans cost:

(51,800,000,000 / 1,000,000) * $0.10 = $5,180 / month in trace ingestion fees alone

What makes this bill particularly frustrating is that 99.4% of those spans represent identical, boring HTTP 200 OK health check polls (GET /healthz), cron pings, and static asset fetches. Paying thousands of dollars each month to store repetitive success traces is pure cloud waste.

Head-Based vs Tail-Based Sampling

Head-based sampling makes an uneducated sampling guess before a request executes, while tail-based sampling makes an informed decision after the request has finished.

HEAD-BASED SAMPLING:
Incoming Request ---> [Random Dice Roll: 5%] ---> Sampled? ---> Execute & Ingest
                                                |
                                                +---> Dropped? ---> Lost forever
Problem: If a rare 500 error happens in the 95% dropped bucket, you have zero visibility!

TAIL-BASED SAMPLING:
Incoming Request ---> Execute All Spans ---> Collector Buffers Trace (3-5s)
                                                      |
                                                      v
                                        Evaluate Completed Trace:
                                        - Was status >= 500? --> KEEP 100%
                                        - Was duration > 1,500ms? --> KEEP 100%
                                        - Was status 200 OK? --> KEEP 1%

With naive head-based sampling set to 5%, if your application encounters a rare race condition that causes 1 in every 1,000 requests to fail with a 500 Internal Server Error, there is a 95% probability that the failed request will be discarded at the front door. You pay for 5% of all traffic, but miss the exact distributed trace needed to diagnose the production outage.

Tail-based sampling solves this trade-off completely. All spans are emitted to an OpenTelemetry Collector deployed within your VPC. The Collector buffers spans in memory for a short window (typically 3 to 10 seconds) until the root span terminates. It then evaluates the trace against sampling rules:

1. Did any span in the trace report an error (status.code == ERROR or http.status_code >= 500)? Sample at 100%.
2. Did the trace exceed our P99 latency threshold (e.g. duration > 1200ms)? Sample at 100%.
3. Did the trace touch a business-critical VIP customer account? Sample at 100%.
4. Was it a routine 200 OK response under 200ms? Sample at 1% or 0.1%.

Configuring the OpenTelemetry Collector for Cost Optimization

The OpenTelemetry Collector provides built-in processors that strip unwanted cardinality and enforce tail sampling before payloads are transmitted to external vendors.

Below is a battle-tested production otel-collector-config.yaml using official components from the OpenTelemetry Collector Contrib repository:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # Protect the collector container against out-of-memory crashes
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20

  # Strip high-cardinality attributes from metric labels via OpenTelemetry Transformation Language (OTTL)
  transform/metrics:
    metric_statements:
      - context: datapoint
        statements:
          # Remove dynamic IDs accidentally placed in metric labels
          - delete_key(attributes, "user_id")
          - delete_key(attributes, "email")
          - delete_key(attributes, "order_id")
          - delete_key(attributes, "jwt")

  # Enforce tail-based sampling on distributed traces
  tail_sampling:
    decision_wait: 5s # Wait up to 5s for child spans to assemble
    num_traces: 50000 # Buffer size in memory
    expected_new_traces_per_sec: 2500
    policies:
      # Policy 1: Always retain errors and exceptions
      - name: errors-policy
        type: status_code
        status_code: { status_codes: [ ERROR ] }

      # Policy 2: Always retain HTTP 5xx errors
      - name: http-5xx-policy
        type: numeric_attribute
        numeric_attribute:
          key: "http.status_code"
          min_value: 500
          max_value: 599

      # Policy 3: Always retain slow traces exceeding 1.5 seconds
      - name: latency-outlier-policy
        type: latency
        latency: { threshold_ms: 1500 }

      # Policy 4: Sample healthy traffic at 1%
      - name: probabilistic-healthy-policy
        type: probabilistic
        probabilistic: { sampling_percentage: 1.0 }

  # Batch telemetry packets to optimize TCP network egress
  batch:
    send_batch_size: 8192
    timeout: 5s

exporters:
  # Export clean, cost-reduced telemetry to your APM vendor
  otlp/vendor:
    endpoint: ${APM_ENDPOINT}
    headers:
      "api-key": ${APM_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/vendor]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, transform/metrics, batch]
      exporters: [otlp/vendor]

Review the official OpenTelemetry Tail Sampling Processor documentation for fine-grained routing specifications and composite sampling policies.

Cost and Diagnostic Accuracy Comparison

Choosing the right sampling and filtering strategy determines both your monthly cloud observability expenditure and your ability to debug production incidents.

Operational Best Practices for APM Cost Control

To protect your engineering organization from sudden billing spikes as traffic scales:

1. Deploy Collector as a DaemonSet or Sidecar: Run the OpenTelemetry Collector within your own Kubernetes cluster, ECS task, or VPC network boundary. Sanitizing and dropping telemetry locally before data crosses the public internet eliminates both vendor ingestion fees and cloud provider cross-AZ/NAT gateway egress charges.
2. Implement Metric Dimension Budgets: Establish an architectural rule that no custom metric may exceed 5 total dimensions, and every dimension must evaluate to a known enum. Disallow arbitrary string formatting in metric tags.
3. Automate Cardinality Alerts: Configure billing and cardinality monitors inside your APM dashboard. If any single metric key generates more than 5,000 distinct time-series within a 1-hour window, trigger a PagerDuty or Slack alert to catch runaway deployments before invoices compound.
4. Use Composite Tail Sampling: Pair error-based sampling with VIP routing. If your SaaS operates enterprise tiers with strict SLAs, configure a tail-sampling policy that retains 100% of traces for high-value enterprise accounts while retaining 1% for free-tier users.

By shifting telemetry filtering and sampling to the OpenTelemetry Collector, engineering teams gain complete architectural control over their observability budget without compromising on diagnostic fidelity when outages occur.