Infrastructure
OpenTelemetry Baggage Without Privacy or Cost Surprises
Use OpenTelemetry baggage for low-cardinality routing context without leaking personal data or exploding telemetry costs.
In this article
OpenTelemetry Baggage Without Privacy or Cost Surprises
OpenTelemetry baggage is context that propagates across service boundaries, often through HTTP headers. It is not automatically a span attribute, nor is it a safe place for arbitrary user data. Because downstream services and third parties may receive it, every key needs an explicit purpose, bounded values, and a propagation policy.
The practical model
Treat opentelemetry baggage without privacy or cost surprises as a system with four review points: choose keys with a threat model, control trust boundaries, manage cardinality separately, prove propagation behavior. Give each point an owner, a measurable acceptance condition, and a rollback trigger. Thresholds should come from observed workloads, contracts, or risk tolerance rather than an unexplained default.
Choose keys with a threat model
Good candidates are coarse experiment groups, tenant tiers, or routing flags with small enumerations. Avoid names, email addresses, tokens, raw tenant identifiers, search text, and anything regulated. Ask where the header can travel, which proxies log it, and whether an untrusted caller can set it.
Control trust boundaries
Strip incoming baggage at the public edge unless a key is explicitly accepted. Recreate trusted values from authenticated server-side state. Maintain an allowlist and length limit, and drop unknown members before calls to vendors. Do not let a caller select privileged routing or bypass security by forging a baggage key.
Manage cardinality separately
Copy only selected baggage keys to span attributes. A value can be safe to propagate yet too high-cardinality for indexing. Estimate distinct values per hour and retention cost. Prefer a small category such as plan=pro over customer_id, and use secure logs or a lookup system when detailed correlation is required.
Prove propagation behavior
Test async tasks, message queues, retries, and scheduled jobs because context libraries do not always cross these boundaries automatically. Inspect real outgoing headers in a safe environment. Add metrics for dropped, oversized, and unapproved keys without recording their values.
Common pitfalls
- Treating baggage as encrypted because it is an HTTP header.
- Accepting client-provided values as trusted facts.
- Turning every baggage item into a metric label.
- Propagating internal context to external APIs.
A focused test plan
- Choose keys with a threat model: verify that good candidates are coarse experiment groups, tenant tiers, or routing flags with small enumerations. Then introduce the failure case “treating baggage as encrypted because it is an http header” and confirm that the system fails visibly, stays within its safety boundary, and has a documented recovery action.
- Control trust boundaries: verify that strip incoming baggage at the public edge unless a key is explicitly accepted. Then introduce the failure case “accepting client-provided values as trusted facts” and confirm that the system fails visibly, stays within its safety boundary, and has a documented recovery action.
- Manage cardinality separately: verify that copy only selected baggage keys to span attributes. Then introduce the failure case “turning every baggage item into a metric label” and confirm that the system fails visibly, stays within its safety boundary, and has a documented recovery action.
- Prove propagation behavior: verify that test async tasks, message queues, retries, and scheduled jobs because context libraries do not always cross these boundaries automatically. Then introduce the failure case “propagating internal context to external apis” and confirm that the system fails visibly, stays within its safety boundary, and has a documented recovery action.
Primary references
Conclusion
Baggage should be small, boring, and governed. Allowlist trusted low-cardinality keys, strip it at boundaries, and decide independently which values belong in searchable telemetry.