Trace sampling

Learn more about the sampling options available to you.

Sampling in OpenTelemetry is the mechanism where span data is processed and exported from your application to backend systems. An effective sampling strategy can give you excellent observability of your systems, without needing to store all of your data and so reduces the associated costs.

The most important thing to consider is how much data you need to sample in order to get a representative view of your system. We recommend checking out some of our other docs for more background on this.

For Distributed Tracing, each application could make its own sampling decision, which might mean that you end up with incomplete traces. It is therefore important to select a sampling strategy that gives you the best possible chance of trace completeness. There are two types of sampling – it is possible to use them both together:

• Head-based – a decision is made by the first application in the trace on whether to sample a span and subsequent spans in the trace. This approach cannot account for the full state of the trace, since the trace has not concluded at decision time.
• Tail-based – a decision is made when some or all of the spans in the trace have completed. This is useful where you have specific sampling criteria that depends on some aspect of the trace that is only available once the span(s) of interest have completed.

Tail-based sampling is generally much more complex to implement, and would normally require a collector to store your spans temporarily before making a decision. We give some more guidance in our collector documentation.

Recommended head-based sampling approach

There are several different approaches to head-based trace sampling. We suggest that you read the OpenTelemetry documentation for a more detailed overview.

In order to preserve trace completeness across client and server(s), we recommend using a Parent Based Trace ID Ratio sampler with your server’s OpenTelemetry instrumentation. This sampler respects its parent span’s sampling decision when present, but would otherwise sample spans according to the declared sampling rate. It is usually simplest to configure this sampler by including additional environment variables at runtime:

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=<SAMPLING_RATE>

SAMPLING_RATE can be in the range from 0.0 to 1.0, where 1.0 is equivalent to an always on sampler that samples every span.