Go performance integration guide

Step-by-step instructions for adding performance monitoring and distributed tracing to your Go projects.

BugSnag’s serverside performance monitoring leverages OpenTelemetry. With the wealth of open source OpenTelemetry instrumentation available for Go, you can easily send spans and traces from your service to BugSnag by installing the OpenTelemetry SDK and completing some simple configuration.

OpenTelemetry SDK installation

The steps below are a starting point and may need to be modified to reflect your project’s setup. First install the packages you will need to instrument your application with OpenTelemetry:

go get "go.opentelemetry.io/otel" \
  "go.opentelemetry.io/otel/sdk/trace" \
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" \
  "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

OpenTelemetry SDK configuration

Add the SDK initialization code in an otel.go file, which contains the OpenTelemetry bootstrapping code:

package main

import (
    "context"
    "fmt"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/trace"
)   

func setupOtel(ctx context.Context) error {

    prop := newPropagator()
    otel.SetTextMapPropagator(prop)

    tracerProvider, err := newTraceProvider(ctx)
    if err != nil {
        return err
    }

    otel.SetTracerProvider(tracerProvider)

    return nil
}

// Propagates the trace context in outbound requests
func newPropagator() propagation.TextMapPropagator {
    return propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    )
}

func newTraceProvider(ctx context.Context) (*trace.TracerProvider, error) {
    traceExporter, err := otlptracehttp.New(ctx)
    if err != nil {
        return nil, err
    }
    traceProvider := trace.NewTracerProvider(
        trace.WithBatcher(traceExporter),
    )
    return traceProvider, nil
}

Then use this configuration in your main.go and apply the otelhttp library to instrument inbound requests:

func main() {
    err := setupOtel(context.Background())
    if err != nil {
        return
    }

    mux := http.NewServeMux()
    mux.Handle("/rolldice/", otelhttp.NewHandler(http.HandlerFunc(rolldice), "rolldice"))
    http.ListenAndServe(":8080", mux)
}

func rolldice(w http.ResponseWriter, r *http.Request) {
    // Instrumented code ...
}

Next you need to set some configuration:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<PROJECT_API_KEY>.otlp.bugsnag.com:4318/v1/traces
export OTEL_SERVICE_NAME="your-service-name" 
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=<RELEASE_STAGE>,service.version=<APP_VERSION>" 

• <PROJECT_API_KEY> can be found in project settings in the BugSnag dashboard.
• OTEL_SERVICE_NAME should uniquely identify your service. We recommend using the same name as your project name.
• <RELEASE_STAGE> and <APP_VERSION> must be the same as you are passing to your BugSnag Error SDK. They are case sensitive.

For the full list of configuration options, see the automatic configuration page in the OpenTelemetry documentation.

Running your app and requesting a page should now result in spans arriving in BugSnag Performance.

Custom spans

For higher resolution information into the performance of your application, you can create your own spans whenever you like. To do this, get a tracer:

tracer = otel.GetTracerProvider().Tracer("myapp")

Then wrap some work in a span:

func somethingInteresting(r *http.Request) {
    ctx, span := tracer.Start(r.Context(), "my_operation")
    defer span.end()

    // some interesting work...
}

For more information see the Go OpenTelemetry instrumentation documentation.

If you would like BugSnag to aggregate your custom span to provide you with summary statistics on its performance, then you need to set a bugsnag.span.first_class attribute on the span. BugSnag will then automatically create a grouping for that span based on its name. This will appear under the ‘Custom Spans’ tab of your Performance dashboard.

span.SetAttributes(attribute.Bool("bugsnag.span.first_class", true))

Advanced capabilities

The above guide should allow you to get started with BugSnag Performance quickly. The OpenTelemetry ecosystem has a huge range of possibilities for different capabilites. To find out about the possibilities please browse the OpenTelemetry documentation for Go.

We will explore a few topics briefly here:

Sampling

Spans sent from OpenTelemetry SDKs use up your unmanaged quota. No spans will be ingested by BugSnag once your daily quota is exhausted. To ensure you have span coverage throughout the day, we recommend you use sampling.

By default the OpenTelemetry SDK will use the parentbased_always_on sampler, which will sample according to the sample decision from the incoming network request if there is one, and always sample if there is not. To use this requires clients of your Go server be instrumented in such a way that they pass their sampling information.

To apply a fixed sampling rate to all your Go spans, you can use the traceidratio sampler:

export OTEL_TRACES_SAMPLER="traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.25" # this will cause 25% of your traces to be sampled

go run main.go

Alternatively you can use the parentbased_traceidratio sampler. This will sample at a constant rate, unless the trace context was propagated from a client, in which case it will use the same sample decision that the client used:

export OTEL_TRACES_SAMPLER="parentbased_traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.25" # this will cause 25% of those traces that did not start in a client to be sampled

go run main.go

Wire protocol

Traces can be received by BugSnag via either gRPC or HTTP (protobuf or JSON). In most cases the simplest way to send traces to BugSnag is to export an environment variable with your BugSnag project’s dedicated OpenTelemetry endpoint before running your OpenTelemetry instrumented app:

export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://<PROJECT_API_KEY>.otlp.bugsnag.com:4317"

export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://<PROJECT_API_KEY>.otlp.bugsnag.com:4318/v1/traces"

For more configuration options see the OpenTelemetry OTLP Exporter Configuration documentation.

Collector

An OpenTelemetry Collector is an open source component that OpenTelemetry users can host on their own infrastructure. It can receive telemetry data from multiple sources, process it in various ways, and send it on to multiple telemetry back-ends, including BugSnag.

An important use of Collectors is to enable tail-based sampling, where a trace can be inspected in its entirety before making a sampling decision, for example looking to see whether it contained any errors.

To find out more about the Collector itself, see the OpenTelemetry Collector documentation.

To learn about using a collector with BugSnag, see our dedicated page on using a collector.

Span batch size

The maximum payload size for BugSnag’s OpenTelemetry trace endpoints is 1MB. OpenTelemetry payload sizes can be controlled via the batch size in the SDK or collector configuration.

The appropriate batch size will largely depend on the number of attributes getting added to your spans and therefore you may need to experiment to find the appropriate setting. The batch size can be controlled by setting an environment variable; we generally recommend 200 as a good starting point, which is lower than the default of 512 spans per batch.

export OTEL_BSP_MAX_EXPORT_BATCH_SIZE=200