Intelligent Trace Sampling

As requests are observed in a cloud-based distributed system, services can create large volumes of trace data in the form of spans. Exporting all of that data to a vendor for analysis is expensive, and often unnecessary. Sampling data, rather than exporting it all, is an effective way of reducing costs.

Intelligent Trace Sampling samples spans in real time using dynamic rules that adapt to trace volumes, environments, or cost constraints—without code changes or redeploys. It uses a type of trace sampling called tail sampling.

Deploy the Demo

Make sure that you’re ready to demonstrate Intelligent Trace Sampling by first installing and running the MyDecisive SmartHub.

1. Start the demo.

   mdai use_case tail_sampling --version 0.9.0 --workflow basic

2. Add a role binding for the load balancer.

   mdai apply ./0.9.0/use_cases/tail_sampling/k8s_rbac.yaml

3. Create the load-balancing resources.

   kubectl create rolebinding loadbalancer-endpointslices-reader -n mdai   --role=loadbalancer-endpointslices-reader   --serviceaccount=mdai:loadbalancer

4. Create the scrape job that will get telemetry data to the metrics store.

   mdai apply ./0.9.0/use_cases/tail_sampling/scrape_trace_count_metrics.yaml

   The job creates a custom metric called trace_span_service_count_total that we’ll use later.

Data should now be flowing to the collector.

Validate the Data Flow

We’ll use Grafana, which is already running in the MDAI cluster, to view the trace data flowing through the data pipelines we’ve set up.

1. Port forward Grafana for easier access.

   kubectl port-forward -n mdai svc/mdai-grafana 3000:80

2. Navigate to Grafana at http://localhost:3000/login and log in with username admin and password mdai.

3. Display the OpenTelemetry Collector dashboard.

The span rate panels in the receivers and exporters sections indicate data volumes. The receivers panel displays 2 sources for incoming span data.

The exporters panel displays all of the valid destinations for the span data egressing the pipeline.

This setup simulates a production system that receives and forwards span data to various destinations, including a vendor (debug/to_vendor). There’s no sampling, so the service owner is subjected to the full cost of exporting large volumes of trace data from a cloud provider to vendor storage. Let’s filter out some of the spans and see how the exported volume changes.

Add Static Tail Sampling

Update the MyDecisive SmartHub configuration to introduce static tail sampling.

mdai use_case tail_sampling --version 0.9.0 --workflow static

After several minutes, we can see that the number of exported spans is consistently lower than the number received.

Static tail sampling works on exact matches. To see the static sampling configuration now in effect, we’ll look at the difference between the 2 otel.yaml configuration files.

The mdai-labs/0.9.0/use_cases/tail_sampling/basic/otel.yaml file has this processors section:

processors:
  batch:
    send_batch_size: 50
    send_batch_max_size: 200
    timeout: 10s

The mdai-labs/0.9.0/use_cases/tail_sampling/staic/otel.yaml file has this processors section:

processors:
  tail_sampling:
    decision_wait: 10s # TODO do we have a number for this?
    # expected_new_traces_per_sec: _ # TODO: do we have a number for this?
    policies:
      - name: keep-all-the-errors
        type: and
        and:
          and_sub_policy:
            - name: error-sampling
              type: status_code
              status_code:
                status_codes:
                  - ERROR
      - name: probabilistic-sample-top-talkers
        type: and
        and:
          and_sub_policy:
            - name: services-using-tail-sampling
              type: string_attribute
              string_attribute:
                key: operation
                values:
                  - "high-volume-op"
            - name: probabilistic-policy
              type: probabilistic
              probabilistic:
                sampling_percentage: 10  # sample 10%
      - name: always-sample-non-top-talkers
        type: ottl_condition
        ottl_condition: {
          error_mode: ignore,
          span: [
            "attributes["operation"] != "high-volume-op"",
          ]
        }

  batch:
    send_batch_size: 50
    send_batch_max_size: 200
    timeout: 10s

The static configuration samples 10 percent of services having the attribute operation = high-volume-op. While it’s an improvement over having no sampling, static configuration can’t adapt to the dynamic nature of production environments. It will only ever sample the traces of services that have that attribute set.

Use Dynamic Trace Sampling

To account for the dynamic nature of production systems, we’ll switch from the static sampling policy to a dynamic one. Dynamic sampling uses policies and expressions that help with sampling decisions across all services.

1. Update the MyDecisive SmartHub configuration to switch to dynamic sampling.

   mdai use_case tail_sampling --version 0.9.0 --workflow dynamic

2. Port forward Prometheus for easier access.

   kubectl port-forward -n mdai svc/prometheus-operated 9090:9090

3. After a few minutes, view a graph of the trace spans emitted by the 4 test services in Prometheus. After about 30 minutes, the graph appears similar to this one. The 2 “loud” services are emitting substantially more spans than the other 2 services.

4. Go to the Grafana OpenTelemetry Collector dashboard. Note that the number of received spans peaks at 142 while the number of spans forwarded to the vendor (debug/to_vendor) peaks at an average of about 55.

The MDAI Data Management dashboard shows that sampling reduces the number of spans sent to a vendor. In a cloud-based production environment where exporting trace data can be very expensive, this translates to a reduction in cost.

Understand Intelligent Tail Sampling

1. What makes Intelligent Tail Sampling dynamic is the Kubernetes environment variable LOUDEST_SERVICES, defined in ,the processors section:

   ...
             - name: probabilistic-sample-loudest-talkers
               type: and
               and:
                 and_sub_policy:
                   - name: services-using-tail-sampling
                     type: string_attribute
                     string_attribute:
                       key: operation
                       values: ${env:LOUDEST_SERVICES}
                   - name: probabilistic-policy
                     type: probabilistic
                     probabilistic:
                       sampling_percentage: 90
           ...

   Any services that are listed in LOUDEST_SERVICES have their trace data sampled at the specified sampling percentage. But how is LOUDEST_SERVICES populated?

2. The answer is in the MyDecisive SmartHub configuration file (./0.9.0/use_cases/tail_sampling/dynamic/hub.yaml), where LOUDEST_SERVICES is defined:

   ...
             - key: loudest_services
               serializeAs:
                 - name: "LOUDEST_SERVICES"
                   transformers:
                     - type: join
                       join:
                         delimiter: ","
               dataType: set
           ...
             prometheusAlerts:
               - name: loudest_services
                 expr: 'sum(increase(trace_span_service_total{service_name!=""}[5m])) by (service_name) >= 2000'
                 severity: warning
                 for: 1m
                 keep_firing_for: 1m
           ...

Let’s see how this configuration provides values for the tail sampling.

1. The environment variable LOUDEST_SERVICES is defined as a list of comma-delimited values.

2. The Prometheus alert loudest_services is defined to trigger whenever a service has a trace with total spans equal to or exceeding 2000.

3. When that Prometheus alert fires for a service that’s sending trace data, a rule adds that service to the variable LOUDEST_SERVICES.

Lessons Learned

Exporting large volumes of trace data from your cloud to a vendor is expensive. And since not all spans are equally important, we can filter out spans that aren’t needed for analysis. Prime targets for filtering are loud services producing the noisiest traces.

But since we don’t know which are the loud services at any given time, static sampling would have to be manually updated. This would also cause frequent redeploys that interrupt availability. What we need is configuration that updates itself without the need for redeployment.

Intelligent Trace Sampling does exactly that, and it’s easy to set up. By deciding on acceptable volume limits and using dynamic variables in PromQL statements, a dynamic configuration can keep up with production environments.