Intelligent LogStream

As they operate in a cloud-based distributed system, services create large volumes of log data. Exporting all of that data to a vendor for analysis is expensive, and often unnecessary. Filtering out uninteresting log lines is an effective way of reducing your costs.

Intelligent LogStream filters log telemetry streams in real time using dynamic rules that adapt to services, environments, or cost constraints—without code changes or redeploys.

Deploy the Demo

Make sure that you’re ready to demonstrate Intelligent LogStream by first installing and running the MyDecisive SmartHub.

1. Start the demo.

   mdai use-case data_filtration --version 0.9.0 --workflow basic

2. Deploy synthetic log generators for mock telemetry data, and a fluentD instance to forward log streams to an OpenTelemetry collector.

   helm upgrade --install --repo https://fluent.github.io/helm-charts fluent fluentd -f ./mock-data/configs/loggen_fluent.yaml

Data should now be flowing to the collector.

Validate the Data Flow

We’ll use Grafana, which is already running in the MyDecisive Local Learning cluster, to view the logs flowing through the data pipelines we’ve set up.

1. Port forward Grafana for easier access.

   kubectl port-forward -n mdai svc/mdai-grafana 3000:80

2. Navigate to Grafana at http://localhost:3000/login and log in with username admin and password mdai.

3. Display the OpenTelemetry Collector dashboard.

The log-records rate panels in the receivers and exporters sections should show similar log volumes. All of the received data gets exported.

This setup simulates a production system that receives and forwards all log data, with no filtering. It subjects the service owner to the full cost of exporting large volumes of logs from a cloud provider to vendor storage. Let’s filter out some of the logs and see how the exported volume changes.

Add Static Filtering

Update the MyDecisive SmartHub configuration to introduce a static filter.

mdai use_case data_filtration --version 0.9.0 --workflow static

A static filter works on exact matches. To see the static filter now in effect, diff these otel.yaml files in your cloned mdai-labs repo.

diff ./0.9.0/use_cases/data_filtration/static/otel.yaml ./0.9.0/use_cases/data_filtration/basic/otel.yaml

The static otel.yaml file defines the static filter in this configuration block.

filter/static-non-crit-services:
   error_mode: ignore
   logs:
     log_record:
       - 'IsMatch(attributes["mdai_service"], "service1|service1234") and IsMatch(attributes["level"], "INFO|DEBUG|WARN")'

The filter configuration uses Prometheus Query Language (PromQL) to match log lines with specific levels (INFO, DEBUG, WARN) generated by 2 noisy services (service1, service1234). After several minutes, go back to the OpenTelemetry Collector dashboard and compare the mean logs records rate on the receiver to the mean logs records rate on the exporter. You’ll notice that the latter mean is lower, showing that some logs were dropped.

This filter is optimal if you could be sure that these 2 services would always be the overly noisy ones, and that it can reduce log volumes sufficiently to bring your budget down to an acceptable level. However, over time the behavior of services in production systems is never predictable. A dynamic filter that adjusts to changing conditions is a better choice.

Use Dynamic Filtering

To account for the dynamic nature of production systems, we’ll switch from the static filter to a dynamic one. Dynamic filtering uses expressions that evaluate log output from all services.

1. Update the MyDecisive SmartHub configuration to switch to a dynamic filter.

   mdai use_case data_filtration --version 0.9.0 --workflow dynamic

2. Provision an MyDecisive observer to gather and compute data on the logs flowing from the services in the demo.

   mdai apply ./0.9.0/use_cases/data_filtration/dynamic/observer.yaml -n mdai

   The observer’s computed data is picked up by a Prometheus instance running in the cluster. More about that later.

3. After giving the demo several minutes to collect data, go to the MDAI Data Management. This dashboard shows the total number of services in the demo, the volume of logs incoming (megabytes received), and the volume exported (megabytes sent). It also shows the differential as a percentage. As time goes on, this differential grows.

4. Scroll down to see the table giving volume data for each service. The table is sorted to highlight top talkers, or services outputting the largest log volumes.

The MDAI Data Management dashboard shows that a large—and growing—percentage of logs is being filtered out. In a cloud-based production environment where exporting logs can be very expensive, this translates to a reduction in cost.

Understand the Dynamic Filter

1. The dynamic filter is defined in ./0.9.0/use_cases/data_filtration/dynamic/otel.yaml:

   ...
           filter/dynamic-non-crit-services:
           error_mode: ignore
           logs:
             log_record:
               - '"{env:SERVICE_LIST_REGEX}" != "" and IsMatch(attributes["mdai_service"], "{env:SERVICE_LIST_REGEX}")'
           ...

   Unlike the static filter, there are no hardcoded service names in the dynamic filter. Instead, matching is based on an attribute and a Kubernetes environment variable, SERVICE_LIST_REGEX. By its name, we can guess that this variable contains service names. But how is it populated?

2. The answer is in the MyDecisive SmartHub configuration file, ./0.9.0/use_cases/data_filtration/dynamic/hub.yaml.

   apiVersion: hub.mydecisive.ai/v1
           kind: MdaiHub
           metadata:
             labels:
               app.kubernetes.io/name: mdai-operator
               app.kubernetes.io/managed-by: kustomize
             name: mdaihub-ddf
             namespace: mdai
           spec:
             variables:
               - key: service_list
                 serializeAs:
                   - name: "SERVICE_LIST_REGEX"
                     transformers:
                       - type: join
                         join:
                           delimiter: "|"
                 dataType: set
                 storageType: "mdai-valkey"
   
             prometheusAlert:
               - name: top_talkers
                 expr: 'sum(increase(bytes_received_by_service_total{mdai_service!=""}[1m])) by (mdai_service, data_type) > 800*1024'
                 severity: warning
                 for: 1m
                 keep_firing_for: 1m
               - name: top_listeners
                 expr: 'sum(increase(bytes_sent_by_service_total{mdai_service!=""}[1h])) by (mdai_service, data_type) > 10*1024*1024'
                 severity: warning
                 for: 15m
                 keep_firing_for: 10m
   
             automations:
               - eventRef: top_talkers
                 workflow:
                   - handlerRef: HandleNoisyServiceAlert
                     args:
                       payload_val_ref: mdai_service
                       variable_ref: service_list

Let’s see how this configuration provides values for the dynamic filter.

1. SERVICE_LIST_REGEX is defined as a list of values delimited by pipe characters (“|”).

2. A Prometheus alert named top_talkers is defined to trigger when a service’s log volume (measured in bytes) is above a certain value for 1 or more minutes.

3. When that alert is triggered, an automation adds the guilty service to service_list, which is the key for SERVICE_LIST_REGEX.

4. The MyDecisive observer continues to compute values from the logs being generated by the services in the demo. And Prometheus continues to periodically poll that observer, refreshing the data for its alerting service to evaluate.

Lessons Learned

Exporting large volumes of logs from your cloud to a vendor is expensive. And since not all logs are equally important, we can filter out logs that aren’t needed for everyday monitoring. Prime targets for filtering are noisy services, or top talkers, which produce the most logs.

But since we don’t know who the top talkers are at any given time, a static filter would have to be manually updated. This would also cause frequent redeploys that interrupt availability. What we need is a filter that updates itself without the need for redeployment.

Intelligent LogStream does exactly that, and it’s easy to set up. By deciding on acceptable volume limits and using dynamic variables in PromQL statements, a dynamic filter can keep up with production environments.