Under Construction

This documentation is a work in progress. Visit regularly to see the improvements we're making.

Generate Log Data

As they run, services generate log data to provide visibility into what’s happening. Log data can include anything from benign informational messages to full-blown errors. You want to know about errors, but too many informational messages can become distracting noise.

You’ll be able to use MDAI to filter out unnecessary log lines. First, lets set up the log generators and generate log data.

Get the Configuration Files

In our Configs repo’s synthetics directory are synthetic log generators in the form of configuration files. You’ll use these to generate synthetic log data.

Copy the following files into your working directory:

loggen_services.yaml
loggen_service_xtra_noisy.yaml
loggen_service_noisy.yaml
loggen_fluent_config.yaml

Deploy Log Generators to Your Cluster

Let’s create 3 different log generators.

The first emits logs for random services named service####, where #### is a random number.

kubectl apply -f ./loggen_services.yaml

The second is a noisy log generator for a particular service (service1234 by default).

kubectl apply -f ./loggen_service_noisy.yaml

The third is an excessively noisy version of the second log generator, but for a different service (service4321 by default).

kubectl apply -f ./loggen_service_xtra_noisy.yaml

Verify That Logs Are Being Generated

If the log generators are running, get the pods.

kubectl get pods -n mdai -l app.kubernetes.io/part-of=mdai-log-generator

You should see a list of pods similar to the following.

NAME                                  READY   STATUS    RESTARTS   AGE
mdai-logger-d8bb6f897-f6jt6           1/1     Running   0          101s
mdai-logger-d8bb6f897-tzh6f           1/1     Running   0          101s
mdai-logger-d8bb6f897-w7d4n           1/1     Running   0          101s
mdai-logger-noisy-77fcbf8b9f-g4c9c    1/1     Running   0          84s
mdai-logger-noisy-77fcbf8b9f-tsg2s    1/1     Running   0          84s
mdai-logger-noisy-77fcbf8b9f-xrp8s    1/1     Running   0          84s
mdai-logger-xnoisy-686cb6465f-cdpvm   1/1     Running   0          71s
mdai-logger-xnoisy-686cb6465f-tdjkd   1/1     Running   0          71s
mdai-logger-xnoisy-686cb6465f-xdzvp   1/1     Running   0          71s

Verify that logs are being generated by using the kubectl logs command with any one of the pods.

kubectl logs -n mdai {pod name}

For example, the logs from the pod named mdai-logger-xnoisy-686cb6465f-xdzvp should contain a variety of log levels.

2025-02-06T04:43:50+00:00 - service4321 - INFO - The algorithm successfully executed, triggering neural pathways and producing a burst of optimized data streams.
2025-02-06T04:43:50+00:00 - service4321 - INFO - The algorithm successfully executed, triggering neural pathways and producing a burst of optimized data streams.
2025-02-06T04:43:50+00:00 - service4321 - INFO - The algorithm successfully executed, triggering neural pathways and producing a burst of optimized data streams.
2025-02-06T04:43:50+00:00 - service4321 - ERROR - The algorithm successfully executed, triggering neural pathways and producing a burst of optimized data streams.
2025-02-06T04:43:50+00:00 - service4321 - ERROR - The algorithm successfully executed, triggering neural pathways and producing a burst of optimized data streams.
2025-02-06T04:43:50+00:00 - service4321 - WARNING - The algorithm successfully executed, triggering neural pathways and producing a burst of optimized data streams.
2025-02-06T04:43:51+00:00 - service4321 - ERROR - The algorithm successfully executed, triggering neural pathways and producing a burst of optimized data streams.
2025-02-06T04:43:51+00:00 - service4321 - WARNING - The algorithm successfully executed, triggering neural pathways and producing a burst of optimized data streams.
2025-02-06T04:43:51+00:00 - service4321 - WARNING - The algorithm successfully executed, triggering neural pathways and producing a burst of optimized data streams.
2025-02-06T04:43:51+00:00 - service4321 - WARNING - The algorithm successfully executed, triggering neural pathways and producing a burst of optimized data streams.
2025-02-06T04:43:51+00:00 - service4321 - WARNING - The algorithm successfully executed, triggering neural pathways and producing a burst of optimized data streams.
2025-02-06T04:43:51+00:00 - service4321 - ERROR - The algorithm successfully executed, triggering neural pathways and producing a burst of optimized data streams.
2025-02-06T04:43:51+00:00 - service4321 - ERROR - The algorithm successfully executed, triggering neural pathways and producing a burst of optimized data streams.
2025-02-06T04:43:51+00:00 - service4321 - INFO - The algorithm successfully executed, triggering neural pathways and producing a burst of optimized data streams.
2025-02-06T04:43:51+00:00 - service4321 - INFO - The algorithm successfully executed, triggering neural pathways and producing a burst of optimized data streams.
2025-02-06T04:43:51+00:00 - service4321 - INFO - The algorithm successfully executed, triggering neural pathways and producing a burst of optimized data streams.

Tip

If you installed K9s, you can use it to observe the logs. To launch the K9s application, enter k9s in the terminal window where you launched the cluster (for the correct context). Use the arrow keys to select one of the log generators, then press l (lowercase letter l). Press ESC to exit the log window.

Success

Now that logs are being generated, we can create a collector.