🗺️Distributed Tracing

Distributed Tracing enables tracking of requests as they pass through the microservices in a system. Ddosify captures Traces and Spans out-of-the-box on your Kubernetes Cluster without modifying your code or restarting your services. Ddosify uses a lightweight eBPF Agent (Alaz) to generate Distributed Tracing data and enable auto context propagation.

To do so, on the Ddosify Platform, click "Clusters" under the "Observability" tab first.

After you click on a cluster you want to observe on the clusters page, click on the distributed tracing tab. This will take you to the distributed tracing page. You will see this:

Here, you can filter and view traces in chart and list form.

Filtering

You can apply the following filtering:

Operations (trace names)
Trace count
Tags
Min/max duration (in ms)
Min/max span count
Time interval

Note: Filtering affects both the trace chart and the trace list. They show the same traces but in different formats.

To apply filtering, click the "Filters" button on the upper right.

Operations

Operations filtering allows you to select traces with specific names. It allows multi-select so that you can look for multiple names.

Note: The name of a trace is the name of its first span.

Max Results

This limits the number of traces shown.

Tags

Tags allow you to filter traces by the tags in their spans. Tag filtering accepts the logfmt format.

Note: Span attributes are not allowed to be nested; thus, tag filtering does not support nested filtering either.

Let's see how we can query traces by their tags. Let this be an example tags for a span:

{
    "path": "/", 
    "to_ip": "192.168.20.73", 
    "method": "GET", 
    "to_uid": "d0210c8f-fe44-472c-bbb2-39d64973e5a4", 
    "from_ip": "192.168.30.206", 
    "to_name": "kube-state-metrics-shard", 
    "to_port": "8081", 
    "to_type": "daemonset", 
    "from_uid": "2148b55d-cab8-41c6-ab5e-b12d9630ba0c", 
    "protocol": "HTTP", 
    "from_name": "aws-node", 
    "from_port": "41736", 
    "from_type": "daemonset", 
    "status_code": "200"
}

An example tag filter could look like this:

method=GET protocol=HTTP from_port=41736

While filtering, it is enough for a single span of a trace to satisfy this filter. As a result, the traces with at least one span matching this filter are retrieved.

Min/Max Duration

You can filter the traces by the total time they took in ms.

Min/Max Span Count

You can also filter the traces by their span count.

Time Interval

To apply a time interval, click the "Last 5 min" button next to the "Filters" button. You will see this:

Here, you can specify your desired time interval. You can specify your start-end times yourself or pick predefined time intervals from the "Quick Selections" section.

Note: The time interval between the start and the end could be apart at most 24 hours.

Trace Chart

The trace chart allows you to view the distribution of your filtered traces by their dates and durations. While the x-axis is the date, the y-axis is the duration.

To view more details (start time, name, duration, span count) about a trace, you can hover on it:

Trace List

Under the trace chart, you can view the filtered traces in a listed format. You can delete them here as well. You can also use the page numbers below to see the remaining traces.

Trace Details

After you click on a trace (either on the chart or the list), you can view its details (including its call hierarchy and time).

Each row represents a trace, whereas the columns represent the time.

No matter how long your trace took, the whole hierarchy is always fitted inside your view, so you would not need scrolling/zooming to understand the trace.

You can expand/collapse spans by clicking on their "Name" column.

You can also hover over a span to see its details, such as: