Distributed Tracing: The good, The Bad, and the Ugly

Distributed Tracing: The Good, the Bad, and the Ugly Ricardo Ferreira Senior Developer Advocate Amazon Web Services © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Before we begin: Happy hour: it starts at 4:00 PM. New trailer for Thor: Love and Thunder. I welcome your questions anytime. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

What is distributed tracing anyway? © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

It is a type of logging. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Logging 101 © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Stitching additional data is hard © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

© 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Because logging with distributed systems is hard. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Single-machine threading Log{“/customer/find”, 235, 200, 30} API Thread 2 Thread 1 Log{“/api/find”, 840, 200, 35} Log{“/api/find”, 840, 200, 45} Customer API Database Customer Log{“/customer/find”, 235, 200, 42} © 2022, Amazon Web Services, Inc. or its affiliates. Log{“/db/find”, 450, 200, 5} Database Log{“/db/find”, 450, 200, 3} @riferrei

Correlating network distributed threads is hard © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Reason why is called distributed tracing Distributed Transaction Microservice A Microservice B © 2022, Amazon Web Services, Inc. or its affiliates. Microservice C @riferrei

Black-box versus white-box instrumentation Black-box White-box • Code is not changed • Require code changes • Handled by some runtime • Handled by the application • Minimal execution visibility • Better execution visibility © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Ideally; you should use both of them. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

But what about logs and metrics? © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

The famous trio in the SRE world Metrics: hey, something is not right. Traces: I can reveal the culprit for you. Logs: Let me show you what happened. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Best way to get started with this? © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

High-quality, ubiquitous, and portable telemetry to enable effective observability. https://opentelemetry.io © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

OpenTelemetry: SDKs, protocols, and integration © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

AWS Distro for OpenTelemetry https://aws-otel.github.io/download © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Distributed tracing: the good parts 👍‍ © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Decrease time spent with Incident management 10% 60% 30% • Monitoring metric values • Understanding topologies • Reading logs and events • Catching up with alerts • Isolating the anomalies • Reproducing exceptions • Evaluating trends and changes • Collecting context data • Creating code patches © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Finding bugs between releases V1.12 V1.13 © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Understand what the code (really) does. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Distributed tracing: the bad parts 👎‍ © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

My language. Your language. Their language. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

UI and frontends give the vendors goosebumps. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Asynchronous programming has its challenges. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

To correlate, or not correlate, that is the question. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

eBPF is the future; but not the present. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Distributed tracing: the ugly stuff 🤮 © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

© 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Native binaries may not come in handy for you. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Black-box is great; but not for everybody. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Sampling is a nightmare to get it right. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Mixing open standards with proprietary stuff. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Be prepared to become a data engineer. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei

Thank you! Ricardo Ferreira @riferrei © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei