Hidden Secrets about Instrumenting JVMs for OpenTelemetry
A presentation at JNation 2021 by Ricardo Ferreira
Hidden Secrets about Instrumenting JVMs for OpenTelemetry Ricardo Ferreira @riferrei
OpenTelemetry Crash Source @riferrei
opentelemetry overview @riferrei
Quick Demo https://github.com/riferrei/otel-with-java @riferrei
Much better now đź‘Ť @riferrei
Ricardo Ferreira Developer Advocate q Elastic Community Team q HashiCorp Ambassador q Before working for Elastic: Confluent, Oracle, Red Hat q Distributed Systems, O11y, Streaming Systems, databases q riferrei@elastic.co q riferrei@riferrei.com @riferrei
JVM Instrumentation options java -javaagent:./${OTEL_AGENT} \ -Dotel.traces.exporter=otlp \ -Dotel.metrics.exporter=otlp \ -jar target/app.jar 80% @riferrei API, SDK, SDK-Extensions 20%
Manual Instrumentation @riferrei
Manual Instrumentation <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId> <version>${version}</version> </dependency> @riferrei
Down the river via @WithSpan <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-extension-annotations</artifactId> <version>${version}</version> </dependency> @WithSpan public void someMethod() { } @riferrei
Disabling auto instrumentation -Dotel.instrumentation.common.default-enabled=false @riferrei
Enabling only manual instrumentation -Dotel.instrumentation.opentelemetry-annotations.enabled=true @riferrei
Meaningful names for services -Dotel.resource.attributes=service.name= \ brand-estimator,service.version=1.0 @riferrei
Meaningful names for services export OTEL_RESOURCE_ATTRIBUTES=service.name= \ brand-estimator,service.version=1.0 @riferrei
Making your traces observable @riferrei
Using Text-Map-Propagators @riferrei
Using Text-Map-Propagators @riferrei
Using Text-Map-Propagators @riferrei
Can I tell you a secret? SELECT a FROM b WHERE password=? SELECT a FROM b WHERE password=”secret” @riferrei
Databases can be a bit chatty java.sql.Datasource#getConnection() -Dotel.instrumentation.jdbc-datasource.enabled=true @riferrei
Excluding noisy frameworks @riferrei
Using the batch-Span-Processor @riferrei
Planning your sampling strategy @riferrei
Debugging the Agent -Dotel.javaagent.debug=true export OTEL_JAVAAGENT_DEBUG=true @riferrei
Disabling the agent -Dotel.javaagent.enabled=false export OTEL_JAVAAGENT_ENABLED=false @riferrei
Observable metrics: no pun intended @riferrei
Inspecting telemetry data with jMC <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-sdk-extension-jfr-events</artifactId> <version>${version}</version> </dependency> @riferrei
THANK YOU Ricardo Ferreira @riferrei
OpenTelemetry is the new-kid-on-the-block in API to instrument distributed applications to send telemetry data (logs, metrics, and traces) to backends for visualization and monitoring. It was created to be a de-facto replacement to standards such as OpenTracing and OpenCensus.
The support for Java in OpenTelemetry is generous. It supports both automatic and manual instrumentation. With automatic instrumentation, developers can have known libraries and frameworks instrumented without writing a single code line. Manual support also allows them to decide which parts of the code need instrumentation, and there is a rich set of APIs available to use.
This talk will open the pandora box for both options. It will reveal which knobs are available to use that make a tremendous difference in how the applications report telemetry data — both in terms of performance and details. Knowing these hidden secrets will grant developers fine control over their applications and how observability stacks show them to the world.
Video
Code
The following code examples from the presentation can be tried out live.
Resources
The following resources were mentioned during the presentation or are useful additional information.
Buzz and feedback
Here’s what was said about this presentation on social media.
