Slides from my talk & demo at Go NYC Meeetup 19-Jan-2017. We present Jaeger, Uber’s open source distributed tracing system, featuring Go backend, React based UI, and OpenTracing API support. We show examples of instrumenting application code for tracing and using distributed context propagation to attribute backend resource usage to top level consumers.

1. Confidential. Internal use only
Tracing 2000+ polyglot microservices
at Uber with Jaeger and OpenTracing
Go Language NYC Meetup - January 19, 2017
Yuri Shkuro, Staff Engineer, Uber Technologies

2. Confidential. Internal use only
Agenda
1. What is distributed tracing
‐ Quick intro
2. Why should you care
‐ Demo
3. Why OpenTracing
‐ Code examples
4. Jaeger @ Uber

3. Confidential. Internal use only
What Is Distributed Tracing
(show of hands)

4. Distributed Context Propagation
A
B
C
D
E
{context}
{context}
{context}
{context}
Unique ID → {context}
1. Assign a unique identifier to each
request at the edge service
2. Store it in a context object, along
with other metadata
3. Propagate the context across
process boundaries (in-band)
4. Baggage is arbitrary K/V
5. Capture timing, events, tags and
collect them out of band (async)
6. Re-assemble the call tree from the
storage for the UI
Edge service

5. Confidential. Internal use only
Why Should You Care?
(demo)
https://github.com/uber/jaeger/tree/master/examples/hotrod

6. Confidential. Internal use only
Recap
1. Discover architecture
2. View request timeline & errors
3. Find sources of latency
4. Highly contextualized logging
5. Baggage propagation
6. Compute usage attribution

7. Tracing is
WHAT’S THE CATCH?

8. ● Instrumentation is time consuming
● Vendor lock-in (backend and data model)
● Inconsistent semantics across languages
● Not suitable for open source frameworks
Tracing Instrumentation Has Been Too Hard

9. A vendor-neutral open standard for distributed tracing.
• Addresses instrumentation problem
• Open source, no vendor lock-in
• Consistent semantics across languages
• Available in 7+ programming languages
Who should care:
• Cloud-native / microservice applications
• OSS frameworks, esp. IPC and control flow
• Tracing and/or monitoring systems
http://opentracing.io
An official project of
OpenTracing

10. Announced v1.0 spec in August 2016
Tracer Implementations: Zipkin, Jaeger, Hawkular, LightStep, Appdash,
and a few smaller tracing systems
Supported Frameworks: GRPC, YARPC, Dropwizard, Flask, Django, Go-kit, ...
Some Companies using OpenTracing:
A young, fast growing project

11. OpenTracing Approach

12. TRACER
● StartSpan(operationName) Span
● Inject(SpanContext, Format, Carrier)
● Extract(Format, Carrier) SpanContext
SPAN
● Context() SpanContext
● SetTag(key string, value interface{})
● Log(...)
● Finish()
● SetBaggageItem(key, value string)
● BaggageItem(key string) string
API: Simple, yet expressive

13. ● Off the shelf instrumentation
● Start / Finish
● Inject / Extract
● Tags
● Logs
● Baggage
Code Examples

15. Jaeger (yā′gər - hunter)
Host or Container
Application
Instrumentation
OpenTracing API
jaeger-client
jaeger-agent
(Go)
jaeger-collector
(Go)
memory queue
Data Store
(Cassandra)
jaeger-query
(Go)
jaeger-ui
(React)
Control Flow
Trace
Reporting
Thrift over
TChannel
Control Flow Trace Reporting
Thrift over UDP
Adaptive
Sampling

16. Confidential. Internal use only
Thanks!
References:
http://uber.github.io/jaeger
http://opentracing.io - @opentracing