1.
Confidential. Internal use only
Tracing 2000+ polyglot microservices
at Uber with Jaeger and OpenTracing
Go Language NYC Meetup - January 19, 2017
Yuri Shkuro, Staff Engineer, Uber Technologies

2.
Confidential. Internal use only
Agenda
1. What is distributed tracing
‐ Quick intro
2. Why should you care
‐ Demo
3. Why OpenTracing
‐ Code examples
4. Jaeger @ Uber

3.
Confidential. Internal use only
What Is Distributed Tracing
(show of hands)

4.
Distributed Context Propagation
A
B
C
D
E
{context}
{context}
{context}
{context}
Unique ID → {context}
1. Assign a unique identifier to each
request at the edge service
2. Store it in a context object, along
with other metadata
3. Propagate the context across
process boundaries (in-band)
4. Baggage is arbitrary K/V
5. Capture timing, events, tags and
collect them out of band (async)
6. Re-assemble the call tree from the
storage for the UI
Edge service

5.
Confidential. Internal use only
Why Should You Care?
(demo)
https://github.com/uber/jaeger/tree/master/examples/hotrod

6.
Confidential. Internal use only
Recap
1. Discover architecture
2. View request timeline & errors
3. Find sources of latency
4. Highly contextualized logging
5. Baggage propagation
6. Compute usage attribution

7.
Tracing is
WHAT’S THE CATCH?

8.
● Instrumentation is time consuming
● Vendor lock-in (backend and data model)
● Inconsistent semantics across languages
● Not suitable for open source frameworks
Tracing Instrumentation Has Been Too Hard

9.
A vendor-neutral open standard for distributed tracing.
• Addresses instrumentation problem
• Open source, no vendor lock-in
• Consistent semantics across languages
• Available in 7+ programming languages
Who should care:
• Cloud-native / microservice applications
• OSS frameworks, esp. IPC and control flow
• Tracing and/or monitoring systems
http://opentracing.io
An official project of
OpenTracing

10.
Announced v1.0 spec in August 2016
Tracer Implementations: Zipkin, Jaeger, Hawkular, LightStep, Appdash,
and a few smaller tracing systems
Supported Frameworks: GRPC, YARPC, Dropwizard, Flask, Django, Go-kit, ...
Some Companies using OpenTracing:
A young, fast growing project

11.
OpenTracing Approach

12.
TRACER
● StartSpan(operationName) Span
● Inject(SpanContext, Format, Carrier)
● Extract(Format, Carrier) SpanContext
SPAN
● Context() SpanContext
● SetTag(key string, value interface{})
● Log(...)
● Finish()
● SetBaggageItem(key, value string)
● BaggageItem(key string) string
API: Simple, yet expressive

13.
● Off the shelf instrumentation
● Start / Finish
● Inject / Extract
● Tags
● Logs
● Baggage
Code Examples

14.
Jaeger (yā′gər - hunter)
Host or Container
Application
Instrumentation
OpenTracing API
jaeger-client
jaeger-agent
(Go)
jaeger-collector
(Go)
memory queue
Data Store
(Cassandra)
jaeger-query
(Go)
jaeger-ui
(React)
Control Flow
Trace
Reporting
Thrift over
TChannel
Control Flow Trace Reporting
Thrift over UDP
Adaptive
Sampling

15.
Confidential. Internal use only
Thanks!
References:
http://uber.github.io/jaeger
http://opentracing.io - @opentracing