Lyft open sources several infrastructure projects including Confidant for securely storing secrets, Discovery for service registration and lookup, Ratelimit for rate limiting requests, and Envoy as an edge and internal proxy. Envoy handles all service to service communication at Lyft and provides observability, load balancing, and integration with other services. These projects help Lyft build and operate a microservices architecture at large scale.

1. Open-source Infrastructure at Lyft
Constance Caramanolis
Daniel Hochman July 2017

2. Overview of Lyft Architecture
Open-source Infrastructure Projects
- Confidant
- Discovery
- Ratelimit
- Envoy
Q&A
Agenda

3. Architecture (simplified)
Front Envoy
Application
Envoy
DiscoveryConfidant
>100 Clusters
Ratelimit

4. Python
lyft / confidant
Your secret keeper. Stores secrets in Dynamo, encrypted at rest.
1,105
12 contributors
November 2015

5. How is a service configured?
lyft / location-service Private
common:
PORT: 8080
TIMEOUT_MS: 15000
development:
USE_AUTH: False
staging:
API_KEY: secret_key_igjq3i494fqq234qbc
production:
API_KEY: secret_key_ojajf823jj49ij8h
environment.yaml

6. Service
location-service
Confidant to the rescue!
Credential
api_key: password123

7. Behind the scenes
Application
IAM Role
EC2 Instance
Credential
api_key: password123
api_key = os.getenv('CREDENTIAL_API_KEY')
KMS
DynamoDB
Confidant

8. Server-blind secrets
Highly sensitive secrets are encrypted and decrypted by the end-users.
Confidant stores but can't read them.
Confidant
KMS
IAM Role
EC2 Instance

9. lyft / discovery
Provides a REST interface for querying for the list of hosts that belong to a microservices
54
6 contributors
Python
August 2016

10. POST /v1/registration/location-service
{
"ip": "10.0.0.1",
"port": 80,
"revision": "da08f35b",
"tags": {
"id": "i-910203",
"az": "us-east-1a",
"canary": true
}
}
Tracking hosts
* * * * *

11. - Hosts are stored in DynamoDB
- Storage support is abstract
- Hosts removed if not reporting since now - HOST_TTL
- Ecosystem designed to tolerate eventual consistency
unlike Zookeeper, etcd, Consul
- Pair with active healthchecks
Storage
DynamoDB

12. GET /v1/registration/<service>
{
"hosts": [
{
"ip": "10.0.0.1", "port": 80, "revision": "da08f35b",
"tags": {"id": "i-910203", "az": "us-east-1a", "canary": true}
},
...
{
"ip": "10.0.0.2", "port": 80, "revision": "da08f35b",
"tags": {"id": "i-121286", "az": "us-east-1d"}
}
]
}
Fetching hosts

13. Services list the hosts they want to talk to!
internal_hosts:
- jobscheduler
- roads
external_hosts:
- dynamodb_iad
- kinesis_iad
Envoy per-service configuration
location-service/envoy.yaml
/etc/envoy.conf
(on the box)

14. Active Healthcheck
Application
Envoy
Discovery
jobscheduler roads
GET /healthcheck
Application
Envoy
GET
GET
Every host healthchecks every host in a destination cluster
location-service

15. lyft / ratelimit
Go/gRPC service designed to enable generic rate limit scenarios
224
6 contributors
Go
January 2017

16. Why rate limit?
- Control flow
- Protect against attacks
- Bad actors
- Accidents happen
oops!

17. Rate Limit Service
- Written in Go
- Enable generic rate limit
scenarios
- Decisions based on a domain
and set of descriptors
- Settings configured at runtime
- Backed by Redis
Ratelimit
?
INCR

18. Domains and descriptors
Domain
Defines a container for a set of rate limits
Globally unique
e.g. "envoy_front"
Descriptors
Ordered list of key/value pairs
Case sensitive
e.g. ("destination_cluster", "location-service"), ("user_id", "1234")

19. Limit definition
Runtime Setting
Defines the request per unit for a descriptor.

20. Request flow example
Rq1: (“user_id”, “1234”)
Redis state: user_id_1234 : 1
Rs1: RateLimitResponse_OK
Rq2: (“user_id”, “9876”)
Redis state: user_id_1234: 1, user_id_9876 : 1
Rs2: RateLimitResponse_OK
Rq3: (“user_id”, “1234)
Redis state: user_id_1234: 2, user_id_9876 : 1
Rs3: RateLimitResponse_OVER_LIMIT
Definition
domain: test_domain
key: user_id
rate_limit:
unit: hour
requests_per_unit: 1

21. Ratelimit Client
from lyft_idl.client.ratelimit.ratelimit_client import RateLimitClient
ratelimit_client = RateLimitClient(settings.LYFT_API_USER_AGENT)
# Determines whether or not to limit jsonp_messages_post according to ratelimit service.
def should_allow_jsonp_messages_post(ip_address, phone_number):
domain = settings.get('RATE_LIMIT_DOMAIN')
ip_descriptors = [(('jsonp_messages_post_from_ip_address', ip_address), )]
phone_descriptors = [(('jsonp_messages_post_from_phone_number', phone_number), )]
return (
ratelimit_client.is_request_allowed(domain, ip_descriptors) and
ratelimit_client.is_request_allowed(domain, phone_descriptors)
)

22. lyft / envoy
Front/service L7 proxy
1,924
62 contributors
C++
September 2016

23. Why Envoy?
Service Oriented Architecture
- Many languages and frameworks
- Protocols (HTTP/1, HTTP/2, databases, caching, etc…)
- Partial implementation of SoA best practices (retries, timeouts, …)
- Observability
- Load balancers (AWS, F5)

24. What is Envoy?
The network should be transparent to applications.
When network and application problems do occur it
should be easy to determine the source of the problem.

25. What is Envoy?
- Modern C++11
- Runs alongside applications
- Service discovery integration
- Rate Limit integration
- HTTP2 first (get gRPC!)
- Act as front/edge proxy
- Stats, Stats, Stats
- Logging

26. Observability: Global Health

27. Observability: Service to Service

28. Envoy Client in Python (internal)
from lyft.api_client import EnvoyClient
switchboard_client = EnvoyClient(
service='switchboard'
)
switchboard_client.post(
"/v2/messages",
data={
'template': 'welcome'
},
headers={
'x-lyft-user-id': 12345647363394
}
)

29. Envoy deployment @Lyft
- > 100 services
- > 10,000 hosts
- > 2,000,000 RPS
- All service to service traffic (REST and gRPC)
- MongoDB, DynamoDB, Redis proxy
- External service proxy (AWS and other partners)
- Kibana/Elastic Search for logging.
- LightStep for tracing
- Wavefront for stats

30. Architecture Revisited
Front Envoy
Application
Envoy
DiscoveryConfidant
>100 Clusters
Ratelimit

31. Done!
- Lyft is hiring. If you want to work on large-scale problems in a fast-moving,
high-growth company visit lyft.com/jobs
- Visit github.com/lyft
- Slides available at slideshare.net/danielhochman
- Q&A