Tech roulette presentation given at Warby Parker tech team meeting.

1. Consistent Hashing

2. The problem
•
Distribute items into buckets
•
Fetch an item from its bucket

3. Naive Solution
hash(key) % buckets

4. Naive solution
0
foo
3
4
baz
1
1
2
0
bar
2
parker
3
warby

5. Other considerations
•
Given a resource key and a list of servers, how do you find a
primary, second, tertiary (and on down the line) server for the
resource?
•
If you have different size servers, how do you assign each of them
an amount of work that corresponds to their capacity?
•
How do you smoothly add capacity to the system without
downtime? Specifically, this means solving two problems:
•
How do you avoid dumping 1/N of the total load on a new
server as soon as you turn it on?
•
How do you avoid rehashing more existing keys than
necessary?

9. Benefits
•
Weighted distributions based on the number of
replicas per server.
•
New servers can be brought online gradually.
•
Simple data replication.

10. Real world usage
•
memcached
•
redis
•
graphite
•
Dynamo (Amazon)
•
BigTable (Google)

11. •
Consistent Hashing and Random Trees: Distributed
Caching Protocols for Relieving Hot Spots on the
World Wide Web
http://thor.cs.ucsb.edu/~ravenben/papers/coreos/kll+97.pdf
•
http://www.tomkleinpeter.com/2008/03/17/
programmers-toolbox-part-3-consistent-hashing/
•
https://weblogs.java.net/blog/tomwhite/archive/
2007/11/consistent_hash.html