A brief introduction to the use of Redis SortedSet to optimize mysql performances for large collection of with ranked + sortable columns.

2. ARE YOU REDIS?

3. Emanuele `mekdigital` Tozzato
Sony Computer Entertainment America

4. ....an open-source,
networked, in-memory,
persistent, journaled,
key-value data store...

5. • no significant difference read/write
• strings, hashes, lists, sets and sorted sets.
• stable, safe, actively updated

6. APPLICATION #1
large mysql collections
with sortable columns and
ranking / ordinal index

7. GAME LEADERBOARD
&
PRODUCTS TABLE

8. GAME LEADERBOARD
simplest ever
score player t stamp
score ? √
rank ? score >= my_score

9. • Ordinal 1 2 3 4
• Fractional 1 2.3 2.5 3
• Dense 1 2 2 3
• Standard Competition 1 2 2 4
• Modified Competition 1 3 3 4

10. O F D S S’ score player t stamp
1 2 1 1 3 150 adam 2
2 3 1 1 3 150 mike 5
3 1 1 1 3 150 ronald 1
4 4 2 4 4 121 sam 7
5 6 3 5 7 100 adam 2
6 5 3 5 7 100 mark 1
7 7 3 5 7 100 robert 3
how is rank calculated in SQL ?

11. (D)
SELECT
id, player, score,
@prev := @curr, @curr := score,
@rank := IF(@prev = @curr, @rank, @rank+1) AS rank
FROM
lb,
(SELECT @curr := NULL, @prev := NULL, @rank := 0) sel1
ORDER BY score DESC;
(S)
SELECT a.player, a.points, COUNT(b.id) rank
FROM simple_leaderboard a, simple_leaderboard b
WHERE a.points < b.points OR (a.points = b.points AND a.player =
b.player)
GROUP BY a.player, a.points
ORDER BY a.points DESC, a.player ASC;
(S’)
SELECT a.player, a.score, COUNT(b.id) rank
FROM lb a, lb b
WHERE a.score <= b.score OR (a.score = b.score AND a.player = b.player)
GROUP BY a.player, a.score
ORDER BY a.score DESC, a.player ASC

12. REALTIME RANKING

13. REALTIME RANKING
• even with pagination, need for full table scans
• even with good indexes, high complexity
• large sets increased tied items or larger scores

14. PRECALC RANKING

15. PRECALC RANKING
• table locking due to increased writes
• ‘event • (ranked) players’ complexity
• oh wait! ‘event • player • column’ ...

16. GAME LEADERBOARD
not the simplest ever
no_stiuq_yldneirf ,stiuq_yldneirf ,
9_rep_k_p ,sgninni_rep_stih_sklaw_p ,sllab_no_esab_p ,k_p
,snur_denrae_p ,emag_rep_pu_nevig_snur_p ,pu_nevig_snur_p ,
9_rep_stih_b ,pu_nevig_stih_p ,dehctip_sgninni_p ,sevas_p
,stuotuhs_p ,semag_etelpmoc_p ,are_p ,tcp_dleif_f ,srorre_f
,stuo_tup_f ,stsissa_f ,deyalp_sgninni_f ,hctip_yb_tih_b
,yalp_lbd_otni_dednuorg_b ,tcp_esab_no_b ,seilf_ecifircas_b
,tcp_gnigguls_b ,sesab_latot_b ,sklaw_b ,gnilaets_thguac_b
,sesab_nelots_b ,stuoekirts_b ,emag_rep_ibr_b ,ibr_b
,emag_rep_rh_b ,rh_b ,b3_b ,b2_b ,emag_rep_snur_b ,snur_b
,stih_b ,gva_gnittab_b ,stab_ta_b ,no_stiefrof ,stiefrof
,no_sedecnoc ,sedecnoc ,no_stcennocsid ,stcennocsid
,no_stiuq ,stiuq ,semag ,kaerts ,tcp_niw ,sessol ,sniw

17. REDIS MEETS LEADERBOARDS
Sorted Set
key score member
O(1) ~ O(log(N)+M)
• ZADD key score member
• ZCARD key
• Z(REV)RANK key member
• Z(REV)RANGE key member start stop [ws]
• Z(REV)RANGEBYSCORE key member min max [ws] [limit offset]

18. REDIS MEETS LEADERBOARDS
col_1 ... col_n player t stamp
123 ... 21.23 10 1100
145 ... 19.22 12 1105
on create / update of leaderboard entry:
zadd col_1 123 10
zadd col_n 21.23 10
zadd col_1 145 12
zadd col_n 19.22 12

19. REDIS MEETS LEADERBOARDS
col_1 ... col_n player t stamp
123 ... 21.23 10 1100
145 ... 19.22 12 1105
redis> zrevrange col_1 0 -1
1. "12"
2. "10"
redis> zrevrange col_n 0 -1
1. "10"
2. "12"
redis> zrank col_1 12
(integer) 1
redis> zrank col_1 10
(integer) 0

20. REDIS MEETS LEADERBOARDS
col_1 ... col_n player t stamp
100 ... 21.23 10 1100
100 ... 19.22 12 1105
zadd col_1 100 10
zadd col_1 100 12
zadd col_1 100 12
zadd col_1 100 10
redis> zrevrange col_1 0 -1
1. "12"
2. "10"
The elements having the same score are returned in (reverse)
lexographical order. [...previously: (reverse) insertion order...]

21. REDIS MEETS LEADERBOARDS
col_1 ... col_n player t stamp
100 ... 21.23 10 1100
100 ... 19.22 12 1105
break tie composing a fractional score
using a secondary significant column
zadd col_1 100.1100 10
zadd col_1 100.1105 12

22. QUESTIONS ??
(n.b. skip slide unless someone is still awake)

23. redis_leaderboard.rb 1
after_save :dump_to_redis
after_destroy :remove_from_redis
dump to redis and remove from redis methods know™ which columns
need to be ranked and will perform the ZADD redis command that will
insert or update the score in the SortedSet it belongs.
ranked_columns.each do |col|
# zadd(key, score, member)
sorted_set.zadd(col.name, self.send(col.name), self.id)
end

24. redis_leaderboard.rb 2
:find_leaderboard
find leaderboard accepts pagination params to calculate START/STOP
values for Z(REV)RANGE / LIMIT/OFFSET for Z(REV)RANGEBYSCORE,
sorting column and sort order... (make sure START is >= 0)
# ZRANGE
start = params[:per_page] * (params[:page] - 1)
stop = params[:per_page] * params[:page] - 1
ids = params[:sort_order] == 'desc' ?
sorted_set.zrevrange(params[:sort_column], start, stop) :
sorted_set.zrange(params[:sort_column], start, stop)
# ZRANGEBYSCORE
limit = params[:per_page]
offset = params[:per_page] * params[:page]
ids = params[:sort_order] == 'desc' ?
sorted_set.zrevrangebyscore(params[:sort_column], limit, offset) :
sorted_set.zrangebyscore(params[:sort_column], limit, offset)

25. redis_leaderboard.rb 3
:find_leaderboard ~ cont
once the array of ids is returned by Redis, we just need to fetch those
rows from the database with no effort and `transfer` the rank from the
array to the unsorted collection:
result = all(:conditions => {:id => ids})
ids.each_with_index do |sorted_id, index|
row = result.detect {|lb_row| lb_row.id == sorted_id.to_i}
# attr_accessor :rank
row.rank = index + row_start + 1
end
result = result.sort {|x,y| x.rank <=> y.rank}
# assign result to a WillPaginate collection ...

26. redis_leaderboard.rb 4
:find_around_me
find around me is a classic requirement of ranking tables. It’s really easy to
accomplish this task using just the player id and the cardinality of the
items below and above. first find the rank of the player and use it to
create limit and offset for Z(REV)RANKBYSCORE
num_above_below = 5
player_rank = sorted_set.z(rev)rank(params[:sort_column], player_id)
limit = 2*num_above_below + 1
offset = player_rank - num_above_below
ids = params[:sort_order] == 'desc' ?
sorted_set.zrevrangebyscore(params[:sort_column], limit, offset) :
sorted_set.zrangebyscore(params[:sort_column], limit, offset)

27. redis_leaderboard.rb 5
:find_around_me ~ SQL
find_by_sql(["SELECT lb.* FROM
(SELECT @numrow := @numrow+1 as rn, ldb.*
FROM (SELECT * FROM #{table_name}
WHERE game_type = ?
AND platform = ?
ORDER BY #{sort_column} #{sort_order}) ldb,
(SELECT @numrow := 0) rown) lb,
(SELECT rn FROM
(SELECT @numrow := @numrow+1 AS rn, player_rn.player_id
FROM
(SELECT player_id
FROM #{table_name}
WHERE game_type = ?
AND platform = ?
ORDER BY #{sort_column} #{sort_order}) player_rn,
(SELECT @numrow := 0) rown ) my_pos
WHERE player_id = ?) p_rownum
WHERE lb.rn >= p_rownum.rn-? AND lb.rn <= p_rownum.rn+?
ORDER BY #{sort_column} #{sort_order}",
game_type, platform, game_type, platform, player_id, num_above_below,
num_above_below])