Redis/Lessons learned

Lessons learned: NoSQL/Redis

Tit Petric / Jan 2013

Monotek d.o.o.

NoSQL: why it’s here
SQL:
- Slow query performance
- Concurrency / locking
- Hard to scale (even harder for writes, storage)

Typical problems
- Session storage
- Statistics (high write to read ratio)
- Modifying schema on large data sets

Tit Petric / Twitter @titpetric

NoSQL: memcache

Memcache (RTV 2008-Present)
- Pro: stability, speed
- Pro: simple text protocol (added binary to fuck with us)
- Love/Hate: Scaling out reads/writes
- Con: Persistence
- Con: Replication
- Con: Key eviction not based on LRU/TTL but slab allocation


NoSQL: sharedance

Sharedance (2009-2011)
- Pro: Persistent KV storage
- Pro: Simple text protocol (wrote a client in LUA)
- Con: Had to patch daemon to handle eviction load (1 key = 1
file, filesystems can’t handle this)
- Con: Had to use special ReiserFS filesystem on deployment


NoSQL: redis

Redis (2011-Present)
- Pro: stability, speed
- Pro: simple text protocol, documentation, clients
- Love: Scaling out reads/writes
- Pro: Persistence
- Pro: Replication
- Con/Feature request: key segmentation / grouping.


NoSQL: uses at RTV Slovenia

To-do (2013-?)
- Memcached protocol translator to Redis
- Look at twemcached, avoid client based sharding
- Implement webdis deployment
- Redis scripting with LUA


NoSQL: Redis data types

Redis is still a Key/Value store!

Only we can have different values:
- Strings (essentialy the same as memcache)
- Hashes (nested k/v pairs)
- Lists (simple arrays)
- Sets (unique arrays)
- Sorted sets (weighted unique arrays)


NoSQL: Redis data types

LETS SEE SOME EXAMPLES! 


Redis data types: Strings
Limiting Google bot crawl rate

“A 503 (Service Unavailable) error will result in fairly
frequent retrying. To temporarily suspend crawling, it is
recommended to serve a 503 HTTP result code.”



SETNX – set a key if it doesn’t exist
EXPIRE – expire a key after TTL seconds
INCR – increment value by one


Redis data types: Hashes
News ratings

HMSET – set multiple hash fields / values
HGETALL – get all fields and values
HINCRBY – increment integer value of a hash
field


News ratings


News ratings

Vote data Why Expire?
Race condition.
We need HMSETNX
Could be better.


Other use cases include:

- User data, partial data retrieval
- select username, realname, birthday from users where id=?
- HMGET users:$id username realname birthday

- Using SORT (list|set) BY hash values

- Don’t use HASHes to store session. Eviction policy
works on KEYS not on hash values!


Redis data types: Lists
Any kind of information log (statistics,…)

LPUSH – push values to the beginning of the list
RPUSH – push values to the end of the list
LRANGE – get a range of values
LTRIM – trim a list to the specified range
LLEN – get the length of the list


Collecting statistics

We can skip the database completely


Process data
Into SQL
database


Well, it’s a way to scale writes to SQL 

Processing job can DIE for ages, because:

- Back of the envelope calculation for redis memory use:
100M keys use 16 GB ram
- Logs get processed in small chunks (200 items), avoiding
memory limits. Could increase this by a lot.
- We also use sharding so writes are distributed per $table


Redis data types: Sets
Set values are UNIQUE

SADD – Add one or more members to a set

Perfect use case: set insersection with
SINTERSTORE, find duplicates.

MySQL is too slow for this, even with
indexes…


SET Intersection in MySQL
List1 = first table of data
List2 = second table of data


Bulk transfer MySQL data to redis
Via: http://dcw.ca/blog/2013/01/02/mysql-to-redis-in-one-step/


SET Intersection in Redis

Much faster, without indexes!

0.118 seconds vs. mysql 1.35 (+0.36 for index)
15x speed increase!


Other possible uses for sets:

• Common friends between A and B
• Friend suggestions (You might know…)
• People currently online …


Redis data types: Sets vs. Sorted sets
Ok, typical use case in sql>

select title, content from news order by
stamp desc limit 0,10

#1) Use SORT from redis + HMGET
#2) Use sorted sets (ZSET type)


Redis data types: Sorted sets
Sorted sets by time with a PK
auto_increment? NO!

• Most read news items (sort by views)
• Order comments by comment rating
• Friends by most friends in common


Order comments by rating

ZINCRBY – increase/decrease score of item
ZRANGE – return portion of sorted set, ASC
ZREVRANGE – portion of sorted set, DESC


Sort comments by rating! With pagination!

ZRANGE – return portion of sorted set, ASC
ZREVRANGE – portion of sorted set, DESC


Scaling Redis deployment

SLAVEOF [host] [port]
Starts replicating from [host]:[port], making this instance a slave

SLAVEOF NO ONE

Promote instance to MASTER role


Scaling Redis deployment
Phpredis client does not implement
sharding by itself! But …
- Master / Multi-slave scaling is easy to do
- Failover for reads is easy, node ejection possible
- Client deploys still take time – twemproxy is an option
- Twemproxy also provides sharding support, & Memcached 
- Want to see what Redis is doing? Issue “MONITOR” command.
- Stale data is better than no data, we still consider Redis volatile
- FlushDB = rebuild cache, we tolerate data loss


Redis: Q & A section

Questions and answers!

Follow me on Twitter: @titpetric
Read our tech blog: http://foreach.org


Redis/Lessons learned

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Redis/Lessons learned

Similar a Redis/Lessons learned (20)

Redis/Lessons learned