SlideShare una empresa de Scribd logo
1 de 21
Descargar para leer sin conexión
"Little Server of Awesome"


       2011 Dvir Volk
      Software Architect, DoAT
   dvir@doat.com http://doat.com
Redis: Key => DataStructure Server

● Memcache-ish in-memory key/value store
● But values are complex data types:
   ○ blobs
   ○ lists
   ○ sets
   ○ sorted sets
   ○ hash tables

● And it has persistence.
● Open source; very helpful and friendly community.
● Used in the real world: github, craigslist, bit.ly, engineyard...
● Used heavily in doAT as a front-end database, search, geo
  resolving
Key Features and Cool Stuff
● All data is in memory (almost)

● All data is eventually persistent

● Handles huge workloads easily

● Mostly O(1) behavior, Ideal for write-heavy workloads

● Support for atomic operations, transactions

● Has pub/sub functionality

● Single threaded, uses aync. IO
● Internal scripting with LUA in alpha
A little benchmark

This is on my laptop (core i7 @2.2Ghz)

 ● SET: 187265.92 requests per second
 ● GET: 185185.17 requests per second
 ● INCR: 190114.06 requests per second


 ● LPUSH: 190114.06 requests per second
 ● LPOP: 187090.73 requests per second
 ●
 ● SADD: 186567.16 requests per second
 ● SPOP: 185873.61 requests per second
Scaling it up

 ● Master-slave replication out of the box

 ● Slaves can be made masters on the fly

 ● Redis-cluster in alpha state.

 ● Sharding supported by some clients

 ● Single threaded - run num_cores/2 instances on the same
   machine
Persistence
● All data is synchronized to disk - eventually or immediately

● Pick your risk level Vs. performance

● Data is either dumped in a forked process, or written as a
  append-only change-log (AOF)

● Append-only mode supports transactional disk writes so you
  can lose no data (cost: 99% speed loss :) )

● You can save the state explicitly, background or blocking

● If the data is too big to fit in memory - redis can handle its
 own swap.
Show me the features!
The basics...
Get/Sets - nothing fancy. Keys are strings, anything goes - just quote spaces.
redis> SET foo "bar"
OK
redis> GET foo
"bar"

You can atomically increment numbers
redis> SET bar 337
OK
redis> INCRBY bar 1000
(integer) 1337

Getting multiple values at once
redis> MGET foo bar
1. "bar"
2. "1337"

Keys are lazily expired
redis> EXPIRE foo 1
(integer) 1
redis> GET foo
(nil)
Be careful with EXPIRE - re-setting a value without re-expiring it will remove the
expiration!
Atomic Operations
GETSET puts a different value inside a key, retriving the old one
redis> SET foo bar
OK
redis> GETSET foo baz
"bar"
redis> GET foo
"baz"

SETNX sets a value only if it does not exist
redis> SETNX foo bar
*OK*
redis> SETNX foo baz
*FAILS*
List operations
 ● Lists are your ordinary linked lists.
 ● You can push and pop at both sides, extract range, resize..
 ● Random access and ranges at O(N)
 ● BLPOP: Blocking POP - wait until a list has elements and pop
   them. Useful for realtime stuff.
Set operations
 ● Sets are... well, sets of unique values w/ push, pop, etc.
 ● Sets can be intersected/diffed /union'ed server side.
 ● Can be useful as keys when building complex schemata.
Sorted Sets
 ● Same as sets, but with score per element
 ● Ranked ranges, aggregation of scores on INTERSECT
 ● Can be used as ordered keys in complex schemata
 ● Think timestamps, inverted index, geohashing, ip ranges
Hashes
 ● Hash tables as values
 ● Think of an object store with atomic access to object
   members

 redis> HSET foo bar 1             redis> HINCRBY foo bar 1
 (integer) 1                       (integer) 2
 redis> HSET foo baz 2
 (integer) 1                       redis> HGET foo bar
 redis> HSET foo foo foo           "2"
 (integer) 1
                                   redis> HKEYS foo
 redis> HGETALL foo                1. "bar"
 {                                 2. "baz"
   "bar": "1",                     3. "foo"
   "baz": "2",
   "foo": "foo"
 }
PubSub - Publish/Subscribe
 ● Clients can subscribe to channels or patterns and receive
   notifications when messages are sent to channels.
 ● Subscribing is O(1), posting messages is O(n)
 ● Think chats, Comet applications: real-time analytics, twitter
Transactions
  ● MULTI, ...., EXEC: Easy because of the single thread.
  ● All commands are executed after EXEC, block and return
    values for the commands as a list.
  ● Example:
redis> MULTI
OK
redis> SET "foo" "bar"
QUEUED
redis> INCRBY "num" 1
QUEUED
redis> EXEC
1) OK
2) (integer) 1

  ● Transactions can be discarded with DISCARD.

  ● WATCH allows you to lock keys while you are queuing your
    transaction, and avoid race conditions.
Gotchas, Lessons Learned
● 64 bit instances consume much much more RAM.

● Master/Slave sync far from perfect.

● DO NOT USE THE KEYS COMMAND!!!

● vm.overcommit_memory = 1

● Use MONITOR to see what's going on
Recipe 1: Social Feed
 ● Each User's followers are list: user:$ID:followers => { 1, 2, 4 }

 ● Each User's feed is a sorted set with the time-stamp being the
   score: user:$ID:feed => { msg1234: 124950132, ... }

 ● Each message is either a blob or a HASH object with fields:
   msg1234 => {title, text, ... }

 ● When a message is added, we:
     ○ add the message
     ○ read the posting user's follower list
     ○ add the message id to each follower's feed
 ● When a user reads their feeds, we use ZREVRANGE to get the
   latest item ids, and HGETALL to fetch the message texts.

 ● For bonus points, connected users can receive push notifications
   with PubSub
Recipe 2: Real-Time Analytics
 ● We want to measure page views, per page, one minute resolution,
   in real time.
 ● Each page is represented by a sorted set, where the value is the
   minute, and the score is the number of hits.

 ● On each page hit, we do ZINCRBY <page> <current_minute> 1

 ● Page views for the last N minutes: ZREVRANGE <page> 0 N

 ● Group several pages together: ZUNIONSTORE ... AGGREGATE
   SUM

 ● Trimming for a specific window: ZREMRANGEBYRANK

 ● Again, PubSub can be used to push updates to clients
Recipe 3: Processing Queue
● Each task can be a blob describing it with json, using
  SET/GET/DEL.

● We keep a list of pending tasks, or multiple lists per task
  type/priority ("tasks:urgent")

● Worker procs can work from any machine over the network.

● Workers poll task queues using BRPOP tasks:urgent
● atomicity guaranteed!

● When a task is pulled it can be marked on a different list/set of
  "tasks:inprogress". When a task is done it is moved to a "done"
  or "failed" task list.

● A manager can monitor failed tasks and move them around.
Other use case ideas
● Geo Resolving with geohashing
● Implemented and opened by yours truly https://github.com/doat/geodis

● API Key and rate management
● Very fast key lookup, rate control counters using INCR

● Real time game data
● ZSETs for high scores, HASHES for online users, etc

● Database Shard Index
● map key => database id. Count size with SETS

● Comet - no polling ajax
● use BLPOP or pub/sub

● Machine learning data with automatic persistence.
● sorted sets are your vectors!
More resources

Redis' website:
http://redis.io

Excellent and more detailed presentation by Simon Willison (a
bit old):
http://simonwillison.net/static/2010/redis-tutorial/

Much more complex twitter clone:
http://code.google.com/p/redis/wiki/TwitterAlikeExample

Full command reference:
http://redis.io/commands

Más contenido relacionado

La actualidad más candente

Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
Dvir Volk
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
TO THE NEW | Technology
 
Introduction to Storm
Introduction to Storm Introduction to Storm
Introduction to Storm
Chandler Huang
 

La actualidad más candente (20)

Introduction to redis
Introduction to redisIntroduction to redis
Introduction to redis
 
Redis overview for Software Architecture Forum
Redis overview for Software Architecture ForumRedis overview for Software Architecture Forum
Redis overview for Software Architecture Forum
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Geospatial Indexing at Scale: The 15 Million QPS Redis Architecture Powering ...
Geospatial Indexing at Scale: The 15 Million QPS Redis Architecture Powering ...Geospatial Indexing at Scale: The 15 Million QPS Redis Architecture Powering ...
Geospatial Indexing at Scale: The 15 Million QPS Redis Architecture Powering ...
 
An Introduction to REDIS NoSQL database
An Introduction to REDIS NoSQL databaseAn Introduction to REDIS NoSQL database
An Introduction to REDIS NoSQL database
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Redis 101
Redis 101Redis 101
Redis 101
 
High-Volume Data Collection and Real Time Analytics Using Redis
High-Volume Data Collection and Real Time Analytics Using RedisHigh-Volume Data Collection and Real Time Analytics Using Redis
High-Volume Data Collection and Real Time Analytics Using Redis
 
Introduction to memcached
Introduction to memcachedIntroduction to memcached
Introduction to memcached
 
Cassandra Introduction & Features
Cassandra Introduction & FeaturesCassandra Introduction & Features
Cassandra Introduction & Features
 
Under the Hood of a Shard-per-Core Database Architecture
Under the Hood of a Shard-per-Core Database ArchitectureUnder the Hood of a Shard-per-Core Database Architecture
Under the Hood of a Shard-per-Core Database Architecture
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Redis and it's data types
Redis and it's data typesRedis and it's data types
Redis and it's data types
 
Introduction to Storm
Introduction to Storm Introduction to Storm
Introduction to Storm
 
RedisConf17 - Lyft - Geospatial at Scale - Daniel Hochman
RedisConf17 - Lyft - Geospatial at Scale - Daniel HochmanRedisConf17 - Lyft - Geospatial at Scale - Daniel Hochman
RedisConf17 - Lyft - Geospatial at Scale - Daniel Hochman
 
HTTP Analytics for 6M requests per second using ClickHouse, by Alexander Boc...
HTTP Analytics for 6M requests per second using ClickHouse, by  Alexander Boc...HTTP Analytics for 6M requests per second using ClickHouse, by  Alexander Boc...
HTTP Analytics for 6M requests per second using ClickHouse, by Alexander Boc...
 
HBase Low Latency
HBase Low LatencyHBase Low Latency
HBase Low Latency
 
MongoDB WiredTiger Internals
MongoDB WiredTiger InternalsMongoDB WiredTiger Internals
MongoDB WiredTiger Internals
 
Redis Overview
Redis OverviewRedis Overview
Redis Overview
 
Redis Reliability, Performance & Innovation
Redis Reliability, Performance & InnovationRedis Reliability, Performance & Innovation
Redis Reliability, Performance & Innovation
 

Similar a Introduction to redis - version 2

Docker Introduction + what is new in 0.9
Docker Introduction + what is new in 0.9 Docker Introduction + what is new in 0.9
Docker Introduction + what is new in 0.9
Jérôme Petazzoni
 

Similar a Introduction to redis - version 2 (20)

Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Dragoncraft Architectural Overview
Dragoncraft Architectural OverviewDragoncraft Architectural Overview
Dragoncraft Architectural Overview
 
Logs @ OVHcloud
Logs @ OVHcloudLogs @ OVHcloud
Logs @ OVHcloud
 
Introduction to AWS Big Data
Introduction to AWS Big Data Introduction to AWS Big Data
Introduction to AWS Big Data
 
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
 
Redis - for duplicate detection on real time stream
Redis - for duplicate detection on real time streamRedis - for duplicate detection on real time stream
Redis - for duplicate detection on real time stream
 
Redis for duplicate detection on real time stream
Redis for duplicate detection on real time streamRedis for duplicate detection on real time stream
Redis for duplicate detection on real time stream
 
Ledingkart Meetup #2: Scaling Search @Lendingkart
Ledingkart Meetup #2: Scaling Search @LendingkartLedingkart Meetup #2: Scaling Search @Lendingkart
Ledingkart Meetup #2: Scaling Search @Lendingkart
 
PostgreSQL: present and near future
PostgreSQL: present and near futurePostgreSQL: present and near future
PostgreSQL: present and near future
 
Presentations from the Cloudera Impala meetup on Aug 20 2013
Presentations from the Cloudera Impala meetup on Aug 20 2013Presentations from the Cloudera Impala meetup on Aug 20 2013
Presentations from the Cloudera Impala meetup on Aug 20 2013
 
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQDocker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
 
Docker Introduction + what is new in 0.9
Docker Introduction + what is new in 0.9 Docker Introduction + what is new in 0.9
Docker Introduction + what is new in 0.9
 
Type safe, versioned, and rewindable stream processing with Apache {Avro, K...
Type safe, versioned, and rewindable stream processing  with  Apache {Avro, K...Type safe, versioned, and rewindable stream processing  with  Apache {Avro, K...
Type safe, versioned, and rewindable stream processing with Apache {Avro, K...
 
Ceph Day New York 2014: Future of CephFS
Ceph Day New York 2014:  Future of CephFS Ceph Day New York 2014:  Future of CephFS
Ceph Day New York 2014: Future of CephFS
 
Amazon DynamoDB Lessen's Learned by Beginner
Amazon DynamoDB Lessen's Learned by BeginnerAmazon DynamoDB Lessen's Learned by Beginner
Amazon DynamoDB Lessen's Learned by Beginner
 
Redis
RedisRedis
Redis
 
Docker and Containers for Development and Deployment — SCALE12X
Docker and Containers for Development and Deployment — SCALE12XDocker and Containers for Development and Deployment — SCALE12X
Docker and Containers for Development and Deployment — SCALE12X
 
Amazon Redshift
Amazon RedshiftAmazon Redshift
Amazon Redshift
 
Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)
 

Más de Dvir Volk (8)

RediSearch
RediSearchRediSearch
RediSearch
 
Searching Billions of Documents with Redis
Searching Billions of Documents with RedisSearching Billions of Documents with Redis
Searching Billions of Documents with Redis
 
Boosting Machine Learning with Redis Modules and Spark
Boosting Machine Learning with Redis Modules and SparkBoosting Machine Learning with Redis Modules and Spark
Boosting Machine Learning with Redis Modules and Spark
 
Redis modules 101
Redis modules 101Redis modules 101
Redis modules 101
 
Tales Of The Black Knight - Keeping EverythingMe running
Tales Of The Black Knight - Keeping EverythingMe runningTales Of The Black Knight - Keeping EverythingMe running
Tales Of The Black Knight - Keeping EverythingMe running
 
10 reasons to be excited about go
10 reasons to be excited about go10 reasons to be excited about go
10 reasons to be excited about go
 
Kicking ass with redis
Kicking ass with redisKicking ass with redis
Kicking ass with redis
 
Introduction to Thrift
Introduction to ThriftIntroduction to Thrift
Introduction to Thrift
 

Último

CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
giselly40
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 

Último (20)

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Evaluating the top large language models.pdf
Evaluating the top large language models.pdfEvaluating the top large language models.pdf
Evaluating the top large language models.pdf
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 

Introduction to redis - version 2

  • 1. "Little Server of Awesome" 2011 Dvir Volk Software Architect, DoAT dvir@doat.com http://doat.com
  • 2. Redis: Key => DataStructure Server ● Memcache-ish in-memory key/value store ● But values are complex data types: ○ blobs ○ lists ○ sets ○ sorted sets ○ hash tables ● And it has persistence. ● Open source; very helpful and friendly community. ● Used in the real world: github, craigslist, bit.ly, engineyard... ● Used heavily in doAT as a front-end database, search, geo resolving
  • 3. Key Features and Cool Stuff ● All data is in memory (almost) ● All data is eventually persistent ● Handles huge workloads easily ● Mostly O(1) behavior, Ideal for write-heavy workloads ● Support for atomic operations, transactions ● Has pub/sub functionality ● Single threaded, uses aync. IO ● Internal scripting with LUA in alpha
  • 4. A little benchmark This is on my laptop (core i7 @2.2Ghz) ● SET: 187265.92 requests per second ● GET: 185185.17 requests per second ● INCR: 190114.06 requests per second ● LPUSH: 190114.06 requests per second ● LPOP: 187090.73 requests per second ● ● SADD: 186567.16 requests per second ● SPOP: 185873.61 requests per second
  • 5. Scaling it up ● Master-slave replication out of the box ● Slaves can be made masters on the fly ● Redis-cluster in alpha state. ● Sharding supported by some clients ● Single threaded - run num_cores/2 instances on the same machine
  • 6. Persistence ● All data is synchronized to disk - eventually or immediately ● Pick your risk level Vs. performance ● Data is either dumped in a forked process, or written as a append-only change-log (AOF) ● Append-only mode supports transactional disk writes so you can lose no data (cost: 99% speed loss :) ) ● You can save the state explicitly, background or blocking ● If the data is too big to fit in memory - redis can handle its own swap.
  • 7. Show me the features!
  • 8. The basics... Get/Sets - nothing fancy. Keys are strings, anything goes - just quote spaces. redis> SET foo "bar" OK redis> GET foo "bar" You can atomically increment numbers redis> SET bar 337 OK redis> INCRBY bar 1000 (integer) 1337 Getting multiple values at once redis> MGET foo bar 1. "bar" 2. "1337" Keys are lazily expired redis> EXPIRE foo 1 (integer) 1 redis> GET foo (nil) Be careful with EXPIRE - re-setting a value without re-expiring it will remove the expiration!
  • 9. Atomic Operations GETSET puts a different value inside a key, retriving the old one redis> SET foo bar OK redis> GETSET foo baz "bar" redis> GET foo "baz" SETNX sets a value only if it does not exist redis> SETNX foo bar *OK* redis> SETNX foo baz *FAILS*
  • 10. List operations ● Lists are your ordinary linked lists. ● You can push and pop at both sides, extract range, resize.. ● Random access and ranges at O(N) ● BLPOP: Blocking POP - wait until a list has elements and pop them. Useful for realtime stuff.
  • 11. Set operations ● Sets are... well, sets of unique values w/ push, pop, etc. ● Sets can be intersected/diffed /union'ed server side. ● Can be useful as keys when building complex schemata.
  • 12. Sorted Sets ● Same as sets, but with score per element ● Ranked ranges, aggregation of scores on INTERSECT ● Can be used as ordered keys in complex schemata ● Think timestamps, inverted index, geohashing, ip ranges
  • 13. Hashes ● Hash tables as values ● Think of an object store with atomic access to object members redis> HSET foo bar 1 redis> HINCRBY foo bar 1 (integer) 1 (integer) 2 redis> HSET foo baz 2 (integer) 1 redis> HGET foo bar redis> HSET foo foo foo "2" (integer) 1 redis> HKEYS foo redis> HGETALL foo 1. "bar" { 2. "baz" "bar": "1", 3. "foo" "baz": "2", "foo": "foo" }
  • 14. PubSub - Publish/Subscribe ● Clients can subscribe to channels or patterns and receive notifications when messages are sent to channels. ● Subscribing is O(1), posting messages is O(n) ● Think chats, Comet applications: real-time analytics, twitter
  • 15. Transactions ● MULTI, ...., EXEC: Easy because of the single thread. ● All commands are executed after EXEC, block and return values for the commands as a list. ● Example: redis> MULTI OK redis> SET "foo" "bar" QUEUED redis> INCRBY "num" 1 QUEUED redis> EXEC 1) OK 2) (integer) 1 ● Transactions can be discarded with DISCARD. ● WATCH allows you to lock keys while you are queuing your transaction, and avoid race conditions.
  • 16. Gotchas, Lessons Learned ● 64 bit instances consume much much more RAM. ● Master/Slave sync far from perfect. ● DO NOT USE THE KEYS COMMAND!!! ● vm.overcommit_memory = 1 ● Use MONITOR to see what's going on
  • 17. Recipe 1: Social Feed ● Each User's followers are list: user:$ID:followers => { 1, 2, 4 } ● Each User's feed is a sorted set with the time-stamp being the score: user:$ID:feed => { msg1234: 124950132, ... } ● Each message is either a blob or a HASH object with fields: msg1234 => {title, text, ... } ● When a message is added, we: ○ add the message ○ read the posting user's follower list ○ add the message id to each follower's feed ● When a user reads their feeds, we use ZREVRANGE to get the latest item ids, and HGETALL to fetch the message texts. ● For bonus points, connected users can receive push notifications with PubSub
  • 18. Recipe 2: Real-Time Analytics ● We want to measure page views, per page, one minute resolution, in real time. ● Each page is represented by a sorted set, where the value is the minute, and the score is the number of hits. ● On each page hit, we do ZINCRBY <page> <current_minute> 1 ● Page views for the last N minutes: ZREVRANGE <page> 0 N ● Group several pages together: ZUNIONSTORE ... AGGREGATE SUM ● Trimming for a specific window: ZREMRANGEBYRANK ● Again, PubSub can be used to push updates to clients
  • 19. Recipe 3: Processing Queue ● Each task can be a blob describing it with json, using SET/GET/DEL. ● We keep a list of pending tasks, or multiple lists per task type/priority ("tasks:urgent") ● Worker procs can work from any machine over the network. ● Workers poll task queues using BRPOP tasks:urgent ● atomicity guaranteed! ● When a task is pulled it can be marked on a different list/set of "tasks:inprogress". When a task is done it is moved to a "done" or "failed" task list. ● A manager can monitor failed tasks and move them around.
  • 20. Other use case ideas ● Geo Resolving with geohashing ● Implemented and opened by yours truly https://github.com/doat/geodis ● API Key and rate management ● Very fast key lookup, rate control counters using INCR ● Real time game data ● ZSETs for high scores, HASHES for online users, etc ● Database Shard Index ● map key => database id. Count size with SETS ● Comet - no polling ajax ● use BLPOP or pub/sub ● Machine learning data with automatic persistence. ● sorted sets are your vectors!
  • 21. More resources Redis' website: http://redis.io Excellent and more detailed presentation by Simon Willison (a bit old): http://simonwillison.net/static/2010/redis-tutorial/ Much more complex twitter clone: http://code.google.com/p/redis/wiki/TwitterAlikeExample Full command reference: http://redis.io/commands