8. AVAILABLE DATA STRUCTURES:
•STRINGS (Binary safe, can be anything from "hello world" to a jpeg file)
•LISTS(Collections of string elements sorted according to the order of insertion)
•SETS (Collections of unique, unsorted string elements)
•SORTEDSETS (It's like Sets with a score)
•HASHES (Objects. Maps of fields associated to values. Think non-nested json objects)
•BITMAPS (Manipulate Strings on a bit level)
•HYPERLOGLOGS (Probabilistic data structure used to estimate the cardinality of a set)
9. STRINGS
•Binary safe, can be from "hello world" to a jpeg filekeyname
value •Maximum size = 512MB
•Keynames can be emojis 🐘🦄😀
12. HASHES
keyname •Maps between string fields and string values
•Great for tracking representing objects
field1
field2
field3
field4
value1
value2
value3
value4
•Max length: > 4 billion elements
13. SORTED SETS
keyname •Sets with a floating number score
•Very fast access to elements in the middle
•Perfect for leaderboards
field1
field2
field3
field4
0.6379
6.379
63.79
637.9
•Get ranks of elements
•Get ranges of sorted elements
•Sort by value and by field, indirectly (lex)
14. BITMAPS
•Manipulate Strings on a bit levelkeyname
value •Set / clear individual bits
•Find the first set or unset bit
•Get count of all set/unset bits
17. I loved the service at #statscompany,
30 minutes from ticket creation to
resoltuion. Well done guys, I'm
impressed.
I hate the service at #statscompany, I've
been on hold for the past 20minutes.
18. OUR SCENARIO: TWITTER ANALYZER DASHBOARD
751
TOTAL TWEETS
751Main stream
120sub-stream foo
107sub-stream bar
...
Substream:keyword 1 Substream:keyword 2
Love
Hate
Great
0 30 60 90 120
86
107
120
Total tweets
Scores
Main Stream
22. AVAILABLE DATA STRUCTURES:
•STRINGS (Binary safe, can be anything from "hello world" to a jpeg file)
•LISTS(Collections of string elements sorted according to the order of insertion)
•SETS (Collections of unique, unsorted string elements)
•SORTEDSETS (It's like Sets with a score)
•HASHES (Objects. Maps of fields associated to values. Think non-nested json objects)
•BITMAPS (Manipulate Strings on a bit level)
•HYPERLOGLOGS (Probabilistic data structure used to estimate the cardinality of a set)
23. [2]SETTRACKEDDATA
Use sets to store all the hashtags we'll be looking at and all the keywords as well
$client->sadd('hashtags', 'thestatscompany','statscompany',
'statscampaign');
// hashtags | 'thestatscompany'
// | 'statscompany'
// | 'statscampaign'
$client->sadd('keywords', 'loved', 'amazed', 'frustrated',
'problem', 'terrible', 'amazing', 'great', 'hate');
25. [3]GETTHEDATA
Use Twitter Stream API to receive notifications for tweets containing any of the hashtags we're following
$hashtags = $client->smembers('hashtags');
// array (size=3)
// 0 => string 'thestatscompany' (length=15)
// 1 => string 'statscompany' (length=12)
// 2 => string 'statscampaign' (length=13)
26. Save every new tweet from the stream as a separate string:
$keyname = 'tweet_id:' . $tweet_id;
$tweet_contents = "Amazed by the customer service of #statscompany";
$client->set($keyname, $tweet_contents);
$keyname = 'tweet_id:' . $tweet_id;
$tweet_contents = "Amazed by the customer service of #statscompany";
$client->set($keyname, $tweet_contents);
27. AVAILABLE DATA STRUCTURES:
•STRINGS (Binary safe, can be anything from "hello world" to a jpeg file)
•LISTS(Collections of string elements sorted according to the order of insertion)
•SETS (Collections of unique, unsorted string elements)
•SORTEDSETS (It's like Sets with a score)
•HASHES (Objects. Maps of fields associated to values. Think non-nested json objects)
•BITMAPS (Manipulate Strings on a bit level)
•HYPERLOGLOGS (Probabilistic data structure used to estimate the cardinality of a set)
28. And then push to a queue to be processed asynchronously
$client->lpush('message_queue', $keyname);
// 'message_queue' | 'tweet_id:45645656'
// | 'tweet_id:44645234'
// | 'tweet_id:43645232'
$client->lpush('message_queue', $keyname);
// 'message_queue' | 'tweet_id:45645656'
// | 'tweet_id:44645234'
// | 'tweet_id:43645232'
29. AVAILABLE DATA STRUCTURES:
•STRINGS (Binary safe, can be anything from "hello world" to a jpeg file)
•LISTS(Collections of string elements sorted according to the order of insertion)
•SETS (Collections of unique, unsorted string elements)
•SORTEDSETS (It's like Sets with a score)
•HASHES (Objects. Maps of fields associated to values. Think non-nested json objects)
•BITMAPS (Manipulate Strings on a bit level)
•HYPERLOGLOGS (Probabilistic data structure used to estimate the cardinality of a set)
31. [4]WORKERTOPROCESSTHEQUEUEDJOBS
A separate worker will be grabbing jobs off the top of the queue and processing them:
$message_queue = $client->rpop('message_queue');$message_queue = $client->rpop('message_queue');
Reliable queue: RPOPLPUSH, BRPOPLPUSH
Blocking queue: BLPOP, BRPOP
34. SORTED SETS
keyname •Sets with a floating number score
•Very fast access to elements in the middle
•Perfect for leaderboards
field1
field2
field3
field4
0.6379
6.379
63.79
637.9
•Get ranks of elements
•Get ranges of sorted elements
•Sort by value and by field, indirectly (lex)
40. Redis is TCP server using client/server model. A request to the server is
accomplished in two steps:
•The client sends a query to the server, and reads from the socket, usually in a blocking way,
for the server response
•The server processes the command and sends the response back to the client
Client Server
Networking link
Request from client to server
Response from server to client
41. Redis is a TCP server using the client/server model. A request to the server
is accomplished in two steps:
•The client sends a query to the server, and reads from the socket, usually in a blocking way,
for the server response
•The server processes the command and sends the response back to the client
Client Server
Request from client to server
Response from server to client
RTT: Round Trip Time
42. Number of
requests to
server
•Save the tweet
•Add tweet to main feed
•Increase counter for total tweets
•Get all monitored keywords
•Check tweet for any keywords
•Increase counter for found keywords
•Add tweet to sub-feed
•Update score
9THESTEPS:
43. Number of
requests to
server
•Save the tweet
•Add tweet to main feed
•Increase counter for total tweets
•Get all monitored keywords
•Check tweet for any keywords
•Increase counter for found keywords
•Add tweet to sub-feed
•Update score
13?
} + 4 requests for every found keyword
THESTEPS:
45. 1
Number of
requests to
server</php
$keywords = $client->smembers("keywords");
$responses = $client->pipeline(function ($pipe) use ($keywords) {
$pipe->set(...); // Save the tweet
$pipe->lpush(...); // Save the tweet to main feed
$pipe->ltrim(...); // Trim the main feed to X tweets so it doesn't grow forever
foreach ($keywords as $keyword) {
$pipe->incr(...); // Increase the counter for the keyword
$pipe->lpush(...); // Add to the subbed
$pipe->ltrim(...); // Trim the stream to X tweets
$pipe->zincrby(...); // Update score
}
});
USINGTHEPREDISLIBRARYFORPHP
46. 2
Number of
requests to
server</php
$keywords = $client->smembers("keywords");
$responses = $client->pipeline(function ($pipe) use ($keywords) {
$pipe->set(...); // Save the tweet
$pipe->lpush(...); // Save the tweet to main feed
$pipe->ltrim(...); // Trim the main feed to X tweets so it doesn't grow forever
foreach ($keywords as $keyword) {
$pipe->incr(...); // Increase the counter for the keyword
$pipe->lpush(...); // Add to the subfeed
$pipe->ltrim(...); // Trim the stream to X tweets
$pipe->zincrby(...); // Update score
}
});
USINGTHEPREDISLIBRARYFORPHP
49. hello.lua:
local tweet = ARGV[1]
local key_tweet = KEYS[1]
local key_keywords = KEYS[2]
local key_total_tweets_count = KEYS[3]
local key_scores = KEYS[4]
local key_main_feed = KEYS[5]
redis.call("SET", key_tweet, tweet) -- Save the tweet
redis.call("INCR", key_total_tweets_count) -- Increase the total tweet count
redis.call("LPUSH", key_main_feed, tweet) -- Push the tweet to the main feed
redis.call("LTRIM", key_main_feed, 0, 100) -- Trim the main feed
local keywords = redis.call("SMEMBERS", key_keywords) -- Get the keywords
for i, name in ipairs(keywords) do
if string.find(tweet, name) then
local substream_name = "sub_feed:" .. name
redis.call("LPUSH", substream_name, tweet) -- Push the tweet to the sub feed
redis.call("LTRIM", substream_name, 0, 100) -- Trim the sub feed
redis.call("ZINCRBY", key_scores, 1, name) -- Increment the score in the leaderboard
end
end
return "OK"
50. > redis-cli --eval hello.lua tweet_123 keywords ... , "A tweet about the words foo and bar"
"OK"
1!
Number of
requests to
server
51. SCRIPT
•Evaluates a script cached on the server side by its SHA1 digest. The command is otherwise
identical to EVAL.
> redis-cli SCRIPT LOAD "$(cat hello.lua)"
"6d52847f03028ab1d4620b60dd6ef4a14c8727d7"
> redis-cli evalsha 6d52847f03028ab1d4620b60dd6ef4a14c8727d7 5
> tweet_123 keywords ... "A tweet about the words foo and bar"
"OK"
EVALSHA
•Load a script into the scripts cache, without executing it. Returns a SHA-1 digest of the script.
53. •Dynamically loaded libraries
•Written in C
•Almost as fast as the Redis core
•Let you extend Redis commands, create new data structures, access data almost
as fast as native Redis commands
•Add-ons to Redis
•Coming in version 4.0, currently in Beta (RC)
WHATAREREDISMODULES?
54. •Low-level: Close to native access to core data structures
•High-level: Client-like access to core and modules' commands
LAYERSOFTHEMODULESAPI
55.
56.
57.
58. RedisModule_Call(ctx,"INCR","sc",argv[1],"10");
Command
Format specifier
Arguments
The context object
c -- Null terminated C string pointer.
b -- C buffer, two arguments needed: C string pointer and size_t length.
s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object.
l -- Long long integer.
v -- Array of RedisModuleString objects.
! -- This modifier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of
arguments parsing.
HIGHLEVELAPI
59. Anatomy of a module
#include "redismodule.h"
#include <stdlib.h>
int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
// ...
}
int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
// Register the module or error out
if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR;
// Register the command or error out
if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand,
"readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR;
return REDISMODULE_OK;
}
Definitions of the API
Called whenever the module is loaded
Must be present in each Redis module
Register the module or error out
Register the command or error out
Command Function name
60. Anatomy of a module
#include "redismodule.h"
#include <stdlib.h>
int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
// ...
}
int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
// Register the module or error out
if (RedisModule_Init(ctx, "process_tweet", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR;
// Register the command or error out
if (RedisModule_CreateCommand(ctx, "process_tweet.rand", ProcessTweet_RedisCommand,
"readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR;
return REDISMODULE_OK;
}
The logic of the module
61. How much faster does it get?
Anatomy of a module
#include "redismodule.h"
#include <stdlib.h>
int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
// ...
}
int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
// Register the module or error out
if (RedisModule_Init(ctx, "process_tweet", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR;
// Register the command or error out
if (RedisModule_CreateCommand(ctx, "process_tweet.rand", ProcessTweet_RedisCommand,
"readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR;
return REDISMODULE_OK;
}
int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
// We need EXACTLY 5 arguments
if (argc != 5) return RedisModule_WrongArity(ctx);
RedisModule_AutoMemory(ctx);
RedisModuleCallReply *reply;
reply = RedisModule_Call(ctx, "INCR", "s", argv[3]);
RedisModule_ReplyWithCallReply(ctx, reply);
return REDISMODULE_OK;
}
Enable automatic memory management
Call Redis commands
Reply with a call object
62. RedisModule_AutoMemory(ctx);
RedisModuleKey *key;
key = RedisModule_OpenKey(ctx,argv[1],REDISMODULE_READ|REDISMODULE_WRITE);
if (RedisModule_KeyType(key) == REDISMODULE_KEYTYPE_EMPTY) {
RedisModule_StringSet(key,argv[2]);
}
RedisModule_CloseKey(key);
Open a key. Return a key pointer
Checks the type of the key
Set a string value for a key
Close the key
LOWLEVELAPI