My talk at Topconf.com conference, Tallinn, 1st of November 2012
1. Tarantool/Box: a use case with
serving 2 billion queries a day
November 1st, 2012, Tallinn, Estonia
Konstantin Osipov, Mail.Ru
kostja@tarantool.org
2. Agenda
Tarantool/Box: features and architecture
Case study 1: advanced memcache
Case study 2: a message queue server
Case study 3: a reliable in-memory database
Project future
3. Feature overview
●
Flexible data model, primary and secondary keys
●
Fully cached: 100% of data is cached in RAM
●
Persistent using a Write Ahead Log
●
Log shipping replication and online backup
●
extensible with Lua
4. Data model
●
A game of fields, tuples and spaces
●
HASH, TREE, BITMAP, partial, indexes
●
Secondary keys
●
Single-part, multi-part
●
STRING, NUM and NUM64 data types
10. Use case 1: flexible memcache
●
You can create your own fibers
●
box.fiber.create(), box.fiber.yield()
●
A background fiber performs a customized expire
●
Session store: 20M online users, 200M monthly users,
4 2-CPU units, 96GB RAM each
●
60K RPS, CPU usage is below 20%
11. Use case 2: message queue
●
Reliable queues are a vital ingredient for building
scalable applications
●
In Web Apps queues are used for delayed processing,
load balancing, e-mail notifications
●
Our use case: prefetching avatars
12. Message queues: how
●
Rich Lua stored procedure environment
― box.fiber, box.space, box.cfg
●
Specialized inter-procedure communication API:
― box.ipc
●
Specialized data structures
― Bitmaps and partial keys
13. Use case 3: reliable database
●
User profile: 500 bytes of key/value pairs
●
Math: 200M users * 500 = 100GB
●
At least 2x smaller memory footprint than in a RDBMS
●
Win: predictable response time, significantly higher
performance
14. Example: a FIFO
function fifo_push(name, val)
fifo = find_or_create_fifo(name)
top = box.unpack('i', fifo[1])
bottom = box.unpack('i', fifo[2])
if top == fifomax+2 then -- % size
top = 3
…
end
return box.update(0, name, '=p=p=p', 1, top,
2, bottom, top, val)
end
15. Conclusion
●
Tarantool/Box – a high performance data engine
●
Tarantool/Lua – a building block for your heavily loaded
web applications
●
Tarantool – our approach to an easy to use database
for highly volatile data
16. What's cooking in 1.5
●
Disk-based backend
●
Synchronous master-master replication
●
New data types (array, date, currency, json)
●
authentication