Mongo à la Resque
•
1 recomendación•610 vistas
Talk given at MongoFR 2011 in Paris, France. http://lanyrd.com/2011/mongofr/speakers
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Mongo à la Resque
Similar a Mongo à la Resque (20)
Último
Último (20)
Mongo à la Resque
- 1. Mongo à la Resque Nicolas Fouché http:// .com http://about.me/nfo
- 2. Say “rescue” or “resquioux” in French
- 6. Redis Remote Dictionary Server “memcached on steroids” Initial release was March 2009
- 7. Redis Redis is an open source, advanced key- value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets. http://redis.io
- 8. Redis In RAM Latency: 10ns VS 1ms = 1,000,000ms ≃ 5000 Gets/s ≃ 5000 Sets/s
- 9. Redis Virtual memory keys in memory values as needed in memory replication
- 10. Redis well written ANSI C and clients in all languages
- 11. Redis keep in mind the complexity of each operation
- 12. Resque parent / child fork process
- 13. Resque handles Unix signals => god / monit / bluepill friendly
- 14. Resque Sinatra web UI, known as resque-web
- 15. Queues Priority: $ QUEUES=high,low rake resque:work
- 16. Queues Consume all queues: $ QUEUES=* rake resque:work
- 17. Enqueue jobs Resque.enqueue(Archive, @repo.id, branch)
- 18. Workers class Archive @queue = :file_serve def self.perform(repo_id, branch = nil) repo = Repository.find(repo_id) repo.create_archive(branch || "master") end end
- 19. Persistence Jobs are stored as JSON { "class": "Archive", "args": [ 44, "masterbrew" ] }
- 20. Failure backends redis (for resque-web) hoptoad and others (like exceptional)
- 21. Don’t like Ruby ?
- 23. resque-mongo
- 24. Translating queries atomic operations $set - set a particular value $unset - set a particular value $inc - increment a particular value by a certain amount $push - append a value to an array $pushAll - append several values to an array $pull - remove a value(s) from an existing array $pullAll - remove several value(s) from an existing array $bit - bitwis
- 25. Translating queries Push a job redis.sadd(:queues, queue) redis.rpush("queue:#{queue}", encode(item)) mongo << {:queue => queue, :item => item, :date => Time.now} mongo_queues.update( {:queue => queue }, {'$inc' => {:count => 1}} )
- 26. Translating queries Pop a job redis.lpop("queue:#{queue}") doc = mongo.find_and_modify( :query => {:queue => queue}, :sort => [[:date, 1]], :remove => true) mongo_queues.update( {:queue => queue }, {'$inc' => {:count => -1}})
- 27. Translating queries queue size redis.llen("queue:#{queue}").to_i mongo_queues.find_one(:queue => queue)['count']
- 28. Translating queries Peek a job list_range("queue:#{queue}", start, count) start, count = [start, count].map { |n| Integer(n) } res = mongo.find(:queue => queue).sort([:date, 1]).skip(start).limit(count).to_a res.collect! { |doc| doc['item'] } count == 1 ? res.first : res
- 29. Translating queries Remove queue redis.srem(:queues, queue) redis.del("queue:#{queue}") mongo.remove({:queue => queue}) mongo_queues.remove({:queue => queue})
- 30. Mongo to the rescue ? Sometimes you just know MongoDB better than Redis Most of query features of Redis can be translated in Mongo queries But for Redis addicts, MongoDB has some secrets
- 31. Used to MongoDB ?
- 32. Used to MongoDB ?[lol@cats ~]# bin/mongo MongoDB shell version: 1.6.4 connecting to: test > show dbs admin local monque > use monque switched to db monque > db.getCollectionNames(); [ "delayed_queue", "failures", "job_groups", "monque", "queues", "schedules", "schedules_changed", "stats", "system.indexes", "workers" ] > db.workers.count(); 40 > db.workers.findOne(); { "_id" : ObjectId("4c863c6c89cb535954000001"), "started" : "Tue Sep 07 2010 13:21:48 GMT+0000 (UTC)", "worker" : "myserver:22868:facebook*" }
- 33. Used to MongoDB ? require 'rubygems' require 'mongo' include Mongo db = Connection.new.db('sample-db') coll = db.collection('test') coll.remove 3.times do |i| coll.insert({'a' => i+1}) end puts "There are #{coll.count()} records. Here they are:" coll.find().each { |doc| puts doc.inspect }
- 34. Really used to MongoDB ?
- 35. New features Go take a look in this JSON !
- 37. What’s happening right now ? Use the mongo console !
- 38. Listen on queues by prefix Queues prefix: $ QUEUES=email*,analytics* rake resque:work
- 40. resque-mongo-groups by Catalin Bordianu, aka @omikronn
- 41. resque-mongo-groups Resque.enqueue( EatBreakfast, :with => ['eggs', 'bacon'], :group_id => 'meals:' + Time.now.utc.strftime('%Y%m%d') )
- 43. resque-mongo-groups def self.perform # ... self.class.atomic_op_on_complete = { '$inc' => { 'eggs' => 1, 'bacon' => 1 } } # ... end
- 44. resque-mongo-groups Resque.group_stats("meals:20110316") { '_id' => 'meals:20110316', 'total' => 4, 'delayed' => 2, 'completed' => 3, 'failed' => 1, 'exceptions' => ["#<TooMuchEggs: beware !>"] 'custom' => {:eggs => 12, :bacon => 1, :potatoes => 3} }
- 47. resque-mongo-scheduler breakfast: cron: "0 8 * * *" class: Eat args: "bacon" description: Breakfast
- 48. gem "resque-mongo-scheduler", :require => "resque-scheduler" resque-mongo-scheduler
- 49. resque-mongo ⚡ resque You cannot use resque and resque-mongo in the same project
- 50. resque-mongo ♥ resque migration to resque-mongo => free
- 51. resque-mongo ♥ resque Still have to migrate from to
- 52. resque-mongo ♥ resque easy to merge the last changes from resque to resque-mongo easy to port existing plugins
- 53. Get the gem ! gem install nfo-resque-mongo The maintainer still did not answer to my ownership requests :(
- 54. and use it with Bundler gem "nfo-resque-mongo", :require => "resque"
- 56. Help needed ! Add exclusive features to resque-web Migrate existing plugins Create new plugins Promote the gem Or simply use the gem, you ‘ll surely send pull requests
- 57. That’s all folks Credits for Flickr images: acezebragirl4j, kiwitime, fulbert05, josh_exell, benheine, tudacee