Mongodb railscamphh

on rails
railscamp hamburg, 2010
jan krutisch <jan@krutisch.de>
http://jan.krutisch.de/
Samstag, 23. Oktober 2010

mongodb wtf? lol!?

document database

document database
NoSQLincluded!

10gen

open source
http://github.com/mongodb/mongo

id title descr pos_lat pos_lng

{
"_id" : ObjectId("4c00245062610475a005afcd"),
"address" : "Bernstorffstr. 174n22767 HamburgnDE",
"description" : null,
"position" : {
"lat" : 53.5600912,
"lng" : 9.9596977
},
"tags" : [
"hausarzt",
"naturheilverfahren",
"akupunktur",
"allgemeinmedizin"
],
"title" : "Dr. med. Lilo Eisenbarth",
"loxicon_id" : 808261
}

BSON

BInary Serialized jsON
http://bsonspec.org/

Wire

Storage

rich queries

conceptually close to SQL

easy to grasp

ﬂexible

language integration

on top: map/reduce

Scaling

Master/Slave replication

Replica Sets (1.6)

Primary
Member Member

Primary
Member Primary

Member
Member Primary

Autosharding (1.6)

Durability

No single server
durability!

fsyncs every 60s
(conﬁgurable)

Use Replication!

Use write propagation
locking

Single Server Durability
planned for 1.8

mongo console

$ mongo

> use test
switched to db test
db.quotes.save({
text: "You can observe a lot just by watching.",
from: "Yogi Berra", created_at: new Date()
});
db.quotes.save({
text: "Silence is one of the hardest arguments to refute.",
from: "Josh Billings", created_at: new Date()
});

Indexing

Same concept as with
SQL databases

You want them

Sort order

Unique

Compound

Geospatial

map/reduce

we can haz it, too

function() {
this.tags.forEach(function(z) {
emit(z, {count: 1});
});
}

function(key, values) {
var total = 0;
values.forEach(function(v) { total += v.count });
return {count: total}
}

(it‘s not fast...)

security

simple user/password
auth

per database

read only is possible

one more thing

GridFS

Binary ﬁelds in BSON
< 4MB

GridFS saves ﬁles in
chunks

I‘m in u‘r rubies,
querying teh MongoDB!

core driver

mongo / bson_ext

ODMs / Libs

mongo_mapper

mongoid

Find examples here:
http://github.com/halfbyte/mongo_ruby_examples

Basic driver usage

init

require 'mongo'
@connection = Mongo::Connection.new
@db = @connection.db("test")

@connection = Mongo::Connection.new(
'localhost',
27017,
:pool_size => 5,
:timeout => 20
)

@connection = Mongo::Connection.from_uri(
"mongodb://localhost:27017/test"
)

insert/upsert

doc = {
:text => "You can observe a lot just by watching.",
:from => "Yogi Berra",
:created_at => Time.now
}
@db['quotes'].insert(doc)

doc = @db['quotes'].find_one(id)
doc[:from] = "Yogi Berra, famous baseball player"
@db['quotes'].save(doc)

atomic updates

@db['quotes'].update(
{"from" => "Yogi Berra"},
{"$inc" => {"reads" => 1 } }
)

$inc
$set
$unset
$push
$pushAll
$addToSet
$pop
$pull
$pullAll
$

getting a whole collection

@db['quotes'].find.each do |row|
puts row.inspect
end

exact query

@db['quotes'].find(:from => "Yogi Berra")

more queries

100.times do |i|
db['numbers'].insert({"i" => i})
end

db['numbers'].find("i" => {"$lt" => 2})

$lt <
$gt >
$lte <=
$gte >=
$ne !=

@db['people'].find(:tags => {"$in" => ['cool']})

obj = {
"_id"=>BSON::ObjectID('4c706af16261040680000369'),
"name"=>"Vernon Kreiger",
"address"=>{
"street"=>"536 Haleigh Locks",
"city"=>"Port Kiannahaven",
"zip"=>"80730-0214",
"country"=>"Fakistan"
},
"tags"=>["cool", "weird"]
}

$in IN (2,3,4)
$nin NOT IN
$all [2,3] ~ [1,2,3]

$mod yah, RLY
$size okay
$exists NOT NULL
$type huh?

@db['people'].find("address.city" => /haven/)

Sorting

@db['people'].find().sort("address.street")

Pagination

@db['numbers'].find.sort("i").limit(10)

@db['numbers'].find.sort("i").limit(10).skip(50)

Counting

@db['numbers'].find.count

Distinct

@db['people'].distinct('tags').inspect

Group

Poor mans map/reduce

@db['people'].group(
['created_at'],
{},
{:tags => {}},
reduce,
finalize
)

function(doc, prev) {
for(i in doc.tags) {
if (doc.tags[i] in prev.tags) {
prev.tags[doc.tags[i]]++
} else {
prev.tags[doc.tags[i]] =1
}
}
}

{"created_at"=>2010-09-19 22:00:00 UTC, "tags"=>{"foo"=>11.0, "dumb"=>12.0, "stupid"=>7.0, "bar"=>7.0, "cool"=>14.0, "weird"=>17.0}}
{"created_at"=>2010-09-20 22:00:00 UTC, "tags"=>{"dumb"=>11.0, "stupid"=>5.0, "foo"=>10.0, "cool"=>8.0, "weird"=>9.0, "bar"=>15.0}}
{"created_at"=>2010-09-22 22:00:00 UTC, "tags"=>{"weird"=>15.0, "bar"=>9.0, "stupid"=>17.0, "cool"=>11.0, "dumb"=>10.0, "foo"=>12.0}}
{"created_at"=>2010-09-15 22:00:00 UTC, "tags"=>{"foo"=>11.0, "weird"=>7.0, "stupid"=>10.0, "cool"=>11.0, "bar"=>5.0, "dumb"=>8.0}}
{"created_at"=>2010-09-25 22:00:00 UTC, "tags"=>{"dumb"=>11.0, "weird"=>14.0, "cool"=>8.0, "foo"=>21.0, "bar"=>11.0, "stupid"=>13.0}}
{"created_at"=>2010-09-28 22:00:00 UTC, "tags"=>{"cool"=>11.0, "dumb"=>16.0, "stupid"=>11.0, "weird"=>15.0, "foo"=>9.0, "bar"=>16.0}}
{"created_at"=>2010-09-10 22:00:00 UTC, "tags"=>{"dumb"=>15.0, "weird"=>9.0, "bar"=>8.0, "cool"=>14.0, "stupid"=>11.0, "foo"=>11.0}}
{"created_at"=>2010-09-03 22:00:00 UTC, "tags"=>{"weird"=>14.0, "dumb"=>15.0, "stupid"=>11.0, "foo"=>16.0, "bar"=>20.0, "cool"=>10.0}}
{"created_at"=>2010-09-21 22:00:00 UTC, "tags"=>{"weird"=>15.0, "cool"=>14.0, "foo"=>13.0, "stupid"=>6.0, "bar"=>11.0, "dumb"=>9.0}}
{"created_at"=>2010-09-23 22:00:00 UTC, "tags"=>{"weird"=>15.0, "stupid"=>15.0, "dumb"=>15.0, "foo"=>16.0, "cool"=>10.0, "bar"=>11.0}}
{"created_at"=>2010-09-29 22:00:00 UTC, "tags"=>{"bar"=>9.0, "cool"=>14.0, "weird"=>16.0, "foo"=>8.0, "dumb"=>9.0, "stupid"=>12.0}}
{"created_at"=>2010-09-27 22:00:00 UTC, "tags"=>{"cool"=>13.0, "dumb"=>10.0, "stupid"=>12.0, "bar"=>8.0, "foo"=>16.0, "weird"=>13.0}}
{"created_at"=>2010-09-04 22:00:00 UTC, "tags"=>{"cool"=>11.0, "bar"=>9.0, "stupid"=>6.0, "weird"=>11.0, "dumb"=>8.0, "foo"=>11.0}}
{"created_at"=>2010-09-08 22:00:00 UTC, "tags"=>{"stupid"=>12.0, "dumb"=>11.0, "cool"=>15.0, "foo"=>11.0, "bar"=>9.0, "weird"=>8.0}}
{"created_at"=>2010-10-02 22:00:00 UTC, "tags"=>{"bar"=>8.0, "dumb"=>8.0, "cool"=>10.0, "foo"=>10.0, "stupid"=>8.0, "weird"=>6.0}}
{"created_at"=>2010-09-24 22:00:00 UTC, "tags"=>{"foo"=>13.0, "bar"=>12.0, "stupid"=>15.0, "weird"=>17.0, "dumb"=>7.0, "cool"=>10.0}}
{"created_at"=>2010-09-30 22:00:00 UTC, "tags"=>{"bar"=>10.0, "cool"=>6.0, "stupid"=>14.0, "weird"=>9.0, "dumb"=>12.0, "foo"=>19.0}}
{"created_at"=>2010-09-05 22:00:00 UTC, "tags"=>{"dumb"=>12.0, "foo"=>19.0, "weird"=>8.0, "stupid"=>8.0, "bar"=>7.0, "cool"=>10.0}}
{"created_at"=>2010-09-17 22:00:00 UTC, "tags"=>{"weird"=>13.0, "bar"=>14.0, "dumb"=>12.0, "foo"=>12.0, "stupid"=>10.0, "cool"=>9.0}}
{"created_at"=>2010-09-18 22:00:00 UTC, "tags"=>{"dumb"=>8.0, "cool"=>11.0, "foo"=>6.0, "bar"=>12.0, "weird"=>7.0, "stupid"=>6.0}}
{"created_at"=>2010-09-09 22:00:00 UTC, "tags"=>{"weird"=>11.0, "dumb"=>9.0, "foo"=>6.0, "bar"=>11.0, "cool"=>11.0, "stupid"=>6.0}}
{"created_at"=>2010-09-13 22:00:00 UTC, "tags"=>{"dumb"=>19.0, "stupid"=>9.0, "weird"=>12.0, "cool"=>11.0, "bar"=>10.0, "foo"=>15.0}}
{"created_at"=>2010-09-16 22:00:00 UTC, "tags"=>{"bar"=>6.0, "weird"=>8.0, "dumb"=>9.0, "cool"=>11.0, "stupid"=>17.0, "foo"=>15.0}}
{"created_at"=>2010-09-11 22:00:00 UTC, "tags"=>{"foo"=>10.0, "weird"=>9.0, "bar"=>8.0, "cool"=>4.0, "dumb"=>8.0, "stupid"=>9.0}}
{"created_at"=>2010-09-26 22:00:00 UTC, "tags"=>{"dumb"=>15.0, "weird"=>6.0, "stupid"=>15.0, "bar"=>10.0, "foo"=>13.0, "cool"=>15.0}}
{"created_at"=>2010-10-01 22:00:00 UTC, "tags"=>{"cool"=>7.0, "weird"=>11.0, "stupid"=>11.0, "bar"=>14.0, "foo"=>12.0, "dumb"=>11.0}}
{"created_at"=>2010-09-12 22:00:00 UTC, "tags"=>{"bar"=>7.0, "weird"=>12.0, "stupid"=>11.0, "cool"=>10.0, "foo"=>11.0, "dumb"=>9.0}}
{"created_at"=>2010-09-14 22:00:00 UTC, "tags"=>{"dumb"=>8.0, "foo"=>15.0, "cool"=>15.0, "stupid"=>15.0, "bar"=>7.0, "weird"=>14.0}}
{"created_at"=>2010-09-07 22:00:00 UTC, "tags"=>{"dumb"=>10.0, "cool"=>7.0, "foo"=>14.0, "weird"=>15.0, "bar"=>11.0, "stupid"=>7.0}}
{"created_at"=>2010-09-06 22:00:00 UTC, "tags"=>{"dumb"=>7.0, "bar"=>11.0, "cool"=>16.0, "weird"=>14.0, "foo"=>12.0, "stupid"=>6.0}}

{"created_at"=>2010-09-19 22:00:00 UTC, "tags"=>{"foo"=>11.0, "dumb"=>12.0, "stupid"=>7.0, "bar"=>7.0, "cool"=>14.0, "weird"=>17.0}}
{"created_at"=>2010-09-20 22:00:00 UTC, "tags"=>{"dumb"=>11.0, "stupid"=>5.0, "foo"=>10.0, "cool"=>8.0, "weird"=>9.0, "bar"=>15.0}}
{"created_at"=>2010-09-22 22:00:00 UTC, "tags"=>{"weird"=>15.0, "bar"=>9.0, "stupid"=>17.0, "cool"=>11.0, "dumb"=>10.0, "foo"=>12.0}}
{"created_at"=>2010-09-15 22:00:00 UTC, "tags"=>{"foo"=>11.0, "weird"=>7.0, "stupid"=>10.0, "cool"=>11.0, "bar"=>5.0, "dumb"=>8.0}}
{"created_at"=>2010-09-25 22:00:00 UTC, "tags"=>{"dumb"=>11.0, "weird"=>14.0, "cool"=>8.0, "foo"=>21.0, "bar"=>11.0, "stupid"=>13.0}}
{"created_at"=>2010-09-28 22:00:00 UTC, "tags"=>{"cool"=>11.0, "dumb"=>16.0, "stupid"=>11.0, "weird"=>15.0, "foo"=>9.0, "bar"=>16.0}}
{"created_at"=>2010-09-10 22:00:00 UTC, "tags"=>{"dumb"=>15.0, "weird"=>9.0, "bar"=>8.0, "cool"=>14.0, "stupid"=>11.0, "foo"=>11.0}}
{"created_at"=>2010-09-03 22:00:00 UTC, "tags"=>{"weird"=>14.0, "dumb"=>15.0, "stupid"=>11.0, "foo"=>16.0, "bar"=>20.0, "cool"=>10.0}}
{"created_at"=>2010-09-21 22:00:00 UTC, "tags"=>{"weird"=>15.0, "cool"=>14.0, "foo"=>13.0, "stupid"=>6.0, "bar"=>11.0, "dumb"=>9.0}}
{"created_at"=>2010-09-23 22:00:00 UTC, "tags"=>{"weird"=>15.0, "stupid"=>15.0, "dumb"=>15.0, "foo"=>16.0, "cool"=>10.0, "bar"=>11.0}}
{"created_at"=>2010-09-29 22:00:00 UTC, "tags"=>{"bar"=>9.0, "cool"=>14.0, "weird"=>16.0, "foo"=>8.0, "dumb"=>9.0, "stupid"=>12.0}}
{"created_at"=>2010-09-27 22:00:00 UTC, "tags"=>{"cool"=>13.0, "dumb"=>10.0, "stupid"=>12.0, "bar"=>8.0, "foo"=>16.0, "weird"=>13.0}}
{"created_at"=>2010-09-04 22:00:00 UTC, "tags"=>{"cool"=>11.0, "bar"=>9.0, "stupid"=>6.0, "weird"=>11.0, "dumb"=>8.0, "foo"=>11.0}}
{"created_at"=>2010-09-08 22:00:00 UTC, "tags"=>{"stupid"=>12.0, "dumb"=>11.0, "cool"=>15.0, "foo"=>11.0, "bar"=>9.0, "weird"=>8.0}}
{"created_at"=>2010-10-02 22:00:00 UTC, "tags"=>{"bar"=>8.0, "dumb"=>8.0, "cool"=>10.0, "foo"=>10.0, "stupid"=>8.0, "weird"=>6.0}}
{"created_at"=>2010-09-24 22:00:00 UTC, "tags"=>{"foo"=>13.0, "bar"=>12.0, "stupid"=>15.0, "weird"=>17.0, "dumb"=>7.0, "cool"=>10.0}}
{"created_at"=>2010-09-30 22:00:00 UTC, "tags"=>{"bar"=>10.0, "cool"=>6.0, "stupid"=>14.0, "weird"=>9.0, "dumb"=>12.0, "foo"=>19.0}}
{"created_at"=>2010-09-05 22:00:00 UTC, "tags"=>{"dumb"=>12.0, "foo"=>19.0, "weird"=>8.0, "stupid"=>8.0, "bar"=>7.0, "cool"=>10.0}}
{"created_at"=>2010-09-17 22:00:00 UTC, "tags"=>{"weird"=>13.0, "bar"=>14.0, "dumb"=>12.0, "foo"=>12.0, "stupid"=>10.0, "cool"=>9.0}}
{"created_at"=>2010-09-18 22:00:00 UTC, "tags"=>{"dumb"=>8.0, "cool"=>11.0, "foo"=>6.0, "bar"=>12.0, "weird"=>7.0, "stupid"=>6.0}}
{"created_at"=>2010-09-09 22:00:00 UTC, "tags"=>{"weird"=>11.0, "dumb"=>9.0, "foo"=>6.0, "bar"=>11.0, "cool"=>11.0, "stupid"=>6.0}}
{"created_at"=>2010-09-13 22:00:00 UTC, "tags"=>{"dumb"=>19.0, "stupid"=>9.0, "weird"=>12.0, "cool"=>11.0, "bar"=>10.0, "foo"=>15.0}}
{"created_at"=>2010-09-16 22:00:00 UTC, "tags"=>{"bar"=>6.0, "weird"=>8.0, "dumb"=>9.0, "cool"=>11.0, "stupid"=>17.0, "foo"=>15.0}}
{"created_at"=>2010-09-11 22:00:00 UTC, "tags"=>{"foo"=>10.0, "weird"=>9.0, "bar"=>8.0, "cool"=>4.0, "dumb"=>8.0, "stupid"=>9.0}}
{"created_at"=>2010-09-26 22:00:00 UTC, "tags"=>{"dumb"=>15.0, "weird"=>6.0, "stupid"=>15.0, "bar"=>10.0, "foo"=>13.0, "cool"=>15.0}}
{"created_at"=>2010-10-01 22:00:00 UTC, "tags"=>{"cool"=>7.0, "weird"=>11.0, "stupid"=>11.0, "bar"=>14.0, "foo"=>12.0, "dumb"=>11.0}}
{"created_at"=>2010-09-12 22:00:00 UTC, "tags"=>{"bar"=>7.0, "weird"=>12.0, "stupid"=>11.0, "cool"=>10.0, "foo"=>11.0, "dumb"=>9.0}}
{"created_at"=>2010-09-14 22:00:00 UTC, "tags"=>{"dumb"=>8.0, "foo"=>15.0, "cool"=>15.0, "stupid"=>15.0, "bar"=>7.0, "weird"=>14.0}}
{"created_at"=>2010-09-07 22:00:00 UTC, "tags"=>{"dumb"=>10.0, "cool"=>7.0, "foo"=>14.0, "weird"=>15.0, "bar"=>11.0, "stupid"=>7.0}}
{"created_at"=>2010-09-06 22:00:00 UTC, "tags"=>{"dumb"=>7.0, "bar"=>11.0, "cool"=>16.0, "weird"=>14.0, "foo"=>12.0, "stupid"=>6.0}}
"tags" => {
"foo" => 11.0,
"dumb" => 12.0,
"stupid" => 7.0,
"bar" => 7.0,
"cool" => 14.0,
"weird" => 17.0
}

function(prev) {
var mostPopular = 0;
for(i in prev.tags) {
if(prev.tags[i] > mostPopular) {
prev.tag = i;
prev.count = prev.tags[i];
mostPopular = prev.tags[i];
}
}
delete prev.tags
}

{"created_at"=>2010-09-27 22:00:00 UTC, "tag"=>"stupid", "count"=>18.0}
{"created_at"=>2010-09-12 22:00:00 UTC, "tag"=>"cool", "count"=>11.0}
{"created_at"=>2010-09-03 22:00:00 UTC, "tag"=>"foo", "count"=>15.0}
{"created_at"=>2010-09-24 22:00:00 UTC, "tag"=>"cool", "count"=>11.0}

Map / Reduce

map = <<-END
function() {
this.tags.forEach(function(z) {
emit(z, {count: 1});
});
}
END
reduce = <<-END
function(key, values) {
var total = 0;
values.forEach(function(v) { total += v.count });
return {count: total}
}
END
collection = @db['people'].map_reduce(
map, reduce
)

Indexes

db['people'].create_index("tags")
@db['people'].create_index(
[["tags", Mongo::ASCENDING]]
)
db['people'].drop_index("tags_1")
db['people'].drop_indexes
db['people'].index_information

Geospatial stuff

@db['people'].create_index(
[["latlng", Mongo::GEO2D]]
)

@db['people'].find(
"latlng" => {"$near" => [53.593978, 10.107380]}
)

GridFS usage

grid = Mongo::Grid.new(@db)
id = grid.put("You can put Strings in here",
:filename => 'test.txt')
file = grid.get(id)
file.filename
file.read
grid.delete(id)
grid.put(
File.open("/Users/jankrutisch/Dropbox/Photos/IMGP8989.jpg")
)

fs = Mongo::GridFileSystem.new(db)
fs.open("test.txt", "w") do |f|
f.write "You can put stuff in here"
end
fs.open("test.txt", "r") do |f|
puts f.read
end
fs.delete("test.txt")

Capped collections

@db.create_collection('capped_numbers',
:capped => true,
:max => 50
)
@db.create_collection('capped_numbers',
:capped => true,
:size => 1024 * 64
)

explain

@db['people'].find(
"address.city" => /haven/
).explain

{
"cursor"=>"BasicCursor",
"nscanned"=>1000,
"nscannedObjects"=>1000,
"n"=>39, "millis"=>2,
"indexBounds"=>{},
"allPlans"=>[
{"cursor"=>"BasicCursor", "indexBounds"=>{}}
]
}

{
"cursor"=>"BtreeCursor address.city_1 multi",
"nscanned"=>1000,
"nscannedObjects"=>39,
"n"=>39, "millis"=>1,
"indexBounds"=>{
"address.city"=>[["", {}], [/haven/, /haven/]]
},
"allPlans"=>[
{
"cursor"=>"BtreeCursor address.city_1 multi",
"indexBounds"=>{
"address.city"=>[["", {}], [/haven/, /haven/]]
}
}
]
}

ODMs

John Nunemaker
@jnunemaker

is in production

documentation?

how to

rails initializer

# config/initializers/mongo_mapper.rb
File.open(File.join(Rails.root, 'config/mongodb.yml'), 'r') do |f|
@settings = YAML.load(f)[Rails.env]
end
MongoMapper.connection = Mongo::Connection.from_uri(@settings["connection"]) if @settings["connection"]
MongoMapper.database = @settings["database"]

a simple example

MongoMapper.connection = @connection
MongoMapper.database = "test"
class Quote
include MongoMapper::Document
key :from
key :text
key :views, Integer
timestamps!
end

ﬁnders

Quote.where(:from => 'Yogi Berra').all
Quote.where(:from => 'Yogi Berra').limit(5).sort(:from.desc).all

embedded docs

class Person
key :name
one :address
key :tags, Array
end
class Address
key :street
key :city
key :country
key :zip
end

person = Person.first
address = Person.first.address

scopes

class Person
scope :tagged, lambda { |tag| where(:tags.in => [tag]) }
end
puts Person.tagged('cool').first.inspect

new website coming soon

Durran Jordan
(of Hashrocket)

Two major versions

1.x (1.9.x) targeting
Rails 2.3.x

2.x (2.0beta) targeting
Rails 3.0

Good documentation

File.open(File.join(RAILS_ROOT, 'config/mongodb.yml'), 'r') do |f|
@settings = YAML.load(f)[RAILS_ENV]
end
Mongoid::Config.instance.from_hash(@settings)

class Quote
include Mongoid::Document
include Mongoid::Timestamps
field :from
field :text
field :views, :type => Integer
end

Quote.where(:from => 'Yogi Berra').all
Quote.where(:from => 'Yogi Berra').limit(5).order_by(:from.desc).all

class Person
field :name
embeds_one :address
field :tags, :type => Array
end
class Address
field :street
field :city
field :country
field :zip
end

More features

mongoid tries to be
clever

(using the „dirty“ ﬂags)

(it‘s probably better to
bypass the ODM
sometimes)

external libraries for
both

mongo_mapper > grip

mongoid > mongoid_grid

Other noteworthy
libraries

Other not so noteworthy
libraries

I ♥

thanks for listening.
‣ jan@krutisch.de
‣ http://jan.krutisch.de/
‣ http://github.com/halfbyte/
‣ http://twitter.com/halfbyte
‣ http://www.mindmatters.de/
‣ http://www.mongodb.org/
‣ http://www.mongoid.org/
‣ http://github.com/jnunemaker/mongo_mapper
‣ http://github.com/halfbyte/mongo_ruby_examples

Mongodb railscamphh

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Mongodb railscamphh

Similar a Mongodb railscamphh (16)

Más de jan_mindmatters

Más de jan_mindmatters (12)

Último

Último (20)

Mongodb railscamphh