http://joind.in/2495 PHPBenelux conference January 2011

1. CouchDB
relax

2. CouchDB
relax
Sander van de Graaf
@svdgraaf
Focus -> practical usage examples

3. http://joind.in/talk/view/2495
second talk ever, please provide feedback

5. CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
• Other stuff

6. CouchDB
relax

8. NOSQL

9. IT’S A MOVEMENT
Movement, definitions vary

10. 1998
Back in the day...

13. Lame movie 1

14. Another one

15. And then some more...

16. XML was introduced

17. Some game was published

18. MC Donald’s Happy Meal

19. Carlo Strozzi
Released NOSQL open source DB

20. NOSQL == Not Only SQL

21. “[The NoSQL movement] departs from the relational
model altogether, it should therefore have been called
more appropriately ‘NoREL’, or something to that
effect.”
- Carlo Strozzi

22. CouchDB
relax

27. Ubuntu One, contacts sync

28. NUTSHELL

29. SPEED
Speed Not diskpace (see cleanup)

30. APPEND ONLY
Append only storage, happy cup of coffee!

31. NO REPAIR NEEDED

32. COMPACTING

33. HTTP SERVER
caching, loadbalancing, without extra
costs :D

34. CAP

35. CouchDB
CAP

36. CouchDB
EVENTUALLY CONSISTENT
CouchDB focus is on Availability + Reliability, and will be
consistent after replication.

37. FULL REST API

38. REST
• GET • SELECT
• PUT • UPDATE
• POST • INSERT
• DELETE • DELETE
• COPY • ...

40. JSON
{
total_rows: 2,
offset: 0,
rows: [
{
id: '_design/foobar',
key: '_design/foobar',
value: { rev: '5-982b2fc36835715b2aae54609b5d5f1e' }
},
{
id: 'f0e1fd96eb6e094f74dda8d949000a6a',
key: 'f0e1fd96eb6e094f74dda8d949000a6a',
value: { rev: '1-86bca407fce8234a63c90ff549b56b10' }
},
]
}
Javascript == awesome! :D

41. REPLICATION
Key feature, relaxed about replication
issues, and version conflicts

42. Welcome to Futon, I prefer a UI

43. http-console rocks the socks out of telnet

44. Berkeley

45. CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
• Other stuff

46. PHP USAGE

47. PHP LIBRARIES
• PHPillow (LGPL)
• PHP Object Freezer (BSD)
• PHP On Couch (GPL 2 / 3)
• PHP CouchDB Extension (PHP license)
• SAG for CouchDB (apache)
• Doctrine 2 CouchDB ODM
All are quite nice, doctrine has some rough edges, I use PHP On Couch with custom patch for
Zend autoloader easyness

48. <?PHP
// setup connection for couchdb
$client = new Couchdb_Client('http://ponies.couchone.com:5984','rainbows');
// fetch a document
$doc = $client->getDoc('awesome_pony');
// updating document
$doc->newproperty = array("type","awesome");
try
{
$client->storeDoc($doc);
}
catch (Exception $e)
{
echo "Document storage failed : " . $e->getMessage();
}
PHP On Couch with small ZF autoloader fix

49. CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
• Other stuff

50. REPLICATION

51. DEFINITION
“Replication is the process of sharing information so
as to ensure consistency between redundant
resources, such as software or hardware components,
to improve reliability, fault-tolerance, or accessibility.”
Source: wikipedia

54. CouchDB
relax

55. CouchDB
relax
CouchDB
relax

56. CouchDB

57. CouchDB
relax
CouchDB
relax
CouchDB
relax
CouchDB
relax
Mysql can do this

58. CouchDB
relax
CouchDB
relax
Master, Master replication

59. CouchDB
relax
CouchDB
relax
CouchDB
relax

60. US NL
CouchDB
relax
CouchDB
relax
BE
CouchDB
relax
Not only locally

61. P2P WEB

62. “World Domination”

63. CLUSTERING
“The fun stuff ”

64. Couchdb doesn’t support partitioning (sharding) itself, couchdb -> http based -> lots of
possibilities

65. loadbalancer
...n
CouchDB
relax
CouchDB
relax
The basics are all the same: easy => couchdb instances 1..n => loadbalancer

66. CHALLENGES
• Large amounts of data
• Large views (with big/long map/reduce queries)
• LOTS of traffic
• Location based partitions
• For fun and profit

67. MAP/REDUCE

68. INPUT
IP Bytes
212.122.174.13 18271
212.122.174.13 191726
212.122.174.13 198
74.119.8.111 91272
74.119.8.111 8371
212.122.174.13 43
Map/Reduce example

69. MAPPER => REDUCER
IP Bytes
18271
191726
212.122.174.13
198
43
91272
74.119.8.111
8371

70. AFTER REDUCE
IP Bytes
212.122.174.13 210238
74.119.8.111 99643

71. PARTITION INPUT
Partition IP Bytes
0 212.122.174.13 18271
0 212.122.174.13 191726
0 212.122.174.13 198
1 74.119.8.111 91272
1 74.119.8.111 8371
0 212.122.174.13 43
Map/Reduce example

72. MAPPER => REDUCER
Partition IP Bytes
18271
191726
0 212.122.174.13
198
43
91272
1 74.119.8.111
8371
If data is big enough, you could even need a re-re-re-reducer

73. AFTER REDUCE
IP Bytes
212.122.174.13 210238
74.119.8.111 99643

74. CLUSTERING OPTIONS
• CouchDB Lounge
• Pillow
• BigCouch

75. LOUNGE
• partitioning/clustering
• Nginx module
• meebo.com
• ‘easy’
• http://tilgovi.github.com/couchdb-lounge/

76. LOUNGE
• dumb_proxy => proxy for simple PUT/GET’s
• smart_proxy => proxy for map/reduce over shards
• replicator => updates all copies, redudantly
it can make sure that there are N copies of a document at every moment

77. nginx
dumb_proxy
...n
CouchDB
relax
CouchDB
relax
dumb_proxy == ONLY GET/PUT

78. nginx
smart_proxy
...n
CouchDB
relax
CouchDB
relax
smart_proxy takes care of the map/reduce and re-reducers over multiple nodes

79. Bonus:
other nginx modules work too
mod_cache, mod_expire, etc.

80. PILLOW
• Erlang based
• router/rereducer (map/reduce over multiple
systems)
• In development (but promising!)
• https://github.com/khellan/Pillow

81. BIGCOUCH
• Fork
• 100% api compatible
• Open Source/Commercial
• https://cloudant.com/#!/solutions/bigcouch

82. CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
• Other stuff

83. BACKEND USAGE

84. PROXIED
CouchDB
relax
proxied via middleware, or via mod_proxy or similiar

85. DIRECT
CouchDB
relax
or direct, because http based, content is directly available in javascript

86. NOSQL && SQL HYBRID
• onSave, onCommit hooks available in every major framework
• onSave -> make a JSON representation of your object, and
PUT it to couchdb (#protip: only ‘public’ data)
• sql db is leading, you don’t care about versioning in couchdb
• youcan use your data directly from couchdb within your
frontend javascript

87. MODEL
<?php
class Pony extends Application_models
{
public function toArray()
{
$data = $this->_getData();
unset($data['created_on']);
unset($data['created_by']);
unset($data['access_level']);
unset($data['private_data']);
$data['tags'] = $this->getTags();
$data['categories'] = $this->getCategories();
$data['rainbows'] = 'double';
return $data;
}
}

88. AFTER_SAVE
<?php
class article_module extends admin_module
{
public function after_save()
{
parent::after_save();
$data = $this->toJson();
$res = CouchDB::put($data);
$this->_id = $res->_id;
$this->_rev = $res->_rev;
}
}

89. PROXY
RewriteEngine On
RewriteRule /data/(.*) http://127.0.0.1:5984/db/$1 [P,L]
Proxy the calls (work around sandbox/other domain error), or use jsonp

90. JAVASCRIPT
<script type="text/javascript">
$.getJSON("/db/ponies/_design/ponies/_view/best-ponies?include_docs=true",
function(res){
for(i in res.rows)
{
doc = res.rows[i].doc;
// do stuff
}
}
);
</script>

91. CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
• Other stuff

92. COUCHAPP
CouchDB has it’s own structure for “distributed, scalable web applications” called couchapps

93. “Distributed, scalable, web
applications you say?
omgwtfbbq!?!1!!!11!1!eleven”

94. _attachments
the magic is in _attachments

95. CouchDB
relax
CouchDB
relax
CouchDB
relax
distribution via replication

96. INSTALLATION
Couchapp 0.7.0
installation is easy

97. $ couchapp init
init a project

98. LAYOUT
creates a default folder

99. $ couchapp push http://ponies.couchone.com:5984/rainbows

100. https://github.com/brandon-beacher/couchapp-tmbundle
couchapp push on save -> textmate

101. CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
• Other stuff

102. OTHER STUFF

103. REWRITES

104. _REWRITE

105. $ curl "http://ponies.couchone.com/rainbows/_design/ponies/_view/best-
ponies?descending=true&limit=5&key=”foobar”

106. such urls make us a sad panda

107. {
....
"rewrites": [
{
"from": "/best-5-ponies",
"to": "ponies/_view/best-ponies",
"method": "GET",
"query": {
"descending": true,
"limit": 5,
"key": "foobar"
}
}
]
}

108. $ curl "http://ponies.couchone.com/rainbows/_design/ponies/_view/best-
ponies?descending=true&limit=5&key=”foobar”
rewrite this

109. $ curl "http://ponies.couchone.com/rainbows/_design/ponies/_rewrite/best-5-ponies"
to this

110. [vhosts]
awesomeponies.com = /rainbows/_design/ponies/_rewrite

111. $ curl "http://ponies.couchone.com/rainbows/_design/ponies/_rewrite/best-5-ponies"
rewrite this

112. $ curl "http://awesomeponies.com/best-5-ponies"
to this

114. _CHANGES

116. $ curl -X GET "http://ponies.couchone.com/rainbows/_changes"

117. {"results":[
],
"last_seq":0}

118. curl -X PUT http://ponies.couchone.come/rainbows/foobar -d '{"type":"awesome"}'

119. {"results":[
{"seq":1,"id":"foobar","changes":[{"rev":"1-aaa8e2a031bca334f50b48b6682fb486"}]}
],
"last_seq":1}

120. {"results":[
{"seq":1,"id":"foobar","changes":[{"rev":"1-aaa8e2a031bca334f50b48b6682fb486"}]},
{"seq":2,"id":"foobar2","changes":[{"rev":"1-e18422e6a82d0f2157d74b5dcf457997"}]}
],
"last_seq":2}

121. _CHANGES OPTIONS
• ?since
• Longpolling
• Continuous

122. $ curl -X GET "http://ponies.couchone.com/rainbows/_changes?since=20"

123. curl -X GET "http://ponies.couchone.com/rainbows/_changes?feed=longpoll&since=2"
Longpolling: good for little updates, connections stays open until change, then gets closed
and you need to reconnect, lots of reconnects for lots of updates

124. curl -X GET "http://ponies.couchone.com/rainbows/_changes?feed=continuous&since=2"
Connections stays open, and you get updates on the fly!

126. FILTERS
filters can be used to filter documents from output

127. function(doc, req)
{
if(doc.priority == 'high')
{
return true;
}
return false;
}
we only want high priority documents

128. curl -X GET
"http://ponies.couchone.com/rainbows/_changes?feed=continuous&filter=app/important

129. function(doc, req)
{
if(doc.name == req.query.name) {
return true;
}
return false;
}
you can use req for request based filters

130. curl -X GET
"http://ponies.couchone.com/rainbows/_changes?feed=continuous&filter=app/name&name=foobar

131. SHOWS

132. function(doc, req) {
return {
body: "Hello World"
}
}

133. curl -X
"http://ponies.couchone.com/rainbows/_design/foobar/_show/showfunction/docid"

134. function(doc) {
return {
"code": 302,
"body": "See other",
"headers": {
"Location": doc.target
}
};
}
You can also define http headers, we used this for translating public id’s into private storage
id’s. In this way, couchdb took care of all the headers and http stuff, and we could use a
regular nginx proxy module

135. LUCENE

136. [external]
fti=/path/to/python /path/to/couchdb-lucene/tools/couchdb-external-hook.py
[httpd_db_handlers]
_fti = {couch_httpd_external, handle_external_req, <<"fti">>}

137. function(doc) {
var ret=new Document();
ret.add(doc.message);
ret.add(new Date(doc.datetime));
return ret;
}

138. curl -X GET
"http://ponies.couchone.com/rainbows/_fti/_design/unicorns/by-query?q=foobar"

139. GEOCOUCH
https://github.com/vmx/couchdb

140. See Dericks talk yesterday

141. GEOCOUCH
• Supports bbox
• fork
• outputs via lists, georss possible
• directly useable by google maps
• can read GIS data
• combined with _changes makes interesting usecase
- bbox => all items withing a certain bounding box, polygon is in the works
- currently a fork of couchdb, in the works as external module
- output can be setup seperately
- google maps can use georss
- GIS: Geographic Information System (used worldwide?)

142. SPATIAL INDEX
in spatial/points.js
function(doc)
{
if (doc.geo && doc.geo.latitude != '' && doc.geo.longitude != '')
{
emit(
{
type: "Point",
coordinates: [parseFloat(doc.geo.latitude), parseFloat(doc.geo.longitude)]
},
[doc._id, doc]
);
}
}

143. Worldwide search
http://ponies.couchone.com/rainbows/_design/unicorns/_spatial/points?bbox=0,0,180,90
{"update_seq":3,"rows":[
{
"id":"augsburg",
"bbox":[10.898333,48.371667,10.898333,48.371667],
"value":["augsburg",[10.898333,48.371667]]
}
]}

144. GEORSS && GOOGLE MAPS
if (GBrowserIsCompatible())
{
map = new GMap2(document.getElementById('map'));
var geoXML = new GGeoXml('http://ponies.couchone.com/rainbows/url-to-georss-view');
map.addOverlay(geoXML);
}

145. curl -X GET "http://ponies.couchone.com/rainbows/_design/alarmeringen/_spatial/points?
bbox=51.711369,4.218407,52.136520,4.745740";

146. Q?

147. http://www.couchone.com/get

148. http://joind.in/talk/view/2495
second talk ever, please provide feedback