A talk I did at phpday.it in Verona, Italy. Using some more advanced CouchDB functionality, like filters, changes feed, shows, geolocation (GeoCouch), and other interesting stuff. It also covers some real use cases.

1. CouchDB
relax

2. CouchDB
relax
Sander van de Graaf
@svdgraaf

3. http://joind.in/talk/view/3009

4. RELAX

5. CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
• Other stuff (GeoCouch!)

6. CouchDB
relax

8. NOSQL

9. IT’S A MOVEMENT

10. 1998

19. Carlo Strozzi

20. NOSQL == Not Only SQL

21. “[The NoSQL movement] departs from the relational
model altogether, it should therefore have been called
more appropriately ‘NoREL’, or something to that
effect.”
- Carlo Strozzi

22. CouchDB
relax

31. NUTSHELL

32. SPEED

33. APPEND ONLY

34. NO REPAIR NEEDED

35. COMPACTING

36. HTTP SERVER

37. CAP

38. CouchDB
CAP

39. CouchDB
EVENTUALLY CONSISTENT

40. FULL REST API

41. HTTP METHODS
• GET • SELECT
• PUT • UPDATE
• POST • INSERT
• DELETE • DELETE
• COPY • ...

43. {
"total_rows": 2,
"offset": 0,
"rows": [
{
"id": "abc089b91fadfcf67a438e73f426f71d",
"key": [
"2011-05-07 13:52:15",
"abc089b91fadfcf67a438e73f426f71d"
],
"value": {
"_id": "abc089b91fadfcf67a438e73f426f71d",
"_rev": "2-a38e08996586cd774b0b5c3006659de0",
"message": "A2 HUISARTSENPOST BARNEVELD NIJKERKERWEG 119 3771LA 119"
}
},
{
"id": "abc089b91fadfcf67a438e73f4286fd5",
"key": [
"2011-05-07 14:00:16",
"abc089b91fadfcf67a438e73f4286fd5"
],
"value": {
"_id": "abc089b91fadfcf67a438e73f4286fd5",
"_rev": "2-691929b5f3d6cde58f27d62681c7b599",
"message": "A1 LEEUWERIKSWEIDE 183 WAGENINGEN 6708LH 183"
}
}
]
}

44. REPLICATION

47. CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
• Other stuff

48. RELAX

49. PHP USAGE

50. PHP LIBRARIES
• PHPillow (LGPL)
• PHP Object Freezer (BSD)
• PHP On Couch (GPL 2 / 3)
• PHP CouchDB Extension (PHP license)
• SAG for CouchDB (apache)
• Doctrine 2 CouchDB ODM

51. PHP ON COUCH
<?PHP
// setup connection for couchdb
$client = new Couchdb_Client('http://ponies.couchone.com:5984','rainbows');
// fetch a document
$doc = $client->getDoc('awesome_pony');
// updating document
$doc->type = "awesome";
try {
$client->storeDoc($doc);
}
catch (Exception $e) {
echo "Document storage failed : " . $e->getMessage();
}

52. CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
• Other stuff

53. REPLICATION

54. DEFINITION
“Replication is the process of sharing information so
as to ensure consistency between redundant
resources, such as software or hardware components,
to improve reliability, fault-tolerance, or accessibility.”
Source: wikipedia

56. CouchDB
relax

57. CouchDB
relax
CouchDB
relax

58. CouchDB
relax
CouchDB
relax
CouchDB
relax
CouchDB
relax

59. CouchDB
relax
CouchDB
relax

60. CouchDB
relax
CouchDB
relax
CouchDB
relax

61. US NL
CouchDB
relax
CouchDB
relax
BE
CouchDB
relax

62. USE CASE
IDG Publishers

66. IDG US IDG AUSTRALIA
CouchDB
relax
CouchDB
relax
CouchDB
relax
CouchDB
relax
CouchDB
relax
CouchDB
relax
CouchDB
relax
IDG NL IDG DE

67. IDG DE IDG Italy IDG Poland IDG Hungary
IDG NL IDG US IDG Romania IDG Sweden
IDG Brazil IDG UK IDG Spania IDG Portugal
IDG Ghana IDG Japan IDG Thailand IDG Vietnam

68. IMPLICATION
Every single bit of IDG content published
worldwide, is available anywhere, anytime in msec.

69. P2P WEB

70. “World Domination”

71. HOW DO I DO THAT?

72. CLUSTERING
“The fun stuff ”

74. loadbalancer
...n
CouchDB
relax
CouchDB
relax

75. CHALLENGES
• Large amounts of data
• Large views (with big/long map/reduce queries)
• LOTS of traffic
• Location based partitions
• For fun and profit

76. MAP/REDUCE

77. INPUT
IP Bytes
212.122.174.13 18271
212.122.174.13 191726
212.122.174.13 198
74.119.8.111 91272
74.119.8.111 8371
212.122.174.13 43

78. MAPPER => REDUCER
IP Bytes
18271
191726
212.122.174.13
198
43
91272
74.119.8.111
8371

79. AFTER REDUCE
IP Bytes
212.122.174.13 210238
74.119.8.111 99643

80. PARTITION INPUT
Partition IP Bytes
0 212.122.174.13 18271
0 212.122.174.13 191726
0 212.122.174.13 198
1 74.119.8.111 91272
1 74.119.8.111 8371
0 212.122.174.13 43

81. MAPPER => REDUCER
Partition IP Bytes
18271
191726
0 212.122.174.13
198
43
91272
1 74.119.8.111
8371

82. AFTER REDUCE
IP Bytes
212.122.174.13 210238
74.119.8.111 99643

83. CLUSTERING OPTIONS
• CouchDB Lounge
• Pillow
• BigCouch

84. LOUNGE
• partitioning/clustering
• Nginx module
• meebo.com
• ‘easy’
• http://tilgovi.github.com/couchdb-lounge/

85. LOUNGE
• dumb_proxy => proxy for simple PUT/GET’s
• smart_proxy => proxy for map/reduce over shards
• replicator => updates all copies, redundantly

86. nginx
dumb_proxy
...n
CouchDB
relax
CouchDB
relax

87. nginx
smart_proxy
...n
CouchDB
relax
CouchDB
relax

88. Bonus:
other nginx modules work too

89. PILLOW
• Erlang based
• router/rereducer (map/reduce over multiple
systems)
• In development (but promising!)
• https://github.com/khellan/Pillow

90. BIGCOUCH
• Fork
• 100% api compatible
• Open Source/Commercial
• https://cloudant.com/#!/solutions/bigcouch

91. CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
• Other stuff

92. BACKEND USAGE

93. PROXIED
CouchDB
relax

94. DIRECT
CouchDB
relax

95. NOSQL && SQL HYBRID
• onSave, onCommit hooks available in every major framework
• onSave -> make a JSON representation of your object, and
PUT it to couchdb (#protip: only ‘public’ data)
• sql db is leading, you don’t care about versioning in couchdb
• youcan use your data directly from couchdb within your
frontend javascript

96. MODEL
<?php
class Pony extends Application_models
{
public function toArray()
{
$data = $this->_getData();
// we don't want any private data pushed
unset($data['created_on']);
unset($data['created_by']);
unset($data['access_level']);
unset($data['private_data']);
$data['tags'] = $this->getTags();
$data['categories'] = $this->getCategories();
$data['rainbows'] = 'double';
return $data;
}
}

97. AFTER_SAVE
<?php
class article_module extends admin_module
{
public function after_save()
{
parent::after_save();
$data = $this->toJson();
// store the document in couchdb
$res = CouchDB::put($data);
// set the local id and revision, so we can
// use it the next time
$this->_id = $res->_id;
$this->_rev = $res->_rev;
}
}

98. PROXY
RewriteEngine On
RewriteRule /data/(.*) http://127.0.0.1:5984/db/$1 [P,L]

99. JSONP
?callback=foobar
foobar(...)

100. JAVASCRIPT
<script type="text/javascript">
$.getJSON("/db/ponies/_design/ponies/_view/best-ponies?include_docs=true",
function(res){
for(i in res.rows)
{
doc = res.rows[i].doc;
// do stuff
}
});
</script>

101. CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
• Other stuff

102. COUCHAPP

103. “Distributed, scalable, web
applications you say?
omgwtfbbq!?!1!!!one1!1!eleven”

104. _attachments

105. CouchDB
relax
CouchDB
relax
CouchDB
relax

106. INSTALLATION
Couchapp 0.7.5

107. $ couchapp init

108. LAYOUT

109. $ couchapp push http://ponies.couchone.com:5984/rainbows

110. get your own copy:
$ couchapp clone http://svdgraaf.couchone.com/couchappdemo/_design/↩
phpday phpday

111. https://github.com/brandon-beacher/couchapp-tmbundle

112. CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
• Other stuff

113. OTHER STUFF

114. REWRITES

115. _REWRITE

116. STEP 1
$ curl "http://ponies.couchone.com/rainbows/_design/ponies/
_view/best-ponies?descending=true&limit=5&key=”foobar”"

118. STEP 1
{
....
"rewrites": [{
"from": "/best-5-ponies",
"to": "ponies/_view/best-ponies",
"method": "GET",
"query": {
"descending": true,
"limit": 5,
"key": "foobar"
}
}]
}

119. STEP 1
$ curl "http://ponies.couchone.com/rainbows/_design/ponies/
_view/best-ponies?descending=true&limit=5&key=”foobar”"

120. STEP 1
$ curl "http://ponies.couchone.com/rainbows/_design/ponies/
_rewrite/best-5-ponies"

121. STEP 2
[vhosts]
awesomeponies.com = /rainbows/_design/ponies/_rewrite

122. STEP 2
$ curl "http://ponies.couchone.com/rainbows/_design/ponies/
_rewrite/best-5-ponies"

123. STEP 2
$ curl "http://awesomeponies.com/best-5-ponies"

125. _CHANGES

126. RELAX

127. $ curl -X GET "http://ponies.couchone.com/rainbows/_changes"

128. {"results":[
], "last_seq":0}

129. curl -X PUT http://ponies.couchone.come/rainbows/foobar -d '{"type":"awesome"}'

130. {"results":[
{
"seq":1,
"id":"foobar",
"changes":[{"rev":"1-aaa8e2a031bca334f50b48b6682fb486"}]
}
], "last_seq":1}

131. {"results":[
{
"seq":1,
"id":"foobar",
"changes":[{"rev":"1-aaa8e2a031bca334f50b48b6682fb486"}]
},
{
"seq":2,
"id":"foobar2",
"changes":[{"rev":"1-e18422e6a82d0f2157d74b5dcf457997"}]
}
], "last_seq":2}

132. _CHANGES OPTIONS
• ?since
• Longpolling
• Continuous

133. SINCE
$ curl -X GET "http://ponies.couchone.com/rainbows/_changes?since=20"

134. LONGPOLLING
curl -X GET "http://ponies.couchone.com/rainbows/_changes?feed=longpoll&since=2"

135. CONTINOUS
curl -X GET "http://ponies.couchone.com/rainbows/_changes?feed=continuous&since=2"

137. FILTERS

138. FILTERS/IMPORTANT.JS
function(doc, req) {
if(doc.priority == 'high')
{
return true;
}
return false;
}

139. $ curl -X GET
"http://ponies.couchone.com/rainbows/_changes?feed=continuous&filter=app/important

140. FILTERS/CITY.JS
function(doc, req) {
if(doc.city.name == req.query.name) {
return true;
}
return false;
}

141. curl -X GET
"http://ponies.couchone.com/rainbows/_changes?feed=continuous&filter=app/name&name=verona

142. SHOWS

143. function(doc, req) {
return {
body: "Hello World"
}
}

144. curl -X
"http://db.example.com/db/_design/foobar/_show/showfunction/docid"

145. function(doc) {
return {
"code": 302,
"body": "See other",
"headers": {
"Location": doc.target
}
};
}

146. USE CASE
CDN

147. From public hash to private file
/m/m1gz4l4a1clb.jpg
a2d85de382c8672925174e731a1d2a89

148. storage
(http based)
5
Varnish proprietary middleware
1 2 webserver
3
4
mysql

149. storage
(http based)
4
Varnish nginx + mod_cache
1 2 3
middleware

150. function(doc) {
return {
"code": 302,
"body": "See other",
"headers": {
"Location": doc.target
}
};
}

151. {
"_id": "m1gz4l4a1clb",
"_rev": "2-d7855bba9445f446c0b71f044e796c28",
"content-type": "image/jpeg",
"target": "http://10.16.10.1/a2d85de382c8672925174e731a1d2a89",
}

152. LUCENE

153. [external]
fti=/path/to/python /path/to/couchdb-lucene/tools/couchdb-external-hook.py
[httpd_db_handlers]
_fti = {couch_httpd_external, handle_external_req, <<"fti">>}

154. FULLTEXT/BY_MESSAGE/BY_QUERY.JS
function(doc) {
var ret=new Document();
ret.add(doc.message);
ret.add(new Date(doc.datetime));
return ret;
}

155. curl -X GET
"http://db.example.com/db/_fti/_design/app/by-query?q=foobar"

156. GEOCOUCH
https://github.com/vmx/couchdb

158. GEOCOUCH
• Supports bbox, polygon, lines, etc.
• fork (for now)
• outputs via lists, georss possible
• directly useable by google maps
• can read GIS data
• combined with _changes makes interesting use cases possible

159. SPATIAL INDEX
in spatial/points.js
function(doc) {
if (doc.geo && doc.geo.latitude != '' && doc.geo.longitude != '') {
emit( {
type: "Point",
coordinates: [
parseFloat(doc.geo.latitude),
parseFloat(doc.geo.longitude)
]
},
[
doc._id,
doc
]
);
}
}

160. SPATIAL INDEX
in spatial/points.js
function(doc) {
if (doc.geo && doc.geo.latitude != '' && doc.geo.longitude != '') {
emit( {
type: "Point",
coordinates: [
parseFloat(doc.geo.latitude),
parseFloat(doc.geo.longitude)
]
},
[
doc._id,
doc
]
);
}
}

161. SPATIAL INDEX
in spatial/points.js
function(doc) {
if (doc.geo && doc.geo.latitude != '' && doc.geo.longitude != '') {
emit( {
type: "Point",
coordinates: [
parseFloat(doc.geo.latitude),
parseFloat(doc.geo.longitude)
]
},
[
doc._id,
doc
]
);
}
}

162. Worldwide search
http://db.example.com/couchappdemo/_design/phpday/_spatial/points?bbox=0,0,180,90
{"update_seq":3,"rows":[
{
"id":"Verona",
"bbox":[10.991748,45.438367,10.991748,45.438367],
"value":[
"Verona",
[10.991748,45.438367]
]
}]
}

163. bbox=west,south,east,north

165. COUCHDB FOR IOS
https://github.com/couchbaselabs/iOS-Couchbase

166. http://www.iriscouch.com/

167. http://www.couchbase.com/downloads

168. Q?

169. http://joind.in/talk/view/3009