Presentation on CouchDB and its usage at VPRO. Given at Magnolia Conference 2010 in Basel and NoSQL-NL Meetup #4 at VPRO in Hilversum.

1. HTTP API for Free?
Check out CouchDB
Nils Breunese, VPRO
N.Breunese@vpro.nl
@breun

2. HTTP API
HTTP is simple
Every OS and programming language supports HTTP
Lots of tooling available: proxying, caching, monitoring, etc.
“Cool! So, how do I get one for my data?”

3. DIY?

4. Relax!

5. Apache CouchDB
Document-oriented database
Built-in HTTP server
Map/Reduce
Replication
Written in Erlang, fast and reliable

6. CAP Theorem
Consistency: all nodes see the same data at the same time
Availability: node failures do not prevent survivors from
continuing to operate
Partition tolerance: the system continues to operate despite
arbitrary message loss
Choose any two; you can’t have all three

7. Eventual Consistency
Consistency
Partition
Availability
tolerance

8. CouchDB at VPRO
- Broadcasts
- Trailers
- Clips
Supplier 1 - Trailers
- Etc.
Media
Supplier 2
Service Websites
Etc. CouchDB
ActiveMQ CouchDB Apps
Twitter Publisher
Mashup Etc.
Flickr
Harvester
Etc. - Articles
- Photos
- Tweets
- News
- Etc.

9. Document

10. What’s a Document?
JSON
Schemaless
Attachments

11. HTTP Document API
Create POST /db
Read GET /db/id
Update PUT /db/id
Delete DELETE /db/id
(Also: HTTP Bulk Document API)

12. Views
{“_id”: “myid1”, keywords: [“foo”,”bar”], title: “title1”, ...}
{“_id”: “myid2”, keywords: [“foo”,”baz”], title: “title2”, ...} docs
function(doc) {
{key: “foo”, value: “title1”}
doc.keywords.forEach(function(keyword) {
{key: “foo”, value: “title2”}
emit(keyword, doc.title);
{key: “bar”, value: “title1”}
});
{key: “baz”, value: “title2”}
} map
_count reduce {key: “foo”, value: 2}
{key: “bar”, value: 1}
{key: “baz”, value: 1}

13. Performance
Append-only B-trees (crash proof)
Views are pre-calculated indexes
Incremental map/reduce
Trade disk space for performance
Very low RAM usage

14. HTTP View API
/{db}/_design/{ddoc}/_view/{view_name}
Query parameters:
key=”foo”
startkey, startkey_docid, endkey, endkey_docid
limit, descending, skip, group, group_level, reduce
/db/_design/unconf/_view/by_keyword?key=”foo”

15. Using the API
Client-side: AJAX from browser (JSONP support in CouchDB)
Invisible for search engines
Server-side
You can use a library like jcouchdb
Indexable!

16. Formatting Data
Show function: transform a document into...
SVG, HTML, MP3, etc.
List function: transform a view result
RSS, etc.

17. Next Episode

18. Media Archive

19. Mashups

20. Public API
Rewrite handlers for prettier URL’s
API keys, monitoring, etc.?
You might need an extra layer

21. CouchApp
Replicatable HTML/JavaScript application
Without any middleware!
CouchApp tool to manage design documents

22. More CouchDB
Website: http://couchdb.apache.org/
Book: http://guide.couchdb.org/
BigCouch: sharding
GeoCouch: support for geospatial queries
couchdb-lucene: full-text indexing and querying
Myriad third-party libraries

23. VPRO Digital
http://weblogs.vpro.nl/digitaal/ (Dutch)