6. Mid Period: 2000s (The “Vendor CMS era”)
Vignette / AOLserver
TCL, Apache, Oracle
Platform for online
publishing
Initially scales well with
acceleration in delivery
of features
7. Mid Period: 2000s (The “Vendor CMS era”)
Surprise! Vendor’s CMS
doesn’t do what we want!
Mish-mash in templates:
HTML, JavaScript, TCL,
SQL, PL-SQL
No model in app tier, only
in RDBMS schema created
in Oracle Designer
10. Mid Period: 2000s (The “Vendor CMS era”)
After a few years, very
difficult to extend
Database schema
becomes fixed due to
dependencies in
templates
11. Mid Period: 2000s (The “Vendor CMS era”)
If you can’t change the
system:
12. Modern Period
circa ’05-09
The “J2EE Monolithic” era
13.
14. Web server Web server Web server
I bring you NEWS!!!
App server App server App server
Oracle
CMS Data feeds
15. Web server Web server Web server
Modern java app
I bring you NEWS!!!
App server App server App server
Spring / Hibernate
DDD / TDD
Strong Oracle in java
model
Database abstracted away with ORM
CMS Data feeds
18. Complexity still increasing:
300+ tables,
10,000 lines of hibernate XML config
1,000 domain objects mapped to database
70,000 lines of domain object code
Very tight binding to database
19. ORM not really masking complexity:
Database has strong influence on domain model: many
domain objects made more complex mapping joins in
RDBMS
Complex hibernate features used, interceptors, proxies
Complex caching strategy
Lots of optimisations
And:
We still hand code complex queries in SQL!
21. Partial NoSQL
circa ’09-10
The “Sticking Plaster” era
22. Introduce yet more caching to patch up load problems
Text
Introduction of memcached
23. Decouple applications from database by building APIs
Power APIs using alternative, more scalable technologies
APIs used to scale out database reads
Writes still go to RDBMs
24. Content API
Mutualised news!
http://content.guardianapis.com
Read API delivered using Apache Solr
Hosted in EC2
Document oriented search engine
Scales well for read operations
25. Core
Api
Web servers
Solr/API
App server
Solr/API
Memcached (20Gb)
Solr/API
rdbms Solr
Solr/API
Solr/API
CMS Cloud, EC2
31. MutualisedAPI is very simple
JSON news!
Multiple domain concepts expressed in single document
Can be designed in forwardly extensible way
What if the JSON API was our primary model?
32. Full NoSQL
in development
The “It’s the future!” era
33. Database selection
Simple keystore. Too simple?
Huge scalability. Do we need it?
Schema design difficult.
Simple to use, can execute similar
queries to RDBMs
34. MongoDB
Mutualised news! database
Document oriented
Stores parsed JSON documents
Can express complex queries
Can be flexible about consistency
Malleable schema: can easily change at runtime
Can work at both large & small scales
37. Flexible Schema
Mutualised news!
Can easily represent different classes of tag as
documents
Both documents can be inserted into
same collection
Far simpler than equivalent hibernate
mapped subclass configuration
43. The first project: Identity
Current login/registration system still in TCL/PL-SQL
3M+ users in relational database
Very complex schema + PL-SQL
New system required
Can we migrate from Oracle to NoSql?
44. Build API that can support both backends
Registration app guardian.co.uk
API This bit is hard!
Oracle
45. Build API that can support both backends
Registration app guardian.co.uk
API
MongoDB Oracle
46. Migrate using API & decommision
Registration app guardian.co.uk
API
MongoDB
47. Add new stuff!
Registration app guardian.co.uk
API
MongoDB Solr? Redis?
48. MongoDB
Simple, flexible schema with similar query & indexing to
RDBMS
Great at small or large scale
Easy for developers to get going
Commercial support available (10Gen)
One day may power all of guardian.co.uk
No transactions / joins: developers must cater for this
Produces a net reduction in lines of code / complexity
Very standard 3 tier application\nScale application servers on load\nCaching local to application server at first. Memcached added later\nRead heavy, broadcast model. Almost no writes compared to reads\n\n
Very standard 3 tier application\nScale application servers on load\nCaching local to application server at first. Memcached added later (in next era!)\nRead heavy, broadcast model. Almost no writes compared to reads\n\n
\n
\n
\n
\n
\n
Talk: beginning to use NoSql in real organisation. Change in journalism affecting platform\n\n
We don’t have a scale problem with current application & model\n(Interesting fact: small dip at end is actually period of very high load. Caching works)\n\n
Talk: beginning to use NoSql in real organisation. Change in journalism affecting platform\n\n
Most of our new features - and partners - drive from the content api\n
Introduction of memcached & Solr\nSolr hosted in the cloud (EC2)\n
“Out” service\n
\n
Most recent content\n\n
Most recent content with tags, fields\n(this is pretty well how we went live with the content api)\n\n\n
Single article with media\nExtensible schema, eg: adding geotagging to images. Hard in DB, easy in JSON\nThis document represents at least 30 database tables!\n\n\n
\n\n
\n
Couch used at BBC. To simple.\nCassandra: Impressive. Do we need it? Schema design tricky.\nMongoDB: Not a huge mindset change. Devs working in a few days\n
Not a million miles from a RDBMS\nSimpler\n
Experiments with mongodb & content API\nGuardian site categorises content with tags\nTone tag represents “editorial tone” of content\n(SKIP IF LESS THAN 10 MINS TO GO!)\n\n
Different tag types can have different schemas\nKeywords (subjects) are in a section, music / madonna\n\n
\n\n
\n\n
\n\n
Suppose we want to add external musicbrainz ID to tag?\nAn update can modify the schema at runtime. No downtime.\n\n
Where clause: id\n$push atomically ads external reference onto tag\n\n
Resulting document now looks like this\n\n
Migration project, not green fields\n
REST API\nMapped initially just to oracle, then (next slide) to both datastores\nIntegration tested\n\n
API supports both data stores - lazy migration\nCurrently writing this - so far 60-70% less code for mongo version\n\n