NoSQL Now 2013 Presentation

Service Entities Using
MongoDB
NoSQL Now 2013, San Jose
Arjen Schoneveld
Friday, August 23, 13

Arjen Schoneveld
• co-Founder of Dexels,Amsterdam based
software company
• Architect of Sportlink Sports ERP system
• PhD in Computer Science
• arjen@dexels.com, http://nl.linkedin.com/in/
aschoneveld/

Our Story
• How it started: Sportlink Sport Management System
• Navajo: a Service Oriented Application Framework
• Limitations of monolithic persistence
• MongoDB use case: Sportlink Digital Match Report
• Using Navajo Entity Services as a Schema
• RESTfull Entities

Sportlink Architecture (2003)
Sport Club
Application
Federation
Application
Federation
Website
Database (Oracle) Crystal
Reports
Comp.
Planning

Data Model
Object Model
Service Facade
Web Services
JSP/HTML/
Javascript/CSS
ORM
JAX-WS
SOAP/WSDL
Service Proxy
A Service Oriented Application Stack
Object Model

How to simplify?

Services First
• Changes are “the rule” in many environments
• Most Services are CRUD and instances have
very short live-span
• Many services do not need “OOP models”
• Use OOP for complex algorithms (e.g. match
scheduling) only
• Build, not generate, Services

Usable Services
• Loose Coupling: no client stub generation!
• Metadata is key ingredient: self describing
• Type, description, input validation, subsequent
services (protocol)
• Self describing Abstract User Interface → enables
automatic creation of usable ﬁrst order UIs
• Use a Canonical Service Language as the “lingua
franca” between the services and the client

What we came up with

Data Model
The Navajo Application Framework
UIs
[Platform Agnostic
Declarative UI Language]
Canonical
Service
Language
Services
[Service Script Language]

How we implemented it

fd
[Resources]
Database
Mail Server
FTP Server
SOAP Service
[Adapters]
[Tipi]
UIW
idgets
UIEvents
Service
Listeners
CallServices
UIActions
Layout +
CSS
[Services]
Client State
N
avajo
Expressions
[Logging]
[Work
Flows]
Use Adapters
[Authentication]
Validation
Service
Links
JDBC
MongoDB
[Entities]
SOAP
CallService
Joinmessage
Email Crystal
[Service
Queue]

How Sportlink Evolved...

Sportlink Ecosystem (2013)
Club
(Tipi/Swing)
Association
(Java/Swing)
Website
(Third Party)
Oracle
Crystal
FTP
~4000 services
Comp.
Planning
E-Mail
Digital
Match Report
(iOS/Android/Tipi)
Internet
Payment
(PHP)
Facility Mngtmt
(GWT+G. Earth)
Course
Registration
(GWT+G. Maps)
Ofﬁcial Portal
(Tipi/VaadIn)
MongoDB
MySQL
Website
Plugins
(PHP)
Financial
Admin
ServiceWorkﬂow
Enterprise
DB

Growing Pains
• More end user applications
• More functionality
• More sport associations with different
requirements
• More end users
• Single database has become the bottleneck

Oracle Limitations
• License Costs!
• Unpredictable query behavior
• Only partly/expensive High Availability
• Poor affordable(!) horizontal scalability in
combination with good single node
performance
• Service/Persistence impedance mismatch!

Oracle
Mongo
Neo4j
MySQL
Dynamo
Cassandra
PostgreS
QL
MatchReport
Web Plugin
Mobile App
Club App
Association
App
Association
App
Courses App Facility
Mngmnt
Polyglot Persistence

Our MongoDB Use Case
• Digital Match Report (Mobile,Web and
Desktop)
• Player Roster for home- and away teams
(role, captain, base player)
• Match Result, Match Events, Player
Validation, Digital Players Pass
• 40.000 matches per weekend (ﬁxed times:
11.00, 12:30, 14:00, 15:30, ...)
• typical: 3 users per match (ofﬁcial, two
captains)

Why MongoDB
• MatchReport application is introduced
gradually → Scalability → NoSQL
• MatchReport can be fetched and updated
like a Data Aggregate → Document Based
NoSQL

Home Away Ofﬁcial Mongo Oracle
Fetch Match Report
MatchReportReport Fetch Match Report
Report
Populate Home Players Report
Populate Away Players Report
Fetch Match Report
Report
Report Report
Fill In Report Report
Finalize Report Report Match
Players
Events
Players
Events

How do we integrate Navajo and
MongoDB?

Navajo Services
• A simple XML/HTTP/POST-based Web Service
• Uses a Canonical Service Language: Navajo Document
• Consumes and produces a Navajo Document (Everything is a Document)
• May contain additional (Service) links to next-step services (implementing an
application protocol, like hyperlinks)
• Navascript consumes and produces a Navajo Document, using expressions
and access (Java) adapters for: database access, calling other Navajo
Services, calling SOAP, manipulating messages (e.g. joins), etc.
• JavaScript and JRuby are also supported Navajo Service implementation
languages
• Navajo Services are (hot) deployed in a Navajo Server/OSGi runtime using
realtime generation of OSGi bundles

Navajo Document
• Navajo Services use Navajo Documents for all
message processing (Canonical Service Language)
• All data required by and delivered from a Navajo
Service is contained in a Navajo Document
• A Navajo document consists of the following
elements:
• messages, for grouping:
• properties and other messages
• methods (links to relevant subsequent services)

Properties
• A property has metadata:
• name. Name must be unique within a message
• type. Natively supported types are: string, integer, ﬂoat, boolean, date,
binary, selections(!).
• value.The value of the property.
• description. Used to indicate the semantical meaning of the property
• subtype. May contain additional formatting hints for clients, like matching
regular expressions for strings or the mime-type of a binary property
• direction. Indication for a client to render a property read-only or not.

Serializations
• The default serialization of Navajo Document is XML
• Different other serialization are supported, e.g. JSON
• Navajo Document contains a lot of metadata
automatic User Interface Generation is supported (an
Eclipse Plugin is provided)
• Serialization of binary type properties is Base64.
Navajo XML parser supports transparent streaming of
binary properties

A Navajo Document<tml>
<message name=”Person”>
<property name=”Lastname” length=”100” description=”Lastname of the person” type=”string”
subtype=”regexp=[A-z]*” direction=”out” value=”Kant”/>
<property name=”FirstName” type=”string” subtype=”regexp=[A-z]*” length=”50” direction=”out”
value=”Immanuel”/>
<property name=”DateOfBirth” type=”date” direction=”out” value=”1724-04-22”/>
<property name=”Photo” type=”binary” direction=”in”>
</property>
</message>
</tml>

Collections:Array Messages
<tml>
<message name=”Persons” type=”array”>
<message name=” Persons” type=”deﬁnition”>
<property name=”Lastname” type=”string” subtype=”regexp=[A-z]*” length=”100” direction=”out”
description=”Last name of this Person”/>
<property name=”Firstname” type=”string” subtype=”regexp=[A-z]*” length=”50” direction=”out”
description=”First name of this Person”/>
</message>
<message name=” Persons” index=”0”>
<property name=”Lastname” value=”Kant”/>
<property name=”Firstname” value=”Immanuel”/>
</message>
<property name=”Lastname” value=”Godel”/>
<property name=”Firstname” value=”Kurt”/>
</message>
<property name=”Lastname” value=”Turing”/>
<property name=”Firstname” value=”Alan”/>
</message>
</message>
</tml>

Club
Application
[Tipi]
QueryMatch
Oracle
2. Request
<tml rpc_name=”QueryMatch”>
<message name=”Match”>
<property name=”Id” value=”1”/>
</message>
</tml>4. Response
<tml rpc_name=”QueryMatch”>
<property name=”Id”/>
<property name=”Date”/>
<property name=”HomeTeam”/>
<property name=”AwayTeam”/>
</message>
<methods>
<method name=”UpdateMatch”/>
<method name=”DeleteMatch”/>
</methods>
</tml>
3. SELECT * FROM match WHERE matchid = ?
<navascript>
[see next slide]
</navascript>
<tipi>
<button>
<onPush>
(1) <callservice
name=”QueryMatch”>
</onPush>
</button>
(5) <panel listener=”QueryMatch”>
<label name=”Calendar date”
value=”{navajo:/QueryMatch/
MatchDate}”/>
</panel>
</tipi>

<navascript>
<validations>
<check code=”Empty Id”>
[/Match/Id] != ‘’
</check>
</validations>
<map.sqladapter>
<sqladapter.query>
SELECT * FROM MATCH WHERE matchid =?
</sqladapter.query>
<sqladapter.parameter value=”[/Match/Id]”/>
<property name=”FormattedMatchDate”>
<expression value=”FormatDate($columnValue(‘Date’),’dd-MMM-yyy'”/>
</property>
</message>
</map.sqladapter>
<methods>
<method name=”ProcessUpdateMatch”/>
<method name=”ProcessDeleteMatch”/>
</methods>
</navascript>

MongoDB
• MongoDB is a document-based NoSQL database
• MongoDB is schema-less, unlike SQL databases that
rely on a predeﬁned schema (or metadata) to deﬁne
e.g. tables and table structures
• MongoDB serializes data as JSON objects
• MongoDB supports a sharded setup → scalability
• Advantage: ease of use due to lack of structure
• Disadvantage: unmaintainable due to lack of structure

Navajo and MongoDB
• Store Navajo Document directly in
MongoDB
• Navajo Document can be easily
transformed to JSON
• No need for writing “mapping code” (no
impedance mismatch)

JSONify Navajo Document
<message name=”MatchReport”>
<property name=”MatchId” value=”1” type=”integer”/>
<property name=”HomeTeamId” value=”1234” type=”integer” description=”Unique id of the home team”/>
<property name=”AwayTeamId” value=”4321” type=” integer” description=”Unique id of the away team”/>
<property name=”Date” value=”2013-08-24” type=”date” description=”Match date”/>
<message name=”HomePlayers” type=”array”>
<message name=”HomePlayers” type=”arrayelement”>
<property name=”PersonId” type=”string” value=”CHGP12Y”/>
<property name=”Role” type=”selection” cardinality=”1” type=”selection”>
<option name=”Defender” value=”DEF” selected=”1”/>
<option name=”Keeper” value=”KPR” selected=”0”/>
</property>
</message>
</message>
[... same for AwayPlayers ... left for brevity ]
</message>
The Navajo Document XML above can be converted to JSON as follows:
{ “MatchId” : 1,“HomeTeamId” :“1234”,“AwayTeamId” :“4321”, “Date” : Date ( 23242873 ), “HomePlayers” :
[ {“PersonId” :“CHGP12Y”,“Role” :“DEF”}] }

Query MongoDB
• Entity in MongoDB can be queried using MongoDB query syntax
• Entity in MongoDB can be queried using a Navajo Document -> will be converted
to a JSON query:
<property name=”MatchId” value=”1”/>
</message>
can be converted to Mongo query (message name maps to Collection name):
MatchReport.ﬁnd({“MatchId” : 1})

<navascript>
</message>
<map.navajomap useCurrentOutDoc=”true”>
<navajomap.callwebservice name=”mongo/ProcessQueryMongo” append=”’/’”/>
</map.navajomap>
</navascript>
Mongo Query Service
Construct Query
Call Query Service and append result

Result of Query
<message name=”MatchReport”>Unique Id
<property name=”_id” value=”aabf3fdde323g”/>
<property name=”HomeTeamId” value=”1234” type=”integer”/>
<property name=”AwayTeamId” value=”4321” type=”integer”/>
<property name=”Date” value=”2013-08-24” type=”date”/>
<message name=”HomePlayers”>
<property name=”__id” value=”abb344gf67”/>
<property name=”Role” type=”string” value=”DEF”/>
</message>
<property name=”__id” value=”ffhggab3232b”/>
<property name=”PersonId” type=”string” value=”MJGP845”/>
<property name=”Role” type=”string” value=”KEP”/>
</message>
</message>
</message>
What is Lost?
What is Added?
Unique Id
Unique Id

How can we resolve this?

Query using Entity
Service
• Deﬁne a Navajo Service for producing the Entity
Navajo Document: Entity Service as a Schema
• Expects no input
• Simply returns Navajo Document describing the
Entity
• Pass Entity Service as a “parameter” to the Navajo
Service that queries or mutates Mongo
• Entity service is used to decorate the “raw” Mongo
document

Example Entity Message
<property name=”MatchId” type=”integer”/>
<property name=”HomeTeamId” type=”integer” description=”Unique id of the
hometeam”/>
<property name=”AwayTeamId” type=”integer” description=”Unique id of the
person”/>
<property name=”Date” type=”date” description=”Match date”/>
<message name=” HomePlayers” type=”deﬁnition”>
<property name=”PersonId” type=”string” value=”Unique person id”/>
<property name=”Role” type=”selection” cardinality=”1” type=”selection”>
<option name=”Defender” value=”DEF”/>
<option name=”Keeper” value=”KPR”/>
</property>
</message>
</message>
</message>
Navajo Document above describes the Entity type “MatchReport”

Mongo Query Service
(revisited)
<navascript>
<message name=”__Mongo__”>
<property name=”EntityService” value=”entity/MatchReport”/>
</message>
</message>
<navajomap.callwebservice name=”mongo/ProcessQueryMongo” append=”’/’”/>
</map.navajomap>
</navascript>

Result of Query (revisited)
<property name=”HomeTeamId” value=”1234” type=”string” description=”Unique id of the home team”/>
<property name=”AwayTeamId” value=”4321” type=”string” description=” Unique id of the away team”/>
<property name=”Date” value=”2013-08-24” type=”date” description=”Match date”/>
<property name=”PersonId” type=”string” value=”Unique person id”/>
<property name=”Role” type=”string” type=”selection”>
<option name=”Defender” value=”DEF”/>
<option name=”Keeper” value=”KPR”/>
</property>
</message>
</message>
</message>
</message>
</message>

Flexibility Harnessed
• Entity adds missing metadata
• Entity can be used for type validation
• Adding properties is easy, simply add it to the
Entity, next time it will appear in the query
• Deleting properties is easy, simply delete it from
the Entity, next time it won’t appear in the query.

Insert Example
<navascript>
</message>
<property name=”HomeTeamId” value=”9787” type=”string” description=”Unique id of the home team”/>
<property name=”AwayTeamId” value=”6543” type=”string” description=” Unique id of the away team”/>
<property name=”PersonId” type=”string” value=”BBFW63X”/>
</message>
</message>
</message>
<navajomap.callwebservice name=”mongo/ProcessInsertMongo”/>
</map.navajomap>
</navascript>
This will fail!

Update Example<navascript>
</message>
<property name=”HomeTeamId” value=”1234” type=”integer”/>
<property name=”AwayTeamId” value=”4321” type=”integer”/>
<property name=”BogusAttribute” value=”FooBar” type=”string”/>
</message>
</message>
</message>
</message>
<navajomap.callwebservice name=”mongo/ProcessUpdateMongo”/>
</map.navajomap>
</navascript>
This will be removed

Delete Example
Assume that we accidentally inserted the following document:
<property name=”_id” value=”aaa666”/>
<property name=”HomeTeamId” value=”809832” type=”integer” />
<property name=”AwayTeamId” value=”321311” type=”integer” />
<message name=”HomePlayers” type=”array”/>
</message>
This can be deleted using following script:
<navascript>
<property name=”_id” value=”aaa666”/>
</message>
<navajomap.callwebservice name=”mongo/ProcessDeleteMongo”/>
</map.navajomap>
</navascript>

What about more ﬁne grained mutations?

Simple Modify Example
Let’s just change the date
<navascript>
</message>
</message>
<navajomap.callwebservice name=”mongo/ProcessModifyMongo”/>
</map.navajomap>
</navascript>

Inserting new Array
Element
Insert a new player for the home team:
<property name=”Insert” type=”boolean” value=”true”/>
<property name=”PersonId” type=”string” value=”BBFW06E”/>
<property name=”Role” type=”string” value=”DEF” />
</message>
</message>
</message>
Note that the properties _id and Insert (set to true) are required
Id is not required,
it will be assigned

<property name=”Modify” type=”boolean” value=”true”/>
</message>
</message>
</message>
Id is required
Set Modify ﬂag
Modifying an Array
Element
Change the role of a player:

Deleting an Array
Element
<property name=”Delete” type=”boolean” value=”true”/>
<property name=”__id” value=”agb3f4g4468”/>
</message>
</message>
</message>
Id is required
Set Delete ﬂag

What have we reached?
• A Flexible Schema to manage MongoDB
document data
• Schema is managed outside Mongo
• Basically a metadata template to ﬁlter/
validate both incoming and outgoing data
• No need to write any “Mapping Code”

Normal Services
Entity Services
... Symmetry at last!

What is missing?
• Notion of entity instances (to enable client
push)
• Relations between entities
• Deﬁnition of key/required properties
• Data consistency
• Entity deﬁned persistence

Entity First
• Thus far: Operation-Oriented Services:
DoUpdateMatchReport, QueryMatchReport,
FinalizeMatchReport, etc.
• Resource-Oriented, the Way of the Web
• RESTful resources can be viewed as First Class
Citizens (everything is a resource)
• REST method semantics enable use of Cache

REST in 1 Slide
• A REST architecture relies fully on HTTP as an
Application Protocol
• REST revolves around URI resources (nouns) and
HTTP methods (actions) and ...
• Rely on hypermedia to drive the client(!) state
machine deﬁned by the application

RESTFul Navajo
• Navajo Entity == Resource
• Extend to support inheritance and referencing
other Navajo Entities
• Add key properties
• Navajo Entity support one or more HTTP methods
(GET, PUT, POST and DELETE)
• Entity Operations (HTTP methods) are
implemented by Navajo Services

Entity Service<validations>
<check code=”A MatchReport should have a date unless it is a DRAFT
MatchReport”>
[/Match/Status] != ‘DRAFT’ OR CheckDate([/MatchReport/Date)</check>
</validations>
<message name=”MatchReport” extends="navajo://Report">
<property name="_id" type="string" key="true,auto"/>
<property name="MatchId" type="string" key="true" reference=”navajo://
Match/MatchId”/>
<property name="Status" type="string"/>
<property name="HomeTeamId" type="integer" reference=”navajo://Team/
TeamId”/>
<property name="AwayTeamId" type="integer" reference=”navajo://Team/
TeamId”/>
<property name=”PersonId” type=”string” reference=”navajo://
Person”>
<property name=”Role” type=”string”/>
</message>
</message>
</message>

Entity Operations
<navascript>
<operations>
<operation service=”mongo/ProcessQueryMongo” method=”GET”/>
<operation service=”mongo/ProcessModifyMongo” method=”POST”/>
</operations>
<message name=”MatchReport” extends="navajo://Report">
[ .... ]
</message>
</navascript>
[NOTE: operation == method]

Pure HTTP GET
http://dexels.example.org/MatchReport?MatchId=33

Backwards Compatibility
<navascript>

<map.entity entity="'MatchReport'">

<entity.createproperty name="'MatchId'" value="3" />

<entity.get append="'/'" />

</map.entity>
</navascript>

Entity Insert
<navascript>

<message name="MatchReport">

<property name="MatchId">

<expression value="‘31232’"/>

</property>
<property name="HomeTeamId">


</property>
<property name="AwayTeamId">


</property>

</message>

<map.entity useCurrentOutDoc="true" entity="'MatchReport'">

<entity.insert/>

</map.entity>
</navascript>

Caching and Consistency
• ETag can help us with
• Preventing that multiple GET operation
return same data (send 304 instead)
• Preventing that Stale data is used for
updating an Entity (check ETag on
PUT,DELETE,POST) (send 409)

Polyglot Persistence
• Entity Operations can implement different
persistence methods
• Some Entities can “live” inside MongoDB while other
reside in e.g. Oracle
• Non-functional data requirements (e.g. consistency,
availability) can drive persistence decision
• “Transparent” migration of Data to other datasources

Entity drives Service drives Entity drives Service drives Entity
drives Service drives Entity drives Service drives Entity drives
Service
• Role of Normal Service and Entity Service
are reversed
• Instead of QueryMatchForm,
DoInsertMatchForm,
DoDeleteMatchForm, we have MatchForm
(GET, DELETE, INSERT)

Normal Services
Entity Services
... from process- to resource oriented

NoSQL Now 2013 Presentation

Recomendados

Recomendados

Más contenido relacionado

Similar a NoSQL Now 2013 Presentation

Similar a NoSQL Now 2013 Presentation (20)

Último

Último (20)

NoSQL Now 2013 Presentation