SQL NoSQL ORM ODM Delphi MongoDB OpenSource mORMot

1. Software Architecture & Design
 Architecture
 From n-Tier to SOA
 From SOAP to REST
 Technical Debt
 Design
 From SQL to ORM, NoSQL and ODM
 From RAD to MVC
 SOLID principles
 Domain Driven Design (DDD)
Applying patterns on Delphi code using mORMot
Software Architecture & Design

2. From SQL to ORM
Arnaud Bouchez

3. From SQL to ORM
 SQL
 NoSQL
 ORM
 ODM
 practice
From SQL to ORM

4. SQL
 De-Facto standard for data manipulation
 Schema-based
 Relational-based
 ACID by transactions
 Time proven and efficient
 Almost standard
From SQL to ORM

5. SQL
 De-Facto standard for data manipulation
 Schema-based
 Relational-based
From SQL to ORM

6. Not Only SQL
 Two main families of NoSQL databases:
 Aggregate-oriented databases
 Graph-oriented databases
Martin Fowler
From SQL to ORM

7. Not Only SQL
 Graph Database
 Store data by their relations / associations
From SQL to ORM

8. Not Only SQL
 Aggregate
 is a collection of data
that we interact with
as a unit
 forms the boundaries
for ACID operations
in a given model
(Domain-Driven Design modeling)
From SQL to ORM

9.  Aggregate Database families
 Document-based
e.g. MongoDB, CouchDB
 Key/Value
e.g. Redis
 Column family
e.g. Cassandra
From SQL to ORM
Not Only SQL

10. Not Only SQL
 Are designed to scale for the web
 Examples: Google farms, Amazon
 Should not be used as scaling RDBMS
 Can be schema-less
 For document-based and key/value
or column-driven
 Can be BLOB storage
From SQL to ORM

11. Not Only SQL
 RBMS stores data per table
 JOIN the references to get the aggregate
From SQL to ORM

12. Not Only SQL
 NoSQL stores aggregate as documents
 Whole data is embedded
From SQL to ORM

13. Not Only SQL
 NoSQL stores aggregate as documents
 Whole data is embedded
(boundaries for ACID behavior)
From SQL to ORM

14. Not Only SQL
 Data modeling
From SQL to ORM

15. Not Only SQL
 Data modeling
SQL
 Normalization
 Consistency
 Transactions
 Vertical scaling
NoSQL
 Denormalization
 Duplicated data
 Document ACID
 Horizontal scaling
From SQL to ORM

16. Not Only SQL
 SQL > NoSQL
 Ubiquitous SQL
 Easy vertical scaling
 Data size (avoid duplicates and with no schema)
 Data is stored once, therefore consistent
 Complex ACID statements
 Aggregation functions (depends)
From SQL to ORM

17. Not Only SQL
From SQL to ORM

18. Not Only SQL
From SQL to ORM
SELECT SUM(price) AS total FROM orders
db.orders.aggregate( [
{
$group: {
_id: null,
total: { $sum: "$price" }
}
}
] )

19. Not Only SQL
 NoSQL > SQL
 Uncoupled data: horizontal scaling
 Schema-less: cleaner evolution
 Straight-To-ODM
 Version management (e.g. CouchDB)
 Graph storage (e.g. Redis)
From SQL to ORM

20. Object Relational Mapping (ORM)
 ORM
 gives a set of methods (CRUD)
 to ease high-level objects persistence
 into an RDBMS
 via mapping
From SQL to ORM

21. Object Relational Mapping (ORM)
 Since decades
 Classes are the root of our OOP model
 RDBMS is (the) proven way of storage
 Some kind of "glue" is needed to let class
properties be saved into one or several tables
From SQL to ORM

22. Object Relational Mapping (ORM)
 In fact
 Classes are accessed via Delphi/Java/C# code
 RDBMS is accessed via SQL
 SQL by itself is a full programming language
 With diverse flavors (e.g. data types)
 Difficult to switch the logic
 Error prone, and difficult to maintain
From SQL to ORM

23. Object Relational Mapping (ORM)
 Sometimes,
 there will be nothing better
than a tuned SQL statement
 But most of the time,
 You need just to perform
some basic CRUD operations
(Create Retrieve Update Delete)
From SQL to ORM
object
instance
SQL
RDBMS
object
instance
ORM
CRUD
operations
SQL
mapping
RDBMS

24. Object Relational Mapping (ORM)
myObject.Value := 10;
myContext.Update(myObject);
UPDATE OBJTABLE SET …
01010101001110011111
From SQL to ORM

25. Object Relational Mapping (ORM)
 Benefits
 Stay at OOP level
 Manage SQL flavors
 Persistence Ignorance
 Optimize SQL
 Cache
From SQL to ORM
 No silver bullet
 Hard task
 Hidden process
 Legacy (tuned) SQL
 Performance cost

26. Object Relational Mapping (ORM)
 Benefits
 Stay at OOP level
 Manage SQL flavors
 Persistence Ignorance
 Optimize SQL
 Cache
From SQL to ORM
 No silver bullet
 Hard task
 Hidden process
 Legacy (tuned) SQL
 Performance cost
(or not)

27. Object Relational Mapping (ORM)
 Mapping may be created:
 Code-first
 Objects are defined,
then persisted (dehydrated)
 Usually the best for a new project
 Model-first
 From an (existing) database model
 Sometimes difficult to work with legacy structures
From SQL to ORM
ORM
class type data model
object instance RDBMS

28. Object Relational Mapping (ORM)
 Mapping may be defined:
 by configuration
 via code (attributes or fluent interface)
 via external files (XML/JSON)
 by convention
 from object layout
 from database layout
From SQL to ORM

29. Object Relational Mapping (ORM)
 Objects may be:
 Any business class (POJO/POCO/PODO)
 Eases integration with existing code
 But may pollute the class (attributes)
 Inherit from a common class
 Get a shared behavior
 Tend to enforce DDD’s persistence agnosticity
From SQL to ORM

30. Object Relational Mapping (ORM)
 ORM advantages
 Compile time naming and types check
(strong types and names)
 One language to rule all your logic
(no mix with SQL nor LINQ syntax)
 Database abstraction
(the ORM knows all dialects, and can switch)
 Able to cache the statements and the values
 You still can write hand-tuned SQL if needed
From SQL to ORM

31. Object Relational Mapping (ORM)
 Don't think about…
 tables with data types (varchar/number...),
but objects with high level types
 Master/Detail,
but logical units (your aggregates)
 writing SQL,
but writing your business code
 "How will I store it?",
but "Which data do I need?".
From SQL to ORM

32. Object Document Mapping (ODM)
 Aggregate = all data in a given context
Such documents do map our objects!
From SQL to ORM

33. Object Document Mapping (ODM)
 CRUD operation on Aggregates documents
mORMot’s ODM is able to share code with ORM
From SQL to ORM

34. mORMot’s RESTful ORM/ODM
From SQL to ORM

35. mORMot’s RESTful ORM/ODM
 mORMot’s client-server RESTful ORM/ODM
 code-first or model-first ORM/ODM
 ORM for RDBMS: generates SQL
 ODM for NoSQL: handles documents and queries
 convention (TSQLRecord) over configuration
 can be consumed via REST
 over HTTP or WebSockets, via JSON transmission
 or locally, on the server side, e.g. in SOA services
From SQL to ORM

36. mORMot’s RESTful ORM/ODM
From SQL to ORM
UI Components
DataBase
RAD
UI
Delphi classes
code mapping
DataBase
Handwritten
SQL
UI
ORM
MVC
binding
Database
Generated
SQL
UI 1
Client 1
MVC/MVVM
binding
Server
Secure
protocol
(REST)
UI 2 (web)
Client 2
MVC
binding
ORM
Persistence
layer
Database
Generated
SQL

37. mORMot’s RESTful ORM
 Defining the objects
TSQLRecord
 ID: Int64 / TID primary key
 A lot of useful methods
 Define a data model gathering objects
TSQLModel
 Defining the REST instance
TSQLRestServer TSQLRestClient …
 CRUD (and SOA) methods
From SQL to ORM

38. mORMot’s RESTful ORM
 Defining the object
From SQL to ORM
/// some enumeration
// - will be written as 'Female' or 'Male' in our UI Grid
// - will be stored as its ordinal value, i.e. 0 for sFemale, 1 for sMale
// - as you can see, ladies come first, here
TSex = (sFemale, sMale);
/// table used for the Babies queries
TSQLBaby = class(TSQLRecord)
private
fName: string;
fAddress: string;
fBirthDate: TDateTime;
fSex: TSex;
published
property Name: string read fName write fName;
property Address: string read fAddress write fAddress;
property BirthDate: TDateTime read fBirthDate write fBirthDate;
property Sex: TSex read fSex write fSex;
end;

39. mORMot’s RESTful ORM
 Defining the object
From SQL to ORM
/// some enumeration
// - will be written as 'Female' or 'Male' in our UI Grid
// - will be stored as its ordinal value, i.e. 0 for sFemale, 1 for sMale
// - as you can see, ladies come first, here
TSex = (sFemale, sMale);
/// table used for the Babies queries
TSQLBaby = class(TSQLRecord)
private
fName: RawUTF8;
fAddress: RawUTF8;
fBirthDate: TDateTime;
fSex: TSex;
published
property Name: RawUTF8 read fName write fName;
property Address: RawUTF8 read fAddress write fAddress;
property BirthDate: TDateTime read fBirthDate write fBirthDate;
property Sex: TSex read fSex write fSex;
end;

40. mORMot’s RESTful ORM
 Defining the Model and the Server:
Model := TSQLModel.Create([TSQLBaby],'rootURI');
ServerDB := TSQLRestServerDB.Create(Model,'data.db'),true);
ServerDB.CreateMissingTables;
Server := TSQLHttpServer.Create('8080',ServerDB);
 Defining the Model and the Client:
Model := TSQLModel.Create([TSQLBaby],'rootURI');
Client := TSQLHttpClient.Create('127.0.0.1','8080', Model);
 Both will now communicate e.g. over
http://server:8080/rootURI/Baby
From SQL to ORM

41. mORMot’s RESTful ORM
 Use CRUD methods
 TSQLRest.Add()
 TSQLRest.Retrieve()
 TSQLRest.Update()
 TSQLRest.UpdateField()
 TSQLRest.Delete()
 TSQLRecord.Create*()
 and some dedicated methods for BLOB process
From SQL to ORM

42. mORMot’s RESTful ORM
From SQL to ORM
var Baby: TSQLBaby;
ID: integer;
begin
// create a new record, since Smith, Jr was just born
Baby := TSQLBaby.Create;
try
Baby.Name := 'Smith';
Baby.Address := 'New York City';
Baby.BirthDate := Now;
Baby.Sex := sMale;
ID := Client.Add(Baby);
finally
Baby.Free;
end;
// update record data
Baby := TSQLBaby.Create(Client,ID); // retrieve record
try
assert(Baby.Name='Smith');
Baby.Name := 'Smeeth';
Client.Update(Baby);
finally
Baby.Free;
end;
// retrieve record data
Baby := TSQLBaby.Create;
try
Client.Retrieve(ID,Baby);
// we may have written: Baby := TSQLBaby.Create(Client,ID);
assert(Baby.Name='Smeeth');
finally
Baby.Free;
end;
// delete the created record
Client.Delete(TSQLBaby,ID);
end;

43. mORMot’s RESTful ORM
 Returning several objects
 Only one object instance is allocated and filled
 See also function TSQLRest.RetrieveList(): TObjectList
 Results are transmitted as a JSON array
 Results can be cached at client or server side
 Full WHERE clause of the SELECT is at hand
From SQL to ORM
aMale := TSQLBaby.CreateAndFillPrepare(Client,
'Name LIKE ? AND Sex = ?',['A%',ord(sMale)]);
try
while aMale.FillOne do
DoSomethingWith(aMale);
finally
aMale.Free;
end;

44. mORMot’s RESTful ORM
 JSON per-representation transmission layout
 Expanded / standard (AJAX)
JSON array of JSON objects
[{"ID":1},{"ID":2},{"ID":3},{"ID":4},{"ID":5},{"ID":6},{"ID":7}]
 Non expanded / faster (Delphi)
JSON object of JSON array of values, first row being idents
{"fieldCount":1,"values":["ID",1,2,3,4,5,6,7]}
In mORMot, all JSON content is parsed and processed in-place,
then directly mapped to UTF-8 structures.
From SQL to ORM

45. mORMot’s RESTful ORM
 « One to one » / « One to many » cardinality
From SQL to ORM
TSQLMyFileInfo = class(TSQLRecord)
private
FMyFileDate: TDateTime;
FMyFileSize: Int64;
published
property MyFileDate: TDateTime read FMyFileDate write FMyFileDate;
property MyFileSize: Int64 read FMyFileSize write FMyFileSize;
end;
TSQLMyFile = class(TSQLRecord)
private
FSecondOne: TSQLMyFileInfo;
FFirstOne: TSQLMyFileInfo;
FMyFileName: RawUTF8;
published
property MyFileName: RawUTF8 read FMyFileName write FMyFileName;
property FirstOne: TSQLMyFileInfo read FFirstOne write FFirstOne;
property SecondOne: TSQLMyFileInfo read FSecondOne write FSecondOne;
end;

46. mORMot’s RESTful ORM
 Automatic JOINed query
 At constructor level
 With nested instances memory management
 See also CreateAndFillPrepareJoined()
From SQL to ORM
var MyFile: TSQLMyFile;
begin
MyFile := TSQLMyFile.CreateJoined(Client,aMyFileID);
try
// here MyFile.FirstOne and MyFile.SecondOne are true instances
// and have already retrieved from the database by the constructor
// so you can safely access MyFile.FirstOne.MyFileDate or MyFile.SecondOne.MyFileSize here!
finally
MyFile.Free; // will release also MyFile.FirstOne and MyFile.SecondOne
end;
end;

47. mORMot’s RESTful ORM
 « One to one » / « One to many » cardinality
From SQL to ORM
TSQLMyFileInfo = class(TSQLRecord)
private
FMyFileDate: TDateTime;
FMyFileSize: Int64;
published
property MyFileDate: TDateTime read FMyFileDate write FMyFileDate;
property MyFileSize: Int64 read FMyFileSize write FMyFileSize;
end;
TSQLMyFileInfoID = type TID;
TSQLMyFile = class(TSQLRecord)
private
FSecondOne: TSQLMyFileInfoID;
FFirstOne: TSQLMyFileInfoID;
FMyFileName: RawUTF8;
published
property MyFileName: RawUTF8 read FMyFileName write FMyFileName;
property FirstOne: TSQLMyFileInfoID read FFirstOne write FFirstOne;
property SecondOne: TSQLMyFileInfoID read FSecondOne write FSecondOne;
end;

48. mORMot’s RESTful ORM
 « One to one » / « One to many » cardinality
 Properties store true Int64 values, not “fake pointers”
 This is the preferred way when working with Aggregates
From SQL to ORM
TSQLMyFileInfoID = type TID;
TSQLMyFile = class(TSQLRecord)
private
FSecondOne: TSQLMyFileInfoID;
FFirstOne: TSQLMyFileInfoID;
FMyFileName: RawUTF8;
published
property MyFileName: RawUTF8 read FMyFileName write FMyFileName;
property FirstOne: TSQLMyFileInfoID read FFirstOne write FFirstOne;
property SecondOne: TSQLMyFileInfoID read FSecondOne write FSecondOne;
end;

49. mORMot’s RESTful ORM
 Server or Client Cache
 Disabled by default
 May be activated
 For a given table
 For a given set of IDs
 On a given TSQLRest
 In conjunction with
Server DB cache
From SQL to ORM

50. mORMot’s RESTful ORM
 Object Time Machine
ServerDB.TrackChanges([TSQLInvoice]);
 Every TSQLInvoice modification will be tracked
and stored in a separated TSQLRecordHistory table.
 i.e. Add / Update / Delete ORM commands
 Stored in an optimized JSON/binary format
 History depth, and storage details are customizable
 Previous TSQLInvoice state could then be retrieved from
this TSQLRecordHistory table
From SQL to ORM

51. mORMot’s RESTful ORM
 Object Time Machine
ServerDB.TrackChanges([TSQLInvoice]);
aInvoice := TSQLInvoice.Create;
aHist := TSQLRecordHistory.CreateHistory(ServerDB,TSQLInvoice,400);
try
writeln('History Count: ',aHist.HistoryCount);
for i := 0 to aHist.HistoryCount-1 do begin
aHist.HistoryGet(i,aEvent,aTimeStamp,aInvoice);
writeln('Event: ',ord(aEvent))^);
writeln('TimeStamp: ',TTimeLogBits(aTimeStamp).ToText);
writeln('Identifier: ',aInvoice.Number);
end;
...
From SQL to ORM

52. mORMot’s RESTful ORM
 Object Time Machine
ServerDB.TrackChanges([TSQLInvoice]);
aInvoice := TSQLInvoice.Create;
aHist := TSQLRecordHistory.CreateHistory(Client,TSQLInvoice,400);
try
writeln('History Count: ',aHist.HistoryCount);
for i := 0 to aHist.HistoryCount-1 do begin
aHist.HistoryGet(i,aEvent,aTimeStamp,aInvoice);
writeln('Event: ',ord(aEvent))^);
writeln('TimeStamp: ',TTimeLogBits(aTimeStamp).ToText);
writeln('Identifier: ',aInvoice.Number);
end;
...
From SQL to ORM

53. mORMot’s RESTful ORM
 Master/Slave Replication
 Hidden monotonic version number will be maintained
to monitor Add/Update/Delete ORM commands
 Deletions stored in a separated table
From SQL to ORM
TSQLRecordPeopleVersioned = class(TSQLRecordPeople)
protected
fFirstName: RawUTF8;
fLastName: RawUTF8;
fVersion: TRecordVersion;
published
property FirstName: RawUTF8 read fFirstName write fFirstName;
property LastName: RawUTF8 read fLastName write fLastName;
property Version: TRecordVersion read fVersion write fVersion;
end;

54. mORMot’s RESTful ORM
 Master/Slave Replication
MasterServer := TSQLRestServerDB.Create(MasterModel,'master.db3');
HttpMasterServer := TSQLHttpServer.Create('8888',[MasterServer]);
MasterClient := TSQLHttpClientHTTP.Create(
'127.0.0.1',HTTP_DEFAULTPORT,MasterModel);
SlaveServer := TSQLRestServerDB.Create(SlaveModel,'slave.db3');
SlaveServer.RecordVersionSynchronizeSlave(
TSQLRecordPeopleVersioned,MasterClient);
From SQL to ORM

55. mORMot’s RESTful ORM
 Master/Slave Replication
MasterServer := TSQLRestServerDB.Create(
MasterModel,'master.db3');
HttpMasterServer := TSQLHttpServer.Create(
'8888',[MasterServer]);
MasterClient := TSQLHttpClientHTTP.Create(
'127.0.0.1',HTTP_DEFAULTPORT,MasterModel);
SlaveServer := TSQLRestServerDB.Create(
SlaveModel,'slave.db3');
SlaveServer.RecordVersionSynchronizeSlave(
TSQLRecordPeopleVersioned,MasterClient);
From SQL to ORM
Master
Slave
MasterServer
(MasterModel)
master.db3
MasterClient
(MasterModel)
HTTP
SlaveServer
(SlaveModel)
On Demand
Replication
slave.db3

56. mORMot’s RESTful ORM
 Master/Slave Real-Time Replication
 Push ORM modifications over WebSockets
From SQL to ORM

57. mORMot’s RESTful ORM
 Master/Slave Real-Time Replication
MasterServer := TSQLRestServerDB.Create(MasterModel,'master.db3');
HttpMasterServer :=
TSQLHttpServer.Create('8888',[MasterServer],'+',useBidirSocket);
HttpMasterServer.WebSocketsEnable(Server,'PrivateAESEncryptionKey');
MasterServer.RecordVersionSynchronizeMasterStart;
MasterClient := TSQLHttpClientWebSockets.Create(
'127.0.0.1',HTTP_DEFAULTPORT,MasterModel);
MasterClient.WebSocketsUpgrade('PrivateAESEncryptionKey');
SlaveServer := TSQLRestServerDB.Create(SlaveModel,'slave.db3');
SlaveServer.RecordVersionSynchronizeSlaveStart(
TSQLRecordPeopleVersioned,MasterClient);
From SQL to ORM

58. mORMot’s RESTful ORM
 Master/Slave Real-Time Replication
SlaveServer := TSQLRestServerDB.Create(
SlaveModel,'slave.db3');
SlaveServer.RecordVersionSynchronizeSlaveStart(
TSQLRecordPeopleVersioned,MasterClient);
From SQL to ORM
Master
Slave
MasterServer
(MasterModel)
master.db3
MasterClient
(MasterModel)
WebSockets
TCP/IP
SlaveServer
(SlaveModel)
Replication
slave.db3

59. mORMot’s RESTful ORM
 Master/Slave Replication – Multi Offices Synch
From SQL to ORM
Main Office
Office A Office B
Main
Server
External DB
Local
Server A
HTTP
Local
Server B
HTTP
Client 1 Client 2 Client 3
local
network
Client 1 Client 2 Client 3 Client 4
local
network

60. mORMot’s RESTful ORM
 Master/Slave Replication – Multi Offices Synch
From SQL to ORM
Main Office
Office A Office B
Main
Server
Local Data A
Reference
Read Only
Local Data B
Reference
Read Only
Local Data A
Business
Read/Write
Replication
Local Data B
Business
Read Only
Local Data A
Business
Read Only
Replication
Local Data B
Business
Read/Write

61. mORMot’s RESTful ORM
 Automated Data Sharding
 On production servers, databases may store:
 Roaming / user input data with mostly updates
resulting in a reasonable growing size of storage
 Events which are mostly added
resulting in an always increasing database
 You may reach out of disk space
and/or encounter performances issues
 Purge via periodic deletion is possible,
but may slow down the process
From SQL to ORM

62. mORMot’s RESTful ORM
 Automated Data Sharding
 On production servers,
you may define a table as “sharded”
 A TSQLRecord will be stored in its own set of databases
 Each database will contain up to a given number of items
(e.g. 500,000 entries)
 Only up to a given number of databases will be active
 And older databases may be archived/deleted instantly
just by packing or deleting the corresponding file
From SQL to ORM

63. mORMot’s RESTful ORM
 Automated Data Sharding
const SHARD_RANGE = 500000;
function TTestMemoryBased.CreateShardDB(
maxshard: Integer): TSQLRestServer;
begin
result := TSQLRestServer.CreateWithOwnModel(
[TSQLRecordTest],false,'shardtest');
Check(result.StaticDataAdd(TSQLRestStorageShardDB.Create(
TSQLRecordTest,result,SHARD_RANGE,[],'',maxshard)));
result.CreateMissingTables;
…
Will setup sharding for the TSQLRecordTest class
using local SQlite3 databases.
From SQL to ORM

64. mORMot’s RESTful ORM
 Automated Data Sharding
R := TSQLRecordTest.Create;
try
for i := 1 to 50 do begin
R.FillWith(i);
Check(db.AddWithBlobs(R)=i);
R.CheckWith(self,i);
end;
finally
R.Free;
end;
Data will be sharded on disk, using Test####.dbs files.
By convention, TSQLRecordTest →Test####.dbs
From SQL to ORM

65. mORMot’s RESTful ORM
 ORM² for DDD persistence service
 ORM for plain Delphi class storage
 Create TSQLRecord from any domain class
 Fields mapping (name, data)
 Aggregate flat / un-normalized orientation (no join)
From SQL to ORM

66. mORMot’s RESTful ORM
 ORM² for DDD persistence service
 ORM for plain Delphi class storage
 Create TSQLRecord from any domain class
 Fields mapping (name, data)
 Aggregate flat / un-normalized orientation (no join)
 Isolate persistence from business logic
 Built-in CQRS dual-phase commit service
 Object-level tuning from domain logic to storage logic
From SQL to ORM

67. mORMot’s RESTful ORM
 Persistence over External RDMS
 SynDB optimized data access layer
From SQL to ORM
SynDB
ZDBC ODBC OleDB Oracle SQLite3
DB.pas
TDataset
NexusDB BDE DBExpress
FireDAC
AnyDAC
UniDAC

68. mORMot’s RESTful ORM
 SynDB classes
 By-pass the DB.pas unit
avoids TDataSet bottleneck, works with Starter Edition
… but can still use it (e.g. FireDAC)
 Simple KISS designed API
 Less data types, interface-based, no legacy
 Statement cache, ORM aware
 Array binding (bulk insert / batch mode)
 Direct JSON generation
 Optional remote access via HTTP
From SQL to ORM

69. mORMot’s RESTful ORM
 SynDB integration with ORM
From SQL to ORM
TSQLRestServerDB.Add
TSQLRestServerDB.EngineAdd
internal
table
TSQLRestServerStaticExternal.EngineAdd
external
table
REST
TSQLRequest
INSERT INTO...
SQlite3 engine
internal engine
SQLite3 file
ISQLDBStatement
INSERT INTO...
External DB client
ODBC/ZDBC/OleDB...
External DB server

70. mORMot’s RESTful ORM
 SQLite3 Virtual Tables
From SQL to ORM

71. mORMot’s RESTful ORM
 SQLite3 Virtual Tables
 Mapping of external field names
 JOIN several external databases
 Very slight performance penalty
 Access TObjectList instances
 Access NoSQL databases
 By-passed if possible
 Full SQL-92 engine at hand
From SQL to ORM

72. mORMot’s RESTful ORM
 SQLite3 Virtual Tables
From SQL to ORM
mORMot
ORM
SQLite3
direct
TObjectList
direct
External DB
direct
virtual
Oracle SQLite3 ODBC OleDB ZDBC
direct
FireDAC AnyDAC UniDAC BDE DBExpress NexusDB
DB.pas
TDataSet

73. mORMot’s RESTful ORM
From SQL to ORM

74. mORMot’s RESTful ORM
From SQL to ORM

75. mORMot’s RESTful ORM
 Defining the object
From SQL to ORM
type
TSQLRecordPeopleExt = class(TSQLRecord)
private
fData: TSQLRawBlob;
fFirstName: RawUTF8;
fLastName: RawUTF8;
fYearOfBirth: integer;
fYearOfDeath: word;
fLastChange: TModTime;
fCreatedAt: TCreateTime;
published
property FirstName: RawUTF8 index 40 read fFirstName write fFirstName;
property LastName: RawUTF8 index 40 read fLastName write fLastName;
property Data: TSQLRawBlob read fData write fData;
property YearOfBirth: integer read fYearOfBirth write fYearOfBirth;
property YearOfDeath: word read fYearOfDeath write fYearOfDeath;
property LastChange: TModTime read fLastChange write fLastChange;
property CreatedAt: TCreateTime read fCreatedAt write fCreatedAt;
end;
ID : integer
Data : TSQLRawBlob
FirstName : RawUTF8
LastName : RawUTF8
YearOfBirth : integer
YearOfDeath : word
ID : INTEGER
Data : BLOB
FirstName : NVARCHAR(40)
LastName : NVARCHAR(40)
YearOfBirth : INTEGER
YearOfDeath : INTEGER

76. mORMot’s RESTful ORM
 Defining the Model and the Server:
Props := TOleDBMSSQLConnectionProperties.Create(
'.SQLEXPRESS','AdventureWorks2008R2','','');
Model := TSQLModel.Create([TSQLRecordPeopleExt],'root');
VirtualTableExternalRegister(Model,TSQLRecordPeopleExt,Props,'Test.People');
ServerDB := TSQLRestServerDB.Create(Model,'application.db'),true);
ServerDB.CreateMissingTables;
HttpServer := TSQLHttpServer.Create('8080',ServerDB);
 Defining the Model and the Client:
Model := TSQLModel.Create([TSQLRecordPeopleExt],'root');
Client := TSQLHttpClient.Create('localhost','8080', Model);
 Both will now communicate e.g. over
http://server:8080/root/PeopleExt
From SQL to ORM

77. mORMot’s RESTful ORM
 Refining the mapping Model on the Server:
Props := TOleDBMSSQLConnectionProperties.Create(
'.SQLEXPRESS','AdventureWorks2008R2','','');
Model := TSQLModel.Create([TSQLRecordPeopleExt],'root');
VirtualTableExternalRegister(Model,TSQLRecordPeopleExt,Props,'Test.People');
Model.Props[TSQLRecordPeopleExt].ExternalDB.
MapField('ID','Key').
MapField('YearOfDeath','YOD');
ServerDB := TSQLRestServerDB.Create(Model,'application.db'),true);
ServerDB.CreateMissingTables;
Server := TSQLHttpServer.Create('8080',ServerDB);
From SQL to ORM
ID : integer
Data : TSQLRawBlob
FirstName : RawUTF8
LastName : RawUTF8
YearOfBirth : integer
YearOfDeath : word
Key : INTEGER
Data : BLOB
FirstName : NVARCHAR(40)
LastName : NVARCHAR(40)
YearOfBirth : INTEGER
YOD : INTEGER

78. mORMot’s RESTful ORM/ODM
 Data Sharding / Denormalization pattern
 Store the whole aggregate as once
 Without JOIN
 From ORM to ODM
 Object Relational Mapping (ORM)
 Object Document Mapping (ODM)
From SQL to ORM

79. mORMot’s RESTful ODM
 Object Document Mapping
 ODM over regular RDBMS
 NoSQL storage via TDocVariant property
 Stored as JSON, handled as a variant
 ODM over dedicated NoSQL engine
 TObjectList
 MongoDB direct access
From SQL to ORM

80. mORMot’s RESTful ODM
 NoSQL storage via TDocVariant property
 Document is indexed:
 by TSQLRecord.ID: integer
 by Name: RawUTF8
From SQL to ORM
TSQLRecordData = class(TSQLRecord)
private
fName: RawUTF8;
fData: variant;
public
published
property Name: RawUTF8 read fTest write fTest stored AS_UNIQUE;
property Data: variant read fData write fData;
end;

81. mORMot’s RESTful ODM
 NoSQL storage via TDocVariant custom type
 Data: variant will store any document
 As JSON in the RDBMS
 Accessed via late-binding in Delphi code
From SQL to ORM
TSQLRecordData = class(TSQLRecord)
private
fName: RawUTF8;
fData: variant;
public
published
property Name: RawUTF8 read fTest write fTest stored AS_UNIQUE;
property Data: variant read fData write fData;
end;

82. mORMot’s RESTful ODM
 Data: variant will store a TDocVariant
 Schema-less data
{ name : "Joe", x : 3.3, y : [1,2,3] }
{ name : "Kate", x : "abc" }
{ q : 456 }
 Real-world evolving data
{ name : "Joe", age : 30, interests : "football" }
{ name : "Kate", age : 25 }
From SQL to ORM

83. mORMot’s RESTful ODM
From SQL to ORM
var aRec: TSQLRecordData;
aID: integer;
begin
// initialization of one record
aRec := TSQLRecordData.Create;
aRec.Name := 'Joe'; // one unique key
aRec.data := _JSONFast('{name:"Joe",age:30}'); // create a TDocVariant
// or we can use this overloaded constructor for simple fields
aRec := TSQLRecordData.Create(['Joe',_ObjFast(['name','Joe','age',30])]);
// now we can play with the data, e.g. via late-binding:
writeln(aRec.Name); // will write 'Joe'
writeln(aRec.Data); // will write '{"name":"Joe","age":30}' (auto-converted to JSON string)
aRec.Data.age := aRec.Data.age+1; // one year older
aRec.Data.interests := 'football'; // add a property to the schema
aID := aClient.Add(aRec); // will store {"name":"Joe","age":31,"interests":"footbal"}
aRec.Free;
// now we can retrieve the data either via the aID created integer, or via Name='Joe'
end;

84. mORMot’s RESTful ODM
From SQL to ORM
var aRec: TSQLRecordData;
aID: integer;
begin
// initialization of one record
aRec := TSQLRecordData.Create;
aRec.Name := 'Joe'; // one unique key
aRec.data := _JSONFast('{name:"Joe",age:30}'); // create a TDocVariant
// or we can use this overloaded constructor for simple fields
aRec := TSQLRecordData.Create(['Joe',_ObjFast(['name','Joe','age',30])]);
// now we can play with the data, e.g. via late-binding:
writeln(aRec.Name); // will write 'Joe'
writeln(aRec.Data); // will write '{"name":"Joe","age":30}' (auto-converted to JSON string)
aRec.Data.age := aRec.Data.age+1; // one year older
aRec.Data.interests := 'football'; // add a property to the schema
aID := aClient.Add(aRec); // will store {"name":"Joe","age":31,"interests":"footbal"}
aRec.Free;
// now we can retrieve the data either via the aID created integer, or via Name='Joe'
end;
 We just mutated a classical RDBMS
into a NoSQL schema-less storage engine!

85. mORMot’s RESTful ODM
 NoSQL Engines
 TObjectList - TSQLRestStorageInMemory
 Fast in-memory storage
 Supports hash indexes on (unique) properties
 Binary or JSON persistence as file
 Could embed some unpersisted live process
 MongoDB
 SynMongoDB.pas for direct access
 mORMotMongoDB.pas for ODM integration
From SQL to ORM

86. mORMot’s RESTful ODM
 SynMongoDB.pas
 BSON types, including BLOB, ObjectID, date/time…
 TBSONVariant with late-binding
 (Extended) JSON as input or output
 Document-level access
 Bulk document insertion
 Latest 3.2 compatibility (WiredTiger, scram-sha1)
 Could be used standalone, without the ODM
From SQL to ORM

87. mORMot’s RESTful ODM
 mORMotMongoDB.pas
 Full integration with mORMot’s REST
 Share the same logic code with RDBMS
 Just one line to change on the server side
to switch from an ORM to an ODM
 Same exact RESTful business code
From SQL to ORM

88. mORMot’s RESTful ODM
uses SynMongoDB, mORMotMongoDB;
Model := TSQLModel.Create([TSQLORM]);
ServerDB := TSQLRestServerDB.Create(Model,nil,':memory:');
MongoClient := TMongoClient.Create('localhost',27017);
MongoDatabase := MongoClient.Database[DB_NAME];
StaticMongoDBRegister(TSQLORM,ServerDB,MongoDatabase);
ServerDB.CreateMissingTables;
… usual ORM code
From SQL to ORM

89. mORMot’s RESTful ODM
 mORMotMongoDB.pas
 On-the-fly REST GET query translation
 SQL WHERE clause translated
to MongoDB query pipelines
 Including aggregate functions, limits, offset
aClient.OneFieldValue(TSQLOrders,'sum(price)','',[],aTotal);
SELECT SUM(price) AS total FROM orders
db.orders.aggregate([{$group:{_id: null,total:{$sum:"$price"}}}])
 Returned BSON is used to fill the TSQLRecord
From SQL to ORM

90. mORMot’s RESTful ODM
 mORMotMongoDB.pas
 On-the-fly REST GET query translation
 SQL WHERE clause translated
to MongoDB query pipelines
 Including aggregate functions, limits, offset
 Returned BSON is used to fill the TSQLRecord
 BATCH support
 As bulk document insertion
From SQL to ORM

91. mORMot’s RESTful ORM/ODM
 BATCH process
From SQL to ORM
// start the BATCH sequence
Check(ClientDist.BatchStart(TSQLRecordPeople,5000));
// delete some elements
for i := 0 to n-1 do
Check(ClientDist.BatchDelete(IntArray[i])=i);
// update some elements
nupd := 0;
for i := 0 to aStatic.Count-1 do
if i and 7<>0 then
begin // not yet deleted in BATCH mode
Check(ClientDist.Retrieve(aStatic.ID[i],V));
V.YearOfBirth := 1800+nupd;
Check(ClientDist.BatchUpdate(V)=nupd+n);
inc(nupd);
end;
// add some elements
V.LastName := 'New';
for i := 0 to 1000 do
begin
V.FirstName := RandomUTF8(10);
V.YearOfBirth := i+1000;
Check(ClientDist.BatchAdd(V,true)=n+nupd+i);
end;
// send the BATCH sequences to the server
Check(ClientDist.BatchSend(Results)=200);
ORM CRUD operation
ORM HTTP Client
In-process
no latency
ORM HTTP Server
Internet
100 ms latency
ORM database core
In-process
no latency

92. mORMot’s RESTful ORM/ODM
 BATCH process
 Truly ACID, even with several RDBMS backends
 Automatic transaction handling
 Avoids slow Client-Server roundtrips
 Unit-Of-Work pattern, even on Server side
From SQL to ORM

93. mORMot’s RESTful ORM/ODM
 BATCH process
 Optimized SQL
 Parameter Array Binding
 e.g. Oracle, ZDBC, FireDAC
 Multiple INSERT statement
 Depending on the database dialect
 Re-use of prepared statements
 NoSQL (MongoDB) bulk insertion
 Sent as array of BSON documents
 With ID pre-computation on client side
From SQL to ORM

94. mORMot’s RESTful ORM/ODM
 Asynchronous BATCH process
 Add pending writes into a list using:
TSQLRest.AsynchBatchAdd
TSQLRest.AsynchBatchUpdate
TSQLRest.AsynchBatchDelete
 A background thread will perform the BATCH
 when the internal list reaches a limit
or after some ms period
From SQL to ORM

95. mORMot’s RESTful ORM/ODM
 Asynchronous BATCH process
 Add pending writes into a list using:
TSQLRest.AsynchBatchAdd
TSQLRest.AsynchBatchUpdate
TSQLRest.AsynchBatchDelete
 A background thread will perform the BATCH
 when the internal list reaches a limit
or after some ms period
 Perfect for transparent event store
 May be associated with local Automated Data Sharding
or a remote MongoDB Big-Data store
From SQL to ORM

96. mORMot’s RESTful ORM/ODM
 Security
 Rooted on framework’s authentication(s)
 Secured by SHA-256 challenge to avoid MIM / replay
 Alternatives: basic, digest, SSPI/Kerberos, custom class
 Users belong to Groups
 Per-table CRUD access right for each group
 To be used as firewall, above application-level rights
 Optional JSON compression and encryption
 WebSockets bi-directional binary transmission e.g.
From SQL to ORM

97. mORMot’s RESTful ORM/ODM
 RESTful Client-Server
 In-process
 Stand-alone client, fast server-side access
 Named pipes or GDI messages
 Stand-alone client, fast server-side access
 HTTP/1.1 via kernel-mode http.sys API
 Part of the OS since Windows XP SP2, used by IIS and WCF
 Kernel-mode execution, IOCP driven
 System-wide URI registration: share root and port
 Socket-based server is also available, e.g. under Linux
 Optional query/answer REST emulation over WebSockets
From SQL to ORM

98. mORMot’s RESTful ORM/ODM
 Cross-platform Clients
 Generated client code using Mustache templates
 Delphi: Windows, MacOS, Android, iOS
 FPC: Windows, MacOS, Android, iOS, Linux, …
 (Cross)Kylix: Linux
 SmartMobileStudio, EWB: Ajax / HTML5
 Featuring almost all framework abilities
 JSON, security, TSQLRest, TSQLRestBatch
From SQL to ORM

99. From SQL to ORM