Data Modeling in the API Economy

Data Modeling in theAPI Economy
PRESENTATION TO DATA MODELING ZONE, OCTOBER 2014
TED EPSTEIN, FOUNDER AND CEO
MODELSOLV, INC.
COPYRIGHT © 2014, MODELSOLV, INC. | ALL RIGHTS RESERVED. 1

All Eyes on APIs
Paradigm shift with integration architecture, happening on two fronts:
◦REST as a recommended architectural style for internal services
◦Emergence of the API Economy

Internal REST
Web-Style REST APIs are replacing SOAP for internal services.
REST brings some useful design constructs for data-oriented APIs.
REST is another wave of change in integration architecture
◦New trends bring new projects, modernization programs
◦A chance to improve on whatever we did before.
◦In this case, a chance to improve on SOA
We need you!
◦Large scale integration needs data modelers!
◦We need to make the case for that, and get our IT organizations on a better path.
◦This talk will help get you started.

Emergence of the API Economy
APIs have entered the mainstream of business and marketing awareness.
Business can profit from APIs
◦As products and services.
◦As sales channels.
◦As points of customer engagement.

Massive Integration on Web and Mobile
Everything integrating with everything else
◦Mashups
◦Common Services
◦Identity (Google, Facebook, Twitter, OpenID)
◦Transactions (PayPal, Google Wallet, Amazon Payments)
◦Location (Google Maps)
◦Document Sharing (Dropbox, Google Drive, SkyDrive)
◦Social (LinkedIn, Facebook, Twitter)

Integration Trends in Business and Government
Cloud/SaaS
Big Data
Mobility
Internet of Things
Open Government

Mapping the Service Landscape

Internal and External Services
Internal: We're somewhere on the REST adoption curve.
◦REST is the new SOA
◦No, it's an evolution of SOA
◦No, it's different, for this reason or that reason
◦Just don't call it SOA, because we don't like SOA
◦Fine, but it's still the new SOA
◦... and that's OK. Really.
External: We're jumping on the API bandwagon.
◦May be headed by marketing
◦May be a separate technology team

Meeting in the Middle
source: http://www.soatothecloud.com/
There is no hard line:
◦We know that some private APIs will become public.
◦We don’t know which ones, or when.
◦We should be converging towards a common architecture for internal & external services.

What's govgot to do with it?
Why do service APIs need “governance”?
APIs are data conduits:
◦Services account for a significant portion of data flow
Major technology initiatives are integration-intensive:
◦Mobile, Big Data, Cloud Computing /SaaS
◦Regulatory compliance
Integration is the #2 challenge in Mobile and SaaS:
◦#1 is data security, #2 is integration
sources: MuleSoftand SnapLogicsurveys

Why is integration such a headache?
Our systems are not speaking the same language.
Every time a new team builds a new API, the IT landscape gets more complex.
Billions spent on remedial solutions to aggregate disparate data, but we have a leaky roof.
◦Increasing complexity
◦Poor data quality
◦High integration cost
◦Slow time to market

What are we doing to stop these leaks?
This is why data modelers need to get involved.
We need unified canonical data models in our APIs.
We need data quality constraints.
Governance and Collaboration: two complementary paths to convergence.

REST Standards - Exploring the Stack

A Simple Data Model
structureTaxFilingid : stringyear : gYearnetTax: decimalcurrency : stringtaxpayer : referencetoPersonstructurePerson taxpayerID: stringlastName: stringfirstName: stringaddresses : referencetoAddress* structureAddress street1 : stringstreet2 : stringcity : stringstateOrProvince: stringpostalCode: string

REST: An Architectural Style
Some Essential Modeling Constructs:
◦Resources
◦Identifiers
◦Representations
◦Media Types
◦Methods
◦Hyperlinks

Fielding Disclaimer
Strict REST vs. REST in Practice
◦URIs: opaque vs. transparent
◦Out-of-Band Information: media types, link relations vs. documentation
◦Processing Model: hypertext-driven vs. client-driven
http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven

Data Model as Resource Model
CollectionResource
Data Type
ObjectResource

HTTP
Some Essential HTTP Modeling Elements:
◦URIs
◦Methods
◦Headers
◦Status Codes

REST/HTTP Resource Model
URI Template
Method
Request
Response
Status Code

Media Types
About Media Types
◦MediaType(Resource) --> Representation
◦Appears in the content-type HTTP header
◦Used for content negotiation
◦Standard media types registered at: http://www.iana.org/assignments/media-types/media-types.xhtml

Low-Level Media Types
Essentially Wire Formats
◦Technology-specific
◦Domain-neutral
Examples:
◦application/xml
◦application/json
many others
◦text/html
◦image/jpeg
◦application/protobuf
◦…

Generic Structured Media Types
What are they?
◦Data-oriented media types a level above basic wire formats
◦Many of these basic formats are not hypermedia-aware
◦Intended to express generic structured data, but with some additional abstractions: Hyperlinks, embedded representations, collections, items…
◦Different media types will emphasize different things

HAL

Sidebar: Hyperlinks
◦Hyperlinks make REST genuinely different.
◦Hyperlinks are your friend.
◦A better way to model relationships
◦Tends to de-emphasize big, deeply nested messages
◦Helps neutralize some of the perspective mismatch problem
◦Hides identifiers in hyperlinks
◦Isn't that expensive?
◦Cache can go a long way
◦When to embed vs. hyperlink:
◦For network efficiency (even with caching)
◦For client developer convenience, if the embedded data will most often be required
◦For encapsulation (Think DDD Composites)

Hyperlinked Resource Model
Hyperlink

Domain-Specific Metadata
Options
◦Domain-Specific Media Types (Sun Cloud, Atom)
◦Schemas (XSD, JSON, etc.)
◦Profiles
◦Microformats
◦Documentation

Diagram with Domain-Specific Data

Sidebar: Schemas and REST
A cultural issue: uncomfortable relationship between REST and formal domain-specific data contracts.
XML Schema was a sore spot for SOA.
Domain-specific data contracts present a dilemma for strict REST:
◦Out-of-band information is supposed to be concentrated in media types and link relations,
◦... but we don't want an explosion of media types.
REST is silent on the question of what to do with domain-specific data.
REST is silent on whether your payload can or should include a schema.
An opinionated perspective:
◦There's always a schema, whether you document it or not.
◦There are new options (JSON Schema, technology-independent schemas).
◦Defining a data contract formally just makes sense, much of the time.

API Description Languages
Options
◦Swagger
◦API Blueprint
◦RAML
What they have in common
◦Allow modeling of resources, data, methods, links, etc.
◦Public domain specifications, open-source web-based tooling.
◦Generate interactive documentation / testing sandbox.
◦Integration libraries and code generators.
◦Highly recommended as learning tools, to help you explore how REST APIs are put together.

Beyond REST as a Wrapper

REST as a Silver Bullet
The fantasy:
◦Exactly one resource for everything
◦The golden source is the only source
◦Everything perfectly hyperlinked
The Reality
◦One size fits none
◦Need to balance centralization and agility
◦Different APIs for different functions

API as a Domain Model

API as a workflow model
Affordances:
◦Action-oriented hyperlinks
◦Transaction-oriented resources
◦Navigational links to related resources
Workflow and affordances vary by service context.
Variations often require different APIs for different clients.

API as a Client View
It’s not necessarily just a domain model dump.
It’s the domain as the client sees it: Respect the Client’s vocabulary
Points of Divergence:
◦Property subsets
◦Flattening
◦Renaming
◦Change of perspective.
◦Reification: example, refund

Canonical Models and Variations
There’s acommon theme (canon) underneath these variations.
See how far you can go with canonical models and realizations:
◦Can we describe the theme and variations separately?
◦Can we model the variations as adaptations, augmentations of the theme?
◦If so, maybe we don’t needAPI-specific data types

Realization: Property Subset
API requests or responses may only need a subset of properties defined in the canonical model.
Realization model may specify a list of included properties.

Realization: Perspective
Message and resource structures project different views from the same logical data model
Canonical model should support bi-directional references.
Realization model should allow embedded or linked representations.

Realization:
Metadata
Business Information Model
Account ID
Balance
Margin
Status
Account
Party ID
Name
Party
1
0..n
Data Aspects
· Deltas
· Data Source
· Data Security
· Explicit Null Values
...
Message Structure
<party dataSource=“MSDB”>
<partyId>123</partyId>
<partyName xsi:nil=“true” nullValue=“Not Available” />
<accounts>
<account dataSource=“A2” transType=“insert”>
<accountId>XYZ</accountId>
<balance xsi:nil=“true” isRestricted=“true” />
…
</account>
…
</accounts>
</party>
APIs may need to augment
essential data with descriptive
metadata.
Data aspects are cross-cutting
concerns that may be woven
together with canonical data
as part of the interface
realization.

Realization: Contextual Constraints
<=$10MM
Asset Class = Bond
Trade
Services may have specific constraints that are not intrinsic to the data definitions.
Realization model may specify constraints on requests or responses. Constraints may take different forms: range, subtype, logical expression, etc.

Conclusion: Get involved!
We need your modeling skills.
Done right, new API initiatives can have huge positive impact.
Done poorly, they can actually make things worse.
◦Net Negative Contributing API(NNCAPI)
◦… or Convergence-Resistant API ;-)

Resources
REST:
◦Basic REST in action: http://www.youtube.com/watch?v=7YcW25PHnAA
◦HTTP and REST: http://code.tutsplus.com/tutorials/a-beginners-guide-to-http-and-rest--net-16340
◦REST API Design Principles: http://www.infoq.com/articles/rest-introduction
◦REST Idioms and Patterns: http://www.narwhl.com/
API Blogs
◦http://apievangelist.com
◦http://programmableweb.com

Resources (continued)
Canonical Models:
◦http://www.slideshare.net/TedEpstein/canonical-modeling-for-api-interop
◦http://www.digitalml.com/resources/Service_Design_Forum
API Description Languages
◦http://swagger.io/
◦http://apiblueprint.org/
◦http://raml.org/
◦http://modelsolv.com/reprezen

Questions
THANK YOU!
ted.Epstein@modelsolv.com

Data Modeling in the API Economy

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (19)

Destacado

Destacado (8)

Similar a Data Modeling in the API Economy

Similar a Data Modeling in the API Economy (20)

Último

Último (20)

Data Modeling in the API Economy