Web frameworks help you build an API quickly but most have little support for dealing with an API that needs to evolve, forcing you to prematurely version your API. But many industry professionals are telling us not to version. How can we avoid it? Take back control of the content you send over the wire. API responses are the "user interface" of your API and should be crafted with same attention to detail that cause designers to fret over color choices, shadows and highlights. In this talk I’ll show techniques that can be used to build responses that are easier to evolve and highlight the types of practices that encourage breaking changes and force you to version your API.
5. We have been here before
• CORBA, DCOM
• SOAP, WSDL
• DTOs
• JSON
6. The ASP.NET Web API Project Template
public class ValuesController : ApiController
{
// GET api/values
public IEnumerable<string> Get() { return new string[] { "value1", "value2" };}
// GET api/values/5
public string Get(int id) { return "value"; }
// POST api/values
public void Post([FromBody]string value) { }
// PUT api/values/5
public void Put(int id, [FromBody]string value) { }
// DELETE api/values/5
public void Delete(int id) { }
}
7. The ASP.NET Web API Starter Tutorial
public class ProductsController : ApiController
{
//…
public IEnumerable<Product> GetAllProducts()
{
return products;
}
public IHttpActionResult GetProduct(int id)
{
var product = products.FirstOrDefault((p) => p.Id == id);
if (product == null)
{
return NotFound();
}
return Ok(product);
}
}
http://www.asp.net/web-api/overview/getting-started-with-aspnet-web-api/tutorial-your-first-web-api
13. Which objects to map?
• Domain objects
• How do we hide content we don’t want to expose?
• How do we create different views of data?
• Changes to domain model cause changes to API
• DTOs
• whole lot of busy work
• Only indirect control over serialization process
14. Automatic Serialization
• All the properties
• Missing values, unrecognized properties
• Non standard data types: datetime, timespan
• Does null mean unspecified, or explicitly null?
• Empty collection or no collection
• Capitalization
• Links
• Cycles
• Changes to behavior in the frameworks
• Security Risks
15. The whole app is useless
because the API added a new
preference
21. Anatomy of an HTTP representation
200 OK HTTP/1.1
Server: Microsoft-HTTPAPI/2.0
Content-Length: 44
Content-Type: text/plain
Acme-perf-database-cost: 120ms
The quick brown fox jumped over the lazy dog
24. {
"description" :"There is a hole in my bucket",
"steps_to_reproduce" : "Pour water in bucket. Lift bucket off
ground. Look for water dripping",
"date_reported": "2012-04-21T18:25:43-05:00"
}
Just enough data to solve the problem
25. {
"description" :"The font is too big",
"application_name" : "Wordament",
"application_version" : "1.2.0.2392",
"environment_osversion" : "NT4.0",
"environment_memory_free" : "100 MB",
"environment_diskspace_free" : "1.5 GB",
"reported_by_user" : “Bob Bing”
}
Why do they need that data?
26. {
"description" :"The font is too big",
"application_name" : "Wordament",
"application_version" : "1.2.0.2392",
"environment" : { "osversion" : "NT4.0",
"memory_free" : "100 MB",
"diskspace_free" : "1.5 GB"
}
}
Attribute Groups
27. {
"description" :"The font is too big",
"history" : [
{"status" : "reported", "date" :"2014-02-01"},
{"status" : "triaged", "date" :"2014-02-04"},
{"status" : "assigned", "date" :"2014-02-12"},
{"status" : "resolved", "date" :"2014-02-19"},
]
}
Attribute Groups for multiple instances
28. {
"description" :"The font is too big",
"reported_by_user" : { "name" : "Bob Bing",
"email" : "bob@acme.com",
"twitter" : "@bobbing",
"date_hired" : "2001-01-21"
}
}
Attribute Groups for related data
29. {
"description" :"The font is too big",
"reported_by_user_url" : "http://api.acme.com/users/75"
}
Linking related data
30. {
"description" :"The font is too big",
“links" : [
{ "href" :"http://api.acme.com/users/75", "rel": "reportedbyuser" },
{ "href" :"http://api.acme.com/users/24", "rel": "assignedtouser" }
]
}
Multiple Links
31. {
"issues" : [
{
"description" :"There is a hole in my bucket"
}
]
}
Beware of structural changes
{
"issue" :
{
"description" :"There is a hole in my bucket"
}
}
Item in a Array
Item as a Property
32. Naming Your Conventions
• Empty arrays
• Nulls
• Single item as array or object
• Camel case, snake case
• Vocabulary
• Protocols
36. {
"collection": { "links": [],
"items": [
{
"data": [
{ "name": "Title",
"value": "rntttLearning from Noda Time: a case study in API design and open source (good, bad and ugly)rntt“ },
{ "name": "Timeslot",
"value": "04 December 2013 16:20 - 17:20“ },
{ "name": "Speaker",
"value": "Jon Skeet“ }
],
"links": [ {
"href": "http://conference.hypermediaapi.com/speaker/6",
"rel": "http://tavis.net/rels/speaker" }, {
"href": "http://conference.hypermediaapi.com/session/133/topics",
"rel": "http://tavis.net/rels/topics" }
],
"href": "http://conference.hypermediaapi.com/session/133"
}
],
"query": [], "template": { "data": [] },
"version": "1.0"
}
}
Meet application/vnd.collection+json
37. Version as a Last Resort
• Version the payload
• HTML DOCTYPE, Collection+Json
• Version the resource
• /api/documents/invoice.v2/758
• Version the media type
• application/vnd.github.v3+json
• Version the API
• /api/v2/documents/invoice/758
38. Wrap up
• Understand the limitations of “objects over the wire”
• Consider taking back control of your representations
• Think in terms of messages, instead of objects
• Build software that is designed to survive change
• Believe that versioning is an admission of failure
- Developer advocate for Runscope.
- Cloud based solutions for API performance monitoring
Microsoft MVP
Book
Two parts to API design, resource identification and representation design
designing representations aka HTTP response.
- Versioning sucks
So many ways to do it
Starting to hear the message, don’t version
Roy says middle finger
Layer7 say don’t.
The problem is we all make mistakes.
- Why do we think we need it
One reason we think we need versioning is because we do
Objects over the wire
Pass object graph
Infrastructure converts it
Hope the client understands it
Pre-arranged agreement between client/server
The details are key to avoiding versioning.
But Web APIs are not the same as local APIs
Objects over the wire. The idea is that you construct some kind of object graph in your favourite language in your web application, that contains the data that you want to deliver to the clients. And then you pass that object graph to some piece of infrastructure code that converts it into a wire format that hopefully will be consumable by the client. The challenge is, for this to work, there has to be some kind of pre-arranged agreement between sender and receiver. The details of that agreement are the key to avoiding versioning.
- Trying to do this since the mid 1990’sCORBA, DCOM
- When the web won, SOAP was invented
simplified to DTOs
- REST was rediscovered and redefined to use JSON to send objects over the wire
JSON has helped. It makes the easy stuff really easy. But the more work we do in JSON the more we are starting to see the re-invention of the complexity. JSON-Schema, JSON Pointer, JSON Path, patch formats, namespacing.
Fallacy of REST has no contracts. We are just pretending they are there, but they are implicit and unwritten.
So what does “objects over the wire” look like in code, in the current incarnation.
We have been trying to do this since the mid 1990’s. CORBA, DCOM were attempts to allow access to remote objects.
When it was recognized that the web has won, SOAP was invented to try and do Objects over the wire on HTTP
Some smart people realized that “objects over the wire” was never really going to succeed so they simplified to DTOs over the wire.
Types over the wire, but sharing types creates coupling. SOAP used a similar concept but called them data contracts
Only works if you have tight control over both sides of the wire.
- The default behavior is to return CLR types
let the framework decide how to convert it to representation on the wire
What does Ienumerable<string> look like on the wire? Does every platform do it the same way?
What does string look like on the wire? JSON spec has changed recently for simple values.
POSTing string. So many StackOverflow questions on posting string of XML that doesn’t work.
The worlds simplest API and multiple interoperability issues.
In this example we move from returning a native CLR data type to a custom object. Product
Note IHTTPActionResult
Starts to break down when the information we are trying to send is not in the payload
Returning a object limits how we can craft the response.
But this is not just Microsoft doing silly things
What are the odds that List<Reqstars> creates the same wire representation as List<Reqstars> in Web API
Returning objects is hiding a critical information needed to ensure interoperability
Even the really cool frameworks.
I wonder what this will look like on the wire.
It is a very common pattern.
Every framework provider will tell you, “oh but you can customize the response if you want”
…but that’s not where they lead you.
It is also not a problem that is limited to the .net space.
This python example is slightly better, because at least we know it is going to be sent as a JSON representation
and the mapping from objects to JSON is fairly straight forward
And just as one last example, rails does the same thing.
So what, you say… most of those are minor implementation details.
What’s the fundamental problem with objects over the wire?
Even with DTOs, you only have indirect control over the serialization process
If you address supports suite no you are going to get it whether there is a value or not.
Sending data, what does it expect? Mandrill
How do you format dates? iJson was a new standard developed to help interoperability
In a geneology app, if Date of death is null, are we missing that data or is he just not dead yet?
How you handle links in your server implementation is your business, how you format them on the wire is everyone’s business
.Net objects don’t follow capitalization rules of JSON. What about word separation?
Cycles can cause some serializers to die randomly.
The great YAML fiasco
When dealing with Automatic Serialization there are so many unknowns
Being aware of what are the rules of your representations is absolutely critical.
When you know it is a rule, you start to think about the long term effects and the cross platform concerns.
When you are “just serializing your objects as JSON” you get a false sense of security.
This is what happens when you are not clear about your contract.
The poor guys writing the serializers. They break people when they fix things.
What’s a DOM….
- Here is just one example of using the JSON.NET Jobject and dynamic.
- There are many ways to do this.
- helper methods allow you to establish conventions in your representation formats.
- Instead of domain objects to DTOs,
- Do domain objects/linq projections to DOM.
- You define your own conventions instead of a serializer doing it for you.
And maybe you don’t even have to define your own conventions, Maybe the conventions of an existing media type meet your needs
The problem with not using serializers, now you have the problem of having think about how to structure your responses.
So, let’s start with some fundamentals.
Clear up confusion around terms.
Mapping Resources to entities can be limiting to your design.
A different URI should almost always be a different resource.
Ideally if you want two URLs for the same resource then one should redirect to the other.
Prevents cache pollution
A Resource can have multiple representations, or just a single one.
There is nothing wrong with creating distinct resources for available formats
Resources can be completely dynamic concepts
You can change that reason phrase…
Headers are metadata about the response
Performance – Headers can be processed and interpreted without having to interpret the entire request body. Cross cutting concerns
Timing – Headers arrive first and can actually be used to abort the reception of the body, or prepare for the body, whilst downloading bytes
Naming – don’t use x-, do use company- for custom headers
Content - Links, json, Unicode?
In the future http/2 will support header compression
Often at the root of an API. Another example : Atom service document
Building an Issue Tracking System
Really simple minimal documents make it really easy to get started
- Optional attributes can be added to the root object
- Consider the use-case, not the server object model.
- What would a serializer do?
- Objects include everything. Guide client devs down the right path
- You can always add, it’s much hard to take away.
- There are reasons to group attributes, until you have a reason, probably not worth it. Structural changes are tricky.
- Clients shouldn’t depend on order
- Name it based on most natural conventions for the format
How much semantics does the client really need? Is it going to do calcs on the memory free?
Consider cultural issues, like formatting, language.
Should a date be pretty formatted and localized?
Should an address be concatenated or separate fields
Should a name be two fields, or formatted firstname + lastname or last name comma first name
Grouping together can have a variety of advantages
saved a few bytes
potentially more human readable
allows definition of mandatory fields within a group, without the group itself being mandatory
- group becomes a candidate for linking instead of embedding.
It might map to an object, it might not.
Groups are useful for supporting multiple instances of a set of data.
Embed related data to reduce round trips
When updating an issue, we would not update the contents of the reportedByUser.
- Related information can also be included by providing a link.
- Consider volatility and reusability of data.
- beware of embedding short lived data inside long lived data.
// Links can be very handy for pointing at information that changes rarely and can easily be cached locally.
// There are many ways of representing links. Sometimes a link is presented as an object.
Multiple objects can be displayed like this…
Making a your concept a property allows for a issue.v2,
…or allows for defining a media type that can hold many different concepts
Sometimes you might want to make something a collection, even though today there is only one.
Don’t make the root of your JSON object an array (must Ignore policy)
“Media Type” is the name you give to your conventions
Content-Type is the header to identify conventions to the client
There is a lot of nitty gritty details to think about when defining a media type from scratch
Better to layer new types on old ones.
Profile is an attempt to separate the efforts.
Distinction between what is here and what is elsewhere