Discovering, composing, testing and modeling web APIs , especially OpenAPI, REST and OData ones

1. Model-driven Round-trip
Engineering of REST APIs
Hamza Ed-douibi
May 13, 2019
Supervised by : Jordi Cabot and
Javier Luis Cánovas Izquierdo

2. 1
AGENDA
3
4
5
MOTIVATION
APPROACH
CONTRIBUTIONS
SUMMARY & FUTURE WORK
2 PROBLEM STATEMENT

3. MOTIVATION

4. REST
Representational state transfer (REST)
ProgrammableWeb:
more than 21 000 APIs

5. 5 Motivation

6. 6 Motivation
Not easy to understand
Hard to maintain
No automatization

7. 7 REST APIs
D e s c r ip t i o n f o r m a t s / p r o t o c o l s
OData

8. …consortium of forward-looking
industry experts who recognize
the immense value of
standardizing on how REST APIs
are described…
…is focused on creating,
evolving and promoting a vendor
neutral description format…
https://www.openapis.org

9. 9 OpenAPI
E x a m p l e http://petstore.swagger.io/v2/pet/123GET
"info": {
"description": "This is … ",
"version" : "1.0.0",
"title": “…",
"contact": {…},…
}
Metadata
{
"swagger": "2.0",
"info": {
"version": "1.0.0",
"title": "Swagger Petstore",
"license": {
"name": "MIT"
}
},
"host": "petstore.swagger.io",
"basePath": "/v1",
"schemes": [
"http"
],
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"paths": {
"/pets": {
"get": {
"summary": "List all pets",
"operationId": "listPets",
"tags": [
"pets"
],
"parameters": [
{
"name": "limit",
"in": "query",
"description": "How many items to return at one
time (max 100)",
"required": false,
"type": "integer",
"format": "int32"
}
],
"responses": {
"200": {
"description": "An paged array of pets",
"headers": {
"x-next": {
"type": "string",
"description": "A link to the next page of
responses"
}
},
"schema": {
"$ref": "#/definitions/Pets"
}
},
"default": {
"description": "unexpected error",
"schema": {
"$ref": "#/definitions/Error"
}
}
}
},
OpenAPI definition
"paths":{
"/pet/{petId}":{
"get":{
"parameters":[
{ "name":"petId",
"in":"path",...
}
],
"responses":{
"200":...,
},...
},...
},
Operations
"definitions":{
"Pet":{
"type":"object",
"properties":{
"id":...,
"category":...,
"name":...,
...
} }
Schema
{
"id": 123,
"category": {
"id": 1,
"name": "dogs"
},
"name": "doggie",
"photoUrls": [
"http://example.com"
],
"tags": [
{
"id": 1,
"name": "black"
}
],
"status": "available"
}

10. 10 OpenAPI
A d v a n t a g e s
Generate code
Generate
documentation
Swagger UISwagger Codegen
Holistic view of
functionality
Automatization

11. PROBLEM
STATEMENT

12. 12 Problem statement
Composition
Awesome!….but
Many resources without APIs
Many APIs without OpenAPI
definitions
Testing REST APIs is hard
Limited support for generating
REST APIs
Limited composition approaches
NO
APIfication
Discovery
Testing
Generation
Composition

13. APPROACH

14. 14 Approach
uses

15. 15 Model-driven Engineering
M D E - b a s e d s o f t w a r e d e v e l o p m e n t p r o c e s s
Original model
… CODE
Software code1 refinementst n refinementth
Model-to-model
Transformation
Model-to-text
Transformation
CODE
CODE

16. 16 Approach
uses

17. 17 OpenAPI
T h e O p e n A P I m e t a m o d e l

18. 18 OpenAPI
T h e O p e n A P I p r o f i l e

19. CONTRIBUTIONS

20. APIDiscoverer
Example-driven discovery of Web API
specifications

21. 21 APIDiscoverer

22. 22 APIDiscoverer
M o t i v a t i o n
Machine-readable description
(barely) Human-readable
descriptions

23. 23 APIDiscoverer
A p p r o a c h
APIDiscoverer
Discovery &
Enrichment
Generation
OpenAPI
model
Example 1
call 1
response 1
Example
provision
Example 2
call 2
response 2
Example
provision
Example n
call n
response n
Example
provision
OpenAPI
definition

24. 24 APIDiscoverer
G e n e r a l v i e w o f t h e a p p r o a c h
OpenAPI
definition
Structural Discoverer
Behavioral Discoverer
OpenAPI metamodel
OpenAPI model
UML modelJSONDiscoverer UML2Schema
Conforms to
OpenAPI
generator
Example Provision Discovery Generator
API call
examples
{
"request": {
"method" : "POST" ,
"url" : "http://petstore... ",
"body" : { ... }
},
"response" : {
"status" : 200,
"body" : { ... }
}
}
{
"request": {
"method" : "GET" ,
"url" : "http://petstore... ",
"body" : { ... }
},
"response" : {
"status" : 200,
"body" : { ... }
}
}
{
"request": {
"method" : "GET" ,
"url" : "http://petstore... ",
"body" : { ... }
},
"response" : {
"status" : 200,
"body" : { ... }
}
}

25. 25 APIDiscoverer
D i s c o v e r y
OpenAPI
definition
Structural Discoverer
Behavioral Discoverer
OpenAPI metamodel
OpenAPI model
UML modelJSONDiscoverer UML2Schema
Conforms to
OpenAPI
generator
Example Provision Discovery Generator
API call
examples

26. 26 APIDiscoverer
D i s c o v e r y & E n r i c h m e n t
Behavioral Discoverer
Source Target
host
basepath
protocol
API
relativePath Path
httpMethod
request-body
response-body
Operation
Query
relative-Path
request-Body
Parameter
response-Code Response
request-body
response-body
Schema
http://petstore.swagger.io/v2/pet/123GET
{
"id": 123,
"category": {
"id": 1,
"name": "dogs"
},
"name": "doggie",
"photoUrls": [
"http://example.com"
],
"tags": [
{
"id": 1,
"name": "black"
}
],
"status": "available"
}

27. 27 APIDiscoverer
D i s c o v e r y & E n r i c h m e n t
Structural Discoverer
JSONDiscoverer
Behavioral Discoverer
Source Target
host
basepath
protocol
API
relativePath Path
httpMethod
request-body
response-body
Operation
Query
relative-Path
request-Body
Parameter
response-Code Response
request-body
response-body
Schema
{
"id": 123,
"category": {
"id": 1,
"name": "dogs"
},
"name": "doggie",
"photoUrls": [
"http://example.com"
],
"tags": [
{
"id": 1,
"name": "black"
}
],
"status": "available"
}

28. 28 APIDiscoverer
D i s c o v e r y & E n r i c h m e n t

29. 29 APIDiscoverer
O p e n A P I g e n e r a t o r
OpenAPI
definition
Structural Discoverer
Behavioral Discoverer
OpenAPI metamodel
OpenAPI model
UML modelJSONDiscoverer UML2Schema
Conforms to
OpenAPI
generator
Example Provision APIDiscoverer Generator
API call
exxamples
{"swagger":"2.0",
"info":{ },
"host":"petstore.swagger.io","basePath":"/v2",
"tags":[ "pet" ],"Schemes":[ "http" ],
"paths":{
"/pet/{petId}":{
"get":{
"produces":["application/json"],
"parameters":[{"name":"petId","in":"path","type":"integer"}],
"responses":{
"200":{
"description":"OK",
"schema":{"$ref":"#/definitions/Pet"
}}}}
}},
"definitions":{
"Pet":{
"type":"object",
"properties":{
"id":{"type":"integer"},
"category":{"$ref":"#/definitions/Category"},
"name":{"type":"string"},
"photoUrls":{"type":"array","items":{"type":"string"}},
"tags":{"type":"array","items":{"$ref":"#/definitions/Tag"}},
"Status":{"type":"string"}},
}}}

30. 30 APIDiscoverer
T o o l s u p p o r t

31. 31
Discover other aspects: non-functional properties,
semantic definitions, security mechanisms
Discovery
extensions
Textual analysis of API documentation websites to
infer call examples
Generation of
call examples
Support OpenAPI 3OpenAPI 3
APIDiscoverer
F u t u r e w o r k

32. APITester
Specification-based generation of test cases
for REST APIs
John Greim—Getty Images

33. 33 APITester

34. 34 APITester
M o t i v a t i o n

35. 35 APITester
M o t i v a t i o n
Leveraging OpenAPI for
testing
• High coverage level
• Parameter input generation
Limited coverage level
Limited parameter input
generation

36. 36 APITester
A p p r o a c h
OpenAPI
definition
OpenAPI metamodel
TestSuite metamodel
Testing
preparation
Model
Extraction
Model
trans.
Code
Gen.
CODE
CODE
Java
Junit
tests

37. 37 APITester
A p p r o a c h
OpenAPI
definition
OpenAPI metamodel
TestSuite metamodel
Testing
preparation
Model
Extraction
Model
trans.
Code
Gen.
CODE
CODE
Java
• Simple parameter (examples, default values, enums)
• Dummy parameter
• Complex parameter (response of an operation)
Inference rules
Junit
tests

38. 38 APITester
A p p r o a c h

39. 39 APITester
A p p r o a c h
OpenAPI
definition
OpenAPI metamodel
TestSuite metamodel
Testing
preparation
Model
Extraction
Model
trans.
Code
Gen.
CODE
CODE
Java
Junit
tests

40. 40 APITester
T e s t S u it e m e t a m o d e l

41. 41 APITester
A p p r o a c h
OpenAPI
definition
OpenAPI metamodel
TestSuite metamodel
Testing
preparation
Model
Extraction
Model
trans.
Code
Gen.
CODE
CODE
Java
Junit
tests

42. 42 APITester
A p p r o a c h
GR 1. Nominal test case GR 2. Faulty test
case

43. 43 APITester
A p p r o a c h
GR 1. Nominal test
case
GR 2. Faulty test
case

44. 44 APITester
A p p r o a c h
OpenAPI
definition
OpenAPI metamodel
TestSuite metamodel
Model
Extraction
Model
Extraction
Model
trans.
Code
Gen.
CODE
CODE
Java
Junit
tests

45. 45 APITester
A p p r o a c h

46. 46
Increase coverage levels by improving our parameter
inference technique
Coverage
Support OpenAPI 3
Integration
testing
APITester
F u t u r e w o r k

47. APIComposer
Model-based composition of REST APIs
John Greim—Getty Images

48. 48 APIComposer

49. 49 APIComposer
OData request
OData response
End-users
OpenAPI
OData

50. 50 OData
OData is a standardized protocol for
creating and consuming data APIs
Resources are defined in
a data model
D a t a m o d e l
URIs to identify resources
and query data
Q u e r y
JSON and XML/Atom to
de/serialize data
S e r i a l i z a t i o n
OData

51. 51 APIComposer
P r o c e s s

52. GET http://restcountries/alpha/ES
GET Battuta.medunes.net/region/ES/all
{"@odata.context":"$metadata#Country",
"name": "Spain",
"code": "ES",
"population": 46538422,....
"regions":[
{"name": "Andalucia"},
{"name": "Aragon"},
{"name":"Canary Islands"},,...]
}
"name": "Spain",
"code": "ES",
"population": 46538422,....
{"name": "Andalucia"},
{"name": "Aragon"},
{"name":"Canary Islands"},...]

53. 53
Extend the matching strategy: Semantic, Non-functional
properties
Matching
strategy
Support data modification operations
Data
modification
Support other formats: GraphQL, OpenAPI 3Other formats
APIComposer
F u t u r e w o r k

54. 54 Side effects
O p e n A P It o U M L
Visualization of OpenAPI
definitions as UML
Class diagrams
Better visualization and
understanding of the data
model and operations of a REST
API.
Easier integration of REST APIs
in all kinds of model-based
processes.

55. 55 Side effects
O p e n A P It o U M L

56. SUMMARY &
FUTURE WORK

57. 57 Summary

58. 58 Publications
ICWE 2018
OData profile ICWE 2017
SAC 2016
ECFMA 2017
RCIS 2018
ESOCC 2018
EDOC 2018
International conferences
(full research papers)
4
International conferences
(short and demo papers)
3
National conferences 1

59. 59
A case study to validate the contributions of the thesis in
a single industrial scenario
All-in-One
Smart cities are a potential direction to apply to the
work presented in this thesis
Smart cities
Extend our work to support non-functional requirements
Non-functional
requirements
Extend our work to support GraphQLGraphQL
Future work

60. Thank you!
Model-driven Round-trip
Engineering of REST
APIs
Hamza Ed-douibi
Supervised by : Jordi Cabot and
Javier Luis Cánovas Izquierdo