1. The document discusses the need for an integration specification to manage APIs that use multiple protocols and styles at large companies. 2. An integration specification would provide a single source of truth for technical metadata about APIs, including how they are structured, the data they manage, and how they relate to each other and enterprise data. 3. This would enable automated API discovery, integration, and management processes by describing APIs and their access to data products at an enterprise level.

1. Multi-Protocol
APIs at Scale in
Jesus de Diego
adidas API Platform
2021
API

2. Bio Jesus de Diego
Software Engineer
Software Architect
Worked with APIs for 15 years
Right now working with the adidas API Platform

3. What is this about?

4. Agenda
1. API Landscape
2. Multi-Style APIs?
3. The Gateway to APIs
4. Issues
5. One specification to gather all APIs?
6. Putting all together
7. Take Aways

5. API Landscape

7. +400HTTP based APIs
+300API Owners
+153m requests a day
+1000API Consumers

8. Full Picture

9. API Lifecycle

10. API Guidelines

11. Multi-Style APIs?

12. MSA
Microservices-based applications overcome the biggest limitations of traditional, monolithic
applications.
A microservices-based application architecture (MSA) breaks a monolith into its component
services and running each component as an autonomous application (microservice).
Some basic rules have to be followed: Single responsibility, well defined autonomous functional
area, non-shared data source, isolated data boundaries, and others.
Each microservice exposes one or more APIs. MSAs have increased the need to count on having
HTTP-based APIs as the primary (there are other options like Kafka) way of communication
between microservices.

13. RPC (Remote Procedure Call)
● REST predecessor from 1970s
● Invoke a function on a remote server in a particular format and receive a response in
the same format
● With an RPC API, the method is found in the URL. The arguments that invoke the
methods are found in the query string.
POST /deleteResource
(with a query string that says { “id”: 3 } )

14. REST (Representational State Transfer)
The key concepts are RESOURCE, STATE (resource representation) and DATA (media) FORMAT.
According to Roy Fielding, an API qualifies as “RESTful” when it meets the following constraints:
• Uniform interface
• Client-server independence
• Stateless
• Cacheable
• Layered
The client does not build URLs (mimicking the web browser that just follows the URL to find a
resource).
API clients never have to understand the format of the URLs. These formats are not part of the API
spec provide to the API consumers.

15. We can face
certain problems
in our REST APIs
• Long Running Tasks
• Blocked connections
• No server notifications
• No streaming
• No broadcasting
information (subscriptions)
• Etc..

16. Websockets
It’s a protocol (ws/wss) in the OSI model (7) compatible with HTTP
Supported by most browsers
Full Duplex communication
Streams of messages on top of TCP
Note: Supported by Kong API Gateway (ws/wss)

17. High Performance
• Compact Binary Exchange Format
• Make API Calls to the Server like they were Client local
• Cross Language/Cross Platform
• Auto-generate API definitions for idiomatic client and server –just
implement the interfaces
Uni/Bi-Directional Streaming
• Pluggable support for streaming with HTTP/2 transport

18. gRPC
Capabilities
Request/Response
service NotificationsService {
rpc GetNotifications(Empty) returns () {}
}
Unidirectional stream
service NotificationsService {
rpc SubscribeNotifications(Empty) returns (stream Event) {}
}
Bidirectional stream
service NotificationsService {
rpc ExchangeNotifications(stream Event) returns (stream Event) {}
}

19. gRPC Pros/
Cons
Remarkable
• Microservices, Web Apps, Mobile apps can produce and/or consume
streams of data.
• Regular Request/Response is faster and more efficient when handling
heavy payloads. Ideal for IoT or low-bandwidth networks.
PROs
Good tooling.
Multi language supported.
Fast, efficient, two-ways streaming, etc.
CONs
Not so popular in the development community.
API Contract is strictly service description.
CONCLUSION
Not needed if you handle low/medium payload size and you do not need
streaming.

20. gRPC
Fast
Readings
gRPC Official Site
Protocol Buffers discussion group
Defining the API Contract: protobuf/gRPC
gRPC Kafka Streaming Demo

21. What’s this?
• Based on a data graph (a hierarchy of types with
fields).
• The data graph is populated by other services.
• The data graph can be queried by using a SDL
(Schema Definition Language) which is defined by
the API Producer.
• The SDL defines a collection of types and the
relationships between those types.
• It supports mutations on the data (updates and
inserts).
• Focused on the needs of the client. API consumers
get the data they need in the shape they want.

22. • Data Schema -> API Contract
• Queries and Mutations
• Arguments (scalars and objects)
• Variables
• Single endpoint
• Introspection
• Streaming is supported for subscriptions
(long-lasting GraphQL read operations that
can update their result whenever a particular
server-side event occurs) .
• Different implementations (server library
and client)
pos
t
What’s this?

23. Possible
Architecture
(REST Wrapper)

24. Possible
Architecture
(Data Aggregator)

25. • Avoid the REST waterfall
• Get the right amount of data
• Tolerate incremental changes
in the data schema
Promises

26. Pros
Cons
• Overfetching, underfetching? Avoid the REST waterfall
• Limited amount of tooling for GraphQL.
• Is the schema (the .graphqls file) the API Contract?
• Better adaptation to changes in the API contract for API
consumers.
• Need to define the data schema, mutations and implement
resolvers for mutations and queries.
• Sources: REST APIs, Data sources.
• Supported by many programming languages.
• Possible added latency. Is cache broken?
• Proxy caching and rate limiting on top of GraphQL is key.

27. GraphQL
Fast
Readings
GraphQL Official Site
GraphQL Specification
GraphQL Contract Testing with Pact
Getting started with GraphQL Java and Spring Boot
GraphQL Subscriptions

28. What is
this? • An API technical specification for Event-Driven
Architectures (EDAs)
• It is not specifically about Streaming (it also exists in
gRPC and GraphQL)
• It is based on OAS. So, we’ll find similarities.

29. OAS
similarities
adidas API Platform

30. Main
Features
• Supported protocols
• AMQP, Websockets, KAFKA, MQTT, HTTP/x
• Servers define the URL of the broker or server
• Security
• User/Password
• Certificates
• API Keys
• OIDC
• OAuth2

31. AsyncAPI
Fast
Readings
Official web site
Blog
Twitter
Podcasts

32. Take Aways

33. When is
each API
style good
for?
REST
• It is still the default option for HTTP based APIs
• Non-blocking REST APIs are a good solution to improve
performance and consumption of resources
gRPC
• You need real-time-like flows
• You handle really big payloads
• You work with low bandwidth scenarios
• You need streams
GraphQL
• You need subsets of data
• You frequently change the sub-set of data you need
• You need streams
AsyncAPI
• Related to Event Driven architectures
• You need streams

34. API Styles
• They are a part of the architectural layout, especially MSAs
• Best adaptation to Business requirements (Functional and Non-Functional)
• Availability of resources (memory, CPU, bandwidth)
Tools
• Adhere to a standard better than to a tool or toolset
Protocols
• HTTP is not 1.1 any longer
• New HTTP versions (2, 3) are more capable and we have to take advantage
of their features as they are stable
Event Driven APIs
• Preferably using AsyncAPI
• Ideal for Event Streaming Engines: Kafka, Spark Streaming…
Boundaries
and Clear
Principles

35. Boundaries
and Clear
Principles

36. 1. Foundation for Data Product APIs
2. HTTP 2/3-based APIs provide streaming of data
with balance in a resilient way
3. API Gateways work well with HTTP 2/3-based
APIs
4. Streaming Engines (Kafka) offer production/
consumption of events in EDAs
5. HTTP 2/3-based APIs can extend the propagation
of events as streams beyond Kafka boundaries
6. The solution is part of the architectural design,
stakeholders, client apps and systems
7. It makes sense to think of this landscape as a
unitary point of view: Data Product integration
across different systems and boundaries
Multi-Protocol
Integration?

37. API Registry
• A fundamental tool to register API
descriptions for Multi-Protocol APIs.
• The API Registry has to be compliant with
tech API specs (OAS, protobuf, AsyncAPI).
• It can include design capabilities and a
lifecycle for versions.
• It allows integrations (testing, source code,
mocked server, etc).
• API descriptions have to be available via API
to be aggregated and handled.

38. API Registry
The reality is, support to non-OAS APIs is still partial, non-
intuitive, not supporting design functionalities.
The target is to have an API Registry with all these features
supporting first OAS and AsyncAPI, gRPC and GraphQL at a
later stage.

39. The Gateway
to APIs

40. Why to use an API Gateway?

41. API Gateway as an Architectural Pattern
● Isolates the consumer from the service
● Applies specific features to specific consumers
● Applies security restrictions before touching the services
● It is highly available and uses load balance strategy
● Reduces the number of requests/roundtrips
● Better adaptation of APIs to consumers
● It allows strategies like Blue/Green deployments or Canary
releases

42. API TEAM / CREATING APIS @ADIDAS
Big Picture
3-JUN-21
42
Git
API Registry
CI/CD
API Gateway
K8s Platform
API Consumer
API Description
API Description
Service Implementation
Style Guide
Dredd Plug-in
Git integration
adidas team APIs
Documentation
Mock Service
Analytics
Admin
API GatewayToolbelt
2
3
4
5
6
7
1
1 Design 2 Develop 3 Deploy 4 Publish 5 Use 6 Analyze 7 Update
Security Checks
Runscope
ELK
SpectralPlug-in
Spectral

43. API Authentication and Authorization
● API Key (Key Auth)
● OIDC (integration with Azure AD and
others ID Providers)

44. API Security Criteria
● Check your API is not vulnerable
● Alignment with Load Balancer other
services
● Use the OWASP Security Top 10 as
the main reference
● Create specific tests to meet these
rules

45. API Security Measures
● Authentication/Authorization checks (all possible ways)
● Never rely on client for anything important
● Define response schemas
● Set up a short life for your tokens!
● Limit the rate of requests
● Never return raw data directly from the database
● Sanitize all inputs, even from internal consumers
● Use CORS filtering
● Use always WAF & Bot detection

46. ● ---
● apiVersion: configuration.konghq.com/v1
● kind: KongPlugin
● plugin: rate-limiting
● metadata:
● name: xxx-rate-limiting
● annotations:
● kubernetes.io/ingress.class: adidas-kong
● config:
● second: 100
● hour: 10000
● policy: redis
● redis_host: ratelimit.kong.svc.cluster.stg
API Rate Limiting
Consumer Quota
Burst Limit
Resources
Security

47. ● ---
● apiVersion: configuration.konghq.com/v1
● kind: KongPlugin
● plugin: proxy-cache
● metadata:
● name: pc-att-marking-code-api-cache
● annotations:
● kubernetes.io/ingress.class: adidas-kong
● config:
● cache_ttl: 120
● strategy: memory
API Performance + Cache

48. 48
API Transformation

49. ● ---
● apiVersion: configuration.konghq.com/v1
● kind: KongPlugin
● plugin: request-transformer
● metadata:
● name: pc-att-markingapi-movementsapi-tagsapi-reqt
● annotations:
● kubernetes.io/ingress.class: adidas-kong
● config:
● replace:
● uri: /$(uri_captures[1])$(uri_captures[2])

50. API Schema Validation

51. API Request Size

52. Enterprise API
features are
implemented by
the API Gateway

53. The Gateway to APIs

54. BASICS
Tool: the API Gateway (Kong)
Features: Non-Functional Requirements (from Business Analysis)
Extra cost: Fixed cost and cost per incoming request

55. API Producer
Team
Team
API Gateway
Deployment Model
adidas API Platform

56. https://tools.adidas-group.com/bitbucket/projects/API/repos/kong-managed/

57. Ingress
ACL Plugin
Key Auth (API Key)
Plugin

58. OOOK…..

59. Issues

60. API Registry & API Description Management:
● APIs are managed as individual isolated items
● The defined data domain in each API is not correlated to the Data
Catalog
● They do not describe how they can be consumed
● They do not include relationships to other APIs
● They do not involve Enterprise API features
Issues

61. Issues
API Gateway Operations:
● Description of APIs do not include Enterprise API features based on Non-
Functional Requirements
● They do not include integration with other layers in the network (e.g. Load
Balancers, WAF)
● API Consumers are managed manually
● There is not a unique source of truth agreed by all stakeholders which is the
very base for automatic processes
● Enterprise API information is spread across multiples files as annotation
values

62. Business
● Integration by API consumers (internal and external) is a slow
process (KPI: Consumer Engagement Time)
● Rolling Out a new Enterprise API is slow (KPI: API Time To Market).
● Integration does not cover everything, just partial areas of the surface
of a Data Domain (KPI: Operational Scope Coverage)
● API Consumer development cost increases (KPI: API Consumer
Cost)
Issues

63. In summary, a great number of APIs in a company is a success but it is
dangerous.
The number of API producers and consumers grows and it can not be
handled easily
The API Gateway provision is the right way but it requires a lot of
information that API tech specs do not consider.
Issues

64. One
Specification
to rule all
APIs?

65. What is Metadata?
Metadata is structured information that describes, explains, locates, or otherwise makes it easier to retrieve,
use, or manage an information resource.
What is Technical Metadata?
Technical metadata (also called structural metadata) describes how the data is arranged and displayed to
users by describing the structure of the data objects—such as tables, columns, rows, indexes, and
connections. Technical metadata tells data professionals how they will need to work with the data—for
example, if they can work with it as is, or if they need to transform it for analysis or integration.

66. Integration Specification
Based on API Technical Metadata
API Technical Metadata describes what one or several APIs are structured, the data
domain they manage and the connections to dictionaries, the ways of usage,
connections to other APIs and how can they work together.
Why?
To allow integration, API discovery, quick engagement, uniform application of
configuration to allow APIs to be released in an Enterprise API way.

67. Integration Specification
Targets
To have the unique source of truth for API Automation processes.
To allow the description of access to Data Products

68. Integration Specification

69. Integration Specification

70. Integration Specification

71. Integration Specification

72. Integration Specification
Technical Contract
● Non-Functional
● Accessibility
● Availability

73. Integration Specification Main Features
Multiple Dimensions: Individual API, Group of APIs
API Style Seamless: Composing groups of APIs of different styles
Consistent: Data Domains are compliant with company dictionary (EDC)
Non-Functional Behavior: Capacity, scalability, performance, rate limits, etc.
Security: Authentication, Authorization, restrictions
Consumer Oriented: Discovery, Engagement, Usage Semantics

74. Integration Specification
Sample

75. Putting
Everything
Together

76. Integration
Landscape

77. What about
the Tooling?

78. The
Integration
Hub App
● Unique entry point for Data Product and API inter-actions.
● Connected to the Data Catalog, API Registry, ID Provider,
LeanIX Inventory.
● It manages the changes in the Integration Specification
files.
● These changes trigger the build pipeline to make the
changes in the API Gateway configuration

79. The Integration Hub App

80. Use Case: New Consumer for 1 or multiple APIs

81. Use Case: New API

82. Use Case: Changing NFRs in an existing API

83. Take Aways

84. Take Aways
● Large companies using APIs are increasingly using multiple API
styles and protocols as a better fit to meet business expectations and
produce better APIs, more adapted to Business requirements, being
more efficient in resource consumption.
● Using an API Registry and an API Gateway has to allow the usage of
Multi-Protocol/Style APIs (e.g. gRPC endpoints, GraphQL cache)
● API Specifications (OAS, AsyncAPI, etc) are just technical references
and they do not cover all the operational aspects in the API lifecycle.
● API Specifications handle APIs as isolated items, not covering
integration and operations.

85. Take Aways
● We need the Integration Specification to fill all these gaps in APIs
description.
● The Integration Specification is useless without the Integration Hub
Application, including:
● Interpreter
● API Browser
● API Search
● Subsequent changes in manifest files
● CD Pipelines

86. Next Steps
Our agenda with the Integration Specification:
● Establish the links between APIs using multiple protocols and styles to
describe access to greater entities (Data Products).
● Link APIs with Enterprise Data Domains and Dictionaries (Data Catalog) to
enable Data Product and API Search and Discovery.
● Apply Security constraints and Non-Functional Requirements and protection
uniformly via API Gateway configuration.
● Make the Integration Hub Application and the whole Enterprise API Lifecycle
● In summary, the Integration Specification is going to be the base for
automatic provision and management of Adidas APIs.

87. Thanks!
jesusjavier.dediego@adidas.com
Personal Blog
Medium
#adidoescode
API