Building Real-time Push APIs Using Kafka as the Customer Facing Interface with Javier Moreno Molina

Internal
Building Real-time Push APIs Using Kafka
as the Customer Facing Interface
Javier Moreno Molina

Javier Moreno Molina
Building Real-time Push APIs Using Kafka as the Customer
Facing Interface

Internal
3
Mercedes-Benz Connectivity Services
- Introduction
- Event-Driven APIs
- Push vs. Pull
- An Event-Driven API Ecosystem
- Conclusion & Outlook
- Q & A
Outline

Mercedes-Benz Connectivity Services 4
JAVIER MORENO MOLINA
Solutions Architect | since 2018
Mercedes-Benz Connectivity Services GmbH
- Focus on DevOps, Platform & Infrastructure
- Interest on Distributed Systems and
Communication
About me.

Internal
We develop, operate and offer connectivity services that provide various
vehicle data points & vehicle indicators.
Introduction
Mercedes-Benz Connectivity Services GmbH
WHO WE ARE
As a connectivity and data
service provider in the
automotive sector,
Mercedes-Benz Connectivity
Services GmbH
offers innovative data-based
product solutions.
WHAT WE OFFER
We create digital solutions
based on real-time data from
measuring points and signals
of connected Mercedes-Benz
vehicles worldwide. This data
is used to generate added
value for products and
processes in companies from
a wide range of industries.
HOW WE HELP
We help our customers to
increase their process
quality and efficiency,
improve customer
experiences, act more
sustainably or develop new
business models.
WHO WE SERVE
•car & van fleet operators
and their service providers •
•car rental companies•
•leasing companies•
•insurance companies•
•neutral servers•
• pp & software developers•

Internal
Our history: coming from a fleet management portal to an API with
flexible data packages for cars and vans.
Introduction
2015 – 2016 IDEATION
- Fleet management
product: portal for fleet
managers and app for
drivers, as well as portal
for dealers
- Target groups: small
and middle size
companies, leasing
companies, rentals,
dealers
2017 GO LIVE CONNECT
BUSINESS
- 3 services: vehicle
management, tracking
and driver´s log
- 2 customer frontends
and 1 web based portal
2018 GO LIVE CONNECT
BUSINESS-API
- Request of middle size
and large customers to
consume the portal
services additionally via
an API
- Pull API Partner concept
enabled
2019 ADAPTION OF TARGET
GROUPS AND STRATEGY
- The demand for APIs is
growing continuously
- the services at that time
were not flexible
enough to meet the
requirements of our
customers
- Push API is found to be
more attractive for
customers
2020 CONNECT YOUR
BUSINESS API
- 12 service packages with
several data signals and
data points cover wide
range of use cases
- Service packages can be
activated on a vehicle-
level
- Target group: Third party
providers to enrich their
own products & large
customers to manage
their own fleet

Internal
Our connect your business API is highly recognized within the industry
and was selected #1 at the 2023 DEVIES Awards & 2022 API Awards.
Introduction
With the connect your business API, we aim to create a holistic ecosystem around the connected car
together with various partners. The API enables third party providers to enrich own products and
business models with Mercedes-Benz vehicle data, without the need for a retrofit solution. This network
of partners will be the cornerstone for the development of many new databased solutions in the future.
The awards were both presented during the world’s largest
developer and microservices conferences & expos in California.
The Advisory Board to the Awards has selected our technology
based on three criteria:
1. attracting notable attention and awareness in the API & software
industry,
2. general regard & use by the developer & engineering community,
3. being a technical leader in its sector for innovation.

Internal
We know that secure and responsible handling of data is the basis for the acceptance of connected driving. We
therefore pursue new technical developments with three clear principles in mind and integrate them into our
products (privacy-by-design):
Privacy is a high priority for us.
Introduction
1
2
3
Transparency: The customer must know when which data is collected and for which purpose. In the
sales information, the Mercedes me App, the Operator's Manual and the Terms of Use, we inform the
customer comprehensively about the data processing. Where possible, even directly in the vehicle.
Self-determination: The customer decides which services he actually uses and which data he wants to
share either by consent, contract or by touching a button.
Data security: The high safety requirements of our customers apply equally to the data security of the
connected vehicle. Mercedes-Benz protects customer data against manipulation and misuse. With
regard to IT technological progress, we continuously develop data security.

Internal
9
Event-Driven APIs

Internal
APIs are everywhere today and have been for a while.
Most of them are Request-Based (REST).
Many IoT use cases demand Event-Based communication:
- Real-Time Data Delivery
- Delivery on relevant ocurrence
However, there is no clear or standard way to offer event-driven
interfaces as it occurs in REST.
IoT and Event Driven APIs
Event-Driven APIs

Internal
Event Driven APIs challenges
Event-Driven APIs
SPECIFICATIONS
- There are widely accepted
Interface Specifications for REST
- OpenAPI
- RAML
- No clear specification standard
for event-based interfaces
- AsyncAPI?
COMMUNICATION COMPLEXITY
- Uncertainty and variability of
- Data Volume
- Message Rate
- Wrong coordination may destroy
use case feasibility
- Superfluous data processing
- Latency
ROLE OF CONSUMER DEPENDABILITY
- Availability
- REST: From request to
response
- Unavailable consumers may
impact the service provider
- Security and Access
- REST: authorizing at request
time
- Difficult after data processing

Internal
12
Push vs. Pull

Internal
Push Examples: Webhooks
Push vs. Pull
HETEROGENEITY
- No Standards
- Payload Ownership
- Assumptions on
consumer capacity
OPERATIONAL COMPLEXITY
- Failures on consumer
side need specific
actions on sender side
- Retries
- Replays
SCALABILITY
- HTTP Post requests
typically blocking
- Retry management
HTTP
WEBHOOK SERVER
ON CONSUMER
INFRASTRUCTURE
SECURITY
- More difficult to
enforce from client
side

Internal
- Throughput limitations (routing, dequeuing, etc.)
- Delivery Guarantees limitations (exactly once not possible at a system level)
- Messages availability based on delivery status:
- A retention policy is anyway required in case consumers are not available for a long time
- No replays possible
Push Examples: MQTT/AMQP
Push vs. Pull

Internal
Push vs. Pull
Push vs. Pull
Intuitively we think of event-driven APIs as Push APIs, but … can a Pull-based system be a good option in
some scenarios?
PULL
• Requires polling
• Allows freedom in consumption rate
• Optimal throughput
PUSH
• Optimal bandwidth & latency
• Limitations on Delivery Guarantees
• Assumptions on Customer side

Internal
What about Kafka?
Push vs. Pull
Kafka offers a Push API-like experience:
- Publish/Subscribe
- „Data Sending“
… but is a Pull-based implementation.
1 2 3
EFFICIENT LONG POLLING
Kafka client hides polling logic.
STATELESS MESSAGE MANAGEMENT
Messages managed by log size or
retention not consumption state.
CLIENT INITIATED COMMUNICATION
Data Consumers get messages matching
their needs.
No unavailable consumers overhead.

Internal
B2B Context
Push vs. Pull
Bandwidth Utilization
Mitigated by long polling
Not relevant in Server to Server communication
Smart Broker/Dumb Client Paradox: dumb clients are not easier to implement for the customer
- Availability: Unavailable consumers have no operational burden (consume no resources)
- vs. retry strategy and storage and memory size increase
- Rate: Consumption is optimized to the available consumer resources
- vs. broker can overload the client
- Delivery: Consumer can decide on itself if a message delivery is successful or not and go back
in time or request message replays if it detects any data loss.
- vs. delivery ends with client acknowledgement. Support is difficult for any failure in the
triggered logic after ack

Internal
18
An Event-Driven API Ecosystem

Internal
Complete the REST API Puzzle
AUTHENTICATION
OpenID Connect
DOCUMENTATION
OpenAPI
AUTHORIZATION
OAuth

Internal
Authentication
STANDARDS
REST APIs already have well-
established standards, such as
OpenID Connect.
SASL OAUTHBEARER
Kafka supports OAUTHBEARER SASL
authentication mechanism:
OAuth 2.0 bearer tokens can be
used for client authentication.
COMMON AUTHENTICATION
Most Event Driven APIs will also
contain REST APIs.
Using OAUTHBEARER there is no
need for additional authentication
mechanisms.
No production ready OAuth 2.0 token management implementation in Kafka.
We set up a repository with client code samples for OAuth consumers:
- Available at: https://github.com/mercedes-benz/kafka-integration-samples
- Currently on golang, C# and Python

Internal
Authorization
KAFKA ACLS
Once authenticated, Kafka ACLs
can be used to define permissions
CONTENT AUTHORIZATION
Only based on Topic
CONCURRENT CONSUMERS
Can be controlled based on Group
Resources:
- Cluster
- Topic
- Group
- Transactional ID
- Delegation token
Cannot authorize for specific messages inside a topic.
Every specific data access policy requires a topic.
Number of required topics must be analyzed.
Consumer may only join to in ACLs allowed Consumer Groups.
For a consumer group, the maximum number of effective clients is the
number of topic partitions. If more are created they will remain idle.

Internal
There are many specifications for REST APIs documentation like OpenAPI, RAML, etc. with OpenAPI 3.0
being currently the most popular option.
OpenAPI belongs to The Linux Foundation.
We already use OpenAPI to document our REST APIs.
AsyncAPI is also a Linux Foundation initiative to specify asynchronous APIs.
It is compatible with OpenAPI schemas.
https://developer.mercedes-benz.com/products/connect_your_business/specifications/push_api
Event API Documentation - AsyncAPI

Internal
23
Conclusion & Outlook

Internal
Recap & Outlook
Event Driven APIs are still far from the maturity and adoption of REST APIs.
1
Push-Based implementations are complex, specially when there are many consumers that are not
under your control and in scenarios that require high througput.
2
A complete Event Driven API ecosystem can be built on top of Kafka, with REST API similar
authentication (OpenID Connect/OAuth 2.0) and Documentation (AsyncAPI).
3
Last-mile B2C delivery is in many cases not suitable for Pull-based approach, even with Kafka
optimizations.
4
Pull-based can be a good option, in particular in B2B contexts. Kafka is already inside most real-
time data architectures.
5

Internal
WE ARE
HIRING!
OPEN
POSITIONS
:
- SOLUTION ARCHITECT
- SOFTWARE DEVELOPMENT ENGINEER
- SITE RELIABILITY ENGINEER
- TEST ENGINEER
J O I N O U R T E A M
APPLY NOW
www.connectivity.mercedes-benz.com
25

Internal
Q & A
Thank you … Questions?

Building Real-time Push APIs Using Kafka as the Customer Facing Interface with Javier Moreno Molina

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Building Real-time Push APIs Using Kafka as the Customer Facing Interface with Javier Moreno Molina

Similar a Building Real-time Push APIs Using Kafka as the Customer Facing Interface with Javier Moreno Molina (20)

Más de HostedbyConfluent

Más de HostedbyConfluent (20)

Último

Último (20)

Building Real-time Push APIs Using Kafka as the Customer Facing Interface with Javier Moreno Molina