Manage the Digital Transformation with Machine Learning in a Reactive Microservices-Based Architectural Approach

Manage the Digital Transformation with
Machine Learning in a Reactive
Microservices-Based Architectural
Approach
Patrick de Vries

THE COMPANY
About me
Patrick de Vries (45 years) is an OSS manager
(Demand), IT architect with more than 15 years
experience in the mobile networks. Has a passion for
data management and processing.
Maanplein 55| 2500 GC The Hague | The Netherlands
Mobile: +31653102171 | E-mail:
patrick.devries@kpn.com
https://nl.linkedin.com/in/patrick-de-vries-570a8469

THE COMPANY
This is KPN
We make life more free, easy and more fun
by connecting people. We are passionate
about offering secure, reliable and future-
proof networks and services, enabling people
to be connected anytime, anywhere, whilst at
the same time creating a more prosperous
and cleaner world. We’ve been doing this on
the basis of a strong vision. Every day, for
more than 130 years. Below, we show how
our employees make this possible.

SUSTAINABILITY
Every day a little
greener
We see every day as a
chance to do better. Such as
playing our part in tackling the
climate issue by finding the
right tone for our transition to
sustainable growth. KPN is
the Green Connection and
one of the world’s most
sustainable telecom
companies.
In 2017, KPN was again
declared world climate leader.

FOR INTERNAL USE5 UDEX | VALUE DRIVEN EXECUTION |
PROPLEM
Modern for more than a century
Proactively detecting, controlling and
communicating cable faults speeds
up the solution and simultaneously
reduces tickets and calls. In addition
to shortening the cable-outage by
accelerated detection and in 1x good
control, outages can be monitored
until complete resolution. The current
process is reactive because network
monitoring is limited to the active
network elements. Proactively
informing clients about malfunctions
and the handling contributes
additionally to customer satisfaction.

KPN Refresh Classificatie7
PROBLEM
From reactive to preventive
DiagnoseDiagnose RecoveryRecovery Restore
Maen Time to Repair
(MTTR)
Maen Time to Repair
(MTTR)
Detect Repair
Maen Time to Restore Services
(MTTRs)
(MTTRs)
Event
Maen Time Between Failure
(MTBF)
(MTBF)
Event
Failure Duration Time
(FDT)
Event
(MTBF)
(MTBF)
Event
--- Preventive operations (step 2) ------ Preventive operations (step 2) ---
--- Reactive operations ------ Reactive operations ---
RecoveryRecovery Restore
Maen Time to Repair
(MTTR)
(MTTRs)
Event
(MTBF)
(MTBF)
Event
(FDT)
(FDT)
Detect
Diagnose
Repair
--- Reactive operations (step 1) ------ Reactive operations (step 1) ---
Maen Time to Repair
(MTTR)
Event
Detect
Diagnose
Repair

PROBLEM
communicating
Engineer Comm. Reset

PROBLEM
communicating
Total run time of the flow > than 15min
Waiting for 70% of the data
before start processing
Engineer solved problem
A small change > 3 moths delay
No End 2 End control

Sagrada Família
In 1882, construction of Sagrada Família
started under architect Francisco de Paula
del Villar. In 1883, when Villar
resigned,[6] Gaudí took over as chief
architect, transforming the project with his
architectural and engineering style,
combining Gothic and curvilinear Art
Nouveau forms. Gaudí devoted the
remainder of his life to the project, and he
is buried in the crypt. At the time of his
death at age 73 in 1926, when he was run
down by a tram, less than a quarter of the
project was complete. ..etc. https://en.wikipedia.org/wiki/Sagrada_Fam%C3%ADlia

Any organization that designs a system
will Inevitably produce a design whose
structure is a copy of the organization’s
Communication structure.”
“
BACKGROUND
CONWAY’S LAW
http://melconway.com/Home/Conways_Law.htm

Adding manpower to a late software
project makes it later, states that when a
person is added to a project team, and the
project is already late, the project time is
longer, rather than shorter.”
“
BACKGROUND
BROOK’S LAW
https://en.wikipedia.org/wiki/Brooks%27s_law

Out of the Box

Responsive
Elastic Resilient
Message driven
BACKGROUND
The Reactive Manifesto
https://www.reactivemanifesto.org

DEVELOPMENT
Evolution of the Technology
Monolithic
Single unit
SOA
Coarse-grained
Microservices
Fine-grained

DEVELOPMENT (1/3)
Orchestration Benefits and Tradeoffs
Benefits
• Provides a good way for controlling the flow of the application when
there is synchronous processing. For example, if Service A needs to
complete successfully before Service B is invoked.
Tradeoffs
• Couples the services together creating dependencies. If service A is
down, service B and C will never be called.
• If there is a central shared instance of the orchestrator for all
requests, then the orchestrator is a single point of failure. If it goes
down, all processing stops.
• Leverages synchronous processing that blocks requests. In this
example, the total end-to-end processing time is the sum of time it
takes for Service A + Service B + Service C to be called.
orchestrator
Service
A
Service
B
Service
C
request
request
request
response
response
response

DEVELOPMENT (2/3)
Reactive Benefits and Tradeoffs
Benefits
• Enables faster end-to-end processing as services can be executed
in parallel/asynchronously.
• Easier to add/update services as they can be plugged in/out of the
event stream easily.
• Aligns well with an agile delivery model as teams can focus on
particular services instead of the entire application.
• Control is distributed, so there is no longer a single orchestrator
serving as a central point of failure.
Tradeoffs
• Async programming is often a significant mindshift for developers. I
tend to think of it as similar to recursion, where you can’t figure out
how code will execute by just looking at it, you have to think through
all of the possibilities that could be true at a particular point in time.
• Complexity is shifted. Instead of having the flow control centralized
in the orchestrator, the flow control is now broken up and distributed
across the individual services. Each service would have its own flow
logic, and this logic would identify when and how it should react
based on specific data in the event stream.
Event stream
Service
A
Service
B
Service
C
consume produc
e
consume produc
e
consume produc
e

DEVELOPMENT (3/3)
Hybrids Benefits and Tradeoffs
Benefits
• The services are decoupled (but not the services within Service
A).
• Asynchronous processing is enabled by leveraging events
between services.
• The overall flow is distributed. Each service contains its own
flow logic.
Tradeoffs
• Within Service A there is coupling with Services D, E, and F.
• Depending on the design, within Service A there could be
synchronous processing that blocks requests.
Service
B
Service
C
Event stream
Service
A
ServiceD
ServiceE
ServiceF
orchestrator
Benefits
• The services are decoupled (but there is a degree of coupling
between the services and the coordinator).
• Asynchronous processing is enabled by leveraging events
between services.
• The overall flow can be seen in one place in the reactive
coordinator.
Tradeoffs
• The coordinator does have coupling with the services —
specifically with needing to know what commands a service needs
in order to react.
Service
C
Service
A
Coordinator
Service
B
Event stream
Sync block of Async services

BACKGROUND
Kafka is a distributed streaming platform.
A streaming platform has three key capabilities:
• Publish and subscribe to streams of records, similar to a
• message queue or enterprise messaging system.
• Store streams of records in a fault-tolerant durable way.
• Process streams of records as they occur.
A few concepts:
• Kafka is run as a cluster on one or more servers that can span
multiple datacenters.
• The Kafka cluster stores streams of records in categories
called topics.
• Each record consists of a key, a value, and a timestamp.
https://kafka.apache.org/

BACKGROUND
Microservice architecture, or simply
microservices, is a distinctive
method of developing software
systems that tries to focus on
building single-function modules
with well-defined interfaces and
operations. The trend has grown
popular in recent years as
Enterprises look to become more
Agile and move towards a DevOps
and continuous testing.
Microservices can help create
scalable, testable software that can
be delivered weekly, not yearly.
https://martinfowler.com/articles/microservices.ht
ml

https://projects.tmforum.org/wiki/display/AP
I/Open+API+Table?_ga=2.89526814.4494
45334.1549920269-
712967230.1541665147

BACKGROUND
Open communication
{
"type": "record"
"namespace": "com.kpn",
"name": "sg_alarm",
"doc": "Avro Schema for - POST must be used to create a new resource",
"fields": [
{"name": "messageversion", "type": "string"},
{"name": "messageId", "type": "string"},
{"name": "priority", "type": "enum", "symbols": ["HIGHEST", "HIGH", "NORMAL", "LOW", "LOWEST"]},
{"name": "action", "type": "enum", "symbols": ["POST", "PATCH", "DELETE"]},
{"name": "cause": "type": "string"},
{"name": "resourceSpecification", "type": {
"name" : "resourceRecord", "type" : "record",
"fields" : [
{"name":"resourceIdentifier": , "type": "string"},
{"name": "resourceDomain", "type": "enum", "symbols": ["Cable", "Port", "Card"]}
{"name": "resourcePosition": , "type": "string"},
{"name": "serviceIds", "type":{"type": "array", "items":{"name":"serviceId","type":"string"}}}
]
}
},
{"name": "timeSpecification", "type": {
"name" : "timeRecord", "type" : "record",
"fields" : [
{"name": "timestamp", "type": { "type": "long", "logicalType": "timestamp-millis" }},
{"name": "eventStartTime", "type": { "type": "long", "logicalType": "timestamp-millis" }},
{"name": "eventChangeTime", "type": { "type": "long", "logicalType": "timestamp-millis" }},
{"name": "eventFinishTime", "type": { "type": "long", "logicalType": "timestamp-millis" }}
]
}
},
{"name": "eventSpecification", "type": {
"name" : "timeRecord", "type" : "record",
"fields" : [
{"name": "serviceState", "type": "enum", "symbols": ["CableOutage", "PortOutage", "CardOutage"]},
{"name": "eventType", "type": "enum", "symbols": ["Unavailable", "Available"]},
{"name": "testResult", "type": "enum", "symbols": ["UnderGround", "OverGround"]}
]
}
}
]
}

SOLUTION
communicating

OPEN COMMUNICATION
Producer NIFI Kafka Spark Hbase Ticket Comm. Reset
file.received
soap.received
API
soap.read
soap.impactWrite
WriteAllData
soap.impactRead
soap.LineImpactWrite
Request.LineStatus
Response.LineStatus
Write.LineStatus
soap.LineImpactRead
WriteAllData
soap.LineStatusRead
soap.ServicxeImpactWrite
soap.Request. ServiceImpactRead
soap.Response. ServiceImpactRead

BACKGROUND
Machine learning is a type of AI
Artificial Intelligence
Machine Learning
Deep Learning
A program that can
sense, reason, act, and
adapt
Algorithms whose
performance improve
as they are exposed
to more data over
time
Subset of machine
learning in which
multilayered neural
networks learn from
vast amounts of data
1950’s 1960’s 1970’s 1980’s 1990’s 2000’s 2010’s

OPEN COMMUNICATION
Producer NIFI Kafka Spark Hbase Ticket Comm. Reset
file.received
soap.received
API
soap.read
soap.impactWrite
WriteAllData
DiagnoseDiagnose RecoveryRecovery Restore
Maen Time to Repair
(MTTR)
Maen Time to Repair
(MTTR)
Detect Repair
(MTTRs)
(MTTRs)
Event
(MTBF)
(MTBF)
Event
(FDT)
Event
(MTBF)
(MTBF)
Event
--- Preventive operations (step 2) ------ Preventive operations (step 2) ---
--- Reactive operations ------ Reactive operations ---
RecoveryRecovery Restore
Maen Time to Repair
(MTTR)
(MTTRs)
Event
(MTBF)
(MTBF)
Event
(FDT)
(FDT)
Detect
Diagnose
Repair
--- Reactive operations (step 1) ------ Reactive operations (step 1) ---
Maen Time to Repair
(MTTR)
Event
Detect
Diagnose
Repair
Machine learning

"Everything we do is changing
the status quo, by making our
data easy to handle,
simple to use,
and user-friendly”

QaA!
Ask Meh Questions

Manage the Digital Transformation with Machine Learning in a Reactive Microservices-Based Architectural Approach

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Manage the Digital Transformation with Machine Learning in a Reactive Microservices-Based Architectural Approach

Similar a Manage the Digital Transformation with Machine Learning in a Reactive Microservices-Based Architectural Approach (20)

Más de DataWorks Summit

Más de DataWorks Summit (20)

Último

Último (20)

Manage the Digital Transformation with Machine Learning in a Reactive Microservices-Based Architectural Approach