Toyota Financial Services Digital Transformation - Think 2019

Toyota Financial Services and IBM Confidential1
Think 2019
Toyota Financial Services and IBM Confidential
How Toyota Financial Services use ICP for Reaching
New Levels of Innovation and Competitiveness
February 13, 2019; 11:30 – 12:10
Moscone South, Level 2 - San Francisco Ballroom 204
Slobodan Sipcic
Senior Certified Executive IT Architect
IBM GBS | Cloud Application Innovation
James Lang
General Manager – Platform Enablement
Toyota Financial Services | Information & Digital Systems
Ken Schuelke
National Manager – Enterprise API Factory
Toyota Financial Services | Information & Digital Systems

§ Setting the Stage:
• Present State – Summary of Challenges
• Enterprise Integration Platform (EIP) Goal
§ EIP – Reference Architecture
• EIP Reference Architecture View
• EIP PaaS View
§ EIP – Architecture Realization
• EIP Technology Choices:
⎼ Application Platform
⎼ API Gateway
⎼ Data Ingress and Egress
⎼ Data Persistence
⎼ Data Format & Schema Management
⎼ Stream Processing
⎼ Service Aggregation
• EIP Reference Architecture Realized
§ EIP – Physical Architecture
• Low-End Environments
• Production Environments
• Infrastructure as Code (IaC)
§ EIP Automation
• From Empty VPC to Fully Operational EIP in Minutes
⎼ Infrastructure Pipeline
⎼ Technology Components Pipeline
⎼ Services Pipeline
• EIP Automation Benefits:
⎼ Multi-Regional Deployment of EIP
⎼ Disaster Recovery
§ Use Cases:
• EIP Enabled TFS’s Digital Transformation – ESP Use Case
• ESP Benefits – Bringing Mainframe into API Economy
• EIP Batch/Real Time Data Processing – EDP Use Case
• EDP Benefits – Event Based Processing of Streaming Data
Table of Contents

Setting the Stage: Challenges & Goals

Present State – Summary of Challenges
“Toyota Financial Services has a multitude of applications which were implemented over the years by various functional
domain groups to meet “siloed” needs, ranging from lightly configured to heavily customized vendor packaged solutions
and from hybrid to completely purpose-built in-house solutions.”
Summary Challenges:
§ Business processes are overburdened with
manual, redundant data entry
§ Batch processing windows are long and often
delay the business
§ Revitalization and uplift projects suffer from
delays and very high cost
§ Data quality and consistency issues persist
between applications
§ Maintenance of heterogeneous and aging
systems is costing a lot of money
§ Staff intellectual development and innovation is
inhibited by focus on aging technology
Present State is characterized by batch oriented, point-to-
point, movement of large data sets, across ageing legacy
systems, with high data illiteracy across functional
domains and multiple versions of truth.

Enterprise Integration Platform Goal
“The goal is A new Enterprise Integration Platform (EIP), based on modern technology and design patterns, to solve the
business problems and energize the technical staff.”
Summary Goals:
§ Provide access to legacy systems via API and
automate manual business processes
§ Reduce time for daily batch cycles by 75%
§ Eliminate known data quality issues between
the Web, and Customer Service applications
§ Save money and head ache by retiring
numerous integration repositories and
operational data stores as well as mainframe
data processing applications
§ Spur innovation in the technical staff by
leveraging open standards and modern
technologyFuture State is characterized by event based, loosely coupled
microservice acting on streams of messages, minimized movement of
data sets, and harmonized information across functional domains

EIP – Reference Architecture

Enterprise Integration Platform Reference Architecture
“EIP is architected as an Event Processing Digital Center around which all TFS’s current and future data
sources, consumers, services, and processes interact. The purpose of the Platform is to enable business
innovation and agility by providing semantically-cohesive and structurally-flexible harmonized data across
processes and systems and by bringing functions and capabilities to data instead of moving data.”
External Services:
• Bank
• USPS
• CRA
• FICO
• Dealer Daily
• Auto IMS
• Others
• …
TFS Applications:
• Carlos
• Shaw
• LeMans / ITS
• SMG3
• MPS
• UCM
• Others
• …
TMNA Applications:
• Toyota BigData
Platform
• Customer 360
• NVS
• BI
• Others
• …
Systems of Engagement:
• Call (CEP)
• txt
• Web
• Mobile
• eMail
• Chat
• mail
• Other 3rd party
• …

Enterprise Integration Platform – PaaS View
“If we consider EIP as a Platform-as-a-Service then conceptually we can identify three types of
interoperating services: Application Platform Services; Streaming Data Processing Services; and API
Aggregation and Processing services.”
Application Platform Services
Streaming Data Processing View
Enterprise Data Platform (EDP)
API/Service Processing View
Enterprise Services Platform (ESP)
Application Platform Services

EIP – Architecture Realization

EIP Technology Choices
EIP Data Ingress and Egress
EIP Application Platform
2
3
3 3
1
EIP Data Persistence Layer4
5
6
EIP Edge Gateway
1
2
EIP Stream Processing Service Layer5
4
EIP Aggregate Services6
“After a highly competitive selection process TFS chose IBM as the main partner and technology vendor
to take the conceived EIP from a reference model to an operable physical Event Processing Platform.”
The technology choices for the main EIP
components are discussed in the
subsequent sections:

ICP brings the following features and functions to EIP:
§ A unified installer: for rapid set up of a Kubernetes based cluster with master, worker, and proxy nodes by using
an Ansible based installer.
§ Cluster management console: for managing, monitoring, and troubleshooting of EIP applications and cluster from
a single, centralized, and secure management console.
§ Private Docker image registry: to provide a local registry service that functions in the same way as the cloud-
based registry service Docker Hub but with the additional ability to restrict which users can view or pull images.
§ App Catalog: provides a centralized location for browsing and installing packages in the cluster.
§ Isolated Tenant Networks: Calico network overlay allows for improved performance and network isolation inside
the EIP cluster by providing an isolated subnet for each project inside the cluster.
§ Robust monitoring and logging with ELK stack: Elasticsearch, Logstash, and Filebeat are used for the collection,
storage, and querying of logs and metrics.
EIP Application Platform Technology Choice
“IBM Cloud Private is selected as an application platform for developing and
managing EIP’s containerized applications and services. ICP is an integrated
environment for managing containers.”

EIP API Gateway Technology Choices
“EIP implements API Gateways pattern for exposing services’ endpoints to the
consumers. Desired features for the API Gateway are to be open source and
Kubernetes-native.”
§ Several open sources API Gateways were considered including Ambassador, Tyk, and Kong.
§ Ambassador is selected as the most suitable API Gateway technology for EIP for the following
reasons:
• Architecture. Ambassador is Kubernetes-native microservices API gateway built on the Envoy Proxy.
• Self-service. Ambassador is designed so that developers can manage services directly. This requires a
system that is not only easy for developers to use, but provides safety and protection against inadvertent
operational issues.
• Operations friendly. Ambassador has virtually no moving parts, and delegates all routing and resilience to
Envoy Proxy and Kubernetes, respectively. Ambassador stores all state in Kubernetes (no database!).
Multiple Ambassadors can be run in the same cluster, making upgrades easy and seamless.
• Designed for microservices. Ambassador integrates the features teams need for microservices, including
authentication, rate limiting, observability, routing, TLS termination, and more.
• Open Source. Ambassador is an open source API Gateway.

§ Several technology options were considered including NiFi, Sqoop, Flume, and Oozie.
§ NiFi is selected as the most suitable Ingress/Egress technology for the following reasons:
• Platform rather than point solution tool,
• Support for broad source types,
• Usability & Agility (Speed of development),
• Monitoring/Migration and Management of flows,
• Bi-Directional Movement,
• Security,
• Visual Lineage.
EIP Data Ingress and Egress Technology Choices
“EIP use cases focus on managing the flow and transformation of data from
producers to consumers. Ability to Ingress data into EIP and Egress data form EIP
to external consumers is an essential feature of the system.”

EIP Data Persistence Technology Choice
“EIP needs to persists all Raw Source Data and transformed (harmonized and
materialized) data. Several storage architectures (file-based, block-based object-
based) and technologies were considered including storage; .”
§ EIP utilizes Object Storage architecture for persisting the Raw Source Data for several reasons including:
• Its compatibility with cloud computing,
• Its scalability,
• Faster data retrieval and better recovery,
• Fewer limitations compared to the traditional file or block-based systems,
• Cost-effectiveness.
§ EIP utilizes NoSQL document database - MongoDB - for persisting Harmonized and Materialized contract
related information for several reasons including:
• Storage architecture is aligned with the structure of contract information,
• Simplicity of retrieving all attributes for a specific account - no need to join multiple tables in relational database,
• Ability to handle high volumes of data,
• Efficient, scale-out architecture instead of monolithic architecture.

§ EIP standardizes on Apache Avro data format. One of the features of Avro critically important for the
EIP implementation is its ability to define schema for the data.
§ Confluent Schema Registry is used to manage the EIP Avro Schemas. The reasons that inform the
choice of the registry include:
• It provides RESTful interface for managing Avro schemas,
• It allows the storage of a history of schemas which are versioned,
• It supports checking schema compatibility for streaming database,
• It supports compatibility levels (BACKWARDS, FORWARDS, FULL, NONE) setting for the Schema Registry
and an individual subject,
• Specific to Kafka implementation - when using the Confluent Schema Registry - there is no need to send
schema with each record just the schema id from the registry. This increases performance - speeds up the
serialization as only the id of the schema is sent.
EIP Data Format & Schema Management Technology Choices
“Establishing the schema is essential for EIP’s success as a stream processing
platform. The ability to evolve the schema allows TFS to speed operations without
impacting other consumers and producers.”

EIP Stream Processing Technology Choices
“EIP needs to process data in real-time, continuously and concurrently. Stream
processing is an essential feature of EIP.”
§ Options considered for the stream processing include Apache Spark and Kafka Streams.
§ Apache Kafka in conjunction with Kafka Streams is selected as the most suitable streaming technology
for EIP due its ability to directly addresses many difficult problems in stream processing including:
• Event-at-a-time processing (not microbatch) with millisecond latency.
• Stateful processing including distributed joins and aggregations.
• A convenient Domain Specific Language (DSL)
• Windowing with out-of-order data using a DataFlow-like model.
• Distributed processing and fault-tolerance with fast failover.
• No-downtime rolling deployments.

§ The Four-Tier model establishes a foundation for integration of both internal services and applications as
well as external services and APIs – as required by EIP.
§ The Four-Tier Engagement Platform is broken into client, delivery, aggregation, and services layers:
• The client tier generates events that leverage the delivery, aggregation, and services layers to generate
event-specific responses (harmonization, materialization, including responses to invocation of business services).
• The delivery tier is responsible for optimizing delivery of events and data between the platform components. NiFi
and Kafka constitute a scalable event-driven delivery backbone of the EIP architecture.
• The aggregation tier serves as a hub for integrating internal and external services using real-time, bidirectional
communication. Services in this tier follow aggregate design pattern.
• The services tier provides the other tiers with the data and functionality they require. The services in this tier follow
microservice architectural style.
EIP Aggregate Services
“EIP is based on a new architectural approach that facilitates agility, flexibility, and
scalability required by the platform. The approach is described as “The Four-Tier
Engagement Platform” by Forrester Research.”

EIP Reference Architecture Realized
“The TFS & IBM innovative solution for EIP is based on IBM Cloud Private deployed in AWS with several
open-source technologies deployed on top of it. The solution is cloud agnostic and utilizes microservice
architectural style.”
IBM Cloud Private deployed in AWS is the application platform
for developing and managing EIP’s containerized applications
and services.
Edge Gateway is
Ambassador - Kubernetes-
native API Gateway built on
Envoy, designed for
microservices.
Enterprise Data Layer is
based on MongoDB –
document oriented NoSQL
database.
Raw data is persisted in AWS
S3 object storage
Access & Delivery Services
are based on Apache NiFi –
automates the flow of data
between EIP and external
systems.
Enterprise Services Layer
is based on:
• Confluent streaming
technologies (Kafka, Kafka
Streams, Schema
Registry).
• Microservices – all
services are containerized.
• Istio brings service mesh
capabilities to EIP.

EIP – Physical Architecture

EIP Low End Environments
“The EIP low end environments consist of three ICP clusters dedicated to Development, Test and
Integration. Each cluster consists of a single Proxy, Master, Management and Vulnerability Advisor nodes
and nine Worker nodes.”
§ EIP low end ICP environments are deployed in a
dedicated VPC that is peered with the TFS VPC
§ The ICP clusters are deployed in a single AZ and
a single Region.
§ The EIP raw data is persisted in the AWS S3
object storage.
§ The EIP EC2 instances use AWS Elastic Block
Storage (EBS).
§ Each ICP cluster has four classes of nodes (Boot,
Master, Worker, and Proxy) and two optional
nodes (Management and Vulnerability Advisor).
§ Only one Boot node is required for any cluster. A
single node is used as both Master and Boot.
§ Build Node is used to create the AMI that includes
Docker and ICP binaries. This image is then used
by Terraform for the ICP installation on all other
nodes..

EIP Production Environments
“The EIP production environments consist of two highly available ICP clusters dedicated to Staging and
Production. The cluster components are deployed in three Availability Zones for high availability. ”
§ EIP production ICP Clusters are deployed in a
dedicated VPC that is peered with the TFS VPC.
§ Each ICP cluster consists of three instances of
Master, Management, VA and Proxy nodes and
nine Worker nodes.
§ The ICP components are deployed in three AZs
and a single Region. Each AZ contains one
instance of Master, Management, VA and Proxy
nodes. The EIP production clusters tolerate
failure of one AZ – fault tolerance (3-1)/2 = 1.
§ Master nodes share EFS storage per ICP
requirement.
§ Nine worker nodes are distributed across three
AZs – three nodes in each Zone. Kubernetes
scheduler distributes the Pods across the nodes.
We use affinity and anti-affinity rules to direct
how to Pods are placed.

EIP Infrastructure as Code (IaC)
“EIP uses Terraform to automate provisioning of infrastructure. Terraform’s infrastructure as code
approach provides effective, reusable, and safe infrastructure provisioning automation. This approach
enables operators to increase their productivity, move quicker, and reduce human error. ”
Infrastructure as Code approach offers a number of immediate benefits including:
• Speed – Deployment of infrastructure through automation is much faster compared to manually navigating through
interfaces to deploy and connect up resources (deployment time reduces from days to minutes).
• Reliability – Manual deployment of a large set of infrastructure is prone to misconfiguration and provisioning services in
wrong order. With IaC the resources are configured exactly as declared, and implicit/explicit dependencies can are used to
ensure the creation order.
• Execution Plans – Terraform has a planning strategy where it executes a dry run to the configuration. It shows the changes
to be made to infrastructure before making the actual changes.
• Experimentation – With the ease at which the infrastructure can be deployed, experimental changes can be readily
investigated with scaled down resources to minimize the cost. Once approved, everything can be scaled up for production
deployments.
• Resource Graph – The tool creates a graph of all the resources and shows the dependency on one another.
• Dev/Ops & Best Practices – Writing code to design and deploy infrastructure brings code development best practices to
cloud provisioning like writing modular, configurable code committed to version control. This leads to view our infrastructure
as a software application in itself, and shifts the infrastructure deployment in the direction of a DevOps culture.

EIP Automation

From Empty VPC to Fully Operational EIP in Minutes
“Deployment of EIP is fully automated using three pipelines – (1) Infrastructure; (2) Technology; (3) Services”
1. Infrastructure Pipeline deploys infrastructure and ICP
2. Technology Pipeline deploys technology components in ICP
3. Services Pipeline deploys custom developed microservices in ICP

EIP Automation Benefits – Multi-Regional Deployment of EIP
“Although developed and configure as the TFS Digital Center, EIP can be deployed and configured as a
Digital Center in other geographical regions with minimal effort and cost.”

EIP Automation Benefits – Disaster Recovery
“Disaster recovery of EIP is straight-forward and cost effective: Backup regularly Master VM & EBS
snapshots into S3; Configure and perform regular cross-region S3 replications. In the case of a disaster:
Perform automated deployment of the EIP infrastructure at the DR Site; Restore VMs and EBS from
snapshots.”
Primary Site
N. Virginia
DR Site
N. California
Cross-Region S3
Replication

Use Cases

EIP Enables TFS’s Digital Transformation – ESP Use Case
“EIP deploys and integrates technologies that foster TFS’s digital transformation. The TFS Mainframe
resources are exposed as RESTful APIs using z/OS Connect; The APIs are aggregated into reusable
coarse services that capture meaningful bounded business context; Business Service are orchestrated to
deliver high-level business functions.”
1
Due Date Change (DDC) Use Case:
1) CSR Requests DDC,
2) Controller Service receives the request,
puts it on Kafka Topic (guaranteed
delivery) and sends the ACK response,
3) DDC Service picks up the request for
processing,
4) DDC Service is an aggregate of six APIs
exposed by z/OS Connect,
5) DDC Services invokes the APIs and
executes relevant business logic,
6) Once done, DDC Services invokes
Responder Service to deliver the response
to CEP,
7) Responder Service delivers the response
to CEP.
2
3 4
5
6
7

§ ESP implementation of DDC use case illustrates the potential business benefits including:
• Faster implementation of business-functions (compared to the manual implementation the ESP implementation
od DDC takes minutes vs several days),
• Increased CRS productivity,
• Increased customer satisfaction,
• Increased data consistency.
§ The full benefits offered by the platform will come with enriching the API and Business Service
portfolios. The platform facilitates the enrichments through:
• Supporting aggregation on APIs through configuration vs development,
• Reduced cost and simplified operations through containerization and orchestration,
• Support for Dev/Ops and CI and CD.
§ ESP enables cost-effective reuse of mainframe assets for delivery of new services leaving open
potential APIs refactoring of the Mainframe if feasible and justifiable.
ESP Benefits – Bringing Mainframe into API Economy
“ESP provides platform for consuming Mainframe resources exposed as APIs via z/OS Connect and
aggregated into high-level Business Services, to deliver business value above and beyond capabilities of
the constituent APIs.”

EIP Enterprise Data (EDP) Use Case
“EDP is a stream processing platform capable of efficient real-time and batch data processing. Batch files
are streamed by NiFi-based batch Bootstrap Service, and then harmonized and materialized into MongoDB
Collections. Real-time processing differs only in a type of Bootstrap Service used – it needs to be able to
receive the real-time requests.
Batch Processing Use Case:
§ NiFI Bootstrap Service:
1) Receives Batch Files,
2) Persists the Raw Data in S3,
3) Stream the file records to Kafka Harmonization
Topic.
§ Harmonization Service:
4) Picks the records of the Topic,
5) Harmonizes data to a single point of truth and
persist the harmonized data in MongoDB
Collections,
6) Streams data to Kafka Materialization Topic.
§ Materialization Service:
7) Picks the data of the Topic,
8) Transforms the data in a format appropriate for
consumption by the consumers,
9) Persists the data in MongoDB.
§ Delivery Service:
10) Delivers materialized data to the consumers.
1
2
3
4
5
6 8
9
10

The business benefits offered by EDP are manifold including:
• Processing data in motion at speed; processing time for daily CEP batches of over 10M records was reduced
from the hours to minutes,
• Serving the right data with the desired quality at the right time to the right internal and external customers;
Improving data quality,
• Enabling planning and execution of data-oriented projects on-time, on-budget, and on-objectives,
• Lower maintenance and reduced support cost of integration systems and the supporting infrastructure; new
systems can be integrated into EDP via configuration rather then code,
• Support for Dev/Ops and continuous integration (CI) and continuous delivery (CD),
• Increasing business agility by providing semantically-cohesive and structurally-flexible harmonized data
across processes and systems and by bringing functions and capabilities to data instead of moving data.
EDP Benefits – Event-Based Processing of Streaming Data
“EDP implementation is characterized by event based, loosely coupled microservice acting on streams of
messages, minimized movement of data sets, and harmonized information across functional domains.”

Toyota Financial Services to the Power
of IBM
Thank you for your
time!

Toyota Financial Services Digital Transformation - Think 2019

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Toyota Financial Services Digital Transformation - Think 2019

Similar to Toyota Financial Services Digital Transformation - Think 2019 (20)

Recently uploaded

Recently uploaded (20)

Toyota Financial Services Digital Transformation - Think 2019