Increasing usage of sophisticated machine learning algorithms to assure high accuracy in prediction and forecasting created a need for explainable AI. To solve this problem, we designed a framework which gathers the most successful explainable machine learning techniques and incorporate this framework into all machine learning pipelines. Our field practices in supply chain, manufacturing and finance proved that explainability is not just a theoretical discussion around machine learning community but a novel field that presents opportunities to create value in business processes.
2. 1
Gizlilik Sınıflandırması : HİZMETE ÖZEL
Ozgur Akarsu is the AI & Data Analytics Group Manager at KocDigital.
He graduated from ITU Industrial Engineering and finished his Ph.D. in
Organizational Studies in 2016. Ozgur has more than 18 years of
experience in digital transformation and advanced analytics in various
companies. He is also teaching business analytics and value chain
management classes at Istanbul Bilgi University.
Özgür Akarsu
AI & Data Analytics Group Manager
KoçDigital
ozgur.akarsu@kocdigital.com
5. 4
Gizlilik Sınıflandırması : HİZMETE ÖZEL
KoçDigital is a world class Digital Center, combining
strengths of BCG and Koç
• Digital capabilities and
infrastructure
• Investment for future capability
build (e.g. people, infrastructure,
products and tools)
• Commercial relationships
• World class capabilities in
advanced data analytics (GAMMA)
• Access to global network of
industry and topic experts
• State-of-the-art training and
enablement
Offerings
KoçDigital
Academy
Advanced
Analytics
Data Platform
Design & Development
IoT
Solutions
6. 5
Gizlilik Sınıflandırması : HİZMETE ÖZEL
6
COUNTRIES
KoçDigitalProjects&ProductsOverview
Significant capability built in
four years since inception
IN SALES
200M+
TRL
2-9
MONTHS
PAY BACK PERIOD
12
IoT ANDANALYTICS
PRODUCTS
200+
PROJECTS
220+
EMPLOYEES
8. 7
Gizlilik Sınıflandırması : HİZMETE ÖZEL
Demand Planning & Auto Replenishment engine was
built based on three key components
What will be right inventory level in
stores/WHS? What will be Daily
orders from WHS/supplier?
How will be optimised inventory
level in stores with transfer
algorithm?
What will be the demand in
short / long term period?
Demand Forecasting
• Segmented product category
approach
• Tailored forecasting model
• Demand sensing
on complex data
• Model Integration
Auto Replenishment & Inventory Planning
• Parameters Setting (with UI)
• Constraints & Business Rules integration
• Statistical Inventory Control
• Ordering Mechanism Optimization
• KPI (such as lost sales, in-stock rate etc.)
monitoring
S2S Transfer Optimization
• Transfer scenario selection (with UI)
• Constraints & Business rules
integration or selection (wtih UI)
• Profit & logistics cost optimization
• Transfer sales conversion monitoring
9. 8
Gizlilik Sınıflandırması : HİZMETE ÖZEL
A Machine Learning based Forecasting Model to sense the market
demand
Accurate predictions using Machine
Learning algorithms
Scalable for wide range of
SKUs and channels
Sensing the market by wide range of
internal and external variables
1
2
3
10. 9
Gizlilik Sınıflandırması : HİZMETE ÖZEL
We achieved accurate forecast results for an extremely complex
product-channel portfolio
262 Categories
5 Channels
120 Stores
Weekly forecast
4 weeks + 20 weeks
horizon
Store-location level
Lightgbm for short-term
forecast (4 weeks)
Fbprophet for 20 weeks
Error calculation (WAPE) for
category and store level
Scope Forecast Unit Accuracy
70K
Sku
Coverage
44K
Weekly
forecast
74%
Average
Accuracy*
*Results for test period for last 6 months
200+
Tailored
Features
Historic sales
Campaigns-Price
Stock
Special Events
Calendar
Covid Features
Currency rates
Macro-economic indicators
….
Features
11. 10
«How do external factors affect sales?»
«What is the effect of campaigns for model’s outputs in different
locations?»
«Model failed to predict air conditioner sales in May. Why???»
«Why are the models’ predictions lower than actuals for location
X?»
Erosion of trust for model outputs
Resistance to adopting data-driven decision making
12. 11
3 Major
Challenges
in practice
Unexpected weekly deviations
between predictions and actuals
Shortfall of the model for some
products or stores.
1
Difficulty to explain effect of
+100 model features by
analyzing the feature importance
2
3
15. 14
Gizlilik Sınıflandırması : HİZMETE ÖZEL
Our Explainable AI Intelligence Approach- XAI
Explaining the blackbox models High accuracy whitebox models
Local Explanations Counterfactuals Linear Models Trees & Rules
Model Agnostic Explanations Interpretable Models
Explain each prediction
locally by using a local model
that approximates the
predictive model
Set of features that
should be changed in
order to flip a model’s
prediction
Algorithms for extracting or
generating rules by using
linear programming with ML
Learning with subset
stacking (LESS)
algorithm tuned with
lasso
16. 15
Gizlilik Sınıflandırması : HİZMETE ÖZEL
XAI for Retail: Simple Explanations to Complex Problems
Global Local
Complex Non-linear Simple Linear
Shapley
Values
Local Explanations
XAI Dashboard for SCM
Interpretable Linear Ensembles
Lasso
(tuned)
Learning with subset
stacking (LESS)
MSE: 296.42
MSE: 315.51
Only with
20 Features
LightGBM
(tuned)
MSE: 293.59
20. 19
Outcomes
The client SCM team was able to interpret the
weekly model outputs and increased decision-
making quality for stock orders
Strategic insights for product and campaign
management
• Proven effect of changes in air temperature on AC sales for
different regions.
• Visibility of how campaigns effect sales in different regions and
products.
Target increase product availability in stores,
automated order rate, and %10 decrease in
warehouse costs.
XAI as a
business enabler
and
analytics
accelerator!
21. 20
Next Steps
Monitor critical SCM KPIs and apply
necessary revisions.
Apply counterfactual explanations
(CFEs) and rule generator (RUG) and
rule extractor (RUX) modules
Transform XAI solution to a model and
domain agnostic product
KocDgital was founded in 2018 with the cooperation of KocGrup and BCG. We are a technology company developing AA and IOT products and serving end-to-end AI solutions for various industries.
In these 4 years we have successfully completed 200+ projects in 6 different countries. Our product portfolio contains 12 unique products. Our greatest strength is our 220 employees who are specialized in different roles of advanced analytics and AI development cycles.
Last year we finished a Supply Chain digital transformation Project with Koçtaş which is the leading retail company in Turkey.
The Project consist of 3 modules. Demand forecasting was the heart and beginning point whose outputs was used for AutoReplenishment and S2s Transfer optimization
In this Koçtaş Project we built a demand forecasting pipeline based on Machine Learning using a wide range of internal and external variables to ensure accurate forecast especially for the short-term
Our solutions was scalable for wide range of SKUs, channels and product categories.
At the same time it was flexible to include internal and external data which can explain the fluctations in demand.
The Project covered 70K Skus, 5 channels and 120 sales point
Every week it is producing forecasts for 44K units
Using 200+ features, which include internal sales Dynamic +external data such as covid indexes, currency rates and macro-economic indicators such as currency, inflation or interest rate.
The model gave amazing results. Our accuracy score for Lightgbm ML model %74 on average..
The solution was deployed on MS Azure and SCM team started to monitör the outputs and determine the orders
During the incubation period of the Project we started to receive some feedback and questions from the field. To use the outputs of the model in practice, they need to deep dive to the model outputs. And the questions rose.
Our consultants and data Scientist working closely with the client team started to conduct ad hoc analysis to answer each question which is time consuming.
And we identified there is huge risk for erosion of trust for model outputs, that if people are not convinced with how model behaves and predicts, they have a tendency to work as the way they are used to instead of transforming processes using AI.
During the incubation period we saw that the client teams has difficulty to intrepret the affect of features on forecasts.
They was a certain need for explain unexpected weekly deviations for forecasts.
And for some products or stores model was constantly giving lower or higher forecasts than the actuals. We have to analyze the reasons for underfitting.
Answers for those 3 questions was XAI . To accelerate AI adoption, enable accountability for AI Outputs, provide strategic insights for business and to handle the upcoming regulatory compliance requirement, XAI capabilities should be build and embeedded in AI solutions.
Our value proposal includes not only better and fair models but better businesss outcomes as well. XAI has huge potential for business owners to understand how AI systems work and behave.
Our XAI solution portfolio contains 4 dffrerent methods, local explanations, counterfactuals, linear models and trees and rules.
We applied 2 of them for in this koctas case.
Local explanations, transform highly non-linear decision boundaries to locally explained linear models by zooming in every single prediction. But if we zoom in and look at a single prediction, the behaviour of the model in that locality can be explained by a simple interpretable model (mostly linear).
Interpretable linear ensembles, we build several tuned linear models and compared the model power with our deployed model.
So to sum up we could say that XAI has a huge potential to transform organizations. For years our focus was creating more robust and powerful models but we believe that switcing the focus to explanability and interpretability can trigger and fasten the AI revolution.
As data Scientist our focus is business outcomes and XAI can help us to achieve these goals as much as complex and sophisticated algos.