Experteer for ICMA: Applied Machine Learning. Smart Process Automation with AI

© 2017 Experteer GmbH
Experteer for ICMA: Applied Machine Learning
Smart Process Automation with AI
AlexanderChukovski, ExperteerGmbH

2© 2017 Experteer GmbH
Machine Learning – Believe The Hype?

Machine Learning is NOT JUST A TREND.
“Personalization and
recommendations
with Machine
Learning saves us
more than $1B per
year in cancelled
subscriptions.”
2016 - The Netflix Recommender System: Algorithms, Business Value, and Innovation
https://dl.acm.org/citation.cfm?id=2843948

Introduction to Machine Learning
Machine learning is a technology that automatically finds patterns in your
data and uses them to make predictions for new data points as they
become available.
Machine
Learning
Algorithm
Trained
Model Prediction: CAT
Prediction: Hot Dog

Introduction to ML – Only a Hype? No.
Two factors:
1: Cheap cloud storage capable of storing data
Moore law does not only apply to CPUs, it also
applies to Storage.
2: Computer power has exploded
(GPU performance is 10^6 times higher than 1995).
It is a fact: ML has been around (theoretically) since1960. What changed?
Finally we have everything to actually APPLY the theoreticalpart!
And sure, there are some new algorithms (LTSM, Reinforced Learning, etc).

1900 – 1990 Industrian Engineering / Scientific Management
A few steps back in business and management
history…
Taylor applied EngineeringPrinciples to the optimization of Labor
Frederik Taylor
Manager -> Scientific Study of Tasks-> Process & Planning of Wor

1990 – 2016 – Re-Engineering Movement
Data -> Manager -> Rules (Business Logic) -> Developer -> IT System
IT SYSTEMS
Computers did not just speed up processes or automate, they
would allow executives to rethink operations from the ground
up.

Smart Algorithms aim to redesign business process just
like humans did in the original re-engineering movement
Historic Data
Rule 1
Rule 2
Rule 3
SMART ALGORITHMS
Constant Re-Design
2016 – … Machine-Reengineered Business Processes
Data -> Algorithm-> Rules (Business Logic) -> IT Process

What is in it for me?
Industrial Engineering
Efficiency
IT Systems
SMART ALGORITHMS
10%
200%
20%

Early Adopters Confirm!
Harvard Business Review 2016
Study of 30 Early Adopters for machine Learning:
50% Improvement Top-Line;
15-75% Cost Reduction;
https://hbr.org/2016/02/companies-are-reimagining-business-processes-with-algorithms

How we used ML at Experteer - Introduction
Short Introduction
• Exclusive network with 5+ mio senior level candidates;
• 20K+ Recruiters
• 300k+ Job Postings with Salary > €60K/$100K
• 12 Countries, Headquarters in Munich
• Moved to Munich in 2006 from Sofia, Bulgaria;
• Studied Finance/Statistics/Econometryin LMU Munich;
• Responsible for all ML/Deep Learning R&D & Operations at
Experteer;
• Heading the Data Services Business Unit offering HR
AutomationServices to internal and external clients;
• Heading the Job Advertising Business Unit for third-party
advertisers on Experteer.

We experienced the 5x…
2014 2017
Jobs on Experteer: 80K Jobs on Experteer: 300K
OperationalCost: 100K OperationalCost: 45K

How we used ML at Experteer – Our Problem
State2014:
Highly customizedand manualprocess of job aggregation,processingand classification:
• Team of 80+
• Hand-pickingand classifying jobs (90% left-out);
• Extensive Job ClassificationOntology
• 19 Functions
• 631 Industries on 4 Levels
• 8 Career Levels
• Location, education, company and subsidiary, salary,education, travel requirements
• 7 Languages; 12 Countries.
Major asset: Extremely good quality of the positions. 2 mio+ hand-classified jobs in multiple
languages.
JOB COST: 3€/JOB

How we used ML at Experteer – Our Solution
Sourcing Company DB Extraction Classification Filtering
Salary
Benchmark
Yield Quality
User
ATS
Jobboard
Aggregator
B2B Client
Crawler
Import API
Direct
Posting API
Company
Matching
Company
Classification
Job Title
Extraction
Text
Extraction
Salary
Extraction
Location
Extraction
Company
Job Title
Cleaner
Language
Career Level
Industry
Function
Location
Finder
Travel
Requirem.
Real Job
Title
Real Job
White/Blue
Collar
Salary
Calculation
60K + Filter
Deduplication
Job Yield
Setting
Check
Automatic
Classificat.
Retrain
Machine
Breakdown of the whole process in small steps and building a ValueChain
Data Lab – manage/create rules for extraction/classification QC UI

• Building teams – indicatorfor people manager
Job Descriptions are Hard to Read
Manager, App Store Program Management
Job Summary
Appleis seeking a Manager for the App Store Program Management Team. This rolewill lead a
team of engineering program managers responsible for end-to-end delivery of App Store features
across iOS, macOS, and tvOS platforms.
Key Qualifications
8+ years of professional experience in software program/project/product management
Proven experience in building and managing high performing teams and individuals
Proven track record in managing and deploying large,complex programs
Strong relationship management and facilitation skills both within diverseengineering teams and
cross functional organizations
Proven self-starter, who is pro-active and demonstrates creative and critical thinking abilities
Great attention to detail and organized
Excellent written and verbal communication skills
Overall, a highly driven,results-oriented,problem solver who will drive programs to deliver value
quickly to our customers
RequiedExperience
- Understanding of mobile software development
- Understanding of server-based software development
- Knowledge of iTunes Connect,App Store, iOStechnologies
Description
We are looking for a seasoned manager with a proven track record in program management. This
is not just a peoplemanager role, you will need to have hands on experience managing software
releases. You will develop tools and processes to gain efficiencies in the build,development,
testing, and deployment lifecycle. The role requires a combination of program and release
management, strong engineering background,and ability to build collaborative relationships
across various teams in Apple. We are looking for someonewho loves digging into details,building
teams, and driving operational efficiencies under demanding timeframes. You take responsibility;
you feel a personal stake in the product you ship; you communicateresponsibilities and scope
clearly; you value integrity; you manage risk; you need to know how things work; you work for the
success of the entire PM team; you thrivein uncertainty and strive to bring order to it; you have
deep wisdom and judgement; you keep your eye on the ball; you build strong relationships; you are
aware of politics but do not get mired in them.
Education
BS/MS in Computer Science, Engineering or similar technical field
1
• Education is an indicatorfor function.
• Good indicatorfor the function selection
• “building and managinghigh performingteams” – this
is a strong indication fora people managercareerlevel;
• Indicates the industry– Software companies;
• Indicatorfor at least people managercareerlevel
• Indicatorthe function
• “Manager” is a soft indication, has to be looked into
context. It could indicate management responsibilities,
butit dependson the rest ot the responsibilities.
7
1
2
3
4
5
6
8
2
3
4
5 6
7
8
Career Level Function Industry

Details on the Development Process
• Trained multiple classifierand experimentedwith a diverse set of ML Algorithms –
achieved 60% baseline;
• 100K+ business logic rules to compliment the baselineand get to 80% combined
score;
• Automatedeach step one by one – each step is a separate service (assembly line
architecture with microservices and event queue);
• Microservices deployed in AWS for flexible, on-demandscalability;
• Step-by-step roll-out of organizationalchanges in order to support migration from old
to new tech.

0%
20%
40%
60%
80%
100%
120%
140%
160%
0
50000
100000
150000
200000
250000
300000
350000
400000
Live Jobs
Cost Change %
Goal: Double the Jobs/ Half the Costs
LiveJobs
And we managedto build a nice small business on the side…
Unit Cost
3 0€ per Job

Dos and Don’ts
Don‘ts
• Project scope too large
• ... Leading to lots of unknown factors
• ... Leading to lots of moving targets and moving
know how
• ... Leading to a too optimistic roadmap and
expectations
• Mistakes of the first implementation supported
general team scepsis for whole project
• Humans have feelings. Your human generated
data is not a constant quality
Dos
• Big Bold Goals (z.B. „double volume & half cost“) to
radically threat the status quo
• Scoping: less is more
• Think in small and simple modules
• Accept that there are reasons for Politics/People
being sceptic/Fear and try to moderate them
• Unlearning as a growth opportunity
• Follow ML Cloud providers and learn from their
models
• Follow Google and Facebook as industry leaders in
ML/Deep Learning

Technology Management & Technology Know-how is
important
BUT
Change Management & Leadership are the essential
requirements for success

Typical Business Processes You Can Automate with ML
today
A huge list of decision processes for which you have enough labeled data:
• Predict if a subscriber will upsell or not and optimize your marketing campaigns;
• Recommend content, products,
• If a user will churn based on his/her behaviour and automate actions to prevent this:
• Predict the likelihood of a B2B lead to turn into a customer;
• Classify text in different categories;
• Be able to say the sentiment of a text (positive, negative, neutral);
• Summarize long texts into short ones without loss of meaning;
• Classify Images in different categories;
• Predict an answer from a defined set to a question;
• Self-driving cars and computers playing games.

EXCUSES
BUT I NEED A HUGE TEAM OF DATA SCIENTISTS AND TONS OF HARDWARE!
BUT I AM NOT A TECHNICAL PERSON!

Behold…

ML in the Cloud – MVPs in HOURS!
Behold…the power of the Cloud!
- Interactive visual tools that guide you through and help create the model:
- Powerful data transformationtools to maximize the predictive quality of the model;
- Performancevisualizationso that you can fine-tune your model;
- Cover the most common ML-Algorithms that are proven to provide business
value(Classificationand Regression);
- Create an API with a few clicks that can be immediatelyused in production;
- pay per use VS large infrastructure investments;
- fully managed – low devops cost;
- you can trial all services and test wild hypothesis for a few $.

Cloud Providers Overview
Target:
Dev / Broad group
Solving:
Narrowly defined needs;
Mostly predictions
UI:
Very simplistic
Target:
Broad group
Solving:
Wide selection, build your
own + Marketplace
UI:
Azure Studio ML - building
models with a nice and
simple UI – great for non-dev
ppl
Target:
Broad group
Solving:
Fixed selection of closed
algorithms
UI:
APIs and Rest-Services.
Minimalist General PurposeFixed-Offering

Detailed Look: AWS
Three types of ML Models:
Binary Classification Model
•"Is this email spam or not spam?"
Multiclass Classification Model
•"Is this product a book, movie, or
clothing?"
Regression Model
•"What price will this house sell for?"

Detailed Look – IBM Watson
Language
Speech
Conversation
Translation
Classification
Speech to Text
Text To Speech
Chatbots
Empathy
PersonalityTests
Tone Analyzer
Visual Image Recognition

Detailed Look – IBM Watson Knowledge Studio
Train Watson with a WEB UI to understand YOUR BUSINESS

Microsoft Azure ML Studio
SERVERLESS
DRAG-N-DROP / CODE FREE
API IN MINUTES

IT IS PRACTICALLY LIKE LEGO!

Model Precision
In just a few minutes we have a very good working model with an API-
Endpoint that we can use immediately.

Algorithm-as-a-Service
But wait, it can be even easier:
What is algorithm-as-a-service?(HOT TREND)
• Already-deployedand trained models for a range of tasks;
• Test free in browser UI;
• Pay-per-Api-Call.

Algorithm-as-a-Service Example in CS with Algorithmia
Hereis one way you can bring Basic processautomation to your CS:
Complaint
AutomaticAnswer
Distribute to CRM

How Do You Get Started – Building your Model
Define
Problem
Collect
Labeled
Data
Clean
Data
Train
Model
Check
Results
Retrain
Adjust
Data
Lessons learned hard:
1. Get to know your data well. The quality of human-classifiedhistoric data can vary a lot
and this will influence the precision of your prediction.
2. If you don‘t have training data, you can generate some with AmazonTurk (10K Dataset =
$3K) or use RegExp to discover paterns and hand-cleanthe data.
3. Don‘t reinvent the wheel – most common business problems are solved. Get familiar
with the ML-Algorithm cheat sheets.
4. Don‘t jump immediatelyon latestDeep Learning Paper. „Traditional“classifiers almost
always do the job.

What Else Are We Currently Working on?
We experiment with:
• Deep Learning and CNN for Text Classification;
• ML in the Cloud for quick hypothesis evaluation(Azure, AWS, Google Cloud);
• Text Sentiment
• Consumer Churn
• Job PerformancePrediction(Views/Applications)
• User upsell probability
• Support-vector-machines/NeuralNetworks for string classificationand cleaning of job
titles.

Our Offer: Use our Tech or Profit from our Knowledge
API Services
• Automate your job processing stack and enrich your job content;
• Profit from a constant flow of new services;
• Highly scalable, flexible and modular;
• Customize our APIs to suit your data processing needs.
Consulting / Solution
Delivery
• We analyze your value chain and propose ML automation;
• We do complete R&D, tech analysis and feasibility study;
• Use our knowledge, experience and support to build yourself or let
us build it for you;
• Fast prototyping to assess quality of proposed solutions;
• Support in organizational restructuring and roll-out plans.
Scraping / Quality
Job Back-Fill
• High-quality, enriched jobs from all major EU countries/US;
• Dedicated custom scraping of career pages on demand;
• Backfill services based on industry, country, salary, work type;
• Highly-scalable, end-to-end solution.

Our Idea of a Workshop
Our AI Workshop:
• We will show you how your business can profit from AI
and Machine Learning;
• 2 Days Workshop with the leadershipteam of
ExperteerData Services;
• Deep-Dive into your processes and your data;
• Focus on your automationrequests;
• Solution proposals alreadyat the end of the Workshop;
• Technical Whitepaper with concrete analysis,
solutions, optimizationpotential and path to
implementation.

Our Competence
Business
Owner
Data
Scientist
Developer
Binary
Classification
Multiclass
Classification
Regressions
and
Predictions
Problem
Specification
Data
Evaluation
Data
Preparation
Model
Selection
Optimization
and Training
Architecture
Deployment &
APIs
Roll-out and
change mgmt
Quality and
KPIs

How is Our Service Structured
Deliverable 1
Actual Analysis &
Preparation
• First call to assess
the actualsituation
• Extensive
questionnaire and
discussion with client
• Preparing a detailed
workshop agenda
Deliverable 2
Onsite Analysis
& Deep Dive
• Detailed analysis of
existing business
processes
• Problem Definition
• Analysis of existing
data
• Interviews
• IT Architecture
• KPIs
Deliverable 3
Concrete
Proposals
• Discussion ofour
automation proposals
• Prioritization with
client
• Detailed whitepaper
with concrete next
steps
Day 1: Onsite Day 2: OnsitePhone Interview

Our Team
AlexanderChukovski
• Education: LMU Munich,
Statistics/Finance;
• 10+ y. work experience
in the HR industry;
• Built the Data Science
Department of Experteer
from scratch;
• Multiple management
roles at Experteer with
focus on product
development, web
technologies,
automation and data
quality.
Mariia Sendziuk
• Education: Computer
Science, Uni Breslau;
• 3+ in PM role at
Experteer;
• Our data handling
specialist;
• PM for Data Science,
Web Development,
Quality Management;
• Deep knowledge of
Ruby, SQL, Python.
Daniel Sudmann
• Education: Computer
Science, TU Munich;
• 5+ of Back-End
Development at
Experteer with Ruby,
Rails, Python, Javascript;
• Responsible for the
development of the
internal data processing
platform;
• Responsible for the
development of the
machine learning tools.

Example Workshop Structure
Step 1: Scope
In a 1-hour telephone call you will tell us more about your organization and the processes that you want to
automate;
Step 2: Questionnaire
We will follow-up with a list of concrete and specific questions to assess your processes and available data;
Step 3: Workshopoffer from Experteer
Depending on the input of Step 1 and Step 2, we will create an agenda and send an offer for the workshop;
Step 4: 2-3 Day Workshop
Depending on the scope of the project, we usually do a 1-day deep dive into your processes onsite, followed by
a half-day questions and interviews on the next day. In the afternoon, we present and discuss our proposals on
how you can use AI in your processes. If your data allows it, we can build MVPs to show you the potential.
Step 5: Whitepaper
Experteer will provide a whitepaper with detailed results of the evaluation, concrete improvements, technical
planning and an implementation plan for your use or management/investor buy-in.

Contact us!
Alexander Chukovski
Director Data Services
alexander.chukovski@experteer.com
+49 15150444499

Experteer for ICMA: Applied Machine Learning. Smart Process Automation with AI

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Experteer for ICMA: Applied Machine Learning. Smart Process Automation with AI

Similar a Experteer for ICMA: Applied Machine Learning. Smart Process Automation with AI (20)

Último

Último (20)

Experteer for ICMA: Applied Machine Learning. Smart Process Automation with AI