INDUSTRIALIZING INDUSTRIALIZING MACHINE LEARNING IN PHARMA: Challenges, Use Cases, & ROI by Daniel Kinney, Ashish Sharma, Kaiwen Zhong

1. INDUSTRIALIZING MACHINE LEARNING IN PHARMA:
Challenges, Use Cases, & ROI
Presenters: Daniel Kinney (Janssen), Ashish Sharma (Axtria), Kaiwen Zhong (Axtria)
1

2. AGENDA
1 Journey from Traditional Analytics to Industrialized Machine Learning
2 What Does It Take to Industrialize AI / ML?
3 Challenges and Pharma Success Stories: Why some organizations still call AI / ML a “hype”?
4 Future of Machine Learning in Pharma: ROI & sustainable growth of machine learning
2
1 Journey from Traditional Analytics to Industrialized Machine Learning

3. AI / ML Beyond Pharma
3

4. Maturity of ML / AI in Other Industries
BANKING INSURANCE HCPGOVTRETAIL
MANUFACT-
URING
NATURAL
RESOURCES
UTILITIES
37% 48% 38% 26% 44% 40% 35% 33%
% of industry
ready to
deploy or
already
deployed
large scale
AI / ML
In 12 months
Top
Use Cases:
Fraud analysis on
transactional data
Chatbots
Market / Customer
Segmentation
Process
automation
Computer-assisted
diagnostics
Text / Opinion
Mining
Source: Gartner (March 2019)
4

5. Applications of ML / AI in Other Industries
BANKING INSURANCE HCPGOVTRETAIL
MANUFACT-
URING
NATURAL
RESOURCES
UTILITIES
Automation
of
Intelligence
Process
Planning &
Optimization
Real time
monitoring
Public health
surveillance
Disaster
planning
Fraud
detection
Demand
forecasting
Cyber attack
detection
Personalized
medicine
Predictive
optimization
Fault
analysis
Forecasting
Population
health mgmt
Demand
forecasting
Lifestyle
segmentation
Credit ratings
Common
Theme:
1. Enable additional and more accurate recommendations predictions
2. Provide timely insights
3. Automate repeatable steps to reduce SME costs and human errors
Source: Gartner (March 2019)
5
High-speed
trading
Appointment
scheduling
Claims
data entry
Machine
scheduling
Autonomous
drill bits
Autonomous
monitoring
E-service
Engineering
assistants
Scanning /
mapping
Product
design

6. AGENDA
1 Journey from Traditional Analytics to Industrialized Machine Learning
2 What Does It Take to Industrialize AI / ML?
3 Challenges and Pharma Success Stories: Why some organizations still call AI / ML a “hype”?
4 Future of Machine Learning in Pharma: ROI & sustainable growth of machine learning
6

7. PMSA AI / ML Topic Clustering
Marketing (5)
HCP Targeting (13)
Patient Support (1)
NLP (7)
Provider (2)
Adverse Events (1)
New Patient ID (4)
Brand Performance Analyzer: Identifying Opportunities and
Commercial Performance Optimizer
Using AI / ML for More Insightful Pipeline Marketing Decision-Making
Automated Detection of Adverse Drug Reactions Using Social Media Text Data
A New Generation of Patient Support Programs
Application of Predictive Modeling to Define
HCP Potential for Prophylactic Treatment
Delighting Customers w/ Automated Content Personalization
Dynamically Improving NLP & Theme Discovery
Through Contextual Linguistics for Pharma
A ML Approach to Customer Acquisition
Discover latent Patient Insights Using Text Mining & ML
NLP to Understand HCP
Application of NLP & PCA in Life Sciences & Healthcare
ML Techniques Deliver Granular Insights to enable Improved HCP Marketing
Leveraging Event Pathways to Predict Disease
Progression in Contrast to ML Classification Algorithms
Applications of ML in Oncology Analytics: Combining Secondary & APLD
Advanced and Primary Analytics
A Novel Patient-Physician ML Approach on Diagnosing Rare Diseases and
Informing Commercial Strategies
Leverage Informed HCP Migration to Optimize Targeting & Multi-Channel Spend
ML Against the Provider Graph Patient Co-Pay Card Optimization
Use of ML and Patient Level Clinical Encounter Cues to Maximize Values of Sales
Force Activities
An All-Encompassing ML Attribution Model to Plan, Predict and Measure Impact
at the Channel and HCP Level
Unstructured to Structured Data: Can ML Help Make Sense of EMR/EHR Data?Strengthening Patient Journey with Emotional Insights via
Social Analytics
A Novel High-Dimensional Hybrid ML Approach Identifying
High-Value Rare Disease Specialists
Optimizing Physician Targeting to Find Undiagnosed Patients:
An Application of Advanced ML Methods to Hep C
ML to Better ID Physician Targets for New Product Launches
Enhance Accuracy and Effectiveness Of Physician Alert
7

8. Ecosystem Readiness for AI / ML in Pharma
DIGITAL LEADERS
AWS Machine Learning, SageMaker,
Comprehend, Rekognition
Verify, DeepMind Health,
Tesseract, Google Brain
ML Studio
AWS Machine Learning, SageMaker,
Comprehend, Rekognition
Verify, DeepMind Health,
Tesseract, Google Brain
ML Studio
REGULATIONS: FDA
12 ML / AI-based
algorithms approved, e.g.,
DreaMed supports diabetes
treatment decision
Viz.ai analyzes CT results and highlights
patients most likely to experience a stroke
12 ML / AI-based
algorithms approved, e.g.,
DreaMed supports diabetes
treatment decision
Viz.ai analyzes CT results and highlights
patients most likely to experience a stroke
GAME CHANGING TECHNOLOGIES
voted by CIOs
0% 10% 20% 30%
Data Analytics
AI / ML
Digital Transformation
Information Tech
IoT
CRM
Source: Gartner (March 2019)
8

9. PLATFORM
CONFIGURATION
DATA
COLLECTION
FEATURE
EXTRACTION
DATA
VALIDATION MACHINE RESOURCE
MANAGEMENT
ANALYSIS /
VISUALIZATIONPROCESS
MANAGEMENT
TOOLS
CONSUMPTION
INFRASTRUCTURE
MONITORING
ML
ALGORITHMS
PLATFORM
CONFIGURATION
DATA
COLLECTION
FEATURE
EXTRACTION
DATA
VALIDATION MACHINE RESOURCE
MANAGEMENT
ANALYSIS /
VISUALIZATIONPROCESS
MANAGEMENT
TOOLS
CONSUMPTION
INFRASTRUCTURE
MONITORING
ML
ALGORITHMS
De-Coding Industrialization of AI / ML
Most focusLeast focus
9

10. AGENDA
1 Journey from Traditional Analytics to Industrialized Machine Learning
2 What Does It Take to Industrialize AI / ML?
3 Challenges and Pharma Success Stories: Why some organizations still call AI / ML a “hype”?
4 Future of Machine Learning in Pharma: ROI & sustainable growth of machine learning
10

11. Challenges Industrializing Machine Learning
 Downplaying the Role of Iterative Automation
 Data Readiness
 Adoption Barrier
11

12. 3 Myths about Machine Learning
 Inability to show ROI if ML underperforms with poor data and short timeframe
 Inefficient one-off analysis with unnecessarily complicated ML algorithms
1. “BLACK BOX”
Most use cases do not require or are not fit for “black box”
algorithms like neural network.
2.IMMEDIATE
IMPROVEMENT
3. DISRUPTING
TODAY’S PROCESSES
Improvement of processes depends on data availability,
quality, and takes iterations.
Machine learning models do not need to be developed from
scratch. It can incorporate business rules / sit on top of rule-
based, knowledge-based systems.
HOWEVER
HOWEVER
HOWEVER
Consequences
12

13.  Increase use of publicly available data, which can
be labeled quickly and cheaply by online resources.
 ML (reinforcement learning) can also be used to
enrich training data.
It is daunting and time-
consuming to create large
amount of training data.
Data scientists & analysts
spend 80% time cleaning and
prepping data.
Data Readiness – “Garbage In, Garbage Out”
 Organizations loses competitive advantage as they are afraid to adopt machine learning and advanced analytics,.
 Underbudget for data and automation aspect of machine learning, leading to low ROI.
“DATA JANITORIAL WORK”
CREATE TRAINING DATA
SOLUTION
SOLUTION
Consequences
 Cloud & big data tech are
increasingly available for the
entire ML ecosystem.
 Data strategy, governance,
and management
13

14. Ability to Impact Decisions with ML Insights
 Inability to impact decision making eventually impact ROI of machine learning, making sustainable development
of machine learning in organization difficult.
FIELD ADOPTION
 Show relationship between increased sales and adoption.
 Tie adoption of insights into incentive compensation.
NON-PERSONAL
MARKETING
OTHER STRATEGIC
DECISIONS
 Use tools to automatically track target responses to create training data.
 Improve marketing vendor management with clear tracking system.
 Identify pain points: financial, productivity, processes, support.
 Avoid “technology for its own sake”.
SUGGESTION
SUGGESTION
SUGGESTION
Consequences
14

15. SUCCESSFUL INDUSTRIALIZATION OF ML:
Scraping Web Data for
Multiple Vaccines University
Requirements on Multiple Vaccines
Automated Predictive Targeting
with Real World Data, Payer
Insights, & HCP Data
15

16. Case 1: Predictive Targeting (1/4)
We want to identify opportunities where machine learning can be used to leverage real world
data for dynamic targeting across promotional channels
PROBLEM STATEMENT
OPPORTUNITY
IDENTIFICATION
• Call Targeting
• Call Frequency
• Product Priority
• Messaging
Optimization
• Segmentation
PERSONAL
PROMOTIONS
NON PERSONAL
PROMOTIONS
• Email
Targeting
• Media Buys
• Direct Mail
PEER TO PEER
• Speaker
Program
Attendee
Prospects
16

17. Actionable,
frequent
insights as
prediction is
precise and
specific
Case 1: Predictive Targeting (2/4)
TRADITIONAL
APPROACH
MACHINE
LEARNING
AUGMENTATION
Aggregated Physician-
level Rx Data
Longitudinal Patient-
level Data (claims)
Payer & HCP
Features
Rx / Sales
Volume
Physician Prescription
Propensity
Predicted Biologic
Initiation
ML / AI FOR AUTOMATIONDATA READINESS
BUSINESS IMPACT
17

18. Case 1: Predictive Targeting (3/4)
Scale Up & Productionize
Pilot Changes to Test & Learn
Feasibility & PoC1
2
3
 Assess data availability
 Data acquisition & feature engineering
 Minimal viable product & integration
 Conduct analysis to demonstrate POC
 Implement & continuously improve
using test/control approach
 Continuous monitoring of
performance
 Productionize data
management/analytics
processes
 Optimize data infrastructure
Problem & Opportunity
Identification/Ideation
18

19. Case 1: Predictive Targeting (4/4)
FEASIBILITY & PROOF OF CONCEPT PILOT CHANGES TO TEST & LEARN SCALE & SYSTEMIZE CHANGES
PROCESS
Assess data availability and conduct
analysis to demonstrate proof of
concept
Implement & continuously improve
using test/control approach
Productionize data management/
analytics processes & optimize data
infrastructure
MODEL
Baseline model to predict patient
demand to proof out concept
Improved model with machine
learning for improved performance
Continuous tuning & evaluation of
model performance
DATA
Minimal viable product starting with
claims data
Incorporated additional HCP & Payer
features into model
Formalize data management support
model & governance
INFRASTRUCTURE Built locally leveraging Python
Started buildout in
analytics workbench
Operationalize & automate with
integration between analytics
workbench & source systems (CRM,
reporting, etc)
CHANGE
MANAGEMENT
Sales & Marketing leadership alignment
on potential value
Piloted with test/control
districts. Trained pilot users
Scale up to entire salesforce and
declare this is ‘business as usual’ way
of working
1 2 3
19

20. Case 2: Web Scraping for Targeted Marketing (1/5)
• Gather information from 10,000+ global university (English) website domains to see if
they mandate specific vaccinations for students / faculties.
• How to identify large amount of new marketing opportunities for a launch product?
• How to make data available for this project?
• How to extract useful information from the unstructured web data?
PAIN POINTS
PROBLEM
STATEMENT
20

21. Case 2: Web Scraping for Targeted Marketing (2/5)
PLATFORM
CONFIGURATION
DATA
COLLECTION
FEATURE
EXTRACTION
DATA
VALIDATION MACHINE RESOURCE
MANAGEMENT
ANALYSIS /
VISUALIZATIONPROCESS
MANAGEMENT
TOOLS
CONSUMPTION
INFRASTRUCTURE
MONITORING
ML
ALGORITHMS
SCALABILITY ITERATION ML ECOSYSTEM
ML is scaled to gather information from
1 vaccine from 200 domains,
to
1 vaccine from 10,000+ domains,
to
Multiple products in the department
from 10,000+ domains
 ML algorithms are built on top of
rules-based system (iteration 0)
 Time, resources, and data source
domains are managed to run
multiple iterations, with results
improving over time.
All components of industrialized ML
ecosystem were considered at the beginning
of the project to enable
 Automation,
 Efficient data gathering & analysis
 Consumption of information &
decision making
21

22. Case 2: Web Scraping for Targeted Marketing (3/5)
University Domain:
http://www.Princeton.edu
http://www.Harvard.edu
http://www.Columbia.edu
http://www.Yale.edu
http://www.Upenn.edu
http://www.Dartmouth.edu
http://www.Brown.edu
http://www.Cornell.edu
…
https://www.hbs.edu/Pages/default.aspx
https://alumni.harvard.edu/community/volunteer
https://huhs.harvard.edu/sites/default/files/Immunization_pac
ket_C2023_2019.docx.pdf
https://alumni.harvard.edu/programs-events/wtyc
https://alumni.harvard.edu/travel/mode-of-travel/50-years-of-
victory
https://alumni.harvard.edu/programs-events/harvard-alumni-
online-learning
…
Domain Sub URLs: Domain Content:
Illustration
22

23. Case 2: Web Scraping for Targeted Marketing (4/5)
• Data
Gathering /
validation of
10,000+
university
domains
SOURCES
• Set up data
structures in
AWS
• Connect with
scraping-related
APIs and
libraries
CONFIGURE SERVER
• Search all sub
URLs of main
university
domains
• Use existing
and build
custom web-
crawlers using
BS4, Requests,
etc.
SEARCH & CLEAN
DATA READINESSDATA READINESS AI / ML FOR AUTOMATIONAI / ML FOR AUTOMATION
1. FILTER URLS 2. ID REQUIREMENTS
Categorize URLs related
to vs. not related to
“student health” based
on URL and main
contents.
Categorize university
websites that require vs.
not require client’s
vaccines based on
website text contents.
… with the following NLP process:
A. Keyword based categorization
B. Create thesaurus for topics
C. Iterative ML model building & evaluation
BUSINESS IMPACTBUSINESS IMPACT
ANALYZE & VISUALIZE RESULTS
DETERMINE
MARKETING STRATEGY
Different commercial strategies for
universities that
1) Requires the vaccines
2) Recommend but not require
the vaccines
3) Mention but not require the
related disease
4) Neither require nor mention
the vaccines or the related
disease
23

24. Case 2: Web Scraping for Targeted Marketing (5/5)
STEPS
1.
FILTER
URLS
2.
IDENTIFY
REQUIREMENT
Keyword-based 3 5
Traditional ML Algorithms 2 4
BoW + Deep Learning 5 1
Hybrid with Traditional ML 1 (81%) 3
Hybrid with Deep Learning 4 1 (86%)
Ranking Classification Accuracy
on Test Data
Methods
Keyword-based:
Rules-based algorithm using keywords related to “student
health”, and particular vaccine
Traditional ML Algorithms:
Feature engineering for text processing,
Algorithms include Random forest, SVM, and Naïve Bayes
Deep Learning: Bag-of-words for text processing
Hybrid with Traditional ML:
Combining the results from keyword-based model with
results from the best performer of traditional ML algorithms
Hybrid with Deep Learning:
Combining the results from keyword-based model with
results from deep learning.

25. AGENDA
1 Journey from Traditional Analytics to Industrialized Machine Learning
2 What Does It Take to Industrialize AI / ML?
3 Challenges and Pharma Success Stories: Why some organizations still call AI / ML a “hype”?
4 Future of Machine Learning in Pharma: ROI & sustainable growth of machine learning
25

26. Calculating AI / ML ROI: Why is it difficult?
 Both short term and long term values can be
spread across multiple parts of the organizations.
 It is hard to isolate the contribution of AI / ML in
improvements, especially in large business
outcomes.
26

27. Iterations to Reach Agreed Upon Target
Time
Error rate
Average error rate
by human specialists
Current error rate
Iteration 0
Target
error rate
Illustrative example:
Image classification error over time
Iteration 1
Iteration 2
Iteration N
…
Improved
accuracy
Reduced human
specialists costs
and errors
27

28. Considerations for Calculating AI / ML ROI
REVENUECOSTDELIVERY
SPEED
TEAM
EFFICIENCY
COMPETITIVE
ADVANTAGE
BRAND
EQUITY
DIGITAL
MATURITY
More
Tangible
Less
Tangible
Example:
Predict patient
diagnosis accurately
and target
marketing to HCPs
Example:
More productively
utilize highly paid
professionals & sales
reps
Example:
Provide rep NBAs
real time / daily instead
of monthly / quarterly
Example:
Extract more adverse
events per day than when
done by all manually
Example:
Identify un-
discovered side
effects faster
COMPLIANCE
RISK
Example:
Provide cost
efficient and
effective patient
support programs
Example:
Quickly react to positive &
negative media & online
coverage on the brand
Example:
Conduct targeted
and timely digital
campaigns and non-
personal detailing
28

29. Key Takeaway
You / your team does not need
to be machine learning unicorn,
But your organization should
combine three key aspects of
machine learning: data science,
business analytics, and data
engineering
For high ROI and sustainable
growth of machine learning.
MACHINE
LEARNING
Business Analytics
 Ensure ML model is set up to answer
important business question
 Communicate results to realize
impact of ML
 AND CHANGE MGMT
Data Science
 Develop and adjust ML models
 Validate and continuously monitor
model performance
Data Engineering
 Set up infrastructure to enable data
combination, and model run
 Scale models developed by data
scientists in big data environment
PLATFORM
CONFIGURATION
DATA
COLLECTION
FEATURE
EXTRACTION
DATA
VALIDATION
MACHINE
RESOURCE MGMT
CONSUMPTION
INFRASTRUCTURE
MONITORING
ML
ALGORITHMS
ANALYSIS /
VISUALIZATION
PROCESS
MGMT TOOLS
29

30. INDUSTRIALIZING MACHINE LEARNING IN PHARMA:
Challenges, Use Cases, & ROI
Presenters: Danny Kinney (Janssen), Ashish Sharma (Axtria), Kaiwen Zhong (Axtria)
30