This session was recorded in San Francisco on February 5th, 2019 and can be viewed here: https://youtu.be/f4b2Yoe9JEs Combining H2O Driverless AI, H2O-3, and AWS for developing and deploying AI solutions on scale. Martin Stein is a seasoned Product and Marketing executive with a successful track record delivering large-scale advanced analytics and marketing analytics services and products. Martin has served as Board Member, C-Level Executive and subject matter expert in a variety of industries (Marketing, Finance, Real Estate and Media). Currently, Martin as Chief Analytics Officer for g5, a leader in real estate marketing optimization. G5 is a predictive marketing SaaS company that uses AI and other emerging technologies to help marketers amplify their impact.

1. Driving Marketing
Performance with
H2O Driverless AI
Martin Stein
Chief Analytics Officer
G5
@GetG5
#H2OWORLD

2. Introductions
Martin Stein
Chief Analytics Officer
@deepmarketingio
Jeff Hazel
Senior Software Engineer

3. Real Estate Marketing Optimization

4. Paid
Media
Earned
Media
Owned Media
Paid Advertising
LISTINGS
Social Pages
PROMOTE
SOCIAL
ADS
Google My Business
Websites
Search Ads
Display Ads
Remarketing Ads
Facebook Ads
REVIEWS
Blog
SEO
Marketing Channel Efficiency

5. 90% of Interactions are Call based
1 out of 7 Calls has commercial intent
Quick follow up matters
...Calls Matter Most

6. Need to Classify Caller Intent
Benefits For Marketers
• Track Qualified Leads (QL) Including Calls
Automatically!
• Measure Marketing Channel Efficiency
• Rank channel sources (including ads) by
Leads, Qualified Leads and ratios
• Filter and track individual channels
Benefits For Operators
• Separate View for phone agents and operators
that shows ranked Leads by propensity to lease
and recency for quick and prioritized follow up
Use NLP to score and rank calls
with the highest revenue potential

7. Welcome to G5 Call Scoring

8. Turning Vision into Reality
• Project: 3 months (including building a training set)
• Core Team: Martin, Jeff and Kirk
• Data Science Stack: AWS, H2O-3, H2O Driverless AI & Rstudio
• Production Environment: AWS, Python
Evaluation of
Transcription
partners complete,
IBM Watson Selected
Scoring Rate
Stabilizes at 250
leads per day and
about 1600 total calls
per day
Begin Collecting hand
scoring data
(Humans listen to
calls and evaluate
based on criteria)
Production pipeline created
for mass transcription of
hand-scored calls
February 2018 March 2018 April 2018
EMR Cluster created for
feature building using
transcripts and call
metadata
H20 Driverless AI first-run on
call data with Hand Scoring
results, Transcript data, and
Call Metadata
Production pipeline
built using lamba’s
leveraging Call data,
Watson Transcription,
and H20 MOJO

9. Getting from Data to Prediction
AWS: EMR
● Rstudio
● H2O-3
● Spark 2.x
AWS: S3 Hand
Scored
Calls
H2O Driverless AI (on EC2)
w2v
Model
Call Scoring
Model
AWS: Lambda
G5’s
Feature Builder
The Same Lambda for
model building and prod!Features
(meta, w2v)

10. Modeling with H2O Driverless AI

11. Getting from Prediction to Product
For a Predictive Model to Become a Viable
Product, we Need:
Reliability | Reproducibility | Scalability | Iterability
We were able to achieve all of these with H2O on AWS

12. Getting from Prediction to Product
Phone Call (mp3)
AWS: Step Functions
Transcribe
Vectorize
w2v
Model
meta data
Score
Call Scoring
Model
Call Score

13. Getting from Prediction to Product
Key Features of H2O POJOs and MOJOs:
• Model changes without code changes
• Seamless transition from R training to python deployment
• Versioning and accuracy tracking
Key Features of ML on AWS:
• On demand compute in EC2 helps H2O driverless.ai achieve potential
• EMR makes data exploration and model building easy
• S3 integrations across products makes moving data painless
• Lambdas allow for fast deployment of models that SCALE
• Step Functions allow us to organize business logic without sacrificing pipeline
quality

14. Business Outcome: Metrics and Results
97% AUC on
Holdout Test Set
5x faster
Model Building
2 technical staff
Reduce Scoring Time by
vs Competitors
99%
~3 Minutes Time Savings Per Call vs Human Listening
95% Real World
Scoring Accuracy
@1 Million Calls Per Month 50k Hours
=