Market Propensity Logistic Regression Model Using XSTREAMS: A Unified Batch/Streaming and ML Platform for Big Data.

1. Market Propensity Modelling Using
XStreams

2. Copyright © 2015-2017
Exadatum Software Services Pvt. Ltd.
About XSTREAMS
Why XSTREAMS?
XSTREAMS Architecture
Technology Stack
Adv. Market Propensity
Legacy Design & Issues
Feature Engineering
Modelling on XStreams
???
Agenda

3. Copyright © 2015-2017
Exadatum Software Services Pvt. Ltd.
XSTREAMS: Drag and Drop Self Serve Platform
50+ Data Processing
Operators
45+ Transformer and
Estimators for
Feature Engineering
Train, Score and
Evaluate Model in
one Pipeline
Marketplace for
readily available
blueprints.

4. Copyright © 2015-2017
Exadatum Software Services Pvt. Ltd.
Why XSTREAMS?
Sink for Error
Handling
Timeseries and
Aggregated Metrics
Actionable Alerts
Scheduling
Capabilities
Auditing Support
Versioning Support
Granular Role
Based Authorization
Checkpointing
Support.
Common features
automatically
applied to all pipelines
created

5. Copyright © 2015-2017
Exadatum Software Services Pvt. Ltd.
5
Kafka
Pubsub
Flume
RabbitMQ
Amazon S3
HDFS
Kinesis
MQTT
Data Sources
HDFS Files
Hive
Kafka
ElasticSearch
Kinesis
WebSocket
Cassandra
BigQuery
PubSub
Data Sinks
Real Time Self Service ETL Platform
Hadoop Distributions Native / VM / Cloud
Cloudera MapR HDP
High Level Architecture

6. Copyright © 2015-2017
Exadatum Software Services Pvt. Ltd.
6
Source SinkUI/UX
Service Layer
Xstreams Core
Real Time Olap
Technology Stack

7. Copyright © 2015-2017
Exadatum Software Services Pvt. Ltd.
7
.
Marketing Propensity Business Use Case
Predict user purchase trends across different signals like Brand, Price , Size
based on custom and dynamic feature set that is composed on time based
event product category , sub category , age and gender.
Business Applications
Search and Browse for Recommendation
Enhance Browsing Experience
Discount Optimization
Campaign Management
Ad Monetization
Internal R &D (Eg. Acti Mirrors)
Futuristic Shopping Experience
Tagging Using App by Finding Customers

8. Copyright © 2015-2017
Exadatum Software Services Pvt. Ltd.
8
.
Overview of Sources/Features/EndPoints
ClickStream Dataset
CustomerId
ProductId
EventId
Time
Product Dataset
Product Category
Product Sub Category
Brand
Occasion
Age , Gender , Colour , Size
Demographic Dataset
Gender
Household Size
Income
#Children , Marital Status
Education
Source Tables and Attributes
Features
Product Category
Product Sub Category
Event Type
Age
Gender
Time
EndPoints/Singals
Brand
Price
Size
Occasion
Colour

9. Copyright © 2015-2017
Exadatum Software Services Pvt. Ltd.
9
.
Legacy Modelling Steps
Join ClickStream and Product Dataset
Pivot and Vectorize
Join ClickStream and Demographic Dataset
Pivot and Vectorize
Merge Above two Vectors
Apply Binary Classification Logistic Regression Model

10. Copyright © 2015-2017
Exadatum Software Services Pvt. Ltd.
10
.
Legacy Modelling Issues
Pivots creation was taking more than 20 hours
on whole dataset. So sampled 5% dataset was
used.
Default Vector Operator was taking longer time.
Modelling was done on subset of the features
combinations.
Skewed data for Purchase and Non Purchase label.

11. Copyright © 2015-2017
Exadatum Software Services Pvt. Ltd.
11
.
Feature Engineering Optimization on Xstreams
Complete Dataset for 30 million customers was used since
vectorization time was reduced from 18hours to 3 hours due
to custom sparse vectorize operator.
Removed skewed data label by redcing the non
purchase by 1:10 ratio.
Added unknown values for missing demographic
values.
All feature combination were not used instead of top
20.P1/P2 combination varied 60-2400

12. Market Propensity Pipeline on XStreams (Live Demo)

13. Copyright © 2015-2017
Exadatum Software Services Pvt. Ltd.
Thank You!
13