Taken from our BCN Digital Festival last week, for info on attending, speaking or sponsoring our next event on the 29/30th April 2020 email enquiry@digitalenterprisefest.com
Data Analytics in eSports. UbeatCase Study
Building AI & Automate services need a solid base on Data Management, but the current environment is volatile, uncertain, complex and ambiguous so you never know what data will be important in the following months.
The Data Management Platform in an extremely dynamic market like eSports where everything is currently being created, in reinvention and is to be validated is even more challenging.
UBEAT is the leading streaming platform of eSports related content. Created in November 2018, it still hasn’t 12 months of existence but a lot of learnings in its rear mirror and a lot of future to come in its high beam. Especially regarding Data Management.
To apply to speak or sponsor our 2020 events goto www.digitalenterprisefest.com
2. www.mdcloud.es
AGENDA
• What is UBEAT?
• What were the challenges of this project?
• What is the solution that we built?
• What are the results that we achieved?
• What’s ahead?
3. www.mdcloud.es
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Quisque tempus pretium nulla at dapibus. Vivamus venenatis nisi ut neque ultricies, vel iaculis
enim lobortis. In hac habitasse platea dictumst. Praesent aliquet a urna ut suscipit. Fusce elit augue, placerat nec nunc sit amet, sodales hendrerit
libero. Quisque ultrices ligula in leo elementum interdum. In hac habitasse platea dictumst. Quisque tristique eros at metus aliquet consequat.
Pellentesque fermentum gravida lacinia. Pellentesque vulputate lectus odio, et lobortis quam consequat at. Duis in odio viverra diam dictum pulvinar
eu ut ex. Ut sed bibendum est, nec dictum leo. Nulla gravida id neque non fringilla. Mauris in urna interdum, molestie mi at, faucibus quam.
What are eSports?
7. www.mdcloud.es
The first Broadcasting platform
focused in creating
Added Value content for eSports
Interviews
Entertainment
Shows
Finals
Best Outplays
Replays
…
8. www.mdcloud.es
AGENDA
• What is UBEAT?
• What were the challenges of this project?
• What is the solution that we built?
• What are the results that we achieved?
• What’s ahead?
9. www.mdcloud.es
Objectives
Main project objectives and requirements
Extract data from several data
sources and consolidate it in a
central DWH
Central DWH
Provide a 360º customer view
Customer 360
Design new data architecture to
support high variability on data
volume
Cloud Architecture
Develop user friendly reports
for visual data analysis
Visual Reporting
Data aggregation and exporting
for external Marketing
Campaigns
Marketing Campaigns
Provide a suitable environment
to perform multi source
advanced analytics (Customer
segmentation, product
refinement, etc)
Advanced Analytics
10. www.mdcloud.es
Datasources
Overview of Project Datasources
Youbora
Video analytics. Registered user detailed video
consumption: VOD/LIVE, platform, device, session
monitoring, etc. Expected volume 5Gb/y. API
access
Each datasource is owned by different teams, so there is a big challenge for the project to manage everyone and get them aware of others
requirements. Basically, datasource owners must take into account that they are part of something bigger
Google Firebase – Big query
APP Analytics. Telemetry, user actions, user
behavior tracking, app interactions. Expected
volume 20GB/y. API access
Google Analytics
Web Analytics. Telemetry, user actions, user
behavior tracking, web interactions. Aggregated
data. Expected volume 1GB/y. API access
Adobe Campaign
Marketing campaigns. Campaign success
measurement, email metrics, response ratios, etc.
Both source and consumer. Expected volume
1GB/y. File data transfer
BackOffice
Platform internal data. Registered users and
master data. Expected volume 250MB/y. ODBC
access
AppsFlyer
APP market. APP downloads, installations and
uninstallations. Expected volume 100MB/y. API
access
12. www.mdcloud.es
AGENDA
• What is UBEAT?
• What were the challenges of this project?
• What is the solution that we built?
• What are the results that we achieved?
• What’s ahead?
13. www.mdcloud.es
Conceptual Architecture
DWH
Staging Area, Modeled Area and Master Data
Key Concepts
OLAP Analytics
Cube generation for performance purposes
Visual Analytics & Reporting
System edge for user data consumption and visual
analysis
Monitoring
Monitoring facilities for Operation team once in
production
Orchestrator
Automation of data ETL (Extract-Transform-Load)
processes
Data Security
Data encryption and secured communications
Data Governance
Data Quality, Processes and documentation
14. www.mdcloud.es
Physical Architecture
Main Components
Azure Datafactory
PAAS - Acting as orchestrator. Data extraction and
internal movement
Azure Storage (STAs)
PAAS - raw files stored for data ingestion and data
sending to 3rd Parties
Azure SQL (DWH)
PAAS - Microsoft SQL Server Cloud instance acting as
central data repository. Staging / Dimensions + Facts
Integration Runtime
IAAS - Virtual Machine Required for on-premise access
purposes
Key Vault
PAAS - Component for securely storing and accessing
platform, secrets (tokens, passwords, etc)
Azure Analysis Services
PAAS - enterprise-grade data models in the cloud. OLAP
High performance cubes
Excel & PowerBI
Data consumption is done by PowerBI cloud Service and
excel files
15. www.mdcloud.es
Solution highlights
Pay
per Use
Pay as you go
Azure Public cloud infrastructure enables minimum consumption
on money terms. Reduce infrastrcuture costs
Authomatic scalable on demand
Automatic system and infrastructure adaptation on high demand
periods or sudden increases of data flows. No data loss
Security compliance
Platform secrets, passwords and data encryption. Data movement
secured by product
PAAS Services
Major use of PAAS services – Minimum concern on infrastructure
on administration and maintenance. Focus on value
Security
ScalabilityPAAS
16. www.mdcloud.es
AGENDA
• What is UBEAT?
• What were the challenges of this project?
• What is the solution that we built?
• What are the results that we achieved?
• What’s ahead?
20. www.mdcloud.es
Business Results
Goals achieved from business view
Data driven based decisions
Customer Segmentation
Product improvement
Trend detection
Customer 360
Data driven based decisions
Journey helper from: Raw disintegrated data ->
Information -> Knowledge -> Data Based decisions
Customer Segmentation
Significant success in customer targeting.
Higher success rate for marketing campaigns
Product improvement
Video scheduling accuracy -> Product Quality
improved -> Increase customer satisfaction
Trend detection
Aggregated user behavior trend detection ->
Better and quicker response for incoming or
underlying events
Customer 360
Customer (Fan) information completed. New
customer insights for business people (Video
cosumption, campaing response, APP usage, etc)
21. www.mdcloud.es
AGENDA
• What is UBEAT?
• What were the challenges of this project?
• What is the solution that we built?
• What are the results that we achieved?
• What’s ahead?
23. www.mdcloud.es
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Quisque tempus pretium nulla at dapibus. Vivamus venenatis nisi ut neque ultricies, vel iaculis
enim lobortis. In hac habitasse platea dictumst. Praesent aliquet a urna ut suscipit. Fusce elit augue, placerat nec nunc sit amet, sodales hendrerit
libero. Quisque ultrices ligula in leo elementum interdum. In hac habitasse platea dictumst. Quisque tristique eros at metus aliquet consequat.
Pellentesque fermentum gravida lacinia. Pellentesque vulputate lectus odio, et lobortis quam consequat at. Duis in odio viverra diam dictum pulvinar
eu ut ex. Ut sed bibendum est, nec dictum leo. Nulla gravida id neque non fringilla. Mauris in urna interdum, molestie mi at, faucibus quam.
?
Notas del editor
Datacleansing / ETL /
Cloud - and auto scale-up-and-out
Això pasa sempre, silos de coneixement
Dades sensibles encriptades al DWH amb unes claus que estan al KeyVault
Les conexions d'azure van encriptades
Elements
IR (Integration Runtime)
Com que el postgre està protegit per IP, hem d'asegurar una IP estàtica (ADF es un motor al cloud amb les capacitats capades).El cloud està molt bé, pero necesites VMs (IAAS)
Tenir una IP estàtica
Correr un codi CSHARP (functions no ens val)
Conexió ODBC (dll) - per exemple per encirptar (ADF no ho fa de natiu)
STAs
Guardem fitxers Raw (entrada i sortida)
Per fer funcionar functions contra GA necesitem una STA
Guardar token GA (s'hauria de guardar al KeyVault)
SQL
DWH
Taules staging + dimensio/fets (modeled)
ADF
Orquestador de dades
KeyVault
Storage de secrets
Claus d'encriptació, users de BBDD (ADF no té cap credencial)
SSAS
Cubos tabulars per l'excel
Excel/PowerBI
Dades sensibles encriptades al DWH amb unes claus que estan al KeyVault
Les conexions d'azure van encriptades
Elements
IR (Integration Runtime)
Com que el postgre està protegit per IP, hem d'asegurar una IP estàtica (ADF es un motor al cloud amb les capacitats capades).El cloud està molt bé, pero necesites VMs (IAAS)
Tenir una IP estàtica
Correr un codi CSHARP (functions no ens val)
Conexió ODBC (dll) - per exemple per encirptar (ADF no ho fa de natiu)
STAs
Guardem fitxers Raw (entrada i sortida)
Per fer funcionar functions contra GA necesitem una STA
Guardar token GA (s'hauria de guardar al KeyVault)
SQL
DWH
Taules staging + dimensio/fets (modeled)
ADF
Orquestador de dades
KeyVault
Storage de secrets
Claus d'encriptació, users de BBDD (ADF no té cap credencial)
SSAS
Cubos tabulars per l'excel
Excel/PowerBI
Video?
Informes amb powerBI/Informes amb excel
Informes website: (Google Analytics)
Captación
Conversión
Visitas
Fuentes de registro
Horas
Perfil cliente visitante (edad, genero, provincia,…)
Informes de campañas
Datos de apertura, clicks, conversiones,….
Informes APP
Descargas
Geolocalización
Dispositivos
Informes Audiencias APP:
Categorías (tags videos)
Consumos (nº, horas/minutos)
Geolocalización
Canal app/web
Franjas horarias
Top videos (diario, semanal, mensual)
Video?
Informes amb powerBI/Informes amb excel
Informes website: (Google Analytics)
Captación
Conversión
Visitas
Fuentes de registro
Horas
Perfil cliente visitante (edad, genero, provincia,…)
Informes de campañas
Datos de apertura, clicks, conversiones,….
Informes APP
Descargas
Geolocalización
Dispositivos
Informes Audiencias APP:
Categorías (tags videos)
Consumos (nº, horas/minutos)
Geolocalización
Canal app/web
Franjas horarias
Top videos (diario, semanal, mensual)
Video?
Informes amb powerBI/Informes amb excel
Informes website: (Google Analytics)
Captación
Conversión
Visitas
Fuentes de registro
Horas
Perfil cliente visitante (edad, genero, provincia,…)
Informes de campañas
Datos de apertura, clicks, conversiones,….
Informes APP
Descargas
Geolocalización
Dispositivos
Informes Audiencias APP:
Categorías (tags videos)
Consumos (nº, horas/minutos)
Geolocalización
Canal app/web
Franjas horarias
Top videos (diario, semanal, mensual)
Segmentació de clients
Operational Costs/Eficiencia de videos/incrementar qualitat parrilla programació
Uddate information accés
Raw Data - Information - Knowledge - Desitions
No more intuitive (not datadriven) desitions
Cruzar info del fan entre los distintos sistemas (Fan360) - es pot fer diagrama
Youbora te id de user (quins videos ha vist)
Es pot creuar amb Backoffice
Favoritos, Watchlist, etc
Campaign (anada i tornada)