Modernizando plataforma de bi

TriggerDB Consulting SRL | www.triggerdb.com | Info@triggerdb.com | Buenos Aires Argentina
Modernización de MS BI
Maximiliano Accotto

¿Quienes somos?
TriggerDB Consulting SRL es una empresa Argentina fundada por Maximiliano Accotto ,
experto reconocido a nivel mundial en Microsoft SQL Server y BI.
Somos partner certificados de Microsoft en las competencias de SQL Server, Data Analytics y
Powerbi.
Nuestros mas de 15 años de trayectoria , alto expertice y mas de 150 clientes en toda la región,
nos hace lideres en servicios de consultoría y capacitación en tecnologías de SQL Server , analytics y
Powerbi.

Servicios que ofrecemos
Consultoría Soporte Premier Training

Productos en los que trabajamos
SQL Server Analysis Services Azure Data Factory HDInsight PowerBI
Azure
SQL Data Warehouse
Azure
Data Lake
Machine
Learning
Azure SQL Server
Power Pivot
Power Query

Acerca de Maxi Accotto
 Microsoft MVP Data Platform desde el 2005.
• Orador frecuente para eventos y comunidades.
 Especialista en tecnologías de Data y BI con mas de 15 años de
trayectoria
https://twitter.com/maxiaccotto
https://ar.linkedin.com/in/maxiaccotto
maxi@triggerdb.com
https://www.triggerdb.com/blog/

Cualquier tipo y de diferentes orígenes
Social network
analysis
Weather
forecasting
Healthcare
outcomes
Predictive
maintenance
Targeted
advertising
Natural resource
exploration
Fraud
detection
Telemetry data
analysis
Buyer propensity
models
Churn analysis
Life sciences
research
Web app
optimization
Network intrusion
detection
Smart meter
monitoring

Value
Del dato a la decisión y acción

IT Científico de Datos Analista Usuario Final
BI en la era de AI

Arquitectura de BI moderna clásica

Big Data StoresInformation
Management
Data Warehouse Modern BI: Components
Action
People
Automated
Systems
Apps
Web
Mobile
Bots
Event Hubs
Data Catalog
Data Factory
HDInsight
(Hadoop and
Spark)
Stream Analytics
Intelligence
Data Lake
Analytics
Machine Learning
SQL Data
Warehouse
Data Lake Store
& Azure Storage
Data
Sources
Apps
Sensors
and
devices
Data
Intelligence
Dashboards &
Visualizations
Cortana
Bot
Service
Cognitive Services
Power BI
Azure Analysis
Services
IOT Hubs
Cosmos DB
Machine Learning
and Analytics
HDInsight
(Hadoop and
Spark)
Stream Analytics
Data Lake
Analytics
Machine
Learning
Azure Analysis
Services
Azure Databricks

Azure SQL Database

AZURE SQL DATABASE
Best for modernization at
scale with low cost and effort
Elastic PoolSingleSQL
Managed
Instance
SQL
Azure SQL Database
Instance-scoped programming
model with high compatibility
to SQL Server
Standalone managed
database for predictable and
stable workloads
Shared resource model for
greater efficiency through
multi-tenancy

Seamless cloud integration
Easy lift-and-shift migration
Azure SQL Database Managed Instance
facilitates lift and shift migration from on-premises SQL
Server to cloud
Azure Hybrid Benefit for SQL Server
maximizes current on-premises license investments to
facilitate migration
Database Migration Service (DMS)
provides seamless and reliable migration at scale with
minimal downtime
Database Migration
Ser vice (DMS)
Azure SQL Database
Managed Instance
Azure Hybrid Benefit
(AHB) for SQL Ser ver

Azure Data Factory

Azure Data Factory
• Orchestrate, monitor & schedule - Compose data processing,
storage & movement services (on premises & cloud)
• Automatic infrastructure management
• Combine pipeline intent w/ resource allocation & mgmt
• Globally available data movement capability
• Single pane of glass - One place to manage and monitor
your data pipelines

Azure Blob Storage
Game Log Files
Customer Table
On Premises
Data Mart
Game Logs
Customer Table
Azure DB
Customer
Game Usage
Visualize
Data Set
(Collection of files,
DB table, etc)
Activity: a processing step
(Hadoop job, custom code, ML model, etc)
Pipeline: a sequence of
activities (logical group)Data Factory
Concepts
…
Data Sources Ingest Transform & Analyze Publish
Customer
Table
Geocode
Transform,
Combine, etc Analyze Move

Basic Concepts• ADF is Microsoft’s unified platform for ETL/ELT services in the cloud
• ADF allows you to build data pipelines and execute them/schedule their runs
• Data pipeline is a chain/group of activities to be performed on your data, e.g. data movements/transformations
• Some activities are powered by services with data store/compute resources allocated outside ADF, e.g.
HDI/ML/etc.

Azure Data Factory v2 Service
Scheduling | Orchestration | Monitoring
UX & SDK
Authoring | Monitoring/Mgmt
Self Hosted
Integration
Runtime
Cloud Svcs, Apps
& Data
On Prem Apps
& Data
Azure
Integration
Runtime
Command and Control
Data
SSIS
Package
Data Factory
A data integration account.
Location of orchestration, service metadata
Integration Runtime (IR)
ADF’s execution engine
Three core capabilities:
• data movement
• pipeline activity execution
• SSIS package execution
To integrate data flow and control flow across
the enterprises’ hybrid cloud, customer can
instantiate multiple IR instances for different
network environments:
- On premises (similar to DMG in ADF V1)
- In public cloud
- Inside VNet
Pipeline

Azure SQL Data Warehouse

Azure SQL Data Warehouse
PB scale with MPP and independent compute/ storage scale
Grow, shrink, and pause in seconds
Compute-optimized for demanding workloads
Unlimited columnar storage

Fully managed PaaS

Target workload: Analytics
• Store large volumes of data
• Consolidate disparate data into a single location
• Shape, model, transform and aggregate data
• Perform query analysis across large datasets
• Ad-hoc reporting across large data volumes
• All using simple SQL constructs
“SQL on SQL”

Unsuitable workloads
Operational workloads
High frequency reads & writes
Large numbers of singleton selects
High volume of single row inserts
Data Preparation
Row by row processing needs
Incompatible formats (JSON, XML)

Distributing Data
CREATE TABLE [build].[FactOnlineSales]
(
[OnlineSalesKey] int NOT NULL
, [DateKey] datetime NOT NULL
, [StoreKey] int NOT NULL
, [ProductKey] int NOT NULL
, [PromotionKey] int NOT NULL
, [CurrencyKey] int NOT NULL
, [CustomerKey] int NOT NULL
, [SalesOrderNumber] nvarchar(20) NOT NULL
, [SalesOrderLineNumber] int NULL
, [SalesQuantity] int NOT NULL
, [SalesAmount] money NOT NULL
)
WITH
(
DISTRIBUTION = HASH([ProductKey])
)
;

Azure Analysis Services

BI semantic model
Business logic & metrics
Data modeling
Security
Analysis Services
Lifecycle management
In-memory
cache
On-premises
Cloud
Data sources
SQL Database
SQL Data Warehouse
Other data sources
SQL Server
Analytics platform
system
Other data sources
On-premises
Cloud
Client tools
Power BI
Power BI Embedded
Excel
Third party BI tools
Power BI Desktop

SQL Server Analysis Services
SQL Server
Analysis Services
SQL Server
Other
data sources
On-prem data sources
Visual Studio
Authoring & Dev
SSMS
Power BI
Visualizations & insights
Other 3rd party tools & services
SQL Server
Reporting Services

Visual Studio
Authoring & Dev
Power BI
Visualizations & insights
SQL Server
Other
data sources
On-prem data sources
SSMS
Other 3rd party tools & services
Azure
Analysis Services
SQL Database
SQL
Data Warehouse
Cloud data sources

Big Data Store

Big Data Stores
Data
Information
Management
Event Hubs
Data Catalog
Data Factory
Big Data Stores
SQL Data
Warehouse
Data Lake Store
Data
Sources
Apps
Sensors
and
devices
IOT Hubs
Cosmos DB

Big Data Stores
A hyper-scale repository for big data analytics workloads
• A Hadoop Distributed File System for the cloud
• No fixed limits on file size
• No fixed limits on account size
• Unstructured and structured data in their native format
• Massive throughput to increase analytic performance
• High durability, availability, and reliability
• Azure Active Directory access control
LOB
Applications
SocialDevices
Clickstream
Sensors
Video
Web
Relational
HDInsight
ADL Analytics
Machine Learning
Spark
R
ADL StoreSQL Data
Warehouse
Data Lake Store
Cosmos DB

Data lake is the center of a big data solution
A storage repository, usually Hadoop, that holds a vast amount of raw data in its native
format until it is needed.
• Inexpensively store unlimited data
• Collect all data “just in case”
• Store data with no modeling – “Schema on read”
• Complements EDW
• Frees up expensive EDW resources
• Quick user access to data
• ETL Hadoop tools
• Easily scalable
• With Hadoop, high availability built in

Azure Data Lake Store Azure Blob Storage
Purpose Optimized for big data analytics General purpose bulk storage
Use Cases Batch, Interactive, Streaming App backend, backup data, media storage for
streaming
Units of Storage Accounts / Folders / Files Accounts / Containers / Blobs
Structure Hierarchical File System Flat namespace
WebHDFS Implements WebHDFS No (WASB)
Security AD SAS keys
Storage Auto Shared/Files chunked Manually manage expansion/Files intact
Size Limits No limits on account size, file size, # files 500TB account, 4.75TB file
Service State Generally Available Generally Available
Billing Pay for data stored and for I/O Pay for data stored and for I/O
Region Availability Two US regions (East, Central) & North Europe
(Other regions coming soon)
All Azure Regions
ADL Store vs Blob Store

Big Data

CONTROL EASE OF USE
Azure Data Lake
Analytics
Azure Data Lake Store
Azure Storage
Any Hadoop technology,
any distribution
Workload optimized,
managed clusters
Data Engineering in a
Job-as-a-service model
Azure Marketplace
HDP | CDH | MapR
Azure Data Lake
Analytics
IaaS Clusters Managed Clusters Big Data as-a-service
Azure HDInsight
Frictionless & Optimized
Spark clusters
Azure Databricks
BIGDATA
STORAGE
BIGDATA
ANALYTICS
ReducedAdministration
K N O W I N G T H E V A R I O U S B I G D A T A S O L U T I O N S

Azure
HDInsight
Hadoop and Spark
as a Service on Azure
Fully-managed Hadoop and Spark
for the cloud
100% Open Source Hortonworks
data platform
Clusters up and running in minutes
Managed, monitored and supported
by Microsoft with the industry’s best SLA
Familiar BI tools for analysis, or open source
notebooks for interactive data science
63% lower TCO than deploy your own
Hadoop on-premises*
*IDC study “The Business Value and TCO Advantage of Apache Hadoop in the Cloud with Microsoft Azure HDInsight”

Microsoft + Hortonworks
Promoting Open Hadoop
Engineering alignment
Corporate alignment
Field alignment

Spark for Azure HDInsight
In Memory Processing on Multiple Workloads
Azure
HDInsight
In Memory
Spark
• Single execution model for multiple
tasks
• Processing up to 100x faster
performance than Map Reduce
• Developer friendly (Java, Python, Scala)
• BI tool of choice (Power BI, Tabelau,
Qlik, SAP)
• Notebook experience (Jupyter/iPython,
Zeppelin)

HDInsight Supports Hive
SQL-like queries on Hadoop data in HDInsight
HDInsight provides easy-to-use graphical query interface for Hive
HiveQL is a SQL-like language (subset of SQL)
Hive structures include well-understood database concepts such as tables, rows, columns, partitions
Compiled into MapReduce jobs that are executed on Hadoop
Dramatic performance gains with Stinger/Tez
Stinger is a Microsoft, Hortonworks and OSS driven initiative to bring interactive queries with Hive
Brings query execution engine technology from Microsoft SQL Server to Hive
Performance gains up to 100x
Hadoop 2.0
1400s
44.3s
35.1s
Sample Query
Hive
1.0
HDP
1.3 /
Hive
11
HDP
2.0
32x
Speedup 40X
Speedup
HDP
2.1
15s
100x
Speedup

HDInsight Supports HBase
• NoSQL database on data in HDInsight
Data Node Data Node Data Node Data Node
Task Tracker Task Tracker Task Tracker Task Tracker
Name Node
Job Tracker
HMaster
Coordination
Region Server Region Server Region Server Region Server

HDInsight Supports Storm
Stream analytics for Near-Real Time Processing
Consumes millions of real-time events from a scalable event broker (ie. Apache Kafka, Azure Event Hub)
Performs time-sensitive computation
Output to persistent stores, dashboards or devices
Customizable with Java + .NET
Deeply integrated to Visual Studio
Event Queuing
System
Collection Presentation and
action
Event producers Transformation Long-term storage
Event Hubs
Storage
adapters
Stream
processing
Cloud gateways
(web APIs)
Field
gateways
Applications
Search and query
Data analytics (Excel)
Web/thick client
dashboards
Live Dashboards
Apache Storm
on
HDInsight
Devices to take action
Kafka /
RabbitMQ /
ActiveMQ
Web and Social
Devices
Sensors
Azure Stream
Analytics
HDFS
Azure DBs
Azure storage
HBase

Machine Learning Server for HDInsight
• Familiarity of R o Python
• Scalability of Hadoop and Spark
• Up to 7x faster using Spark engine
• Train and run ML models on datasets of any size
• Cloud managed solution (easy setup, elastic, SLA)
Only managed, cloud solution for doing R
or Python

Real-time stream processing in the cloud
• Perform real-time analytics for your Internet of Things solutions
• Stream millions of events per second
• Get mission-critical reliability and performance with predictable
results
• Create real-time dashboards and alerts over data from
devices and applications
• Correlate across multiple streams of data
• Use familiar SQL-based language for rapid development
Event Hubs
Blob Storage
Stream
Analytics
SQL Database
Event Hubs
Power BI
Blob Storage
Table Storage
10
10
10
10
Machine Learning
and Analytics
HDInsight
(Hadoop and
Spark)
Stream Analytics
Data Lake
Analytics
Machine
Learning
Azure Analysis
Services

Maximum
CONTROL
Greatest
SIMPLICITY.
Lowest TCO
Azure Stream Analytics
Fully
managed -
3 9s of
availability
SQL like
query
language
Machine
Learning
integrations
Integrations for
real-time
dashboarding
Custom
code
extensions
Spark Streaming & Storm (HD Insight)
SLA with 3
9s of
availability
Virtual Machines
Full stack
control
Integrationswith
Kafka,ipython
notebookR-
Server
Managed
Spark
Streaming &
Storm
Streaming Options with Microsoft
Cluster ServiceJob Service

Real-time analytics

PowerBI

Any data, any way, anywhere

Power BI product portfolio

Get Data
Easily connect, clean, and mashup data
Free companion authoring tool for the Power BI service
Analyze
Build powerful models and flexible measures
Visualize
Create stunning interactive reports
Publish
Share insights with others
Collaborate
Empower your organization with self-service analytics

Power BI service
Cloud-based SaaS solutions
Get started quickly
Secure, live connection to your data sources,
on-premises and in the cloud
Auto insights and intuitive data exploration using
natural language query
Deliver insights through other services such as
SharePoint, PowerApps and Teams
Pre-built dashboards and reports for popular SaaS
solutions
Live, real-time dashboard updates

Power BI Premium
Dedicated resources in the cloud
Flexibility to license by capacity
Greater scale and performance
Extending on-premises capabilities
Premium capacity – P3
My workspace
User 2
My workspace
User 3
App workspace
Marketing
App workspace
Sales
My workspace
User 1
APIs
Custom app
Power BI service – Contoso organization
Power BI Premium

On-premise reporting solution
Power BI reports and SSRS report on-premises
Connect to data
Over 70+. Data can be imported, queried directly or live connection to SSAS
Power BI reports
Fully interactive reports on-premises to visualize your data and gain insights
SSRS reports
Precisely formatted operational reports
Consume in multiple ways
Mobile Apps, Web Portal, Embedded in your organizational apps
Power BI Report Server

Spend time focusing on your product instead of
building visual analytic features from scratch
Connect to countless data sources so that you can
expose insights to your customers
Take advantage of our existing Custom Visual library
or dream up the right visual for your customers
Embed consistent, easy-to-navigate visualization
experiences across any device
Leverage other familiar services like Visual Studio,
Azure Web Aps and other Azure services
Power BI Embedded
Embed stunning, fully interactive reports and visuals into your applications

Muchas gracias
https://www.linkedin.com/company/triggerdbsrl
info@triggerdb.com
https://www.triggerdb.com

Modernizando plataforma de bi

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Modernizando plataforma de bi

Similar a Modernizando plataforma de bi (20)

Más de Maximiliano Accotto

Más de Maximiliano Accotto (20)

Último

Último (20)

Modernizando plataforma de bi

Notas del editor