In this session, we spend time comparing Microsoft Big Data Technologies for Analytics. he technology includes HDInsight, Apache Spark and Power BI.

1. Workshop Slides Follow-Up:
Comparing Technologies
Jen Stirrup
Data Whisperer,
Data Relish
Level: 300

2. Big Data

3. What is Big Data?
“Big data is a collection of data sets so large
and complex that it becomes awkward to work
with using on-hand database management
tools.
Difficulties include capture, storage, search,
sharing, analysis, and visualization.”
– Wikipedia

5. fully featured RDBMS
transactional processing
rich query
managed as a service
elastic scale
internet accessible http/rest
schema-free data model
arbitrary data formats

6. Apache Spark

7. What is Apache Spark?
Apache Spark solves a problem
There's no need to structure everything as map and reduce operations.

8. Apache Spark
• Interactive manipulation and visualization
of data
– Scala, Python, and R Interactive Shells
– Jupyter Notebook with PySpark (Python) and
Spark (Scala) kernels provide in-browser
interaction

9. Apache Spark
• Unified platform for processing multiple
workloads
– Real-time processing, Machine Learning,
Stream Analytics, Interactive Querying,
Graphing

10. Apache Spark
• Leverages in-memory processing for
really big data
– Resilient distributed datasets (RDDs)
– APIs for processing large datasets
– Up to 100x faster than Hadoop

11. What is Spark?
• an open-source software soIution that
performs rapid caIcuIations on in-memory
datasets
• RDD (ResiIient Distributed Data) is the
basis for what Spark enabIes
– ResiIient
– Distributed

12. Why Spark?
• Native Integration with Hive, HDFS and
any Hadoop File System.
• Faster Development Scala has about three
times less code than Java.
• Execution can be faster for iterative jobs
• Code reuse promotion. APIs and data
types are similar for batch and streaming.

13. Sparking the Action
• New capabilities add dynamic access methods to
feature-rich Spark applications
• Data Frame API Enables common and easy
interchange between Spark components for data
imports and exports Machine Learning Introduces
multiclass classification, clustering, frequent pattern-
mining algorithms
• Enterprise-Ready Consistent operations,
• comprehensive security,
• deployable anywhere Spark SQL [Tech Preview]
• A new module for structured data processing in Spark

14. How does Spark work?
• RDD
• Actions
• Persistance

15. HDInsight

16. HDInsight Cluster Types
• Hadoop: Query workloads
– Reliable data storage, simple MapReduce
• HBase: NoSQL workloads
– Distributed database offering random access to large
amounts of data
• Apache Storm: Stream workloads
– Real-time analysis of moving data streams
• Apache Spark: High-performance workloads
– In-memory parallel processing

17. HDInsight
● Easily run popular open source frameworks – including Apache
Hadoop, Spark and Kafka – using Azure HDInsight, a cost-effective,
enterprise-grade service for open source analytics. Effortlessly
process massive amounts of data and get all the benefits of the broad
open source ecosystem with the global scale of Azure.
● Quickly spin up big data clusters on demand, scale them up or down
based on your usage needs and pay only for what you use.
● Use HDInsight tools to easily get started in your favourite
development environment.

18. HDInsight
● Meet industry and government compliance standards and
protect your enterprise data assets using an Azure Virtual
Network, encryption and integration with Azure Active
Directory.
● HDInsight integrates seamlessly with other Azure
services, including Data Factory and Data Lake Storage,
for building comprehensive analytics pipelines.
● Easy, cost-effective, enterprise-grade and open.

19. HDInsight
● Open-source ecosystem - HDInsight supports the latest open source projects from the
Apache Hadoop and Spark ecosystems. Stay up to date with the newest releases of
open source frameworks, including Kafka, HBase and Hive LLAP.
● Security and compliance - Get enterprise-grade data protection with monitoring, virtual
networks, encryption, Active Directory authentication, authorisation and role-based
access control. HDInsight has more than 30 industry certifications, including ISO, SOC,
HIPAA and PCI, to meet compliance standards.
● Native integration with Azure services - Seamlessly integrate with a wide variety of
Azure data stores and services, including SQL Data Warehouse, Azure Cosmos DB,
Data Lake Storage, Blob Storage, Event Hubs and Data Factory.

20. HDInsight
● Simplified monitoring - HDInsight integrates with Azure Log Analytics
to provide a single interface where you can monitor all your clusters.
● Broad app support - HDInsight supports a broad range of apps from
the big data ecosystem, which you can install with a single click. Pick
from more than 30 popular Hadoop and Spark apps for a variety of
scenarios.
● Multiple languages and tools - Use your preferred productivity tools,
including Visual Studio, Eclipse, IntelliJ, Jupyter and Zeppelin. Write
code in familiar languages such as Scala, Python, R, JavaScript and
.NET.

22. Azure Databricks

23. What is Databricks?
• Databricks provides an end-to-end,
managed Apache Spark platform
optimized for the cloud
• Improved performance of Spark jobs in
the cloud by 10 – 100x
• Cost Efficient to run large-scale Spark
workloads

24. Credit: https://docs.microsoft.com/en-us/azure/azure-databricks/what-is-azure-databricks

25. Databricks for Big Data
• Data Scientists get an interactive
notebook environment
• Good monitoring suite
• Security Controls to facilitate thousands of
users

26. Databricks for Data Engineers
• Databricks Runtime adds increased
performance to Apache Spark workloads
when running on Azure
• Auto-scaling and auto-termination for
Spark clusters to automatically minimize
costs

27. Databricks for Data Science
• Notebooks have real-time collaboration
and are multi-editable for productivity
• Integration with Power BI for data
visualization
• Supported by Azure Database

28. Credit: https://databricks.com/blog/2017/11/15/a-technical-overview-of-azure-databricks.html

29. Why is Databricks in Azure?
• Close integration with Azure services
• Optimized connectors
• One-click management directly from the
Azure console
• Azure Databricks will greatly simplify building
enterprise-grade production data
applications

30. Azure and Databricks together
• Azure launches and manages worker
nodes in the customer subscriptions
• Customer launches a cluster, which
initiates a Databricks appliance
• A managed resource group is deployed
with a Vnet, Security Group and a
Storage account

31. Azure and Databricks Together
• Close Integration to provide an enterprise
platform
• Use all existing VMs
• Security and Privacy remains with
customer
• Network topology is flexible

32. Azure and Databricks together
• Azure Storage and Azure Data Lake
integration
• Azure Power BI
• Azure Active Directory
• Azure SQL Data Warehouse, Azure SQL
DB, Azure Cosmos DB

33. Azure and Databricks Together
• Metadata is stored in an Azure Database
with geo-replication
• Databricks cluster is managed through
Azure Databricks UI

34. Azure and Databricks Together
• Azure Container Services to run the
control plane and data planes via
containers
• Accelerated Networking
• Latest generation Azure hardware for
performance

35. Why Azure Databricks?
Collaboration

36. Why Azure Databricks?
Collaboration
Trusted Cloud

37. Why Azure Databricks?
Collaboration
Trusted Cloud
Scalability

38. Azure
Databricks
Fast, easy and collaborative Apache Spark-based analytics service
https://blogs.microsoft.com/ai/shell-iot-ai-safety-intelligent-tools/
Shell Case Study

39. Shell Case Study

40. Azure Databricks
● Unlock insights from all your data and build
artificial intelligence (AI) solutions with
Azure Databricks
● Azure Databricks supports Python, Scala,
R, Java and SQL, as well as data science
frameworks and libraries including
TensorFlow, PyTorch and scikit-learn.

41. Azure Databricks
● Fast, optimised Apache Spark environment
● Interactive workspace with built-in support
for popular tools, languages and
frameworks

42. Azure Databricks
● Supercharged machine learning on big data
with native Azure Machine Learning
integration
● High-performance modern data
warehousing in conjunction with Azure SQL
Data Warehouse

43. Azure Databricks
● Start quickly with an optimised Apache
Spark environment
● Spin up clusters and build quickly in a fully
managed Apache Spark environment with
the global scale and availability of Azure
● autoscaling and auto-termination to
improve total cost of ownership (TCO)

44. Azure Databricks
● Turbocharge machine learning on big data
● Get high-performance modern data
warehousing

45. Azure Databricks & HDInsight
● Databricks is focused on collaboration, streaming and batch with a notebook
experience for the user. It integrates well with Azure, has AAD authentication, and
can export to SQL DWH, Cosmos DB, Power BI, etc. Databricks’ greatest strengths
are its zero-management cloud solution and the collaborative, interactive
environment it provides in the form of notebooks.
● HDInsight has Kafka, Storm and Hive LLAP, which Databricks doesn’t have. It is
better for processing very large datasets and in a way that allows the user to just “let
it run”.
● Sometimes a mix of both these technologies occurs. Databricks is more user-
friendly and easier to work with, so is better for exploration, whereas HDInsight is
better for processing data.

47. Azure Databricks & HDInsight - Pricing
HDInsight:
● Billed on a per-minute basis, clusters run a group of nodes depending on the
component. Nodes vary by group (e.g. Worker Node, Head Node, etc.), quantity and
instance type (e.g. D1v2).
Component Pricing
Hadoop, Spark, Interactive Query, Kafka,
Storm, HBase
Base price/node-hour
HDInsight Machine Learning services Base price/node-hour + £0.012/core-hour
Enterprise Security Package Base price/node-hour + £0.008/core-hour

48. Azure Databricks & HDInsight - Pricing
Databricks:
● Azure Databricks bills you for virtual machines (VMs) provisioned in clusters and
Databricks Units (DBUs) based on the VM instance selected. A DBU is a unit of
processing capability, billed on a per-second usage. The DBU consumption
depends on the size and type of instance running Azure Databricks.
Workload Standard Tier prices Premium Tier prices
Data Analytics £0.30/DBU-hour £0.410/DBU-hour
Data Engineering £0.12/DBU-hour £0.224/DBU-hour
Data Engineering Light £0.06/DBU-hour £0.164/DBU-hour

49. Azure Databricks & HDInsight - Pricing
Azure Databricks also offers a pre-purchase plan. You can get up to 37% savings
over pay-as-you-go DBU prices when you pre-purchase Azure Databricks Units
(DBU) as Databricks Commit Units (DBCU) for either 1 or 3 years.
HDInsight does not offer a pre-purchase plan.

50. Azure Databricks & HDInsight - Speed
Azure Databricks is even faster than Apache Spark, which can run 100 x faster than
Hadoop MapReduce. It is a very fast system, and provides a series of performance
enhancements on top of regular Apache Spark.
HDInsight is very effective at rapidly collecting large amounts of data, and with it you can
quickly spin up open source projects and clusters, with no hardware to install or
infrastructure to manage. However, some processes can be slightly slower with
HDInsight than with Databricks.

51. Azure Databricks & HDInsight - Hadoop
HDInsight uses Apache Hadoop, which is an open-source distributed
data analysis solution. Hadoop manages the processing of large
datasets across large clusters of computers and it detects and
handles failures.
Why Hadoop?
Azure provides dynamic machines that are billed only when active.
This enables elastic computing, where you can add machines for
particular workloads or projects and then remove them when not
needed. HDInsight can take advantage of this scalable platform. It can
also capitalize on the security and management features of Azure,
integration with Azure Active Directory and Log Analytics.

52. Azure Databricks & HDInsight - Hadoop
You can also make use of Hadoop with
Azure Databricks, but as a storage
function, rather than a function for data
analysis and management.
https://docs.microsoft.com/en-us/azure/azure-databricks/what-is-azure-databricks

53. Azure Databricks & HDInsight - Learning Curve
● Databricks is a good technology to use
regardless of the previous experience that
the user / developer may be going in with.
Databricks’ vision is to make big data
easy for so that every organization can
use it. It aims to make complex systems
easier to work with and manage.
Credit: https://databricks.com/blog/2014/07/14/databricks-cloud-making-big-data-easy.html

54. Azure Databricks & HDInsight - Learning Curve
There is more of a learning curve when it comes to HDInsight. Generally, comprehensive
training is required, and a background knowledge of SQL is very helpful.
There are a number of Microsoft-certified intensive courses that can help to teach
learners how to use HDInsight. Some of these are full-time 5-day courses whereas others
are self-paced. Many courses also contain exams.

55. Azure Databricks & HDInsight - Languages
While Azure Databricks is Spark based, it allows
commonly-used programming languages like
Python, R, and SQL to be used. These languages
are converted in the backend through APIs, to
interact with Spark. This saves users having to
learn another programming language, such as
Scala, for the sole purpose of distributed
analytics.

56. Azure Databricks & HDInsight - Languages
HDInsight clusters, including Spark, HBase, Kafka, Hadoop, and others, support many
programming languages. Some programming languages aren't installed by default. For
libraries, modules, or packages that are not installed by default, you need to use a script
action to install the component.
By default, HDInsight supports:
● Java
● Python
● .NET
● Go
HDInsight also supports Hadoop-specific languages - Pig, HiveQL and SparkSQL.

57. Azure Databricks HDInsight
Pricing Per Cluster Time (VM cost + DBU
processing time)
Per Cluster Time
Engine Apache Spark, optimized for
Databricks
Apache Spark or Apache Hive
Default Environment Databricks Notebooks, R Studio for
Databricks
Ambari, or Zeppelin if using Spark
De Facto Language R, Python, Scala, Java, SQL,
mostly open-source languages
HiveQL, open source
Integration with Data Factory Yes, to run notebooks or Spark
scripts
Yes, to run MapReduce jobs, Pig,
and Spark scripts
Scalability Easy to change machines, allows
autoscaling
Not scaleable
Testing Very easy, notebook functionality is
extremely flexible
Easy, Ambari allows interactive
query execution
Setup and Managing Easy - clusters can be modified
easily and Databricks offers two
main types of services
Complex - must decide cluster
types and sizes
Learning Curve Very flexible Flexible if user knows SQL

58. Azure Databricks and Data Lake Analytics
Both Databricks and DLA can be used for batch processing. How can we decide
which to choose over the other?

59. Azure Databricks and Data Lake Analytics
Data Lake Analytics is a distributed computing resource, which uses its strong U-SQL
language to assist in carrying out complex transformations and loading the data in
Azure/Non-Azure databases and file systems. Data Lake Analytics combines the power
of distributed processing with ease of SQL-like language, making it suitable for Ad-hoc
data processing.
Preferred use cases for DLA:
● Large amounts of data where conversion and loading are the only actions needed
● Processing data from relational databases into Azure
● Repetitive loads with no intermediary action

60. Azure Databricks and Data Lake Analytics
Azure Databricks is a Notebook type resource which allows setting up of high-
performance clusters which perform computing using its in-memory architecture. Users
can choose from a wide variety of programming languages and use their favorite libraries
to perform transformations, data type conversions and modeling. Databricks also comes
with infinite API connectivity options, which enables connection to various data sources
that include SQL/No-SQL/File systems and a lot more.
Preferred use cases for Databricks:
● Processes that require intermediary analysis of data
● ETL that requires more visibility during data transformation and modeling

61. Data Lake Analytics Databricks
Cost Control Pay-as-you-go Manual
Development Tool IDE + SDK based (U-SQL Supported Notebook type
Payment Per job Cluster properties, time duration and
workload
Scaling Auto-scaling based on data Auto-scaling for jobs running on cluster
Data Storage Internal database available Database File System, Direct Access
(Storage)
Manage Usage Portal (preferred); Azure SDK;
Python; Java; Node.js; .NET
Spark framework: Scala, Java, R and
Python; Spark SQL
Monitoring Jobs Azure Portal, Visual Studio Within Databricks
Functionalities Scheduling jobs, inducing in Data
Factory pipelines (U-SQL scripts)
Scheduling jobs, inducing in Data Factory
pipelines (Data Factory notebooks)

62. Data Lake Analytics and HDInsight
Data Lake Analytics is like a cloud-based file system that is basically unlimited in its size.
HDInsight can also do a similar job in the cluster that you make, but when you stop that
cluster, the data also goes away.
How do we decide which one to use?

63. Data Lake Analytics and HDInsight
In the case of these technologies, they can actually be used together.
HDInsight is the analytics service whereas the Azure Data Lake Storage is the storage
service. You most likely need both to have functional analytics cluster.
HDInsight provides the cluster, fully manages the open-source packages for analytics
(Hadoop, Spark, etc), and you set up your cluster to use Azure Data Lake Storage which
support HDFS API (Hadoop FileSystem) on top of Cloud Storage.
Essentially, Hdinsight is a managed Hadoop service to provide compute support, and
DLA is a managed storage service to provide large amount of storage support.

64. Stream Analytics and Event Hubs
Azure Event Hubs is a highly scalable event ingestion service, capable of
processing millions of events per second with low latency and high reliability.
Conceptually, Event Hubs can be thought of as a liaison between “event producers”
and “event consumers”.