HDInsight Hadoop on Windows Azure

•Download as PPTX, PDF•

4 likes•6,022 views

This document provides an overview of Hadoop on Azure and how to work with HDInsight Hadoop clusters on the Microsoft Azure cloud platform. It discusses what Hadoop and HDInsight are, how to set up and configure Hadoop clusters on Azure, how to perform common administration tasks, how data is stored and processed using MapReduce, how to write and run MapReduce jobs in different languages, and how to monitor and query Hadoop jobs and data using tools like Hive, Pig, and Excel.

Technology

Data Expertise / Lynn Langit
Practicing Architect
Cloud Deployments
(Azure, AWS,
Google)
Technical author / trainer
Google Cloud
Developer Series
SQL Server 2012
Developer Series
Cloudera Certified
Developer
2 books on SQL
Server BI
Industry awards
Microsoft – MVP
for SQL Server
Google – GDE for
Cloud Platform
10Gen – Master for
MongoDB
Former MSFT FTE
4 years

What is Hadoop?
S HUGE Hype factor in 2011 / 2012
Apache Hadoop is a software framework that supports data-
intensive distributed applications under a free license
• Uses HDFS storage to enable applications to work with thousands
of nodes and petabytes of data
• Uses MapReduce to process the data
• Inspired by Google
• MapReduce
• Google File System

What is HDInsight?
S Hadoop on Windows
S Azure
S On-premise
S Microsoft worked with Hortonworks to port Hadoop to
Windows (from Linux)

RDBMS vs. Hadoop
RDBMS Hadoop
Data Size Gigabytes (Terabytes) Petabytes (Hexabytes)
Access Interactive and Batch Batch – NOT Interactive
Updates Read / Write many times Write once, Read many times
Structure Static Schema Dynamic Schema
Integrity High (ACID) Low
Scaling Nonlinear Linear
Query
Response Time
Can be near immediate Has latency (due to batch
processing)

MapReduce using Java
S WordCount example

MapReduce using C# Streaming
S WordCount example

MapReduce using JavaScript
S WordCount example

Simple Output Graphing
S WordCount example

Understanding Pig
Load>Transform>Dump
or
Store

Monitoring Job Results
S In the portal
S Main Console
S Job icon (button) status
summary
S Job History
S Interactive Console
S JS quick feedback
S JS detailed feedback (log)
S Using RDP
S Map/Reduce tool
S Hadoop command prompt

Download – ODBC for HIVE
S Includes add-in for Excel

Connecting to PowerPivot
S Create an ODBC connection to HIVE
S Connect to ‘other data source’ in PowerPivot

Hadoop To-Do List
• Use Hadoop when
business needs
designate
• Use other NoSQL if
a better fit
BigData =
Hadoop
• Quick and cheap
• Specialized use cases
• Behavioral data
• dev, test , training
environments
Hadoop on the
cloud • Learn Map/Reduce
• Use HIVE via Excel
• Pay attention to
Impala
Hadoop access
technologies

Keep
Learning
S @LynnLangit
S YouTube – SoCalDevGal
S Hire Me
S Architecture
S Best Practices
S Performance Tuning

What's hot

Interactive query in hadoop

Rommel Garcia

Interactive query using hadoop

Arvind Radhakrishnen

Hadoop Reporting and Analysis - Jaspersoft

Hortonworks

Big data on Azure for Architects

Tomasz Kopacz

Data science holds tremendous potential for organizations to uncover new insights and drivers of revenue and profitability. Big Data has brought the promise of doing data science at scale to enterprises, however this promise also comes with challenges for data scientists to continuously learn and collaborate. Data Scientists have many tools at their disposal such as notebooks like Juypter and Apache Zeppelin & IDEs such as RStudio with languages like R, Python, Scala and frameworks like Apache Spark. Given all the choices how do you best collaborate to build your model and then work through the development lifecycle to deploy it from test into production ? In this session learn the attributes of a modern data science platform that empowers data scientists to build models using all the data in their data lake and foster continuous learning and collaboration. We will show a demo of DSX with HDP with the focus on integration, security and model deployment and management. Speakers: Sriram Srinivasan, Senior Technical Staff Member, Analytics Platform Architect, IBM Vikram Murali, Program Director, Data Science and Machine Learning, IBM

Scaling Data Science on Big Data

DataWorks Summit

Evolution of Big Data at Intel - Crawl, Walk and Run Approach

DataWorks Summit

Hadoop in the cloud – The what, why and how from the experts

DataWorks Summit

Until recently, data was gathered for well-defined objectives such as auditing, forensics, reporting and line-of-business operations; now, exploratory and predictive analysis is becoming ubiquitous, and the default increasingly is to capture and store any and all data, in anticipation of potential future strategic value. These differences in data heterogeneity, scale and usage are leading to a new generation of data management and analytic systems, where the emphasis is on supporting a wide range of very large datasets that are stored uniformly and analyzed seamlessly using whatever techniques are most appropriate, including traditional tools like SQL and BI and newer tools, e.g., for machine learning and stream analytics. These new systems are necessarily based on scale-out architectures for both storage and computation. Hadoop has become a key building block in the new generation of scale-out systems. On the storage side, HDFS has provided a cost-effective and scalable substrate for storing large heterogeneous datasets. However, as key customer and systems touch points are instrumented to log data, and Internet of Things applications become common, data in the enterprise is growing at a staggering pace, and the need to leverage different storage tiers (ranging from tape to main memory) is posing new challenges, leading to caching technologies, such as Spark. On the analytics side, the emergence of resource managers such as YARN has opened the door for analytics tools to bypass the Map-Reduce layer and directly exploit shared system resources while computing close to data copies. This trend is especially significant for iterative computations such as graph analytics and machine learning, for which Map-Reduce is widely recognized to be a poor fit. While Hadoop is widely recognized and used externally, Microsoft has long been at the forefront of Big Data analytics, with Cosmos and Scope supporting all internal customers. These internal services are a key part of our strategy going forward, and are enabling new state of the art external-facing services such as Azure Data Lake and more. I will examine these trends, and ground the talk by discussing the Microsoft Big Data stack.

The Hive Think Tank - The Microsoft Big Data Stack by Raghu Ramakrishnan, CTO...

The Hive

Overview of stinger interactive query for hive

David Kaiser

Data-In-Motion Unleashed

DataWorks Summit

Hadoop and Spark are big data frameworks used to extract useful span a variety of scenarios from ingestion, data prep, data management, processing, analyzing and visualizing data. Each step requires specialized toolsets to be productive. In this talk I will share solution examples in the Big Data ecosystem such as Cask, StreamSets, Datameer, AtScale, Dataiku on Microsoft’s Azure HDInsight that simplify your Big Data solutions. Azure HDInsight is a cloud Spark and Hadoop service for the enterprise and take advantage of all the benefits of HDInsight giving you the best of both worlds. Join this session for practical information that will enable faster time to insights for you and your business.

Build Big Data Enterprise Solutions Faster on Azure HDInsight

DataWorks Summit/Hadoop Summit

Common and unique use cases for Apache Hadoop

Brock Noland

Big Data with Azure

Aaron (Ari) Bornstein

Can you Re-Platform your Teradata, Oracle, Netezza and SQL Server Analytic Wo...

DataWorks Summit

During the Big Data Warehousing Meetup, we discussed options for enabling real-time/interactive queries to support business intelligence type functionality on Hadoop. Also, Hortonworks provided a deep-dive demo of Stinger! You can access that slideshow here: http://www.slideshare.net/CasertaConcepts/stinger-initiative-hortonworks If you would like more information, please don't hesitate to contact us at info@casertaconcepts.com. Or, visit our website at http://casertaconcepts.com/.

Real Time Interactive Queries IN HADOOP: Big Data Warehousing Meetup

Caserta

Big Data Ready Enterprise

DataWorks Summit/Hadoop Summit

Brian Brownlow is an experienced senior analyst programmer for Mayo Clinic. He is made a workshop presentation at the 2014 BDPA Technology Conference on the topic, 'Big Data Implementation - Mayo Clinic Case Study'. This presentation will show part of the Mayo Clinic story on the embarking of an exploration of the application of `Big Data' technologies. `Big Data' is seen as one set of tools that can be used to enhance medical research, medical education and practice management. Mayo Clinic is always searching for better, faster and cheaper ways to use its data to improve patient care and sustain financial outcomes in a challenging reimbursement environment. Our approach uses several components that are open source and combines them with data from various sources to provide information to decision makers in near real time. We have created a center of `Big Data' excellence using in-house staff and vendor engagements. `Big Data' is one element of our Enterprise Data Trust framework.

A Mayo Clinic Big Data Implementation

BDPA Education and Technology Foundation

Big Data in Azure

DataWorks Summit/Hadoop Summit

Databricks is a Software-as-a-Service-like experience (or Spark-as-a-service) that is a tool for curating and processing massive amounts of data and developing, training and deploying models on that data, and managing the whole workflow process throughout the project. It is for those who are comfortable with Apache Spark as it is 100% based on Spark and is extensible with support for Scala, Java, R, and Python alongside Spark SQL, GraphX, Streaming and Machine Learning Library (Mllib). It has built-in integration with many data sources, has a workflow scheduler, allows for real-time workspace collaboration, and has performance improvements over traditional Apache Spark.

Introduction to Azure Databricks

James Serra

Spark is a unified framework for big data analytics. Spark provides one integrated API for use by developers, data scientists, and analysts to perform diverse tasks that would have previously required separate processing engines such as batch analytics, stream processing and statistical modeling. Spark supports a wide range of popular languages including Python, R, Scala, SQL, and Java. Spark can read from diverse data sources and scale to thousands of nodes. In this presentation we discuss Microsoft HDInsight offering of Spark. Azure HDInsight, Microsoft’s managed Hadoop and Spark cloud service that runs the Hortonworks Data Platform. Spark for Azure HDInsight offers customers an enterprise-ready Spark solution that’s fully managed, secured, and highly available and made simpler for users with compelling and interactive experiences.

Spark with Azure HDInsight - Tampa Bay Data Science - Adnan Masood, PhD

Adnan Masood

What's hot (20)

Interactive query in hadoop

Interactive query using hadoop

Hadoop Reporting and Analysis - Jaspersoft

Big data on Azure for Architects

Scaling Data Science on Big Data

Evolution of Big Data at Intel - Crawl, Walk and Run Approach

Hadoop in the cloud – The what, why and how from the experts

The Hive Think Tank - The Microsoft Big Data Stack by Raghu Ramakrishnan, CTO...

Overview of stinger interactive query for hive

Data-In-Motion Unleashed

Build Big Data Enterprise Solutions Faster on Azure HDInsight

Common and unique use cases for Apache Hadoop

Big Data with Azure

Can you Re-Platform your Teradata, Oracle, Netezza and SQL Server Analytic Wo...

Real Time Interactive Queries IN HADOOP: Big Data Warehousing Meetup

Big Data Ready Enterprise

A Mayo Clinic Big Data Implementation

Big Data in Azure

Introduction to Azure Databricks

Spark with Azure HDInsight - Tampa Bay Data Science - Adnan Masood, PhD

Similar to HDInsight Hadoop on Windows Azure

• Capable of processing large sets of structured, semi-structured and unstructured data and supporting system architecture • Implemented Proof of concepts on Hadoop stack and different big data analytic tools, migration from different databases to Hadoop. • Developed multiple Map Reduce jobs in java for data cleaning and pre-processing according to the business requirements, Importing and exporting data into HDFS and Hive using Sqoop. Having Experience in writing HIVE queries & Pig scripts.

Sureh hadoop 3 years t

suresh thallapelly

[Azureビッグデータ関連サービスとHortonworks勉強会] Azure HDInsight

Naoki (Neo) SATO

Big Data with hadoop, Spark and BigQuery (Google cloud next Extended 2017 Kar...

Imam Raza

Srikanth hadoop 3.6yrs_hyd

srikanth K

Poorna Hadoop

Poornachandrarao Kommana

Sunshine consulting mopuru babu cv_java_j2ee_spring_bigdata_scala

Mopuru Babu

Talend Big Data Capabilities Overview

Rajan Kanitkar

Resume_VipinKP

indhuparvathy

Big Data Hoopla Simplified - TDWI Memphis 2014

Rajan Kanitkar

Hadoop - A Very Short Introduction

dewang_mistry

Spark tutorial @ KCC 2015

Jongwook Woo

Deepankar Sehdev- Resume2015

Deepankar Sehdev

Robin_Hadoop

Robin David

Hadoop MapReduce Fundamentals

KOTI_RESUME_(1) (2)

Mukul-Resume

Hadoop in action

Prashanth Kumar_Hadoop_NEW

Prashanth Shankar kumar

Hadoop Frameworks Panel__HadoopSummit2010

Yahoo Developer Network

Hadoop and Big Data: Revealed

Sachin Holla

Similar to HDInsight Hadoop on Windows Azure (20)

Sureh hadoop 3 years t

[Azureビッグデータ関連サービスとHortonworks勉強会] Azure HDInsight

Big Data with hadoop, Spark and BigQuery (Google cloud next Extended 2017 Kar...

Srikanth hadoop 3.6yrs_hyd

Poorna Hadoop

Sunshine consulting mopuru babu cv_java_j2ee_spring_bigdata_scala

Talend Big Data Capabilities Overview

Resume_VipinKP

Big Data Hoopla Simplified - TDWI Memphis 2014

Hadoop - A Very Short Introduction

Spark tutorial @ KCC 2015

Deepankar Sehdev- Resume2015

Robin_Hadoop

Hadoop MapReduce Fundamentals

KOTI_RESUME_(1) (2)

Mukul-Resume

Hadoop in action

Prashanth Kumar_Hadoop_NEW

Hadoop Frameworks Panel__HadoopSummit2010

Hadoop and Big Data: Revealed

Recently uploaded

This presentation explores the impact of HTML injection attacks on web applications, detailing how attackers exploit vulnerabilities to inject malicious code into web pages. Learn about the potential consequences of such attacks and discover effective mitigation strategies to protect your web applications from HTML injection vulnerabilities. for more information visit https://bostoninstituteofanalytics.org/category/cyber-security-ethical-hacking/

HTML Injection Attacks: Impact and Mitigation Strategies

Boston Institute of Analytics

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Scaling API-first – The story of a global engineering organization

Radu Cotescu

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

🐬 The future of MySQL is Postgres 🐘

RTylerCroy

Increase engagement and revenue with Muvi Live Paywall! In this presentation, we will explore the five key benefits of using Muvi Live Paywall to monetize your live streams. You'll learn how Muvi Live Paywall can help you: Monetize your live content easily: Set up pay-per-view access to your live streams and start generating revenue from your content. Increase audience engagement: Provide exclusive, premium content behind the paywall to keep your viewers engaged. Gain valuable viewer insights: Track viewer data and analytics to better understand your audience and tailor your content accordingly. Reduce content piracy: Muvi Live Paywall's security features help protect your content from unauthorized distribution. Streamline your workflow: The all-in-one platform simplifies the process of managing and monetizing your live streams. With Muvi Live Paywall, you can take control of your live stream monetization and create a sustainable business model for your content. Learn more about Muvi Live Paywall and start generating revenue from your live streams today!

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Roshan Dwivedi

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Recently uploaded (20)

HTML Injection Attacks: Impact and Mitigation Strategies

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Scaling API-first – The story of a global engineering organization

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Axa Assurance Maroc - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

🐬 The future of MySQL is Postgres 🐘

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Powerful Google developer tools for immediate impact! (2023-24 C)

Why Teams call analytics are critical to your entire business

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Exploring the Future Potential of AI-Enabled Smartphone Processors

Automating Google Workspace (GWS) & more with Apps Script

HDInsight Hadoop on Windows Azure

1. S Hadoop on Azure @LynnLangit

2. Data Expertise / Lynn Langit Practicing Architect Cloud Deployments (Azure, AWS, Google) Technical author / trainer Google Cloud Developer Series SQL Server 2012 Developer Series Cloudera Certified Developer 2 books on SQL Server BI Industry awards Microsoft – MVP for SQL Server Google – GDE for Cloud Platform 10Gen – Master for MongoDB Former MSFT FTE 4 years

3. What is Hadoop? S HUGE Hype factor in 2011 / 2012 Apache Hadoop is a software framework that supports data- intensive distributed applications under a free license • Uses HDFS storage to enable applications to work with thousands of nodes and petabytes of data • Uses MapReduce to process the data • Inspired by Google • MapReduce • Google File System

4. What is HDInsight? S Hadoop on Windows S Azure S On-premise S Microsoft worked with Hortonworks to port Hadoop to Windows (from Linux)

5. Working with HDInsight

6. RDBMS vs. Hadoop RDBMS Hadoop Data Size Gigabytes (Terabytes) Petabytes (Hexabytes) Access Interactive and Batch Batch – NOT Interactive Updates Read / Write many times Write once, Read many times Structure Static Schema Dynamic Schema Integrity High (ACID) Low Scaling Nonlinear Linear Query Response Time Can be near immediate Has latency (due to batch processing)

7. Setting Up Your Cluster

8. Configuration 1

9. Configuration 2

10. Pricing (during Preview)

11. Demo

12. Basic Administration Connect via RDP

13. NameNode Utility – Top Level

14. NameNode Utility – Drill Down

15. Understanding Storage

16. Using the Azure Storage Viewer

17. What is MapReduce?

18. MapReduce using Java S WordCount example

19. MapReduce using C# Streaming S WordCount example

20. MapReduce using JavaScript S WordCount example

21. Simple Output Graphing S WordCount example

22. Using HIVE

23. Understanding Pig Load>Transform>Dump or Store

24. Monitoring Job Results S In the portal S Main Console S Job icon (button) status summary S Job History S Interactive Console S JS quick feedback S JS detailed feedback (log) S Using RDP S Map/Reduce tool S Hadoop command prompt

25.

26. Monitoring Job Status

27. Download – ODBC for HIVE S Includes add-in for Excel

28. Hadoop Connector to Excel

29. Connecting to PowerPivot S Create an ODBC connection to HIVE S Connect to ‘other data source’ in PowerPivot

30. Connecting with PowerQuery

31. Pulling it Together - Klout

32. Hadoop To-Do List • Use Hadoop when business needs designate • Use other NoSQL if a better fit BigData = Hadoop • Quick and cheap • Specialized use cases • Behavioral data • dev, test , training environments Hadoop on the cloud • Learn Map/Reduce • Use HIVE via Excel • Pay attention to Impala Hadoop access technologies

33. www.TeachingKidsProgramming.org

34. VOTE CONFIRM SHARE

35. Keep Learning S @LynnLangit S YouTube – SoCalDevGal S Hire Me S Architecture S Best Practices S Performance Tuning

Editor's Notes

http://hadoop.apache.org/http://en.wikipedia.org/wiki/Apache_Hadoop
http://www.infosysblogs.com/microsoft/2011/12/isotope_hadoop_on_windows_and.html
OriginalReference: Tom White’s Hadoop: The Definitive Guide (I made some modifications based on my experience)
https://www.hadooponazure.com/AccountDemo - http://www.youtube.com/watch?v=XcHz8aUDDN8 and http://www.youtube.com/watch?v=c7oHntP8HBI
http://www.windowsazure.com/en-us/manage/services/hdinsight/howto-blob-store/
http://blog.gopivotal.com/products/hadoop-101-programming-mapreduce-with-native-libraries-hive-pig-and-cascading
https://www.hadooponazure.com/AccountDemo - http://www.youtube.com/watch?v=XcHz8aUDDN8 and http://www.youtube.com/watch?v=c7oHntP8HBI
https://www.hadooponazure.com/AccountDemo - http://www.youtube.com/watch?v=XcHz8aUDDN8 and http://www.youtube.com/watch?v=c7oHntP8HBI
https://www.hadooponazure.com/AccountDemo - http://www.youtube.com/watch?v=XcHz8aUDDN8 and http://www.youtube.com/watch?v=c7oHntP8HBI
https://www.hadooponazure.com/AccountDemo - http://www.youtube.com/watch?v=XcHz8aUDDN8 and http://www.youtube.com/watch?v=c7oHntP8HBI
https://www.hadooponazure.com/AccountDemo - http://www.youtube.com/watch?v=XcHz8aUDDN8 and http://www.youtube.com/watch?v=c7oHntP8HBI
http://www.windowsazure.com/en-us/manage/services/hdinsight/using-pig-with-hdinsight/
http://www.microsoft.com/en-us/bi/default.aspxhttp://dennyglee.com/Demos - http://www.youtube.com/watch?v=djfpPsGwm6Aand http://www.youtube.com/watch?v=uh9bKWO1K7U
Detailed info - http://dennyglee.com/2012/01/21/connecting-powerpivot-to-hadoop-on-azure-self-service-bi-to-big-data-in-the-cloud/
http://www.youtube.com/watch?v=eRXEA9-l2eQhttp://thinknook.com/architectures-for-running-sql-server-analysis-service-ssas-on-data-in-hadoop-hive-2013-02-25/#!prettyPhoto

HDInsight Hadoop on Windows Azure

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to HDInsight Hadoop on Windows Azure

Similar to HDInsight Hadoop on Windows Azure (20)

More from Lynn Langit

More from Lynn Langit (20)

Recently uploaded

Recently uploaded (20)

HDInsight Hadoop on Windows Azure

Editor's Notes