Riding the Elephant - Hadoop 2.0

•Descargar como PPT, PDF•

0 recomendaciones•753 vistas

Hadoop 2.0, and in particular YARN has opened up a lot of potential applications beyond MapReduce. This presentation explains some of the ways this happened, and what you can now do that you couldn't before. It also introduces some new tools (Spark) and infrastructure pieces (Mesos) to achieve even more efficient cluster use.

Tecnología

Simon Elliston Ball
Head of Big Data - Red Gate Ventures
@sireb
Riding the Elephant:
Hadoop 2.0
http://bit.ly/RidingElephants

Append only distributed file-system
In the beginning…
Map Reduce
Java.

JVM Based (scala, groovy, jython, clojure)
More languages
Streaming (python, whatever)
HDP for Windows and .NET SDK

Abstraction
Photo: https://www.flickr.com/photos/puroticorico/
Hive, Pig
Cascading
Scalding

SQL on Hadoop
Learning to share the toys
HBase
Solr on Hadoop
Sharing HDFS…

Map Reduce v1
JobTracker
Job
Head Node
TaskTracker
Task (Map / Reduce)
Data Node
m slot 1
m slot 2
…
m slot n
Task
r slot 1
r slot 2
…
r slot n
TaskTracker
Task (Map / Reduce)
Data Node
m slot 1
m slot 2
…
m slot n
r slot 1
r slot 2
…
r slot n
TaskTracker
Task (Map / Reduce)
Data Node
m slot 1
m slot 2
…
m slot n
r slot 1
r slot 2
…
r slot n

Map Reduce v1
JobTracker
Job
Head Node
TaskTracker
Task (Map / Reduce)
Data Node
m slot 1
m slot 2
…
m slot n
MR Status
r slot 1
r slot 2
…
r slot n
TaskTracker
Task (Map / Reduce)
Data Node
m slot 1
m slot 2
…
m slot n
r slot 1
r slot 2
…
r slot n
TaskTracker
Task (Map / Reduce)
Data Node
m slot 1
m slot 2
…
m slot n
r slot 1
r slot 2
…
r slot n

Typical Hadoop 1.x setup
HBaseProductionAdhoc

YARN architecture
Container
Application
Master
Container
Data Node
Node Manager
Container ContainerContainer
Data Node
Node Manager
Application
Master
Container Free Slot
Data Node
Node Manager
Resource
Manager
YARN Client

Removing the choke point
Advantages
60%-150% better usage
Long running applications

Not quite…
Operating system for Big Data?
Security
…but a framework for Big Data Apps
Data Access abstraction

Storm on YARN
A whole batch of new applications
HOYA
Tez (Stinger)MapReduce 2
Giraph
<Insert your application here>

Batch applications
Spinning YARNs with Spring
Services
Direct to YARN APIs
Spring Data Hadoop abstraction

Streaming
Why?
Machine Learning
Graphs
Services
Distributed Shell - Anything.

Spark
A higher abstraction
Hadoop based?
… but can run on YARN
In Memory
Distributed
Fault tolerant
Real-time
✓
✓
✓
✓
❌
RRDs
✓

Mesos
Wider sharing
HadoopSparkAurora
Mesos Framework
Hardware
YARN
MapReduce HBase etc
HDFS

Hadoop is more than MapReduce
The new world
YARN opens up new paradigms
Infrastructure maturing: better sharing

Questions?
Simon Elliston Ball
Head of Big Data - Red Gate Ventures
@sireb
simon@simonellistonball.com
http://bit.ly/RidingElephants

Más contenido relacionado

La actualidad más candente

Watch the recording of the speech done at Spark Summit Brussles 2016 here: https://www.youtube.com/watch?v=wyfTjd9z1sY Data Science with SparkML on DataBricks is a perfect platform for application of Ensemble Learning on massive a scale. This presentation describes Prediction-as-a-Service platform which can predict trends on 1 billion observed prices daily. In order to train ensemble model on a multivariate time series in thousands/millions dimensional space, one has to fragment the whole space into subspaces which exhibit a significant similarity. In order to achieve this, the vastly sparse space has to undergo dimensionality reduction into a parameters space which then is used to cluster the observations. The data in the resulting clusters is modeled in parallel using machine learning tools capable of coefficient estimation at the massive scale (SparkML and Scikit Learn). The estimated model coefficients are stored in a database to be used when executing predictions on demand via a web service. This approach enables training models fast enough to complete the task within a couple of hours, allowing daily or even real time updates of the coefficients. The above machine learning framework is used to predict the airfares used as support tool for the airline Revenue Management systems.

Prediction as a service with ensemble model in SparkML and Python ScikitLearn

Josef A. Habdank

The BDAS Open Source Community

jeykottalam

R is a hugely popular platform for Data Scientists to create analytic models in many different domains. But when these applications should move from the science lab to the production environment of large enterprises a new set of challenges arises. Independently of R, Spark has been very successful as a powerful general-purpose computing platform. With the introduction of SparkR an exciting new option to productionize Data Science applications has been made available. This talk will give insight into two real-life projects at major enterprises where Data Science applications in R have been migrated to SparkR. • Dealing with platform challenges: R was not installed on the cluster. We show how to execute SparkR on a Yarn cluster with a dynamic deployment of R. • Integrating Data Engineering and Data Science: we highlight the technical and cultural challenges that arise from closely integrating these two different areas. • Separation of concerns: we describe how to disentangle ETL and data preparation from analytic computing and statistical methods. • Scaling R with SparkR: we present what options SparkR offers to scale R applications and how we applied them to different areas such as time series forecasting and web analytics. • Performance Improvements: we will show benchmarks for an R applications that took over 20 hours on a single server/single-threaded setup. With moderate effort we have been able to reduce that number to 15 minutes with SparkR. And we will show how we plan to further reduces this to less than a minute in the future. • Mixing SparkR, SparkSQL and MLlib: we show how we combined the three different libraries to maximize efficiency. • Summary and Outlook: we describe what we have learnt so far, what the biggest gaps currently are and what challenges we expect to solve in the short- to mid-term.

Using SparkR to Scale Data Science Applications in Production. Lessons from t...

Spark Summit

Apache Spark is a dynamic execution engine that can take relatively simple Scala code and create complex and optimized execution plans. In this talk, we will describe how user code translates into Spark drivers, executors, stages, tasks, transformations, and shuffles. We will then describe how this is critical to the design of Spark and how this tight interplay allows very efficient execution. We will also discuss various sources of metrics on how Spark applications use hardware resources, and show how application developers can use this information to write more efficient code. Users and operators who are aware of these concepts will become more effective at their interactions with Spark.

Apache Spark Performance is too hard. Let's make it easier

Databricks

A Graph-Based Method For Cross-Entity Threat Detection

Jen Aman

H2O World - PySparkling Water - Nidhi Mehta

Sri Ambati

This session will explain how NetApp simplifies the process of analyzing IoT data, using Apache Spark clusters across data centers and the cloud using NetApp Private Storage (NPS) for AWS/Azure, NetApp Data Fabric and NetApp Connectors for NFS and S3. IoT data originates at the edge in different geographical locations, and it can arrive at different data centers or the cloud depending on sensor location. The challenge is how to combine these different data streams across different datacenters to generate wider insights. Learn how NetApp Data Fabric helps solve this challenge. In the Data Fabric architecture, the IoT data is ingested via Kafka into an Apache Spark cluster running in AWS/Azure, but the data is stored in NPS provisioned NFS share through NFS Connector. The IoT data in NPS can then be moved to on-prem datacenters, or on-prem IoT data can be moved to NPS or ONTAP Cloud for processing in AWS/Azure using NetApp SnapMirror Flex Clone or NFS Connector. We’ll also review how NetApp StorageGRID object storage maintains IoT data for archival purposes using S3 Target. The above options allow you to analyze IoT data from AWS, StorageGRID, HDFS or NFS, providing a feasible solution for deploying Spark clusters across datacenters. Takeaways will include identifying Spark challenges that can be remedied by extending your Spark environment to take advantage of NPS; understanding how NPS and StorageGRID can provide a cost-effective alternative for dev/test, DR for Spark analytics; and understanding Spark architecture and deployment options that utilize data from multiple locations, including on-prem and cloud-based repositories.

Analyzing IOT Data in Apache Spark Across Data Centers and Cloud with NetApp ...

Databricks

Extreme Apache Spark: how in 3 months we created a pipeline that can process ...

Josef A. Habdank

Spark Under the Hood - Meetup @ Data Science London

Databricks

Architecting R into the Storm Application Development Process ~~~~~ The business need for real-time analytics at large scale has focused attention on the use of Apache Storm, but an approach that is sometimes overlooked is the use of Storm and R together. This novel combination of real-time processing with Storm and the practical but powerful statistical analysis offered by R substantially extends the usefulness of Storm as a solution to a variety of business critical problems. By architecting R into the Storm application development process, Storm developers can be much more effective. The aim of this design is not necessarily to deploy faster code but rather to deploy code faster. Just a few lines of R code can be used in place of lengthy Storm code for the purpose of early exploration – you can easily evaluate alternative approaches and quickly make a working prototype. In this presentation, Allen will build a bridge from basic real-time business goals to the technical design of solutions. We will take an example of a real-world use case, compose an implementation of the use case as Storm components (spouts, bolts, etc.) and highlight how R can be an effective tool in prototyping a solution.

R + Storm Moneyball - Realtime Advanced Statistics - Hadoop Summit - San Jose

Allen Day, PhD

London Spark Meetup Project Tungsten Oct 12 2015

Chris Fregly

Spark and Ignite are two of the most popular open source projects in the area of high-performance Big Data and Fast Data. But did you know that one of the best ways to boost performance for your next generation real-time applications is to use them together? In this session, Christos Erotocritou – Lead GridGain solutions architect, will explain in detail how IgniteRDD – an implementation of native Spark RDD and DataFrame APIs – shares the state of the RDD across other Spark jobs, applications and workers. Christos will also demonstrate how IgniteRDD, with its advanced in-memory indexing capabilities, allows execution of SQL queries many times faster than native Spark RDDs or Data Frames. Furthermore we will be discussing the newest feature additions and what the future holds for this integration.

Fast Data with Apache Ignite and Apache Spark with Christos Erotocritou

Spark Summit

Introduction to MapReduce

Hassan A-j

LocationTech is an Eclipse Foundation industry working group for location aware technologies. This presentation introduces LocationTech, looks at what it means for our industry and the participating projects. Libraries: JTS Topology Suite is the rocket science of GIS providing an implementation of Geometry. Mobile Map Tools provides a C++ foundation that is translated into Java and Javascript for maps on iOS, Andriod and WebGL. GeoMesa is a distributed key/value store based on Accumulo. Spatial4j integrates with JTS to provide Geometry on curved surface. Process: GeoTrellis real-time distributed processing used scala, akka and spark. GeoJinni mixes spatial data/indexing with Hadoop. Applications: GEOFF offers OpenLayers 3 as a SWT component. GeoGit distributed revision control for feature data. GeoScipt brings spatial data to Groovy, JavaScript, Python and Scala. uDig offers an eclipse based desktop GIS solution. Attend this presentation if want to know what LocationTech is about, are interested in these projects or curious about what projects will be next.

LocationTech Projects

Jody Garnett

Dask: Scaling Python

Matthew Rocklin

In this webcast, Patrick Wendell from Databricks will be speaking about Apache Spark's new 1.6 release. Spark 1.6 will include (but not limited to) a type-safe API called Dataset on top of DataFrames that leverages all the work in Project Tungsten to have more robust and efficient execution (including memory management, code generation, and query optimization) [SPARK-9999], adaptive query execution [SPARK-9850], and unified memory management by consolidating cache and execution memory [SPARK-10000].

Apache® Spark™ 1.6 presented by Databricks co-founder Patrick Wendell

Databricks

# Talk given at PyCon UK 2017 The first step in any data-intensive project is understanding the available data. To this end, data scientists spend a significant part of their time carrying out data quality assessments and data exploration. In spite of this being a crucial step, it usually requires repeating a series of menial tasks before the data scientist gains an understanding ofthe dataset and can progress to the next steps in the project. In this talk I will detail the inner workings of a Python package that we have built which automates this drudge work, enables efficient data exploration, and kickstarts data science projects. A summary is generated for each dataset, including: - General information about the dataset, including data quality of each of the columns; - Distribution of each of the columns through statistics and plots (histogram, CDF, KDE), optionally grouped by other categorical variables; - 2D distribution between pairs of columns; - Correlation coefficient matrix for all numerical columns. Building this tool has provided a unique view into the full Python data stack, from the parallelised analysis of a dataframe within a Dask custom execution graph, to the interactive visualisation with Jupyter widgets and Plotly. During the talk, I will also introduce how Dask works, and demonstrate how to migrate data pipelines to take advantage of its scalable capabilities.

AUTOMATED DATA EXPLORATION - Building efficient analysis pipelines with Dask

Víctor Zabalza

The first step in any data-intensive project is understanding the available data. To this end, data scientists spend a significant part of their time carrying out data quality assessments and data exploration. In spite of this being a crucial step, it usually requires repeating a series of menial tasks before the data scientist gains an understanding of the dataset and can progress to the next steps in the project. In this talk I will present Lens (https://github.com/asidatascience/lens), a Python package which automates this drudge work, enables efficient data exploration, and kickstarts data science projects. A summary is generated for each dataset, including: - General information about the dataset, including data quality of each of the columns; - Distribution of each of the columns through statistics and plots (histogram, CDF, KDE), optionally grouped by other categorical variables; - 2D distribution between pairs of columns; - Correlation coefficient matrix for all numerical columns. Building this tool has provided a unique view into the full Python data stack, from the parallelised analysis of a dataframe within a Dask custom execution graph, to the interactive visualisation with Jupyter widgets and Plotly. During the talk, I will also introduce how Dask works, and demonstrate how to migrate data pipelines to take advantage of its scalable capabilities.

Lens: Data exploration with Dask and Jupyter widgets

Víctor Zabalza

Kubernetes is an open-source containerization framework that makes it easy to manage applications in isolated environments at scale. In Apache Spark 2.3, Spark introduced support for native integration with Kubernetes. Palantir has been deeply involved with the development of Spark’s Kubernetes integration from the beginning, and our largest production deployment now runs an average of ~5 million Spark pods per day, as part of tens of thousands of Spark applications. Over the course of our adventures in migrating deployments from YARN to Kubernetes, we have overcome a number of performance, cost, & reliability hurdles: differences in shuffle performance due to smaller filesystem caches in containers; Kubernetes CPU limits causing inadvertent throttling of containers that run many Java threads; and lack of support for dynamic allocation leading to resource wastage. We intend to briefly describe our story of developing & deploying Spark-on-Kubernetes, as well as lessons learned from deploying containerized Spark applications in production. We will also describe our recently open-sourced extension (https://github.com/palantir/k8s-spark-scheduler) to the Kubernetes scheduler to better support Spark workloads & facilitate Spark-aware cluster autoscaling; our limited implementation of dynamic allocation on Kubernetes; and ongoing work that is required to support dynamic resource management & stable performance at scale (i.e., our work with the community on a pluggable external shuffle service API). Our hope is that our lessons learned and ongoing work will help other community members who want to use Spark on Kubernetes for their own workloads.

Reliable Performance at Scale with Apache Spark on Kubernetes

Databricks

First impressions of SparkR: our own machine learning algorithm

InfoFarm

La actualidad más candente (20)

Prediction as a service with ensemble model in SparkML and Python ScikitLearn

The BDAS Open Source Community

Using SparkR to Scale Data Science Applications in Production. Lessons from t...

Apache Spark Performance is too hard. Let's make it easier

A Graph-Based Method For Cross-Entity Threat Detection

H2O World - PySparkling Water - Nidhi Mehta

Analyzing IOT Data in Apache Spark Across Data Centers and Cloud with NetApp ...

Extreme Apache Spark: how in 3 months we created a pipeline that can process ...

Spark Under the Hood - Meetup @ Data Science London

R + Storm Moneyball - Realtime Advanced Statistics - Hadoop Summit - San Jose

London Spark Meetup Project Tungsten Oct 12 2015

Fast Data with Apache Ignite and Apache Spark with Christos Erotocritou

Introduction to MapReduce

LocationTech Projects

Dask: Scaling Python

Apache® Spark™ 1.6 presented by Databricks co-founder Patrick Wendell

AUTOMATED DATA EXPLORATION - Building efficient analysis pipelines with Dask

Lens: Data exploration with Dask and Jupyter widgets

Reliable Performance at Scale with Apache Spark on Kubernetes

First impressions of SparkR: our own machine learning algorithm

Similar a Riding the Elephant - Hadoop 2.0

Riding the Elephant - Hadoop 2.0

Simon Elliston Ball

With the rise of the cloud, data intensive systems and the Internet of Things the use of distributed systems have become widespread. The first big player was Hadoop, which provided an integral solution to Big Data storage and computation problems. Its popularity empowered many organizations to adopt this technology. However new challenges appeared, like the need to be able to execute iterative, interactive or in-memory algorithms without the disk-intensive burden of MapReduce. For that very reason Hadoop evolved, decoupling its resources manager from the main computation engine: YARN was born. As a result of its vast adoption, YARN has become the de-facto distributed operating system for Big Data. Since early releases, Apache Spark provided a way to be executed on YARN-powered clusters. In this talk we will take a look into that technology, and we will learn what it means having Spark running on this kind of infrastructure.

Yarn by default (Spark on YARN)

Ferran Galí Reniu

Ever wonder what Hadoop might look like in 12 months or 24 months or longer? Apache Hadoop MapReduce has undergone a complete re-haul to emerge as Apache Hadoop YARN, a generic compute fabric to support MapReduce and other application paradigms. As a result, Hadoop looks very different from itself 12 months ago. This talk will take you through some ideas for YARN itself and the many myriad ways it is really moving the needle for MapReduce, Pig, Hive, Cascading and other data-processing tools in the Hadoop ecosystem.

Hadoop past, present and future

Codemotion

Hackathon bonn

Emil Andreas Siemes

This session will give you the architectural overview and introduction in to inner workings of HDP 2.0 (http://hortonworks.com/products/hdp-windows/) and HDInsight. The world has embraced the Hadoop toolkit to solve their data problems from ETL, data warehouses to event processing pipelines. As Hadoop consists of many components, services and interfaces, understanding its architecture is crucial, before you can successfully integrate it in to your own environment.

The Fundamentals Guide to HDP and HDInsight

Gert Drapers

Huhadoop - v1.1

Big Data Joe™ Rossi

How YARN Enables Multiple Data Processing Engines in Hadoop

POSSCON

Hadoop - Past, Present and Future - v1.2

Big Data Joe™ Rossi

Hadoop Essential for Oracle Professionals

Chien Chung Shen

2013 Nov 20 Toronto Hadoop User Group (THUG) - Hadoop 2.2.0

Adam Muise

Hadoop - Past, Present and Future - v1.1

Big Data Joe™ Rossi

Handling not so big data

SATOSHI TAGOMORI

EclipseCon Keynote: Apache Hadoop - An Introduction

Cloudera, Inc.

2014 sept 26_thug_lambda_part1

Adam Muise

BDE-BDVA Webinar: BDE Technical Overview

BigData_Europe

RackN Physical Layer Automation Innovation

rhirschfeld

20140202 fosdem-nosql-devroom-hadoop-yarn

Datalayer

With the rise of the cloud and the Internet of Things the use of distributed systems have become widespread. Hadoop appeared several years ago but new challenges made it reach a new level, evolving into YARN (sometimes also called MapReduce 2.0). This new generation of cluster management services allows applications to run on data centers not only in a more optimized MapReduce, but making other paradigms to share its resources and unleashing its limitation to run iterative or interactive algorithms. The talk will be an introduction to this passionate world. We will take a look into the past, understand how YARN works underneath, and know some of its applications in the industry.

Unleash your cluster with YARN

Ferran Galí Reniu

Overview of slider project

Steve Loughran

Big Data Essentials meetup @ IBM Ljubljana 23.06.2015

Andrey Vykhodtsev

Similar a Riding the Elephant - Hadoop 2.0 (20)

Riding the Elephant - Hadoop 2.0

Yarn by default (Spark on YARN)

Hadoop past, present and future

Hackathon bonn

The Fundamentals Guide to HDP and HDInsight

Huhadoop - v1.1

How YARN Enables Multiple Data Processing Engines in Hadoop

Hadoop - Past, Present and Future - v1.2

Hadoop Essential for Oracle Professionals

2013 Nov 20 Toronto Hadoop User Group (THUG) - Hadoop 2.2.0

Hadoop - Past, Present and Future - v1.1

Handling not so big data

EclipseCon Keynote: Apache Hadoop - An Introduction

2014 sept 26_thug_lambda_part1

BDE-BDVA Webinar: BDE Technical Overview

RackN Physical Layer Automation Innovation

20140202 fosdem-nosql-devroom-hadoop-yarn

Unleash your cluster with YARN

Overview of slider project

Big Data Essentials meetup @ IBM Ljubljana 23.06.2015

Más de Simon Elliston Ball

A streaming architecture for Cyber Security - Apache Metron

Simon Elliston Ball

mcubed london - data science at the edge

Simon Elliston Ball

When to no sql and when to know sql javaone

Simon Elliston Ball

Machine learning without the PhD - azure ml

Simon Elliston Ball

Why Hadoop and SQL just want to be friends - lightning talk NoSQL Matters Dub...

Simon Elliston Ball

Getting your Big Data on with HDInsight

Simon Elliston Ball

Finding and Using Big Data in your business

Simon Elliston Ball

When to NoSQL and when to know SQL

Simon Elliston Ball

Mongo db for c# developers

Simon Elliston Ball

NDC London 2013 - Mongo db for c# developers

Simon Elliston Ball

Mongo db for C# Developers

Simon Elliston Ball

Más de Simon Elliston Ball (11)

A streaming architecture for Cyber Security - Apache Metron

mcubed london - data science at the edge

When to no sql and when to know sql javaone

Machine learning without the PhD - azure ml

Why Hadoop and SQL just want to be friends - lightning talk NoSQL Matters Dub...

Getting your Big Data on with HDInsight

Finding and Using Big Data in your business

When to NoSQL and when to know SQL

Mongo db for c# developers

NDC London 2013 - Mongo db for c# developers

Mongo db for C# Developers

Último

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Dubai, often portrayed as a shimmering oasis in the desert, faces its own set of challenges, including the occasional threat of flooding. Despite its reputation for opulence and modernity, the emirate is not immune to the forces of nature. In recent years, Dubai has experienced sporadic but significant floods, testing the resilience of its infrastructure and communities. Among the critical lifelines in this bustling metropolis is the Dubai International Airport, a bustling hub that connects the city to the world. This article explores the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Orbitshub

💥 You’re lucky! We’ve found two different (lead) developers that are willing to share their valuable lessons learned about using UiPath Document Understanding! Based on recent implementations in appealing use cases at Partou and SPIE. Don’t expect fancy videos or slide decks, but real and practical experiences that will help you with your own implementations. 📕 Topics that will be addressed: • Training the ML-model by humans: do or don't? • Rule-based versus AI extractors • Tips for finding use cases • How to start 👨‍🏫👨‍💻 Speakers: o Dion Morskieft, RPA Product Owner @Partou o Jack Klein-Schiphorst, Automation Developer @Tacstone Technology

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

UiPathCommunity

FWD Group - Insurer Innovation Award 2024

The Digital Insurer

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

MS Copilot expands with MS Graph connectors

Nanddeep Nachan

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

[BuildWithAI] Introduction to Gemini.pdf

Sandro Moreira

DBX First Quarter 2024 Investor Presentation

Dropbox

Six Myths about Ontologies: The Basics of Formal Ontology

johnbeverley2021

Architecting Cloud Native Applications

WSO2

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Whatsapp Number Escorts Call girls 8617370543 Available 24x7 Mcleodganj Call Girls Service Offer Genuine VIP Model Escorts Call Girls in Your Budget. Mcleodganj Call Girls Service Provide Real Call Girls Number. Make Your Sexual Pleasure Memorable with Our Mcleodganj Call Girls at Affordable Price. Top VIP Escorts Call Girls, High Profile Independent Escorts Call Girls, Housewife Women Escorts Call Girl, College Girls Escorts Call Girls, Russian Escorts Call girls Service in Your Budget.

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Deepika Singh

Elevate Developer Efficiency & build GenAI Application with Amazon Q

Bhuvaneswari Subramani

ICT role in 21st century education and its challenges

rafiqahmad00786416

The microservices honeymoon is over. When starting a new project or revamping a legacy monolith, teams started looking for alternatives to microservices. The Modular Monolith, or 'Modulith', is an architecture that reaps the benefits of (vertical) functional decoupling without the high costs associated with separate deployments. This talk will delve into the advantages and challenges of this progressive architecture, beginning with exploring the concept of a 'module', its internal structure, public API, and inter-module communication patterns. Supported by spring-modulith, the talk provides practical guidance on addressing the main challenges of a Modultith Architecture: finding and guarding module boundaries, data decoupling, and integration module-testing. You should not miss this talk if you are a software architect or tech lead seeking practical, scalable solutions. About the author With two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Victor Rentea

Following the popularity of “Cloud Revolution: Exploring the New Wave of Serverless Spatial Data,” we’re thrilled to announce this much-anticipated encore webinar. In this sequel, we’ll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you’re building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Riding the Elephant - Hadoop 2.0

1. Simon Elliston Ball Head of Big Data - Red Gate Ventures @sireb Riding the Elephant: Hadoop 2.0 http://bit.ly/RidingElephants

2. Append only distributed file-system In the beginning… Map Reduce Java.

3. JVM Based (scala, groovy, jython, clojure) More languages Streaming (python, whatever) HDP for Windows and .NET SDK

4. Abstraction Photo: https://www.flickr.com/photos/puroticorico/ Hive, Pig Cascading Scalding

5. SQL on Hadoop Learning to share the toys HBase Solr on Hadoop Sharing HDFS…

6. Map Reduce v1 JobTracker Job Head Node TaskTracker Task (Map / Reduce) Data Node m slot 1 m slot 2 … m slot n Task r slot 1 r slot 2 … r slot n TaskTracker Task (Map / Reduce) Data Node m slot 1 m slot 2 … m slot n r slot 1 r slot 2 … r slot n TaskTracker Task (Map / Reduce) Data Node m slot 1 m slot 2 … m slot n r slot 1 r slot 2 … r slot n

7. Map Reduce v1 JobTracker Job Head Node TaskTracker Task (Map / Reduce) Data Node m slot 1 m slot 2 … m slot n MR Status r slot 1 r slot 2 … r slot n TaskTracker Task (Map / Reduce) Data Node m slot 1 m slot 2 … m slot n r slot 1 r slot 2 … r slot n TaskTracker Task (Map / Reduce) Data Node m slot 1 m slot 2 … m slot n r slot 1 r slot 2 … r slot n

8. Typical Hadoop 1.x setup HBaseProductionAdhoc

9. Typical Hadoop 1.x setup HBaseProductionAdhoc

10. YARN architecture Container Application Master Container Data Node Node Manager Container ContainerContainer Data Node Node Manager Application Master Container Free Slot Data Node Node Manager Resource Manager YARN Client

11. YARN architecture Container Application Master Container Data Node Node Manager Container ContainerContainer Data Node Node Manager Application Master Container Free Slot Data Node Node Manager Resource Manager YARN Client

12. YARN architecture Container Application Master Container Data Node Node Manager Container ContainerContainer Data Node Node Manager Application Master Container Free Slot Data Node Node Manager Resource Manager YARN Client

13. YARN architecture Container Application Master Container Data Node Node Manager Container ContainerContainer Data Node Node Manager Application Master Container Free Slot Data Node Node Manager Resource Manager YARN Client

14. Removing the choke point Advantages 60%-150% better usage Long running applications

15. Not quite… Operating system for Big Data? Security …but a framework for Big Data Apps Data Access abstraction

16. Storm on YARN A whole batch of new applications HOYA Tez (Stinger)MapReduce 2 Giraph <Insert your application here>

17. Batch applications Spinning YARNs with Spring Services Direct to YARN APIs Spring Data Hadoop abstraction

18. Streaming Why? Machine Learning Graphs Services Distributed Shell - Anything.

19. Spark A higher abstraction Hadoop based? … but can run on YARN In Memory Distributed Fault tolerant Real-time ✓ ✓ ✓ ✓ ❌ RRDs ✓

20. Mesos Wider sharing HadoopSparkAurora Mesos Framework Hardware YARN MapReduce HBase etc HDFS

21. Hadoop is more than MapReduce The new world YARN opens up new paradigms Infrastructure maturing: better sharing

22. Hadoop and beyond!

23. Thank you

24. Questions? Simon Elliston Ball Head of Big Data - Red Gate Ventures @sireb simon@simonellistonball.com http://bit.ly/RidingElephants

Riding the Elephant - Hadoop 2.0

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Riding the Elephant - Hadoop 2.0

Similar a Riding the Elephant - Hadoop 2.0 (20)

Más de Simon Elliston Ball

Más de Simon Elliston Ball (11)

Último

Último (20)

Riding the Elephant - Hadoop 2.0