Storm - Altamira University Presentation

•Descargar como PPTX, PDF•

0 recomendaciones•1,262 vistas

Ryan Lanman

Tecnología Empresariales

Objectives
1.Their Motivation
2.Our Motivation
3.Storm Basics
4.Demo

What They Wanted
•
•
•
•
•
•

Guaranteed data processing
Horizontal scalability
Fault-tolerance
No intermediate message brokers!
Higher level abstraction than message passing
“Just works”

Lumify Ingest
Raw
Data

Text Extraction

Entity Extraction

Text Highlighting

Location Extraction

Full Text Indexing

Issues
•
•
•
•
•

No Reducers
High DB Read/Writes
Batch-style processing
M/R Overhead
Zero Fault Tolerance

What We Really Wanted

• Distributed, Stream-type Processing
• Simple Logical DAG
• Better Fault Tolerance

Storm Ingest Workflow
Documents

Raw Data

Content
Sorter

Video

Images

Text
Extraction

Video
Frame
Splitting

Image Text
Extraction

Video
Frame Text
Extraction

Text
…

Storm Basics
What the heck’s a Topology?

Storm Cluster
Supervisor

Zookeeper

Nimbus

Supervisor

Zookeeper

Supervisor

Zookeeper

Supervisor

Supervisor

Storm Data Concepts
•
•
•
•
•

Tuples
Streams
Spouts
Bolts
Topologies

Tuples
• Single unit of data in Storm
• Examples
– Tweet
– User Activity Log Entry
– File Info

Streams

Tuple

Tuple

Tuple

Tuple

Tuple

An unbound sequence of Tuples

Tuple

Tuple

Bolts

Tuple

Process input streams to create new streams

Tuple

Examples
Spout Examples
• HDFS Filesystem Spout
• Kafka Queue Spout

Bolt Examples
• Filtering
• Aggregation
• DB Operations

Demo Topology

Twitter

Twitter
Hosebird
Spout
Sentence
Splitter

Word
Count

Accumulo

Demo Topology

Twitter

Twitter
Hosebird
Spout

Shuffle
Grouping

Field
Grouping

Sentence
Splitter

Word
Count

Accumulo

Más contenido relacionado

Similar a Storm - Altamira University Presentation

CRUD Operation of images through XMLAnshudha Maheshwari

Webinar: Ransomware - Five Reasons You’re Not As Protected As You ThinkStorage Switzerland

Velocity 2016 Speaking Session - Using Machine Learning to Determine Drivers ...SOASTA

Using machine learning to determine drivers of bounce and conversionTammy Everts

InheritedASecurityDeptAmanda Berlin

Watchtowers of the Internet - Source Boston 2012Stephan Chenette

Storm Processing InternalsThirupathi Reddy Guduru

Scalability and AvailabilityAmazon Web Services

Life in a Queue - Using Message Queue with djangoTareque Hossain

Zane lackey. security at scale. web application security in a continuous depl...Yury Chemerkin

Apache Storm InternalsHumoyun Ahmedov

Apachecon Euro 2012: Elastic, Multi-tenant Hadoop on DemandRichard McDougall

Scaling Ruby for Enterprise ApplicationsPro777

Continuous Automated Red Teaming (CART) - Bikash BaraiAllanGray11

TalosMuhammad ilyas

Chirp 2010: Scaling TwitterJohn Adams

Message handler customer deckYves Goeleven

Python for pentestersRashid feroz

Zombie Annihilation Using AWS Big Data - ABD328 - re:Invent 2017Amazon Web Services

Running Yarn at Scale DataWorks Summit

Último

Finding Java's Hidden Performance Traps @ DevoxxUK 2024Victor Rentea

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfOrbitshub

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...apidays

Artificial Intelligence Chap.5 : UncertaintyKhushali Kathiriya

MS Copilot expands with MS Graph connectorsNanddeep Nachan

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays

[BuildWithAI] Introduction to Gemini.pdfSandro Moreira

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot ModelDeepika Singh

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingEdi Saputra

Exploring Multimodal Embeddings with MilvusZilliz

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...apidays

Introduction to Multilingual Retrieval Augmented Generation (RAG)Zilliz

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93

Elevate Developer Efficiency & build GenAI Application with Amazon QBhuvaneswari Subramani

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...Zilliz

DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamUiPathCommunity

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWERMadyBayot

Apidays New York 2024 - The value of a flexible API Management solution for O...apidays

Vector Search -An Introduction in Oracle Database 23ai.pptxRemote DBA Services

Platformless Horizons for Digital AdaptabilityWSO2

Storm - Altamira University Presentation

1. Apache Storm A distributed, real-time computation system Ryan Lanman Some content borrowed from Nathan Marz’ Presentation of a similar name

2. Objectives 1.Their Motivation 2.Our Motivation 3.Storm Basics 4.Demo

3. Their Motivation How Storm Came To Be

10. What They Wanted • • • • • • Guaranteed data processing Horizontal scalability Fault-tolerance No intermediate message brokers! Higher level abstraction than message passing “Just works”

11. Our Motivation Why We Chose Storm ^

12. Lumify Ingest Raw Data Text Extraction Entity Extraction Text Highlighting Location Extraction Full Text Indexing

13. Issues • • • • • No Reducers High DB Read/Writes Batch-style processing M/R Overhead Zero Fault Tolerance

14. What We Really Wanted • Distributed, Stream-type Processing • Simple Logical DAG • Better Fault Tolerance

15. Storm Ingest Workflow Documents Raw Data Content Sorter Video Images Text Extraction Video Frame Splitting Image Text Extraction Video Frame Text Extraction Text …

16. Storm Basics What the heck’s a Topology?

17. Storm Cluster Supervisor Zookeeper Nimbus Supervisor Zookeeper Supervisor Zookeeper Supervisor Supervisor

18. Storm Cluster Supervisor Zookeeper Nimbus Supervisor Zookeeper Supervisor Zookeeper Supervisor Supervisor

19. Storm Cluster Supervisor Zookeeper Nimbus Supervisor Zookeeper Supervisor Zookeeper Supervisor Supervisor

20. Storm Cluster Supervisor Zookeeper Nimbus Supervisor Zookeeper Supervisor Zookeeper Supervisor Supervisor

21. Storm Data Concepts • • • • • Tuples Streams Spouts Bolts Topologies

22. Tuples • Single unit of data in Storm • Examples – Tweet – User Activity Log Entry – File Info

23. Streams Tuple Tuple Tuple Tuple Tuple An unbound sequence of Tuples Tuple Tuple

24. Spouts Spout Producers of Streams

25. Bolts Tuple Process input streams to create new streams Tuple

26. Examples Spout Examples • HDFS Filesystem Spout • Kafka Queue Spout Bolt Examples • Filtering • Aggregation • DB Operations

27. Topologies Spout Spout Spout