Enviar búsqueda
Cargar
Solr+Hadoop = Big Data Search
•
60 recomendaciones
•
32,737 vistas
Cloudera, Inc.
Seguir
From Solr committer Mark Miller
Leer menos
Leer más
Tecnología
Denunciar
Compartir
Denunciar
Compartir
1 de 33
Descargar ahora
Descargar para leer sin conexión
Recomendados
Solr Recipes
Solr Recipes
Erik Hatcher
Rapid Prototyping with Solr
Rapid Prototyping with Solr
Erik Hatcher
Adding Search to the Hadoop Ecosystem
Adding Search to the Hadoop Ecosystem
Cloudera, Inc.
Introduction to Lucene & Solr and Usecases
Introduction to Lucene & Solr and Usecases
Rahul Jain
Integrating the Solr search engine
Integrating the Solr search engine
th0masr
Indexing Text and HTML Files with Solr
Indexing Text and HTML Files with Solr
Lucidworks (Archived)
Solr Distributed Indexing in WalmartLabs: Presented by Shengua Wan, WalmartLabs
Solr Distributed Indexing in WalmartLabs: Presented by Shengua Wan, WalmartLabs
Lucidworks
Lucene for Solr Developers
Lucene for Solr Developers
Erik Hatcher
Más contenido relacionado
La actualidad más candente
Solr 4
Solr 4
Erik Hatcher
Solr Recipes Workshop
Solr Recipes Workshop
Erik Hatcher
Large Scale ETL for Hadoop and Cloudera Search using Morphlines
Large Scale ETL for Hadoop and Cloudera Search using Morphlines
whoschek
Solr Application Development Tutorial
Solr Application Development Tutorial
Erik Hatcher
Battle of the giants: Apache Solr vs ElasticSearch
Battle of the giants: Apache Solr vs ElasticSearch
Rafał Kuć
Solr Flair
Solr Flair
Erik Hatcher
Lucene for Solr Developers
Lucene for Solr Developers
Erik Hatcher
Building Intelligent Search Applications with Apache Solr and PHP5
Building Intelligent Search Applications with Apache Solr and PHP5
israelekpo
Learning to Rank in Solr: Presented by Michael Nilsson & Diego Ceccarelli, Bl...
Learning to Rank in Solr: Presented by Michael Nilsson & Diego Ceccarelli, Bl...
Lucidworks
Apache Solr/Lucene Internals by Anatoliy Sokolenko
Apache Solr/Lucene Internals by Anatoliy Sokolenko
Provectus
Data Engineering with Solr and Spark
Data Engineering with Solr and Spark
Lucidworks
Beyond full-text searches with Lucene and Solr
Beyond full-text searches with Lucene and Solr
Bertrand Delacretaz
Intro to Apache Lucene and Solr
Intro to Apache Lucene and Solr
Grant Ingersoll
Data Science with Solr and Spark
Data Science with Solr and Spark
Lucidworks
Introduction to Apache Lucene/Solr
Introduction to Apache Lucene/Solr
Rahul Jain
New-Age Search through Apache Solr
New-Age Search through Apache Solr
Edureka!
Integrating Spark and Solr-(Timothy Potter, Lucidworks)
Integrating Spark and Solr-(Timothy Potter, Lucidworks)
Spark Summit
State-of-the-Art Drupal Search with Apache Solr
State-of-the-Art Drupal Search with Apache Solr
guest432cd6
Apache Solr crash course
Apache Solr crash course
Tommaso Teofili
Introduction to Apache Solr
Introduction to Apache Solr
Andy Jackson
La actualidad más candente
(20)
Solr 4
Solr 4
Solr Recipes Workshop
Solr Recipes Workshop
Large Scale ETL for Hadoop and Cloudera Search using Morphlines
Large Scale ETL for Hadoop and Cloudera Search using Morphlines
Solr Application Development Tutorial
Solr Application Development Tutorial
Battle of the giants: Apache Solr vs ElasticSearch
Battle of the giants: Apache Solr vs ElasticSearch
Solr Flair
Solr Flair
Lucene for Solr Developers
Lucene for Solr Developers
Building Intelligent Search Applications with Apache Solr and PHP5
Building Intelligent Search Applications with Apache Solr and PHP5
Learning to Rank in Solr: Presented by Michael Nilsson & Diego Ceccarelli, Bl...
Learning to Rank in Solr: Presented by Michael Nilsson & Diego Ceccarelli, Bl...
Apache Solr/Lucene Internals by Anatoliy Sokolenko
Apache Solr/Lucene Internals by Anatoliy Sokolenko
Data Engineering with Solr and Spark
Data Engineering with Solr and Spark
Beyond full-text searches with Lucene and Solr
Beyond full-text searches with Lucene and Solr
Intro to Apache Lucene and Solr
Intro to Apache Lucene and Solr
Data Science with Solr and Spark
Data Science with Solr and Spark
Introduction to Apache Lucene/Solr
Introduction to Apache Lucene/Solr
New-Age Search through Apache Solr
New-Age Search through Apache Solr
Integrating Spark and Solr-(Timothy Potter, Lucidworks)
Integrating Spark and Solr-(Timothy Potter, Lucidworks)
State-of-the-Art Drupal Search with Apache Solr
State-of-the-Art Drupal Search with Apache Solr
Apache Solr crash course
Apache Solr crash course
Introduction to Apache Solr
Introduction to Apache Solr
Similar a Solr+Hadoop = Big Data Search
Using Apache Spark to Tune Spark with Shivnath Babu and Adrian Popescu
Using Apache Spark to Tune Spark with Shivnath Babu and Adrian Popescu
Databricks
Ramping up your Devops Fu for Big Data developers
Ramping up your Devops Fu for Big Data developers
François Garillot
OC Big Data Monthly Meetup #5 - Session 1 - Altiscale
OC Big Data Monthly Meetup #5 - Session 1 - Altiscale
Big Data Joe™ Rossi
Guidance, Code and Education: ScalaCenter and the Scala Community, Heather Mi...
Guidance, Code and Education: ScalaCenter and the Scala Community, Heather Mi...
OW2
Swift as a scripting language iOSDevUK14 Lightning talk
Swift as a scripting language iOSDevUK14 Lightning talk
Diego Freniche Brito
Data Science
Data Science
Ahmet Bulut
Advanced full text searching techniques using Lucene
Advanced full text searching techniques using Lucene
Asad Abbas
php fundamental
php fundamental
zalatarunk
Php introduction with history of php
Php introduction with history of php
pooja bhandari
php
php
Ramki Kv
Small wins in a small time with Apache Solr
Small wins in a small time with Apache Solr
Sourcesense
Get involved with the Apache Software Foundation
Get involved with the Apache Software Foundation
Shalin Shekhar Mangar
Ncku csie talk about Spark
Ncku csie talk about Spark
Giivee The
REST APIs in Laravel 101
REST APIs in Laravel 101
Samantha Geitz
Drupal & Summon: Keeping Article Discovery in the Library
Drupal & Summon: Keeping Article Discovery in the Library
Ken Varnum
Drupal & Apache Solr
Drupal & Apache Solr
Andrei Savu
REST-API introduction for developers
REST-API introduction for developers
Patrick Savalle
Build and maintain large ruby applications Ruby Conf Australia 2016
Build and maintain large ruby applications Ruby Conf Australia 2016
Enrico Teotti
The First Class Integration of Solr with Hadoop
The First Class Integration of Solr with Hadoop
lucenerevolution
Elasticsearch JVM-MX Meetup April 2016
Elasticsearch JVM-MX Meetup April 2016
Domingo Suarez Torres
Similar a Solr+Hadoop = Big Data Search
(20)
Using Apache Spark to Tune Spark with Shivnath Babu and Adrian Popescu
Using Apache Spark to Tune Spark with Shivnath Babu and Adrian Popescu
Ramping up your Devops Fu for Big Data developers
Ramping up your Devops Fu for Big Data developers
OC Big Data Monthly Meetup #5 - Session 1 - Altiscale
OC Big Data Monthly Meetup #5 - Session 1 - Altiscale
Guidance, Code and Education: ScalaCenter and the Scala Community, Heather Mi...
Guidance, Code and Education: ScalaCenter and the Scala Community, Heather Mi...
Swift as a scripting language iOSDevUK14 Lightning talk
Swift as a scripting language iOSDevUK14 Lightning talk
Data Science
Data Science
Advanced full text searching techniques using Lucene
Advanced full text searching techniques using Lucene
php fundamental
php fundamental
Php introduction with history of php
Php introduction with history of php
php
php
Small wins in a small time with Apache Solr
Small wins in a small time with Apache Solr
Get involved with the Apache Software Foundation
Get involved with the Apache Software Foundation
Ncku csie talk about Spark
Ncku csie talk about Spark
REST APIs in Laravel 101
REST APIs in Laravel 101
Drupal & Summon: Keeping Article Discovery in the Library
Drupal & Summon: Keeping Article Discovery in the Library
Drupal & Apache Solr
Drupal & Apache Solr
REST-API introduction for developers
REST-API introduction for developers
Build and maintain large ruby applications Ruby Conf Australia 2016
Build and maintain large ruby applications Ruby Conf Australia 2016
The First Class Integration of Solr with Hadoop
The First Class Integration of Solr with Hadoop
Elasticsearch JVM-MX Meetup April 2016
Elasticsearch JVM-MX Meetup April 2016
Más de Cloudera, Inc.
Partner Briefing_January 25 (FINAL).pptx
Partner Briefing_January 25 (FINAL).pptx
Cloudera, Inc.
Cloudera Data Impact Awards 2021 - Finalists
Cloudera Data Impact Awards 2021 - Finalists
Cloudera, Inc.
2020 Cloudera Data Impact Awards Finalists
2020 Cloudera Data Impact Awards Finalists
Cloudera, Inc.
Edc event vienna presentation 1 oct 2019
Edc event vienna presentation 1 oct 2019
Cloudera, Inc.
Machine Learning with Limited Labeled Data 4/3/19
Machine Learning with Limited Labeled Data 4/3/19
Cloudera, Inc.
Data Driven With the Cloudera Modern Data Warehouse 3.19.19
Data Driven With the Cloudera Modern Data Warehouse 3.19.19
Cloudera, Inc.
Introducing Cloudera DataFlow (CDF) 2.13.19
Introducing Cloudera DataFlow (CDF) 2.13.19
Cloudera, Inc.
Introducing Cloudera Data Science Workbench for HDP 2.12.19
Introducing Cloudera Data Science Workbench for HDP 2.12.19
Cloudera, Inc.
Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19
Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19
Cloudera, Inc.
Leveraging the cloud for analytics and machine learning 1.29.19
Leveraging the cloud for analytics and machine learning 1.29.19
Cloudera, Inc.
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19
Cloudera, Inc.
Leveraging the Cloud for Big Data Analytics 12.11.18
Leveraging the Cloud for Big Data Analytics 12.11.18
Cloudera, Inc.
Modern Data Warehouse Fundamentals Part 3
Modern Data Warehouse Fundamentals Part 3
Cloudera, Inc.
Modern Data Warehouse Fundamentals Part 2
Modern Data Warehouse Fundamentals Part 2
Cloudera, Inc.
Modern Data Warehouse Fundamentals Part 1
Modern Data Warehouse Fundamentals Part 1
Cloudera, Inc.
Extending Cloudera SDX beyond the Platform
Extending Cloudera SDX beyond the Platform
Cloudera, Inc.
Federated Learning: ML with Privacy on the Edge 11.15.18
Federated Learning: ML with Privacy on the Edge 11.15.18
Cloudera, Inc.
Analyst Webinar: Doing a 180 on Customer 360
Analyst Webinar: Doing a 180 on Customer 360
Cloudera, Inc.
Build a modern platform for anti-money laundering 9.19.18
Build a modern platform for anti-money laundering 9.19.18
Cloudera, Inc.
Introducing the data science sandbox as a service 8.30.18
Introducing the data science sandbox as a service 8.30.18
Cloudera, Inc.
Más de Cloudera, Inc.
(20)
Partner Briefing_January 25 (FINAL).pptx
Partner Briefing_January 25 (FINAL).pptx
Cloudera Data Impact Awards 2021 - Finalists
Cloudera Data Impact Awards 2021 - Finalists
2020 Cloudera Data Impact Awards Finalists
2020 Cloudera Data Impact Awards Finalists
Edc event vienna presentation 1 oct 2019
Edc event vienna presentation 1 oct 2019
Machine Learning with Limited Labeled Data 4/3/19
Machine Learning with Limited Labeled Data 4/3/19
Data Driven With the Cloudera Modern Data Warehouse 3.19.19
Data Driven With the Cloudera Modern Data Warehouse 3.19.19
Introducing Cloudera DataFlow (CDF) 2.13.19
Introducing Cloudera DataFlow (CDF) 2.13.19
Introducing Cloudera Data Science Workbench for HDP 2.12.19
Introducing Cloudera Data Science Workbench for HDP 2.12.19
Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19
Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19
Leveraging the cloud for analytics and machine learning 1.29.19
Leveraging the cloud for analytics and machine learning 1.29.19
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19
Leveraging the Cloud for Big Data Analytics 12.11.18
Leveraging the Cloud for Big Data Analytics 12.11.18
Modern Data Warehouse Fundamentals Part 3
Modern Data Warehouse Fundamentals Part 3
Modern Data Warehouse Fundamentals Part 2
Modern Data Warehouse Fundamentals Part 2
Modern Data Warehouse Fundamentals Part 1
Modern Data Warehouse Fundamentals Part 1
Extending Cloudera SDX beyond the Platform
Extending Cloudera SDX beyond the Platform
Federated Learning: ML with Privacy on the Edge 11.15.18
Federated Learning: ML with Privacy on the Edge 11.15.18
Analyst Webinar: Doing a 180 on Customer 360
Analyst Webinar: Doing a 180 on Customer 360
Build a modern platform for anti-money laundering 9.19.18
Build a modern platform for anti-money laundering 9.19.18
Introducing the data science sandbox as a service 8.30.18
Introducing the data science sandbox as a service 8.30.18
Último
201610817 - edge part1
201610817 - edge part1
Jamie (Taka) Wang
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
bruanjhuli
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Safe Software
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
DianaGray10
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
Jamie (Taka) Wang
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
Matsuo Lab
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
dgelyza
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Aijun Zhang
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
Precisely
Designing A Time bound resource download URL
Designing A Time bound resource download URL
Runcy Oommen
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
Matt Ray
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
Bachir Benyammi
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
Adtran
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
Daniel Santiago Silva Capera
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
Eric D. Schabell
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
DianaGray10
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
Liveplex
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
IES VE
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
D Cloud Solutions
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UbiTrack UK
Último
(20)
201610817 - edge part1
201610817 - edge part1
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
Designing A Time bound resource download URL
Designing A Time bound resource download URL
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
Solr+Hadoop = Big Data Search
1.
1 Solr%+%Hadoop%=%Big%Data%Search Mark%Miller
2.
2 Who$Am$I? Cloudera$employee,$Lucene/Solr$committer,$Lucene$PMC$member, Apache$member First$job$out$of$college$was$in$the$Newspaper$archiving$business. First$full$time$employee$at$LucidWorks$G$a$startup$around Lucene/Solr. Spent$a$couple$years$as$“Core”$engineering$manager,$reporting$to the$VP$of$engineering.
3.
3 Very%fast%and%feature%rich%‘core’%search%engine%library.% Compact%and%powerful,%Lucene%is%an%extremely%popular%full>text search%library. Provides%low%level%API’s%for%analyzing,%indexing,%and%searching text,%along%with%a%myriad%of%related%features. Just%the%core%>%either%you%write%the%‘glue’%or%use%a%higher%level search%engine%built%with%Lucene.
4.
4 Solr%(pronounced%"solar")%is%an%open%source%enterprise%search platform%from%the%Apache%Lucene%project.%Its%major%features include%full;text%search,%hit%highlighting,%faceted%search,%dynamic clustering,%database%integration,%and%rich%document%(e.g.,%Word, PDF)%handling.%Providing%distributed%search%and%index replication,%Solr%is%highly%scalable.%Solr%is%the%most%popular enterprise%search%engine. ;%Wikipedia
5.
5 Search'on'Hadoop'History 'Katta 'Blur 'SolBase 'HBASE73529 'SOLR71301 'SOLR71045 'Ad7Hoc • • • • • • •
6.
6 Family'Tree ...
7.
7 Strengthen(the(Family(Bonds No(need(to(build(something(radically(new(8(we(have(the pieces(we(need. Focus(on(integration(points. Create(high(quality,(first(class(integrations(and(contribute the(work(to(the(projects(involved. Focus(on(integration(and(quality(first(8(then(performance and(scale. • • • •
8.
8 SolrCloud
9.
9 Solr%Integration Read%and%Write%directly%to%HDFS First%Class%Custom%Directory%Support%in%Solr Support%Solr%Replication%on%HDFS Other%improvements%around%usability%and%configuration • • • •
10.
10 Read%and%Write%directly%to%HDFS Lucene%did%not%historically%support%append%only%file%system “Flexible%Indexing”%brought%around%support%for%append%only filesystem%support Lucene%support%append%only%filesystem%by%default%since%4.2 • • •
11.
11 Lucene&Directory&Abstraction It’s&how&Lucene&interacts&with&index&files. Solr&uses&the&Lucene&library&and&offers&DirectoryFactory Class&Directory&{ &&&&&&&&listAll(); &&&&&&&&createOutput(file,&context); &&&&&&&&openInput(file,&context); &&&&&&&&deleteFile(file); &&&&&&&&makeLock(file); &&&&&&&&clearLock(file); &&&&&&&&…
12.
12 Putting'the'Index'in'HDFS Solr'relies'on'the'filesystem'cache'to'operate'at'full'speed. HDFS'not'known'for'it’s'random'access'speed. Apache'Blur'has'already'solved'this'with'an'HdfsDirectory'that works'on'top'of'a'BlockDirectory. The'“block'cache”'caches'the'hot'blocks'of'the'index'off'heap (direct'byte'array)'and'takes'the'place'of'the'filesystem'cache. We'contributed'back'optional'‘write’'caching. • • • • •
13.
13 Putting'the'TransactionLog'in'HDFS HdfsUpdateLog'added'9'extends'UpdateLog Triggered'by'setting'the'UpdateLog'dataDir'to'something'that starts'with'hdfs:/'9'no'additional'configuration'necessary. Same'extensive'testing'as'used'on'UpdateLog • • •
14.
14 Running&Solr&on&HDFS Set&DirectoryFactory&to&HdfsDirectoryFactory&and&set&the&dataDir&to&a location&in&hdfs. Set&LockType&to&‘hdfs’ Use&an&UpdateLog&dataDir&location&that&begins&with&‘hdfs:/’ Or&java&FDsolr.directoryFactory=HdfsDirectoryFactory& &&&&&&&&&&&&&&&FDsolr.lockType=solr.HdfsLockFactory &&&&&&&&&&&&&&&FDsolr.updatelog=hdfs://host:port/path&Fjar&start.jar • • • •
15.
15 Solr%Replication%on%HDFS While%Solr%has%exposed%a%plug8able%DirectoryFactory%for%a%long time%now,%it%was%really%quite%limited. Most%glaring,%only%a%local%file%system%based%Directory%would work%with%replication. There%where%also%other%more%minor%areas%that%relied%on%a%local filesystem%Directory%implementation. • • •
16.
16 Future&Solr&Replication&on&HDFS Take&advantage&of&“distributed&filesystem”&and&allow&for something&similar&to&HBase®ions. If&a&node&goes&down,&the&data&is&still&available&in&HDFS&D&allow for&that&index&to&be&automatically&served&by&a&node&that&is&still&up if&it&has&the&capacity. • • Solr&Node Solr&Node Solr&Node HDFS
17.
17 MR#Index#Building Scalable#index#creation#via#map8reduce Many#initial#‘homegrown’#implementations#sent#documents#from#reducer#to SolrCloud#over#http To#really#scale,#you#want#the#reducers#to#create#the#indexes#in#HDFS#and then#load#them#up#with#Solr The#ideal#impl#will#allow#using#as#many#reducers#as#are#available#in#your hadoop#cluster,#and#then#merge#the#indexes#down#to#the#correct#number#of ‘shards’ • • • •
18.
18 MR#Index#Building Mapper: Parse#input#into indexable#document Mapper: Parse#input#into indexable#document Mapper: Parse#input#into indexable#document Index#shard 1 Index#shard 2 Arbitrary#reducing#steps#of#indexing#and#merging End@Reducer#(shard#1): Index#document End@Reducer#(shard#2): Index#document
19.
19 SolrCloud(Aware Can(‘inspect’(ZooKeeper(to(learn(about(Solr(cluster. What(URL’s(to(GoLive(to. The(Schema(to(use(when(building(indexes. Match(hash(E>(shard(assignments(of(a(Solr(cluster. • • • •
20.
20 GoLive After+building+your+indexes+with+map:reduce,+how+do+you deploy+them+to+your+Solr+cluster? We+want+it+to+be+easy+:+so+we+built+the+GoLive+option. GoLive+allows+you+to+easily+merge+the+indexes+you+have created+atomically+into+a+live+running+Solr+cluster. Paired+with+the+ZooKeeper+Aware+ability,+this+allows+you+to simply+point+your+map:reduce+job+to+your+Solr+cluster+and+it+will automatically+discover+how+many+shards+to+build+and+what locations+to+deliver+the+final+indexes+to+in+HDFS. • • • •
21.
21 Flume&Solr&Sync Flume&is&a&distributed,&reliable,&and&available&service&for efficiently&collecting,&aggregating,&and&moving&large&amounts of&log&data.&It&has&a&simple&and&flexible&architecture&based&on streaming&data&flows.&It&is&robust&and&fault&tolerant&with tunable&reliability&mechanisms&and&many&failover&and recovery&mechanisms.&It&uses&a&simple&extensible&data&model that&allows&for&online&analytic&application. =&Apache&Flume&Website
22.
Other Logs 22 Flume.Solr.Sync HDFS Flume Agent Flume Agent Solr
23.
23 SolrCloud(Aware Can(‘inspect’(ZooKeeper(to(learn(about(Solr(cluster. What(URL’s(to(send(data(to. The(Schema(for(the(collection(being(indexed(to. • • •
24.
24 HBase&Integration Collaboration&between&NGData&&&Cloudera NGData&are&creators&of&the&Lily&data&management&platform Lily&HBase&Indexer Service&which&acts&as&a&HBase&replication&listener HBase&replication&features,&such&as&filtering,&supported Replication&updates&trigger&indexing&of&updates&(rows) Integrates&Morphlines&library&for&ETL&of&rows AL2&licensed&on&github&https://github.com/ngdata • • • • • • • •
25.
25 HBase&Integration HDFS HBase interactive&load Indexer(s) Triggers&on&updates Solr&server Solr&server Solr&server Solr&server Solr&server
26.
26 Morphlines A,morphline,is,a,configuration,file,that,allows,you,to,define,ETL transformation,pipelines Extract,content,from,input,files,,transform,content,,load,content,(eg to,Solr) Uses,Tika,to,extract,content,from,a,large,variety,of,input,documents Part,of,the,CDK,(Cloudera,Development,Kit) • • • •
27.
27 Morphlines syslog Flume Agent Solr3Sink Command:3readLine Command:3grok Command:3loadSolr Solr 3Open3Source3framework3for3simple3ETL 3Ships3as3part3Cloudera3Developer3Kit3(CDK) 3It’s3a3Java3library 3AL23licensed3on3github https://github.com/cloudera/cdk 3Similar3to3Unix3pipelines 3Configuration3over3coding 3Supports3common3Hadoop3formats Avro Sequence3file Text • • • • • • •
28.
28 Morphlines +Integrate+with+and+load+into+Apache+Solr +Flexible+log+file+analysis +Single:line+record,+multi:line+records,+CSV+files+ +Regex+based+pattern+matching+and+extraction+ +Integration+with+Avro+ +Integration+with+Apache+Hadoop+Sequence+Files +Integration+with+SolrCell+and+all+Apache+Tika+parsers+ +Auto:detection+of+MIME+types+from+binary+data+using+++Apache+Tika • • • • • • • •
29.
29 Morphlines +Scripting+support+for+dynamic+java+code+ +Operations+on+fields+for+assignment+and+comparison +Operations+on+fields+with+list+and+set+semantics+ +if:then:else+conditionals+ +A+small+rules+engine+(tryRules) +String+and+timestamp+conversions+ +slf4j+logging +Yammer+metrics+and+counters+ +Decompression+and+unpacking+of+arbitrarily+nested+container+file formats • • • • • • • • •
30.
30 Morphlines+Example+Config morphlines+:+[ +{ +++id+:+morphline1 +++importCommands+:+["com.cloudera.**",+"org.apache.solr.**"] +++commands+:+[ +++++{+readLine+{}+}++++++++++++++++++++ +++++{+ +++++++grok+{+ +++++++++dictionaryFiles+:+[/tmp/grokFdictionaries]+++++++++++++++++++++++++++++++ +++++++++expressions+:+{+ +++++++++++message+:+"""<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp}+% {SYSLOGHOST:syslog_hostname}+%{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?:+% {GREEDYDATA:syslog_message}""" +++++++++} +++++++} +++++} +++++{+loadSolr+{}+}+++++ ++++] +} ] Example(Input <164>Feb++4+10:46:14+syslog+sshd[607]:+listening+on+0.0.0.0+port+22 Output(Record syslog_pri:164 syslog_timestamp:Feb++4+10:46:14 syslog_hostname:syslog syslog_program:sshd syslog_pid:607 syslog_message:listening+on+0.0.0.0+port+22.
31.
31 Hue$Integration Hue Simple$UI Navigated,$faceted$drill$down Customizable$display Full$text$search,$standard$Solr API$and$query$language • • • •
32.
32 Cloudera)Search https://ccp.cloudera.com/display/SUPPORT/Downloads Or)Google “cloudera=search=download”
33.
Mark%Miller,%Cloudera @heismark
Descargar ahora