Más contenido relacionado La actualidad más candente (17) Similar a MapR and Lucidworks Joint Webinar 2012 (20) Más de MapR Technologies (20) MapR and Lucidworks Joint Webinar 20121. 1©MapR Technologies - Confidential
Crowd Sourcing Reflected
Intelligence Using Search and Big
Data
Ted Dunning
Grant Ingersoll
2. 2©MapR Technologies - Confidential
Grant’s Background
Co-founder:
– LucidWorks – Chief Scientist
– Apache Mahout
Long time Lucene/Solr committer
Author: Taming Text
Background in IR and NLP
– Built CLIR, QA and a variety of other search-based apps
3. 3©MapR Technologies - Confidential
Ted’s Background
Academia, Startups
– Aptex, MusicMatch, ID Analytics, Veoh
– Big data since before big
Open source
– since the dark ages before the internet
– Mahout, Zookeeper, Drill
– bought the beer at first HUG
MapR
– Chief Application Architect
Founding member of Apache Drill
4. 4©MapR Technologies - Confidential
Agenda
Intro
Search Evolution and Search Revolution
Reflected Intelligence Use Cases
Building a Next Generation Search and Discovery Platform
– MapR
– LucidWorks
1+1=3
5. 5©MapR Technologies - Confidential
Search is Dead, Long Live Search
Search is a system building block
– text is only a part of the story
If the algorithms fit,
use them!
Embrace fuzziness!
Scoring features are everywhere
Content
User
Interaction
Access
Content
Relationships
6. 6©MapR Technologies - Confidential
Search (R)evolution
Search use leads to search abuse
– denormalization frees your mind
– scoring is just a sparse matrix multiply
Lucene/Solr evolution
– non free text usages abound
– many DB-like features
– noSQL before NoSQL was cool
– flexible indexing
– finite State Transducers FTW!
Scale
“This ain’t your father’s relevance anymore”
7. 7©MapR Technologies - Confidential
Add (Lots of) Water
Large-scale analysis is key to reflected intelligence
– correlation analysis
• based on queries, clicks, mouse tracks,
even explicit feedback
• produce clusters, trends, topics, SIP’s
– start with engineered knowledge,
refine with user feedback
Large-scale discovery features
encourage experimentation
Always test, always enrich!
Search
DiscoveryAnalytics
8. 8©MapR Technologies - Confidential
Social Media Analysis in Telecom
Correlate mobile traffic analysis with social media analysis
– events cause traffic micro-bursts
– participants tweet the events ahead of time
Deploy operations faster to predict outages and better handle
emergency situations
– high cost bandwidth augmentation can be marshaled as the traffic appears
– anticipation beats reaction
9. 9©MapR Technologies - Confidential
Provenance is 80% of value
Analysis of social media to determine advertising reach and
response
In one case the same untargeted advertising was worth 5x if sold
with supporting data.
10. 10©MapR Technologies - Confidential
Claims Analysis
Goal
– Insurance claims processing and analysis
– fraud analysis
Method
– Combine free text search with metadata analysis to identify high risk
activities across the country
– Integrate with corporate workflows to detect and fix outliers in customer
relations
Results
– Questions that took 24-48 hours now take seconds to answer
11. 11©MapR Technologies - Confidential
Virginia Tech - Help the World
Grab data around crisis
Search immediately
Large-scale analysis enriches data to find
ways to improve responses and
understanding
http://www.ctrnet.net
12. 12©MapR Technologies - Confidential
Bright Planet - Catch the Bad Guys
Online Drug Counterfeit detection
Identify commonly used language indicating counterfeits
– you know it when you see it
– and you know you have seen it
Feed to analyst via search-driven application
– enrich based on analysts feedback
13. 13©MapR Technologies - Confidential
Veoh - Cross Recommendations
Cross recommendation as search
– with search used to build cross recommendation!
Recommend content to people who exhibit certain behaviors
(clicks, query terms, other)
(Ab)use of a search engine
– but not as a search engine for content
– more like a search engine for behavior
14. 14©MapR Technologies - Confidential
What Platform Do You Need?
Fast, efficient, scalable search
– bulk and near real-time indexing
– handle billions of records with sub-second search and faceting
Large scale, cost effective storage and processing capabilities
NLP and machine learning tools that scale to enhance discovery
and analysis
Integrated log analysis workflows that close the loop between the
raw data and user interactions
15. 15©MapR Technologies - Confidential
Shards
1
2
3 N
Search View
•Documents
•Users
•Logs
Document
Store
Analytic
Services
•View into
numeric/histo
ric data
•Classification
•Recommendation
Personalization &
Machine Learning
Services
Classification Models
In memory
Replicated
Multi-tenant
Discovery &
Enrichment
Clustering, classificat
ion, NLP, topic
identification, searc
h log analysis, user
behavior
Content Acquisition
ETL, batch or near
real-time
Access APIs
Data
• LucidWorks Search
connectors
• Push
Reference Architecture
16. 16©MapR Technologies - Confidential
MapR
MapR provides the technology leading Hadoop distribution
– full eco-system distribution
– integrated data platform
– complete solution for data integrity
MapR clusters also provide tight integration with search
technologies like LucidWorks
– integration is key for effective ops
17. 17©MapR Technologies - Confidential
LucidWorks
LucidWorks provides the leading packaging of Apache Lucene and
Solr
– build your own, we support
– founded by the most prominent Lucene/Solr experts
LucidWorks Search
– “Solr++”
• UI, REST API, MapR connectors, relevance tools, much more
LucidWorks Big Data
– Big Data as a Service
– Integrated LucidWorks Search, Hadoop, machine learning with prebuilt
workflows for many of these tasks
18. 18©MapR Technologies - Confidential
LucidWorks Big Data Architecture
Big Data Operating System
• Administration
• Provisioning
• Monitoring
• Configuration
• Service Management
• Data Management
• Security
System
Management
Uniform ReST API
Search – Discovery – Analytics
• LucidWorks Search
• Machine Learning
(classification, clustering, recommendations)
• Natural Language Processing
• SQL (Hive) Interface
• Data Workflows (ETL, log analysis, common metrics)
• Extensible
• Enterprise
Repository
• Social Media
• Databases
• HDFS
• Cloud (S3)
• Push
Content
Acquisition
Hadoop/HBase Search
Indexes
Search
Logs
19. 19©MapR Technologies - Confidential
Easy Wins
Analyze logs from application stored in MapR
Seamlessly store search indexes in MapR
– and feed to Pig, Mahout and others
– use mirrors + NFS to directly deploy indexes
Snapshots make backups a snap
LucidWorks 2.5 (2013 Q1) easily connects with MapR
21. 21©MapR Technologies - Confidential
Learn More
More information
http://www.mapr.com/company/events/lucidworks-12-13-2012
Vote for this topic for Hadoop Summit EU:
http://bit.ly/128tLQe
Talk to Ted
@ted_dunning
tdunning@maprtech.com
Talk to Grant
@gsingers
MapR and Lucid Works
http://www.mapr.com
http://www.lucidworks.com
Notas del editor TED: We can tighten or loosen as necessary. TED: I think that the agenda needs to go here because it otherwise breaks up some key flow TED: This is a money slide where people should say “Wow man”. They shouldn’t understand the implications of this, but they should be very, very aware that something big just slide into the room.Tech Building Block: Not just textNot just users + queriesEmbrace Fuzziness: Esp. in Big Data, it is the only way you are going to survive.TED: I think that this should make the case for advanced that is still search at its heart. The idea that search can be radically changed should be on the next slide. Search Abuse Can discuss how I started just doing free text, but then a curious thing happened, started to see people using the engine for things like: key/value, denormalized DBs, browsing engines, plagiarism detection, teaching languages, record linkage and much, much moreSearch has added more DB features over the yearsTED: We need to introduce the idea of *REVOLUTION* somewhere in here. All that revolution is good, but what the heck does this have to do w/ Big Data? GSI: needs a bit more meat Service-Oriented ArchitectureStatelessFailover/Fault TolerantLightweight Coordination and MessagingSmart about UpdatesDocument store isDistributedScalableAnalysisBatchNear Real-Time