Presented by Mark Davis, CTO Kitenga - See conference video - http://www.lucidimagination.com/devzone/events/conferences/lucene-revolution-2012 Kitenga's Analyst system uses the LucidWorks Enterprise REST API in a variety of ways, including for configuring collections and managing Solr schema. As part of the Kitenga platform, the ZettaSearch Designer empowers the end-user to dynamically drag-and-drop search widgets to create a specialized search interface. For a user to effectively design search UIs that meet their needs, they need to be able to understand the available schema fields that populate a given collection. ZettaSearch Designer interrogates the Solr infrastructure using the Lucid REST API to provide an overview of the available metadata. It is then easy for the user to build rich, facetted search experiences around the metadata library indexed into the collection. In this implementation overview, I will describe the design of ZettaSearch Designer, how it interacts with big data technologies like Hadoop as part of the indexing pipeline, and how it uses the LucidWorks API to enable user discovery of the metadata needed to create novel search user interfaces on the fly.

1. Kitenga
reinventing information
Mark Davis
Founder/CTO

2. Enabling
Big Data
Search via
the Lucid
ReST API

3. Big
Data
Enormous
transactional
data
Enormous
unstructured
information
Too
big
for
databases
New
tools
are
needed

4. kilobyte (kB) 103 210 kibibyte
(KiB) 210 megabyte (MB)
106 220 mebibyte (MiB) 220
gigabyte (GB) 109 230
gibibyte (GiB) 230 terabyte
(TB) 1012 240 tebibyte (TiB)
240 petabyte (PB) 1015 250
pebibyte (PiB) 250 exabyte
(EB) 1018 260 exbibyte (EiB)
260 zettabyte (ZB) 1021 270
zebibyte (ZiB) 270 yottabyte
(YB) 1024 280 yobibyte (YiB)
280
Volume
Velocity
Variety

5. Indexing
Challenges
Complex,
varied
data
Compute-­‐intensive
metadata
generation
Schema
and
collection
management
Gather
Extract
Metadata
Index
Resources
• Crawl
• Named
• Schema
• Crack
formats
entities
definition
• Categories
• Collection
• Machine
management
learning
• Semantic
analysis

6. Initial
Query
Refine
Query
Evaluate
Relevance
• Keyword
• Analytic
• Read
KWIC
guesses
tools
• Read
• Category
• Facetted
metadata
guidance
guidance
• Read
document
Search
Experience
Challenges
Complex,
varied
data
Resource
discovery
Facetted
search
experience
management

7. The
Solution
Enable fast metadata generation:
Hadoop
Mahout
GPUs
Manage and control collections and schema:
LucidWorks Enterprise API

8. SQL
Search
RDBMS
Documents
Transactional
Data
Text
Classification
BI
Tools
Taxonomies
Ontologies

10. Machine-­‐Learning
Finite
State
Transducer
Finite
State
Transducer
Finite
State
Transducer
Parts-­‐of-­‐Speech
Tagging
Lemmatization
Tokenization

11. Resource
Integration
Facet
Browsing
Facet
Charting
Spellcheck
Autosuggest
Query
Language
Indexing
Metadata
Extraction

12. ¡ Start
to
POC
in
a
week
¡ Open
source
intelligence
problems

13. ZettaSearch
GOAL:
Be
more
competitive
Facetted Search
SOURCES:
Patents,
PR
and Analytics
announcements,
legal
documents,
relationships
whitepapers,
crawled
websites
metadata
entities
ZettaVox
data
ANALYSIS:
Extract
named
entities
and
relationships,
classify
and
label;
visually
understand
relationships
and
trends
Sources
ACTION:
Change
R&D
priorities
and
improve
marketing
approaches
13

14. ¡ Understand
IP
among
competitors
¡ Assist
legal
team
with
litigation
¡ Custom
search
experience
¡ Custom
extractors:
§ Electronic
parts
§ Memory
types
§ Flash
memory
. 5/15/12 14

15. Documents
Size
Dell
102,508
9Gb
EMC
303,678
14Gb
Huawei
11,912
890Mb
Kingston
2,534
134Mb
Lenovo
8,305
542Mb
NEC
3,900
252Mb
Nokia
174,681
22Gb
Panasonic
5,804
473Mb
Rim
181
8Mb
Sharp
USA
31,918
4.9Gb
645,421
60.2Gb
5/15/12 . 15

16. ZettaSearch
GOAL:
Discover
new
drugs,
detect
side-­‐
effects,
speed
R&D
Facetted Search
and Analytics
SOURCES:
Published
research
reports,
relationships
pathways
patents,
adverse
effects
databases,
sequences
entities
ZettaVox
genomics
and
proteomics
databases
data
ANALYSIS:
Extract
named
entities
and
relationships,
classify
and
label;
visually
discover
trends
and
relationships
ACTION:
Change
R&D
priorities
Sources
16

17. ¡ Lousy
search
(Google
Search
Appliance)
¡ Internal
regulators
can’t
find
by
accession
number
¡ Custom
extractors:
§ Accession
number
§ Ontology
of
active
ingredients
§ Drug
names
© 2012 Kitenga Proprietary 17

18. ZettaSearch
GOAL:
Build
“second
screen
Facetted Search
experiences”
and Analytics
SOURCES:
wikipedia,
IMDB,
blogs
relationships
ANALYSIS:
Extract
named
entities
and
metadata
entities
ZettaVox
data
relationships,
preserve
existing
structural
metadata
ACTION:
Enable
new
media
experiences
Sources
18

19. ¡ Crawlers
on
Hadoop
¡ Document
format
crackers
on
Hadoop
¡ Extractors
on
Hadoop
¡ Filters
on
Hadoop
¡ HTTP
documents
to
Solr
sharded
cluster
¡ Intermediary
files
remain
on
HDFS
for
reprocessing

20. ¡ Missing
piece
of
the
puzzle
¡ Addresses
the
impedance
mismatch
between
Big
Data
technologies
and
Solr
search
¡ Manage
collections
¡ Manage
schema

23. ¡ Create
collections
¡ Delete
collections
¡ Update
collection
properties
¡ Create
schema
¡ Modify
schema

24. ¡ Schema
interrogation
¡ Schema
binding
to
user
experience
¡ Facetted
search
¡ Embedded
analytics

27. ¡ Big
Data
search
and
analytics
has
many
challenges:
§ Volume
of
data
§ Variety
of
data
§ Velocity
of
data
§ Extracting
structure
from
unstructured
information
¡ Hadoop
processing
enables
each
of
these
aspects
¡ Controlling
indexing
and
search
is
enabled
by
the
Lucid
Imagination
search
API
¡ We
can
enable
complex
user
interactions
with
Big
Data
on
a
self-­‐serve
basis

28. Analyst
Browser
Enterprise
servers
Cloud
services
Tomcat
App
Server
Amazon
S3
Tomcat
Web
Services
Enterprise
ZettaVoxServices
Cloud
XML
Manager
ZettaVox
+
Author
JSON
GPU
Hadoop
RIA
Search
Indexing
Services
Services
Manager
Manager
ReST
JSON
GPU
MR
Service
Hadoop
Server
Hadoop
Server
Manager
Name
node
Job
Tracker
GPU
GPU
Hadoop
Hadoop
Task
Manager
Hadoop
Task
Manager
Quantum4D
Task
Manager
RDBMS
Entity
Mahout
Crawling
Extraction
©
2012
Kitenga
Proprietary

29. Analyst
Browser
Enterprise
servers
Search
Indexing
• Get
collection
information
• Create
new
collection
• Create
fields
• Delete
fields
• Edit
fields
ZettaVox
ReST
Author
RIA
JSON
Hadoop
Server
Hadoop
Server
Name
node
Job
Tracker
Hadoop
Hadoop
Task
Manager
Hadoop
Task
Manager
Task
Manager
Entity
Mahout
Crawling
Indexing
Extraction
©
2012
Kitenga
Proprietary

30. Questions?