Amazon Cloudsearch Session With Elsevier: re:Invent 2013

Enrich Search User Experience For Different Parts
of Your Application Using Amazon CloudSearch
Jon Handler, CloudSearch Solution Architect
November 15, 2013

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Agenda
• 
• 
• 
• 

Sourcing your documents
Retrieval and ranking
Search user interface
Performance and Scale

•  Developer example:
Peter Simpkin, Solution Architect, Elsevier

Hands-Off Operation
Document Quantity and Size

SEARCH INSTANCE

SEARCH INSTANCE

SEARCH INSTANCE

SEARCH INSTANCE

SEARCH INSTANCE

SEARCH INSTANCE

SEARCH INSTANCE

SEARCH INSTANCE

SEARCH INSTANCE

Index Partition 1
Copy 1

Search
Request
Volume and
Complexity

Index Partition 1
Copy 2

Index Partition 1
Copy n

Index Partition 2
Copy 1

Index Partition 2
Copy 2

Index Partition 2
Copy n

Index Partition n
Copy 1

Index Partition n
Copy 2

Index Partition n
Copy n

Multiple
Sources
Multiple
Functions

Mobile Experience
Cancel

Iron Man!

Iron Man

Done

Iron Man 3 (2013)!
When Tony Stark's world is torn apart by a
formidable terrorist called the Mandarin, he
starts an odyssey of rebuilding and retribution. !

Iron Man 2 (2010)!
Tony Stark has declared himself Iron Man and
installed world peace... or so he thinks. He soon
realizes that not only is there a mad man...!

Iron Man (2008)!

!

When wealthy industrialist Tony Stark is forced
to build an armored suit after a life-threatening
incident, he ultimately decides to use its
technology to ﬁght against evil. !

The Man With The Iron Fists (2012) !
On the hunt for a fabled treasure of gold, a band
of warriors, assassins, and a rogue British soldier
descend upon a village in feudal China, where a
humble blacksmith...!
Movies

Search

Social

Nearby

Account

Movies

Search

Social

Nearby

Account

Agenda
• 
• 
• 
• 


Peter Simpkin, Elsevier Oxford

Developer example

CloudSearch Documents
•  Unique identifier
•  Version
•  Fields
–  Indexed according to configuration
–  Source of matches

Developer example

Amazon RDS
DynamoDB
Amazon S3

Application Content
User actions
Help files
Movie data
Media (clips,
Theater data
images)
User reviews,
Articles
lists etc.

Developer example

Bootstrap Strategy

Amazon
CloudSearch

Amazon EC2

Amazon SQS

Source
System

Processing
Script

Amazon EC2

Queuing Batching

Document Construction
•  One source will be the master
for
each
record

determine
doc
id
and
version

create
fields

for
each
auxiliary
source

gather
additional
data

send
or
queue
the
document

Developer example

Relational DB

Addresses
Street
City

Movie

Theater

Title

Name

Description

AddressesID

Showtimes

TheaterID

ShowtimesID

Date

State

Time
State

Developer example

S3
•  Clips, images, reviews
•  Apache Tika to extract content
•  S3 Metadata for additional fields

Developer example

Developer example

Dynamo DB
DynamoDB

CloudSearch

Table

Item

Domain

Attribute
Attribute
Attribute
Attribute

Field
Field
Field
Field

Document

Developer example
Cancel

Iron Man!

Iron Man

Done

Iron Man 3 (2013)!

Iron Man 2 (2010)!

Iron Man (2008)!

!


Movies

Search

Social

Nearby

Account

Movies

Search

Social

Nearby

Account

Developer example

Searching Show Times
id title

description t_name

t_street date

time

1

Iron
Man

...

Galaxy

Main

11/11 12:30pm

2

Iron
Man

...

Galaxy

Main

11/11 1:15pm

3

Iron
Man

...

Galaxy

Main

11/11 2:45pm

4

Iron
Man

...

Galaxy

Main

11/11 6:00pm

Heterogenous Data

Developer example

Multi Domain

Developer example

Developer example

Updating CloudSearch
Update Processor

Web Server

Users

Amazon EC2

Amazon SQS

Amazon EC2

DynamoDB

Amazon RDS

Amazon
CloudSearch

Amazon S3

Section Summary
•  Multiple sources
•  Bootstrap / Update
•  Heterogeneous data

Developer example

Cancel

Iron Man!
Iron Man 3 (2013)!


Iron Man 2 (2010)!

Good Matches

Iron Man (2008)!

Movies

Search

Social

Nearby

Account

The Search Algorithm
•  Locate documents that satisfy Boolean
constraints
–  Usually intersection

•  Relevance rank those documents
–  Differentiates from databases by relevance

Developer example

Document Structure
Movie
title
description
user_rating
likes
release_date
latitude
longitude

Developer example

Configuring for Search

Developer example

•  Text fields for individual word search
–  User-generated and external text – titles, descriptions

•  Literal fields for exact matches
–  Application-generated text like facets

•  Integer fields for range searching and ranking

Searching Text

Developer example

http(s)://<endpoint>/2011-02-01/search?
•  Simple searches
–  q=<text>

•  Filtering
–  bq= (or title:'iron' (and description:'iron' description:'man'))

•  Filtering with integer ranges
–  bq=(and 'iron man' year:..2010)

•  Geo filtering
–  bq=(and 'iron man' latitude:12700..12900 longitude:5700..5800)

Search Results

Developer example

{"rank":
"-‐text_relevance",

"match-‐expr":
"(label
'iron
man')",

"hits":
{
"found":
204,
"start":
0,

"hit":
[
{
"id":
"sontsst12cf5f88b42"
},

{
"id":
"sopvopr12ab017f082"
},

{
"id":
"sorzrpw12ac468a13b"
},

]
},

...

}

Cancel

Iron Man!
Iron Man 3 (2013)!


Iron Man 2 (2010)!

Relevant Results

Iron Man (2008)!

Movies

Search

Social

Nearby

Account

Customizing Ranking
•  text_relevance and cs.text_relevance
•  Rank expressions
–  Compute a score for each document
–  &rank=<function>

•  Defined in the console
•  Defined at query-time
–  &q='iron-man'&rank-recency=text_relevance + year
&rank=recency

Developer example

Field Weighting
•  Adjust relative importance of fields
•  &rank-title=
cs.text_relevance({"weights":{"title":4.0},
"default_weight":1})

Developer example

Popularity

Developer example

•  Convert floating point to integer
•  Weight by the number of ranks
•  rank-pop=text_relevance +
log10(user-rating * number-user-ranks) * 10 +
metascore * 3

Developer example

Freshness
•  Exponential decay function

r = ce

− λt

•  &rank-decay=text_relevance +
200*Math.exp(-0.1*days_ago)

Location Sort
Iron Man

Done

!

Movies

Search

Social

Nearby

Account

Location Sort
Movie
title
description
user_rating
likes
release_date
latitude
longitude

Developer example

•  Latitude and longitude
expressed as integers
•  Denormalized for particular
theaters with locations

Location Sort
•  Cartesian distance function
(lat − latuser )2 + (lon − lonuser )2

•  &rank-geo=sqrt(pow(latitude - lat, 2) +
pow(longitude - lon), 2)
•  &rank=-geo

Developer example

Rank Expressions: Combined

Developer example

•  &rank-combined=text_relevance + 2.0 * geo +
0.5 * popularity + 0.3 * freshness
•  &rank=combined

Section Summary
•  Search API basics
•  Customizing ranking
–  Field weighting, popularity, freshness, GEO, combined

•  Rank expression comparison tool

Developer example

Simple Faceting: Document

Movie
title
description
genre

Developer example

Simple Faceting: Configuration

Developer example

Simple Faceting: Query
q=iron+man&facet=genre
{"rank":

"match-‐expr":
"(label
'star
wars')",

"hits":
{"found":
7,
"start":
0,
"hit":
[]

},

"facets":
{

"genre":
{

"constraints":
[

{"value":
"Family",
"count":
62},

{"value":
"Action/Adventure",
"count":
21},

{"value":
"Drama",
"count":
5
},

Developer example

Simple Faceting: UI
<div
class='facet'>

<ul
class='facet_list'>

<?php

$genres
=
$resultsObj-‐>facets-‐>genre-‐>constraints;

for
($i
=
0;
$i
<
count($genres);
$i++)
{

$curGenre
=
$genres[$i];
$curCount
=
$thisGenre-‐>count;

?>

<li
class='facet_item'>

<div
class='facet_name'><?=$curGenre?></div>

<div
class='facet_count'><?=$curCount?></div>

</li>

<?php
}
?>

</ul>

</div>

Developer example

Developer example

Document
Movie
title
description
oscar1
oscar2
oscar3

• 
• 
• 
• 
• 

title: Lincoln
description: ...
oscar1: Awards
oscar2: Awards/Best Actor
oscar3: Awards/Best Actor/
Daniel Day Lewis

Query

Developer example

&q=lincoln&facet=oscar1,oscar2,oscar3
{"rank":
"hits":{...},

"facets":
{

"oscar1":
{

"constraints":
[

{"value":
"Awards",
"count":
23},

{"value":
"Nominations",
"count":
124}]},

"oscar2":
{

"constraints":
[

{"value":
"Awards/Best
Actor",
"count":
6},

{"value":
"Awards/Best
Actress",
"count":
3}...]},

"oscar3":
{

"constraints":
[

{"value":
"Awards/Best
Actor/Daniel
Day
Lewis",
"count":
1},

{"value":
"Awards/Best
Actor/Denzel
Washington",
"count":
2}...]},

Drilldown
• 
• 
• 
• 

Developer example

bq=oscar1:'Awards'
bq=oscar2:'Awards/Best Actor'
bq=oscar3:'Awards/Best Actor/Daniel Day Lewis'
bq=(and 'star' oscar2:'Awards/Best Actor')

Section Summary
•  Simple faceting
•  Hierarchical faceting
•  Hierarchical data handling

Developer example

Agenda
• 
• 
• 
• 


Peter Simpkin, Elsevier Oxford

Performance Best Practices

Developer example

•  Match set size
•  Text queries perform better than integer queries
•  Complex relevance functions

Optimizing Index Size

Developer example

•  Trade off literal and uint for cost/performance
•  Result fields matter most
•  Enabling faceting increases size

Wrap Up
• 
• 
• 
• 

Sourcing documents from various locations
Building queries and ranking
UI Components for faceting
Getting the most out of your index

Agenda
• 
• 
• 
• 
• 

Elsevier Intro
Search Problem Statement
Enterprise Content Search
Hints and Tips
CloudSearch Observations

• 
• 
• 
• 

7,000+ employees in 26 countries
2,200 journals / article market
share 25%
$3B revenue
Scientific, Technical & Medical

Customers
Academic
Research
Institutions
Government &
Health
Corporate
Research
Labs
Individual
Researchers

Products

Content Challenges:
•  No central place for consumers
to discover content
• 

Is not currently possible to
search and retrieve atomic
assets

• 

Assets are not reusable across
products

Content Systems

Consumer Platforms

Empower our product development partners
Search Opportunities:
•  Create a comprehensive
inventory to discover easily
content Elsevier owns
• 

Provide access to Granular /
Modular content they want at
will

• 

Assets must be uniquely
addressable

Enterprise Content Search Engine

Enterprise Content Search eco-system

Amazon SWF

SDF metadata

E.U Corporate Data center

Amazon S3

U.S Corporate Data center

Amazon
CloudSearch

DynamoDB

Federated Content Warehouse

Product Platform Data center

Elsevier Technical Drivers & Approach
•  Fully-managed, full featured search service in
the cloud
•  Automatically scales for data & traffic
•  Easy to set up and use
•  PoC created in days
•  Search Engine as a Service
•  Pay-as-you-go pricing model

Hints & Tips
(and issn:'0022-1694'
(and type:'1.2'
(and (not action:'D')
(or (and pubstartdate:..2013176 pubenddate:2005002..)
(or (and pubstartdate:2005001
(and pubstarttime:0.. pubstarttime:..235959))
(or (and pubstartdate:2013177 pubstarttime:..235959)
(or (and pubenddate:2005001 pubendtime:0..)
(and pubenddate:2013177
(and pubendtime:..235959 pubendtime:0..)))))))))

•  Query Response Time = 5 seconds

Optimising Nested Queries
(and issn:'0022-1694' type:'1.2'
(not action:'D')
(or (and pubstartdate:..2013176 pubenddate:2005002..)
(and pubstartdate:2005001 pubstarttime:0..235959)
(and pubstartdate:2013177 pubstarttime:0..235959)
(and pubenddate:2005001 pubendtime:0..)
(and pubenddate:2013177 pubendtime:0..235959)))

•  Response Time = 2.5 seconds

Optimised Nested Query
((not action:'D')
(or (and issn:'0022-1694' and type‘1.2'
and pubstartdate:..2013176 pubenddate:2005002..)
(and issn:'0022-1694' and type‘1.2'
and pubstartdate:2005001 pubstarttime:0..235959)
and pubstartdate:2013177 pubstarttime:0..235959)
and pubenddate:2005001 pubendtime:0..)
and pubenddate:2013177 pubendtime:0..235959)))

•  Response Time = 0.17ms

CloudSearch Observations
facilitate knowledge sharing on content matters
across Elsevier’s product platforms
ability to leverage content infrastructure and
capabilities across Elsevier’s divisions
easy to integrate with existing on-premise
Content Systems
speed to market, allows developers to focus
building other core Content Strategy components
need to spend time optimising queries to
maximise performance

Please give us your feedback on this
presentation

SVC302
As a thank you, we will select prize
winners daily for completed surveys!

Thank You

Amazon Cloudsearch Session With Elsevier: re:Invent 2013

Recomendados

Recomendados

Más contenido relacionado

Similar a Amazon Cloudsearch Session With Elsevier: re:Invent 2013

Similar a Amazon Cloudsearch Session With Elsevier: re:Invent 2013 (8)

Más de Michael Bohlig

Más de Michael Bohlig (9)

Último

Último (20)

Amazon Cloudsearch Session With Elsevier: re:Invent 2013