Multi faceted responsive search, autocomplete, feeds engine & logging

Multi-faceted responsive search,
autocomplete, feeds engine and logging
Remi Mikalsen
Search Engineer, utdanning.no

Multi-facetedMulti-faceted
responsive search,responsive search,
autocomplete,autocomplete,
feeds engine andfeeds engine and
logginglogging

Introduction
Remi Mikalsen
Search engineer, utdanning.no
«Utdanning.no is the official Norwegian national education and
career portal, and includes an overview of education in Norway
and more than 500 career descriptions» - utdanning.no
« [...] Our main goals are to improve the quality of education and
to improve learning outcomes and learning for children, pupils
and students thourgh use of ICT in education» - iktsenteret.no

utdanning.no
Drupal 7 & Solr 3.6
~3 million visitors / year
~12,000 documents
~18,000,000 terms
~260 fields
~1 QPS (~9M searches / year)
~8 ms latency

Universities, colleges and
community colleges
~30 different endpoints
~3500 documents
Folk high schools
(non-academic)
1 national endpoint
~650 documents
Secondary schools
1 national endpoint
~1100 documents
Higher education admissions
(Samordna opptak)
1 national endpoint
~1500 documents
Secondary schools
metadata (Grep)
1 national endpoint
~650 documents
Higher education
metadata (NUS)
1 national endpoint
~3500 documents
Transform &
normalize
Drupal 7
ER-model
Added value
Editorial staff
Professions, interviews,
education summaries, etc.
~1500 documents
Professions metadata
(STYRK)
2 national endpoints
~1000 documents
Fetch data
Solr 3.6
De-normalized
Searchable

Drupal 7
Apache Solr Search
Integration 7.x-1.1
Customized
business logic
Solr 3.6
Pros
Basic Drupal integration
Track document changes
Some facet support
Easily extendable
Cons
Lacks deep introspecting
Little de-normalization
Hacky hierarchies (Drupal)
Note
Custom config files!
schema.xml
(mainly dynamic fields)
solrconfig.xml
(mainly a drupal request handler)
We added
Deep introspecting
Data de-normalization
Solid hierarchy support
Pivot facet support
Atomization
Manual partial re-index
schema.xml
- field types (auto)
- various copy fields
- better spell
- bucket fields
- autocomplete

Organization
(school)
Study programStudy program
Study program
Organization
(school)
+
all its
Study programs
Drupal DB Solr documents
Study program
+
Organization

<doc>
<str name="id">394353</str>
<bool name="bs_mainsearch">true</bool>
<str name="bundle">org</str>
<str name="bundle_name">Organization</str>
<str name="label">ACME University</str>
<str name="atom">[XML]</str>
<arr name="related_nodes">
<str>ACME Rocket Science</str>
<str>Study program 2</str>
<str>Study program N</str>
</arr>
<arr name="sm_geography_hierarchy">
<str>1>California</str>
<str>2>California>San Diego</str>
<str>3>California>San Diego>Gaslamp Quarter</str>
</arr>
<str name="ss_menu_1">orgmenu</str>
<str name="ss_menu_2">org</str>
</doc>

<doc>
<str name="id">394354</str>
<bool name="bs_mainsearch">true</bool>
<str name="bundle">he</str>
<str name="bundle_name">Higher Education</str>
<str name="label">ACME Rocket Science</str>
<str name="atom">[XML]</str>
<arr name="sm_offered_by">
<str>ACME University</str>
</arr>
<arr name="sm_study_area">
<str>Engineering</str>
<str>Science</str>
</arr>
<long name="its_field_semesters">8</long>
<str name="ss_menu_1">edumenu</str>
<str name="ss_menu_2">he</str>
</doc>

Searching
- Site search
- Embedded search
- Feeds engine

Our goal
Students, councelors and teachers must find what they look for
How?
- Interaction design (IxD) vs graphical design
- User testing, user testing and user testing (and experience)
- Resulting in a GUI specification we must implement

Ajax-Solr is our JS framework:
https://github.com/evolvingweb/ajax-solr/wiki/reuters-tutorial
- manages all querying
- widgets for interaction with and displaying results
- events fire search requests which updates widgets
We extended it heavily
- Developed all our widgets (10+)
- Added logging (async, via ajax, local and GA)
- Distributed configuration (server + client)
- Simplified initialization script
But it also works out of the box!

Logger
~200 lines
JS library
~1700 lines
Solr 3.6
Our Website
Solr proxy
~85 lines
ajax-solr
evolvingweb
SolrPhpClient
r60
Default config
Initialize
(config)
JS library
(copy)Search
ACME Engineering
Lorum sollicitudin nunc id nibh
blandit pellentesque ipsum.
ACME Law
Cras nunc id nibh blandit
pellentesque sollicitudin.
ACME Med
Ipsum ollicitudin nunc id blandit
nibh pellentesque nibh.
- Include JS library
- Initialize
- Set up HTML
- Search! (and log)

Site search – widgets & faceting
Ajax Solr allows defining N widgets
«Everything» is a widget
A facet is an instance of a FacetWidget
Interaction with widgets may fire query
All facetation is piped into one query
All widgets are updated after Solr response

Some facet widgets we have developed
- Plain
Facet values and facet counts in a list
Multiple (AND) or single choice
- Hierarchical
Facet values and facet counts in a list
Clicking on a facet value drills down into the hierarchy; facet.prefix + fq
- Dropdown
Displays facet values in a dropdown list
Useful for mobile devices in our responsive theme
- Tagcloud
Facet values in a tagcloud
- Pivot facet
Our menu system

Adding facets
Config
facets['interests'] = new facetobject('tagcloud', 'field_interests', '#interests');
facets['ispublic'] = new facetobject('plain', 'field_ispublic', '#ispublic');
config['facets'] = facets;
HTML
<ul id="interests"></ul>
<ul id="ispublic"></ul>
INITIALIZE
Manager.addFacets(config);

Example widget code
AjaxSolr.PlainFacetWidget = AjaxSolr.AbstractFacetWidget.extend({
multivalue: true,
target: null, // HTML target id
field: null, // Solr-field
facet_display_limit: 5, // Max facets to display before «See more»
facet_field_sort: null, // Optional facet sort
dependencies: null, // Conditional display of facet
facet_display_more: 'See more',
facet_display_less: 'See less',
...
init: function() { ...}
beforeRequest: function() { ... }
afterRequest: function() { ... }
});

Pivot faceting allows you to facet within the results of the parent facet
- http://wiki.apache.org/solr/SimpleFacetParameters
Slight problem; we don't run Solr 4.x!

Problem
Menu facets shouldn't affect each other, but affect search result and other facets

Our solution
Solr document 1
<str name="ss_menu_1">orgmenu</str>
<str name="ss_menu_2">org</str>
Solr document 2
<str name="ss_menu_2">higher_ed</str>
Solr document 3
<str name="ss_menu_2">secondary</str>
Solr query when a top level menu tab is selected
fq={!tag=ss_menu_1}ss_menu_1:edumenu&
facet.field={!ex=ss_menu_1}ss_menu_1
Solr query when a sub-level menu tab is selected
fq={!tag=ss_menu_1}ss_menu_1:edumenu&
fq={!tag=ss_menu_1,ss_menu_2}ss_menu_2:higher_ed&
facet.field={!ex=ss_menu_1}ss_menu_1&
facet.field={!ex=ss_menu_2}ss_menu_2

Drawbacks
- Can be VERY slow on large indexes with many unique terms in the facet
Why do we do it?
- Small index; 18M terms, 12K documents
- Pivot facet fields have very few distinct values (5-8)!

Our goal
Give our users the feeling that we've implemented a mind-reader
How?
With relevant, grouped suggestions* as they type in a search query
Do we succeed?
50% of our «clicks to content» from searches comes from autocomplete

Implementing autocomplete is «easy»
1) Ajax
2) Detect keystrokes
3) Send one request per keystroke
4) Receive results, populate result list
Techniques we employ
- Minimal payload (reduced fl)
- But same boosts and qf as «normal» queries
- group=true, group.field=, group.limit=
- start_label^1.5 wild_label^1 wild_other^0.25
- Caching (jsonp, cache=true)

Define field type
<fieldType name="startsWith" class="solr.TextField">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.PatternReplaceFilterFactory" pattern="([^a-z])" replacement="" replace="all"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="2" maxGramSize="25" />
</analyzer>
<analyzer type="query">
<filter class="solr.PatternReplaceFilterFactory" pattern="([^a-z])" replacement="" replace="all"/>
</analyzer>
</fieldType>
Define fields
<field name="start_label" type="startsWith" indexed="true" stored="false" multiValued="false"/>
Copy fields
<copyField source="label" dest="start_label"/>

Define field type
<fieldType name="wildCardType" class="solr.TextField" omitNorms="true">
<analyzer type="index">
<filter class="solr.EdgeNGramFilterFactory" minGramSize="2" maxGramSize="70" side="front"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="2" maxGramSize="70" side="back"/>
</analyzer>
<analyzer type="query">
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt">
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt" ignoreCase="false"/>
<filter class="solr.NorwegianLightStemFilterFactory"/>
</analyzer>
</fieldType>
Define fields
<field name="wild_label" type="wildCardType" indexed="true" stored="false" multiValued="false"/>
<field name="wild_other" type="wildCardType" indexed="true" stored="false" multiValued="true"/>
Copy fields
<copyField source="label" dest="wild_label"/>
<copyField source="teaser" dest="wild_other"/>
<copyField source="body" dest="wild_other"/>
<copyField source="searchwords" dest="wild_other"/>
<copyField source="related_nodes" dest="wild_other"/>

Our goal
Let other sites search our data
How?
The exact same way we do ourselves
Do we succeed?
Two external sites are up and running and a third is on its way

Logger
~200 lines
JS library
~1700 lines
Solr 3.6
ACME Website
Solr proxy
~85 lines
ajax-solr
evolvingweb
ACME config
SolrPhpClient
r60
Default config
Config
(override)
JS library
(copy)Search
ACME Engineering
Lorum sollicitudin nunc id nibh
blandit pellentesque ipsum.
ACME Law
Cras nunc id nibh blandit
pellentesque sollicitudin.
ACME Med
Ipsum ollicitudin nunc id blandit
nibh pellentesque nibh.
- Register with us
- Include our JS library
- Set up config
- Set up HTML
- Search! (and log)

<html>
<head>
<title>ACME Website</title>

<script src="/js/jquery.js"></script>
<script src="http://example.com/solrservice/js-min/solr-search-full-min.js"></script>
<script src="/js/search-init.js"></script>
</head>
<body>

<form>
<input id="query" name="query" type="search" />
<input type="submit" value="Search" />
</form>

<div><ul class="hits" id="hits"></ul></div>
</body>
</html>

Site owners have full control
Add, edit and configure widgets
Query fields, boosts, etc.
Faceting
Styling
Pre-limit search to parts of our index
Because we eat our own dog food!

Our goal
Deliver data in bulk to partner organizations
How?
Restful searchable data endpoint that returns XML (Atom++)
Do we succeed?
Beta-partner up and running with stunning performance

Consumer
Query
Default config
Feeds engine
~300 lines
Solr proxy
~85 lines
Solr 3.6
Logger
~200 lines
SolrPhpClient
r60

Feeds engine
- Parses incoming query
- Loads config (filters, weights, ...)
- Transforms incoming + config to Solr URL
- Sends to Solr proxy
Solr Proxy
- Loads Solr PHP Client library
- Sends search request and parses response
- Returns results to Feeds engine
Feeds engine
- Loads logger and logs results
- Picks out ATOM from response
- Glues result inside an ATOM frame
- Display feed

http://example.com/data/atom/organizations
http://example.com/data/atom/organizations/10/2
http://example.com/data/atom/organizations?fq=type:HE
http://example.com/data/atom/organizations?fq=type:HE&q=law
Consume with feeds reader

How?
Logging back-end written in PHP that writes to a MySQL database
- called asynchronously from JS library
- called inline in Feeds engine
Google Analytics (ga.js)
- called from JS library (searchwords and categories)
What?
- Search terms
- Facets
- User interaction
- List of search results
- Stack latency (JS, PHP, Solr)
- Search domain
- Session

Why?
Most popular queries with no results?
Most popular queries?
How does QPS affect latency?
Follow a user through search (interaction design & user testing)
Displaying logs
Charts are generated with Google Chart Tools in Drupal
Other statistics can easily be explored with Drupal Views

Demo (includes responsiveness)

http://utdanning.no/sok
http://utdanning.no/search
http://utdanning.no/solrservice/utdanning.no

Drupal 7
Apache Solr Search Integration
+ custom indexing
Omega theme (responsiveness with Drupal)
+ custom js
Ajax Solr
+ custom widgets
Solr Php Client r60
+ custom proxy
Bootstrap (responsiveness without Drupal)
jQuery
Google Chart Tools

Remi MikalsenRemi Mikalsen
remi.mikalsen@iktsenteret.noremi.mikalsen@iktsenteret.no
iktsenteret.noiktsenteret.no
Multi-facetedMulti-faceted
responsive search,responsive search,
autocomplete,autocomplete,
feeds engine andfeeds engine and
logginglogging

CONTACT
Remi Mikalsen
remi.mikalsen@iktsenteret.no

Multi faceted responsive search, autocomplete, feeds engine & logging

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Multi faceted responsive search, autocomplete, feeds engine & logging

Similar a Multi faceted responsive search, autocomplete, feeds engine & logging (20)

Más de lucenerevolution

Más de lucenerevolution (20)

Último

Último (20)

Multi faceted responsive search, autocomplete, feeds engine & logging