Rank Your Results with PostgreSQL Full Text Search (from PGConf2015)

Rank Your Results: Using Full Text Search with Natural
Language Queries in PostgreSQL
to get Ranked Results
Jamey Hanson
jhanson@freedomconsultinggroup.com
jamesphanson@yahoo.com
Freedom Consulting Group
http://www.freedomconsultinggroup.com
PGConf US, NYC
March 26, 2015

Full Text Searching (or just text search) provides the
capability to identify natural-language documents that satisfy
a query, and optionally to sort them by relevance to the
query.
}  PostgreSQL 9.4 documentation, section 12.1
What is PostgreSQL Full Text Search?
PGConf US, NYC 26-Mar-20152

}  Focus on semantics, rather than syntax
}  Keep documents within PostgreSQL
}  Apache Solr, Lucene, Sphinx etc. require
their own copy of the data
}  Simple to keep indexes up to date
}  ~20 * faster than SQL search (LIKE,
ILIKE, etc.) with default configuration
}  Fast enough for nearly all applications
What makes Full Text Search so useful?

}  Traditional search reveals document existence,
ranked results indicate relevance
What makes Full Text Search so useful?
}  Customer expectations are that all searches should
include rank
}  FTS includes full suite of PG query tools such as SQL,
regex, LIKE/ILIKE, wildcards and function-based
indexes
}  FTS parser, stop word list, synonym, thesaurus and
language are all customizable at the statement level
}  FTS is extensible with 3rd party dictionaries and more

}  Infrastructure and data wrangling
}  Creating FTS tables with maintenance triggers
}  Compare FTS with traditional SQL searches and run FTSs
on documents from early American History
}  Rank search results on documents from Data Science
}  Generate HTML-tagged fragments with matching terms
}  Customize the stop-word dictionary
}  Suggest spelling options for query terms
}  Re-write queries at run-time
I will move between slides, demonstrations and SQL scripts. We will
not review every slide in the file.
Agenda

Jamey Hanson
Manage a team for Freedom Consulting Group migrating
applications from Oracle to Postgres Plus Advanced Server and
PostgreSQL in the government space. We are subcontracting to
EnterpriseDB
Overly certified: PMP, CISSP, CSEP, OCP in 5 versions of Oracle,
Cloudera developer & admin. Used to be NetApp admin and
MCSE. I teach PMP and CISSP at the Univ. MD training center
Alumnus of multiple schools and was C-130 aircrew
About the author

•  PostgreSQL 9.4.1 EnterpriseDB free package
•  CentOS 7.0VM with 2GB RAM and 2 CPU cores
•  2 sets of documents to search …
•  Primary documents in American History: The American
Revolution and the New Nation
http://www.loc.gov/rr/program/bib/ourdocs/NewNation.html
(Library of Congress)
•  Text from Data Science Girl’s August 15, 2014 blog post
“38 Seminal Articles Every Data Scientist Should Read”
http://www.datasciencecentral.com/profiles/blogs/30-seminal-articles-every-
data-scientist-should-read
Presentation infrastructure …

}  Used pgAdmin3 for SQL
and administration
}  A few Linux shell commands
to manage the files
}  American history documents
were cut & pasted from Web into MS Notepad
}  Data Science .pdf files were downloaded, converted to
text with pdftotext and manually divided into abstract
and body files
… Presentation infrastructure

}  FTS is built on lexemes, which are (essentially) word roots
without tense, possessive, plurality or other ending.
“It is a basic unit of meaning, and the headwords of a
dictionary are all lexemes”
The Cambridge Encyclopedia ofThe English Language
}  For example ...
}  The lexeme of jump, jumps, jumped and jumping are all ‘jump’
}  Excited, excites, exciting and excited are all ‘excit’
}  Lexemes are stored in lower case
(i.e. case insensitive)
How does FTS work?

}  lexemes are organized into TSVECTORs, which are sorted
arrays of lexemes with associated position and (optionally)
weight. Documents are stored as TSVECTORs
}  Query against TSVECTORs using TSQUERYs, which are
arrays of lexemes with BOOLEAN operators but without
position or weight
}  Match a TSQUERY to a TSVECTOR with the @@ operator
How does FTS work?

1.  Parses text document into tokens using white space, non
printing characters and punctuation
2.  Assigns a class (i.e. type) to each token. 23 classes include
word, email, number, URL, etc.
3.  ‘Word’ tokens are normalized into lexemes using a parser
4.  Lexemes are processed to …
a.  Remove stop words (common words such as ‘and’,‘or’,‘the’)
b.  Add synonyms
c.  Add phrases matching
5.  Lexemes are assembled into TSVECTORs by noting the
position, recording weight and removing duplicates
This process is controlled by TEXT SEARCH DICTIONARYs
How does TO_TSVECTOR work?

}  TSVECTORs are compared to TSQUERYs with the @@
operator
}  TSQUERYs are built with the TO_TSQUERY or
PLAINTO_TSQUERY functions …
Never mind … let’s jump to some examples, which are
much easier to understand.
How does FTS match documents?

}  Explore TSVECTORs and TSQUERYs
00_FTS_explore_tsvector_tsquery_v10.sql
GOTO …

}  -- What do lexemes look like?
SELECT TO_TSVECTOR('enumerate') AS enumerate,
TO_TSVECTOR('enumerated') AS enumerated,
TO_TSVECTOR('enumerates') AS enumerates,
TO_TSVECTOR('enumerating') AS enumerating,
TO_TSVECTOR('enumeration') AS enumeration;
-- all forms of the work have the same lexeme, 'enumer'
-- Example tsvector
SELECT TO_TSVECTOR('We hold these truths to be self evident');
-- 'evid':8 'hold':2 'self':7 'truth':4
-- tsvectors are sorted arrays of lexemes with position and
(optionally) weight
-- notice that common words, a.k.a. stop words, like 'to' and
'be' are not included
TSVECTOR and TSQUERY

-- tsquery_s are compared with tsvector_s to find matching documents
-- they are composed of lexemes and logical operators
SELECT TO_TSQUERY('with & liberty & and & justice & for & all');
-- 'liberti' & 'justic'
-- Notice that stop words are not included in tsquery_s either
-- can also use PLAINTO_TSQUERY with plain(ish) text
SELECT PLAINTO_TSQUERY('With liberty and justice for all');
-- 'liberti' & 'justic'
TSVECTOR and TSQUERY

}  Explore TSVECTORs and TSQUERYs
00_FTS_explore_tsvector_tsquery_v10.sql
RETURN from …

}  Created at run-time with TO_TSVECTOR
+ simple, does not require storage
- slower queries than pre-computing
}  Created ahead of time with TO_TSVECTOR
+ fast queries, flexible, does not slow ingestion, less CPU work
- can leave TEXT and TSVECTOR out of sync, may not get done
}  Create ahead of time with a trigger
+ fast queries,TSVECTOR always up to date
-  slows ingestion, UPDATE trigger first on small changes
}  Two trigger functions are included
tsvector_update_trigger & …_column
How do we create TSVECTORS?

}  GIN (Generalized
Inverted iNdex)
}  GiST (Generalized
Search Tree)
How to make FTS
wickedly fast?
GIN GiST
Speed 3 * faster Slower
Size 3 * bigger smaller
WeightedTSV Unsupported Supported
Build speed Slower 3 * faster
Best practice Static data Updated data
See TomasVondra's 17-Mar-15 Planet PostgreSQL post on FTS performance for details

Let’s build our FTS tables
}  Build our FTS tables using 20_FTS_DDL_v10.sql
GOTO …

}  Build our FTS tables using 20_FTS_DDL_v10.sql
RETURN from …

}  Load text documents from the database host using
pg_read_file
}  pg_read_binary_file for .pdf_s
}  Files must be in $PGDATA, but symbolic links work
}  Syntax is:
(SELECT * FROM pg_read_file('path/from/
$PGDATA/file.txt'))
}  Weighted searches require that the document is divided
into sections. Details forthcoming
}  Can dynamically generate SQL load scripts using
pg_ls_dir or run a script from psql
Loading documents for FTS

}  Load our FTS tables using 30_FTS_Load_v10.sql
}  Update title, author and URL fields with
32_FTS_Update_Titles_v10.sql
GOTO …

}  Load files .txt and .pdf files from within $PGDATA
}  We divided the Data Science documents into abstract and
body so that they can we weighted for weighted rank
queries
}  TSVECTORs are created by the BIU trigger
}  Manually updated fts_data_sci.tsv_document
just to show how it is done
}  The update script populates title, author and URL fields.
Load text and .pdf documents

-- Create dynamic SQL to load fts_amer_hist
WITH list_of_files AS (
SELECT pg_ls_dir('Dropbox/FTS/AmHistory/') AS file_name
)
SELECT
'INSERT INTO fts.fts_amer_hist (document, filename) VALUES (
(SELECT * FROM pg_read_file(''Dropbox/FTS/AmHistory/' || file_name || ''')),
''' || file_name || '''); '
FROM list_of_files
ORDER BY file_name;
-- generates --
INSERT INTO fts.fts_data_sci (
abstract, body, document, pdf_file, pdf_filename) VALUES (
(SELECT * FROM
pg_read_file('Dropbox/FTS/DataScience/WhatMapReduceCanDo_AB.txt')),
(SELECT * FROM
pg_read_file('Dropbox/FTS/DataScience/WhatMapReduceCanDo_BD.txt')),
(SELECT * FROM
pg_read_file('Dropbox/FTS/DataScience/WhatMapReduceCanDo.txt')),
(SELECT * FROM
pg_read_binary_file('Dropbox/FTS/DataScience/WhatMapReduceCanDo.pdf')),
'WhatMapReduceCanDo.pdf');
Dynamic SQL to load files

}  Load our FTS tables using 20_FTS_Load_v10.sql
}  Update details with
22_FTS_Update_Titles_v10.sql
RETURN from …

GOTO …
}  See FTS in action with
40_FTS_explore_fts_amer_hist_v10.sql
Enough with the setup … show me FTS!

}  Compare SQL ILIKE searches with FTS*
}  See how ILIKE misses documents with different word
forms, such as 'enumerate' vs. 'enumeration'
}  See how FTS is ~20 * faster than ILIKE
}  Demonstrate that FTS excludes stop words such as 'the',
'and', & 'or'
}  Demonstrate that FTS includes BOOLEAN logic with
simple syntax
}  *ILIKE is “case insensitive LIKE”
Explore fts_amer_hist

}  See FTS in action with
40_FTS_explore_amer_hist_v10.sql
RETURN from …

}  Result rank is 0 (not found) to 1 (perfect match),
calculated at run time based on a search of all documents.
}  That means TOP 5 is slower than LIMIT 5
}  Two rank functions are available, TS_RANK and
TS_RANK_CD
}  Both consider how often search terms appear
}  Both have an optional normalization parameter that weights the
rank by the log of the size of the document
}  TS_RANK_CD also considers the proximity of search
terms to each other
Ranking results

}  Lexemes in tsvectors can be assigned a weight of
A(high) – D(low), with defaults of {1.0, 0.4, 0.2, 0.1}
}  Weighting does not affect which records are returned,
only their rank
}  Weighted tsvectors are typically built by document section
}  title=A, abstract=B, body=D in our example trigger
new.tsv_weight_document :=
SETWEIGHT(TO_TSVECTOR('pg_catalog.english',
COALESCE(new.title, '')), 'A') ||
SETWEIGHT(TO_TSVECTOR('pg_catalog.english',
COALESCE(new.abstract, '')), 'B') ||
TO_TSVECTOR('pg_catalog.english', new.body);
Building weighted tsvectors

}  The example tsvectors were weighted at build-time by the
trigger
}  Can also build weighted tsvectors at query-time
}  More flexible because different queries can use different weights
}  Requires more code because weighting is done for every query
}  Slightly slower because the source tsvectors must be
concatenated
}  SETWEIGHT(TO_TSVECTOR(title),'A') ||
SETWEIGHT(TO_TSVECTOR(abstract,'B') ||
TO_TSVECTOR(body); -- default weight 'D'
Building weighted tsvectors at query-time

What does all this get us?
}  Search for document relevance, not just
existence
}  Customers now expect demand ranked
results
}  The data and the business logic are inside
PostgreSQL, available to any application

}  Generate weighted, ranked document searches with
50_FTS_weighted_ranked_results_v10.sql
GOTO …

}  Top-5 results syntax
SELECT title,
ts_rank(tsv_document, q) AS rank -- value between 0 and 1
FROM
fts_data_sci,
PLAINTO_TSQUERY('corelation big data') AS q
ORDER BY rank DESC
LIMIT 5;
}  Syntax for ts_rank_cd (ts rank with cover density) is the
same
Weighted, ranked document searches

}  Top-5 weighted results syntax
SELECT title,
ts_rank(tsv_weight_document, q) AS rank -- weighted column
FROM
fts_data_sci,
PLAINTO_TSQUERY('corelation big data') AS q
ORDER BY rank DESC
LIMIT 5;
}  The only difference is using the weighted tsvector
}  Could also have built a weighted tsvector at query time.
Weighted, ranked document searches

}  Generate weighted, ranked document searches with
50_FTS_weighted_ranked_results_v10.sql
RETURN from …

}  We have used English language with default parser
(tokenizer), stop-word list and dictionary.
}  The combination is a TEXT SEARCH DICTIONARY
}  The default is pg_catalog.english
}  SELECT default_text_search_config; to see
}  We created tsvectors (weighted and unweighted) using
default and customer triggers plus manually
Pause … with all default configuration

We can highlight matches with ts_headline

}  TS_HEADLINE returns text fragment(s) surrounding
matching terms with HTML tags
}  Default is a single snippet with matching_term
}  Search for PLAINTO_TSQUERY('liberty justice happy')
Display fragments with matching terms

}  How many fragments? MaxFragments
}  What comes between fragments? FragmentDelimiter
}  How many surrounding words? MinWords / MaxWords
}  Which HTML tags highlight terms? StartSel / StopSel
SELECT TS_HEADLINE(document, q,
'StartSel="",
StopSel="",
MaxFragments=10,
MinWords=5, MaxWords=10,
FragmentDelimiter=" ... ..."')
Configure TS_HEADLINE to improve display

Q: Which American history documents contain 'liberty',
'justice' and 'happy'?
SELECT
'Document title: ' || title || ' ' ||
TS_HEADLINE(document, q,
'StartSel="",
StopSel="",
MaxFragments=10,
MinWords=5, MaxWords=10,
FragmentDelimiter=" ... ..."')
FROM fts_amer_hist,
PLAINTO_TSQUERY('liberty justice happy') AS q
WHERE tsv_document @@ q
ORDER BY TS_RANK(tsv_document, q) DESC;
Well formatted ts_headline results
41

Very nice …
GOTO check out the 4 matching documents

Q: What has FTS gotten us right out of the box?
A: Directly loaded documents that are automatically indexed,
weighted and maintained in a form that supports fast natural
language(ish) queries with ranked results plus well-formatted
document fragments with highlighted matches.
Which is to say, a lot!

}  Create a custom TEXT
SEARCH DICTIONARY
}  Customize the stop word list
based on frequency counts
}  Modify queries at run-time
to remove terms and/or use
synonyms with
TS_REWRITE
}  Create a tool to suggest
spelling corrections for
query terms
Customizing FTS

}  Defines the language, stopwords, dictfile and other options
for TO_TSVECTOR and TSQUERY related functions
}  Custom dictionaries based on a template
}  pg_catalog.english is the default
}  SHOW default_text_search_config;
}  Uses files in $PGSHAREDIR/tsearch_data
$PGSHAREDIR=$PG_HOME/share/postgresql
}  Option STOPWORDS=english references
$PGSHAREDIR/tsearch_data/english.stop
}  NOTE: Must 'touch' a TS DICT after each file change with
ALTER TEXT SEARCH DICTIONARY
Custom TEXT SEARCH DICTIONARY

}  TS_STAT(tsvector)returns
}  ndoc the number of documents a lexeme appears in
}  nentry the number of times a lexeme appears
}  This is useful to identify candidate stop words that appear
too frequently to be effective discriminators
}  TS_LEXSIZE('dictionary', 'word')
}  Useful to text if the custom dictionary is working as planned
FTS helpful utility functions

}  TEXT SEARCH DICTIONARYs change the tsvector,
TS_REWRITE changes the tsquery at SQL run-time
}  Look up tsquery substitution values in a table of:
term TSQUERY
alias TSQUERY
}  Used for alias' or stop words, by substituting ''
}  Ex. 'include' as an alias for 'contain'
'data' as a stop word
INSERT INTO fts_alias VALUES
('contain'::TSQUERY, 'contain | include'::TSQUERY),
('data'::TSQUERY, ''::TSQUERY);
Change the TSQUERY w/TS_REWRITE

}  Create custom dictionary and stop words plus query
rewrite
60_FTS_stop_words_custom_dictionary_and
_query_v10.sql
GOTO …

}  Find words that appear frequently and that appear in multiple
document
SELECT word,
nentry AS appears_N_times,
ndoc AS appears_in_N_docs
FROM TS_STAT(
'SELECT tsv_weight_document FROM fts_data_sci')
-- weighted and un weighted tsvector are equiv
ORDER BY
nentry DESC,
word;
Identify candidate stop words

}  Create custom TEXT SEARCH DICTIONARY
-- DROP TEXT SEARCH DICTIONARY IF EXISTS public.stopper_dict;
CREATE TEXT SEARCH DICTIONARY public.stopper_dict (
TEMPLATE = pg_catalog.simple,
STOPWORDS = english
);
}  Add stop words to
$SHAREDIR/tsearch_data/english.stop
}  'Touch' the dictionary after each change to take effect
ALTER TEXT SEARCH DICTIONARY public.stopper_dict (
STOPWORDS = english
);
Create TEXT SEARCH DICTIONARY

CREATE TABLE fts_alias (
term TSQUERY PRIMARY KEY,
alias TSQUERY
);
}  Add term alias' and stop words
INSERT INTO fts_alias VALUES
('contain'::TSQUERY, 'contain | include'::TSQUERY),
('data'::TSQUERY, ''::TSQUERY);
);
}  Use the alias table with TS_REWRITE
SELECT TS_REWRITE(('data & information & contain')::TSQUERY,
'SELECT * FROM fts_alias');
-- 'information' & ( 'include' | 'contain' )
Create table for TS_REWRITE

}  Create custom dictionary and stop words plus query
rewrite
60_FTS_stop_words_custom_dictionary_and
_query_v10.sql
RETURN from…

}  Use TriGrams , the pg_trgm extension, with a list of
words in the documents to identify words that are close
to input queries as suggestions for misspelled terms
}  Step 1: create a table of words
CREATE TABLE fts_word AS
(SELECT word
FROM TS_STAT(
'SELECT tsv_document FROM fts_amer_hist'))
UNION
(SELECT word
FROM TS_STAT(
'SELECT tsv_document FROM fts_data_sci')
);
Query-term spelling suggestions with TriGrams

}  Step 2: create an index
CREATE INDEX idx_fts_word ON fts_word USING
GIN(word gin_trgm_ops);
}  Step 3: query for 'close' terms that exist in the corpus
SELECT word, sml
FROM fts_word,
SIMILARITY(word, 'asymetric') AS sml
WHERE sml > 0.333
-- arbitrary value to filter results
Check it out in action …
Query-term spelling suggestions with TriGrams

}  SELECT word, sml
FROM fts_word,
SIMILARITY(word, 'asymetric') AS sml
-- 'asymmetric' is the correct spelling
WHERE sml > 0.333 -- arbitrary value to filter results
ORDER BY sml DESC, word;
word sml
========== ========
asymetr 0.636364
asymmetri 0.538462
asymmetr 0.461538
Metric 0.416667
Suggested spelling with TriGrams

}  Infrastructure and data wrangling
}  Creating FTS tables with maintenance triggers
and load our data
}  Compare FTS with traditional SQL searches and run FTSs
on documents from early American History
}  Rank search results on documents from Data Science
}  Generate HTML-tagged fragments with matching terms
}  Customize the stop-word dictionary
}  Suggest spelling options for query terms
}  Re-write queries at run-time
All slides, data and scripts are on the on PGConf web site
Summary and review

LIFE. LIBERTY. TECHNOLOGY.
Freedom Consulting Group is a talented, hard-working,
and committed partner, providing hardware, software
and database development and integration services
to a diverse set of clients.
“
”

POSTGRES
innovation
ENTERPRISE
reliability
24/7
support
Services
& training
Enterprise-class
features, tools &
compatibility
Indemnification
Product
road-map
Control
Thousands
of developers
Fast
development
cycles
Low cost
No vendor
lock-in
Advanced
features
Enabling commercial
adoption of Postgres

Are
there
any
Questions or
follow up?
freedomconsultinggroup.com/jhanson

Freedom Consulting Group
www.freedomconsultinggroup.com
Jamey Hanson

Rank Your Results with PostgreSQL Full Text Search (from PGConf2015)

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (6)

Similar a Rank Your Results with PostgreSQL Full Text Search (from PGConf2015)

Similar a Rank Your Results with PostgreSQL Full Text Search (from PGConf2015) (20)

Último

Último (20)

Rank Your Results with PostgreSQL Full Text Search (from PGConf2015)