Technical Overview of FAST Search Server 2010 for SharePoint - SharePoint Saturday Perth

1. Sponsors SharePoint Saturday
Gold
Technical Overview of FAST
Search Server for
SharePoint 2010
Silver Sezai Komur
SharePoint Solutions Architect
Bronze
CSG

2. What is FAST Search Server for
SharePoint 2010?
• Microsoft buys FAST Search and Transfer in
2008 for $1.2 Billion US.
• Port of FAST ESP integrated with SharePoint
2010.
• New enhanced search engine integrated with
SharePoint Server 2010
SharePoint Saturday
Perth 2011

3. How is FAST Search better than
SharePoint Server Search?
• Better search result quality
– Better quality search engine relevancy and use of
linguistics, stemming & lemmatisation in search
processing
• Search Platform Extensibility
• Advanced Content Processing
– Property Extraction
– Document Processing Pipeline Extensibility
• Extreme Scale Search
– Search Billions of documents with sub second search
times
• Deep Refinement
– Exact refinement number, SharePoint refiners aren't over
the entire result set SharePoint Saturday
Perth 2011

4. How is FAST Search better than
SharePoint Server Search?
• Advanced Sorting
– Sort on Managed Properties and Rank Profiles
• Tuneable Relevance with Multiple Rank Profiles
• FQL Query Language
• Contextual Search
– Tailor results and refinement to user profile or audience
• Rich Web Indexing
– Dynamic web content and Javascript, highly customisable connector
• Similar Results Detection & Results Collapsing
• Thumbnails and Previews
– SharePoint 2010 Word and PowerPoint results via Office Web Apps
• Visual Best Bets
SharePoint Saturday
Perth 2011

5. FAST Search Service Applications
Two SharePoint Service Applications
Communicate with FAST Servers
• FAST Content Search Service Application
– Connector Configuration
– Crawling
• FAST Query Search Service Application
– Queries and Results from associated Web
Applications
– Managed Property Mapping configuration
– People Search
SharePoint Saturday
Perth 2011

6. FAST Web Services
Services that SharePoint communicates with
• Content Distributors
• Query Service
• Administration Service
• Resource Store
• Log Server
• See Install_Info.txt in FAST install folder and
look in IIS
SharePoint Saturday
Perth 2011

7. SharePoint Saturday
Perth 2011

8. SharePoint Farm
Web Server Site Collection Admin UI PowerShell Central Administration UI
• Deployment • Schema configuration • Property mapping
Web Parts • User Context Management • Admin configuration • Property extraction
• Promotion/Demotion • Deployment • Spell-checking
configuration
Custom Query Web
front-end Service
FAST Search Server 2010 for SharePoint Farm
Federation
Object Model
Administration
FAST Search Content SSA
Query Content
Query (FAST Search
Query Matching Item Connector)
Processing Processing - SharePoint
- BDC
- Exchange Content
External
- Web
federation
People Search
FAST
sources Indexing
(query/crawl)
FAST Search Indexing Connectors
Authorization Web Link
Analysis Content
(FSA)
Query SSA
(Search Service
Monitoring
Application)
User
Profiles Active Microsoft System Center
!
Directory Operations Manager !
SharePoint Saturday
Perth 2011

9. Simple Conceptual Architecture
SharePoint Saturday
Perth 2011

10. Topology Diagram
SharePoint Saturday
Perth 2011

11. Medium Farm
SP2010 Farm FAST Search for SharePoint 2010 Farm
WFE WFE FAST-ADM-1 FAST-ADM-2 FAST-FSTIDX-11 FAST-FSTIDX-12 FAST-FSTIDX-13
Query SSA Query SSA Admin Content Distributor 2 Index (Search) Index (Search) Index (Search)
Content Distributor 1 Indexing Dispatcher 2 4 Docprocs+ 4 Docprocs+ 4 Docprocs+
Indexing Dispatcher 1 Web Analyzer
Web Analyzer 6 Docprocs+
SP2010 Services Farm (Enterprise Crawler)
FAST-FSTIDX-21 FAST-FSTIDX-22 FAST-FSTIDX-23
(Index) Search (Index) Search (Index) Search
SP Crawl SP Crawl QR Server QR Server QR Server
People Crawl People Crawl
Crawl DB Crawl DB
Search Admin DB
SQL 2008 Cluster
SharePoint Saturday
Perth 2011

12. SharePoint Saturday
Perth 2011

13. Search Engine Basics
• Crawling
– Gathering content to store in an index
• Indexing
– Storing content in an index optimised for
searching
• Querying
– Users execute searches against the index
SharePoint Saturday
Perth 2011

14. Crawling
SharePoint Saturday
Perth 2011

15. Crawling
• Connecting to sources of content to
download files and data for processing
• Downloading documents or files (Items)
• Working through URLs
– List or directory of items to crawl
– Following links to other items
• Extracting information from files
– Converting file formats to text for processing
– Identifying properties or fields of information
SharePoint Saturday
Perth 2011

16. DEMO
FAST SEARCH SERVICE APPLICATIONS
FAST SYSTEM DIRECTORY
FAST WEB SERVICES
CONNECTORS & CRAWLING
SharePoint Saturday
Perth 2011

17. Processing & Indexing
SharePoint Saturday
Perth 2011

18. Item Processing
• Format conversion
– IFilters
– Advanced Filter Pack (Oracle Outside In) - 400+ formats
• Language and encoding detection
• Lemmatizer Optional content pipeline
– linguistics normalization stages:
• Tokenizer
– word breaking
• XML Properties mapper
• Entity extraction • Offensive content filter
– companies, locations • Verbatim (whole word)
• DateTimeNormalizer extractor (loads dictionary for
– Date normalization custom extraction, e.g.
• Vectorizer
product names)
– Create document vector for similarity searching • Field Collapsing
• WebAnalyzer • Entity Extraction (persons)
– anchor text and link cardinality analysis • Document Processing Pipeline
• PropertiesMapper Extension
– Map to crawled properties
• PropertiesReporter
– report detected properties
FAST Search stores data to its Search Index after processing completes
SharePoint Saturday
Perth 2011

19. Mapper
…
…
Entity
Extraction
Lemmatization
Language
Detection
Format
Conversion
Document Processing Pipeline
Perth 2011
SharePoint Saturday

20. Property Extraction
SharePoint Saturday
Perth 2011

21. Document Processing Pipeline
Extensibility
• Items are processed in the Document Processing
Pipeline after they are crawled and before they
are stored in the index.
• Create and alter crawled property data.
• You can run code and pass data to other systems
– ‘Deep’ Search of raw data
– Geocoding
– OCR
– Audio and Video Transcription
… The sky is the limit!
SharePoint Saturday
Perth 2011

22. DEMO
DOCUMENT PROCESSING PIPELINE
CRAWLED PROPERTIES
MANAGED PROPERTIES
SharePoint Saturday
Perth 2011

23. Search UI
SharePoint Saturday
Perth 2011

24. Web Parts
SharePoint Saturday
Perth 2011

25. Refiners
• Refinement Panel Web Part
• Add and edit refiners displayed by changing
filter category definition XML.
• Properties specified in lower case, managed
property must have refinement enabled
SharePoint Saturday
Perth 2011

26. Rank Profiles
• Configure Multiple Rank Profiles
• Allow Selection of Rank Profile in Search UI
to change sorting
• Defaulting based on user profile
SharePoint Saturday
Perth 2011

27. DEMO
SEARCH UI
REFINERS
DOCUMENT PREVIEWS
VISUAL BEST BETS
SharePoint Saturday
Perth 2011

28. SP Saturday Slide Template Example
QUESTION AND ANSWER
SharePoint Saturday
Perth 2011

29. Related Links
Sezai’s Blog
http://sharepoint-sezai-moss-2007.blogspot.com/
Enterprise Search IT Professional Training
http://technet.microsoft.com/en-us/enterprisesearch/ff960998
Debugging and Tracing Pipeline Extensibility Stages
https://blogs.msdn.com/b/thomsven/archive/2010/09/23/debugging-and-
tracing-fast-search-pipeline-extensibility-stages.aspx
http://techmikael.blogspot.com/2010/12/how-to-debug-and-log-fast-
search.html
Shyam Nyaran’s blog – Visual Refiner Web Part
http://www.dotnetbounce.com/archive/2011/02/06/visual-refiners-for-
sharepoint-server-2010-and-fast-search.aspx
SharePoint Saturday
Perth 2011

30. Sponsors
SharePoint Saturday Gold
Thanks for listening!
Remember to submit your feedback so
you can go into the raffle draw at the
end of the day! And don’t forget that
Silver
you have to be at the draw to claim your
prizes!
Bronze