Brian Ritchie will give a presentation on transforming applications with Elastic Search. Elastic Search is an open source, distributed search and analytics engine that can be used to add powerful search capabilities to applications. It allows storing and searching large volumes of data quickly and flexibly scales. The presentation will cover introducing Elastic Search, bringing application data into it, security considerations, and an example of putting it all together to build a searchable application using Elastic Search.
2. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Brian Ritchie
CTO for XeoHealth Corporation
22 years of technology experience developing
software and managing technology teams in various
industries.
Author of RavenDB High Performance published by
PACKT in 2013.
Pursuing a Masters in Computer Science from
Georgia Tech.
@brian_ritchie
brian.ritchie@gmail.com
http://www.dotnetpowered.com
3. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
WELCOME
INTRODUCTION TO ELASTIC SEARCH
BRING YOUR DATA
SECURITY CONSIDERATIONS
PUTTING IT ALL TOGETHER
12. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
INTRODUCING
Elasticsearch is a search engine based
on Lucene. It provides a lightening fast,
distributed, multitenant-capable full-text
search engine with a HTTP RESTful
interface.
Elasticsearch is released as open source
under the terms of the Apache License.
It can be deployed locally or hosted in the
cloud.
13. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
INTRODUCING
Well known for
being part of the
Stack
The Elastic StackNote: this was rebranded
used to analyze log data
generated by your application
14. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
WHY
FAST
Document
Centric
Shard/
Partition
Index
Everything
Extremely Fast. Optimized for search. Flexible Schema.
Distributed
Cluster
FOR YOUR APPLICATION
15. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
WHY
SCALABLE “We can support clusters of massive scale. Well
into the 100s of terabytes. One of our largest
customers is currently at 750TB and should be in
excess of 1PB by the end of the year. “ - Scalefastr
“Today there are 60+ Elasticsearch clusters and
2000+ nodes. The daily ingestion reaches 18 billion
documents, and daily search requests reach 3.5
billion. ” - eBay
FOR YOUR APPLICATION
17. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
That sounds awesome!!
But, how does Elastic Search fit
into my data infrastructure?
BRINGING YOUR DATA INTO
18. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Utilizing a CQRS inspired pattern
BRINGING YOUR DATA INTO ELASTIC SEARCH
Command/Query Responsibility Segregation (CQRS) is the idea that
you can use a different model to update information than the model
you use to read information
Persistent View Model
Transactional Data
19. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
OPTION 1: Add Elastic search into your database pipeline
cluster
Kafka
ActiveMQ
RabbitMQ
….
BRINGING YOUR DATA INTO ELASTIC SEARCH
20. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
clusterKafka
ActiveMQ
RabbitMQ
….
OPTION 2: Replicate database updates to Elastic Search
BRINGING YOUR DATA INTO ELASTIC SEARCH
21. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
cluster
OPTION 3: Batch your database updates on a schedule
BRINGING YOUR DATA INTO ELASTIC SEARCH
22. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Structuring your data for search
DATA MODELING
Conceptually, indexes are like databases.
• SQL => Databases => Tables => Columns/Rows
• Elasticsearch => Indices => Types => Documents with Properties
Loans Payments Inventory
23. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Defining indexes
DATA MODELING:
Like databases, but more powerful. Queries can easily span indexes bringing
data together as needed.
Use indexes for:
• Security – use indexes to support
multi-tenant system
loan_hud loan_fdic
• Partitioning – time or other
data
log_20180501 log_20180502
24. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Defining documents
DATA MODELING:
{
"FirstName": "Jonathan",
"Address": "15 Wanamassa Point Road ",
"City": "Jacksonville",
"State": "FL",
"Children": [
{ "Name": "Michael", "Age": 10 },
{ "Name": "Jennifer", "Age": 8 },
{ "Name": "Samantha", "Age": 5 },
{ "Name": "Elena", "Age": 2 }
]
}
Instead of decomposing
data into separate entities:
Design documents based on what needs to
be searched and displayed together:
25. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Teach Elastic Search about specific types
DATA MODELING:
PUT _template/file_name_mapping
{
"template": ”loan_*",
"settings": { "number_of_shards": 5 },
"mappings": {
"_default_": {
"dynamic_templates": [ {
"filename_string_not_analyzed": {
"match": "edi_filename",
"mapping": {
"index": "not_analyzed",
"type": "string” }
} } ]
} }
}
26. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Giving the right users access to the right data
SECURITY CONSIDERATIONS
cluster
Search Guard
Custom
.NET Core / Java / etc.
Elastic Stack
Security
27. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
PUTTING IT ALL TOGETHER
The Home Mortgage Disclosure Act (HMDA) requires many financial institutions to
maintain, report, and publicly disclose loan-level information about mortgages.
Searching CFPB loan records w/ Elastic Search
28. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
PUTTING IT ALL TOGETHER
107 million
loan records
var record = csv.GetRecord<dynamic>();
var loan = Loan.FromFileLoan(record);
var index_name = string.Format("loan_{0}",
loan.agency_abbr.ToLower());
var response = client.Index<Loan>(
new IndexRequest<Loan>(loan, index_name));
Loading CFPB loan records into Elastic Search
{
"county": "Maricopa County",
"respondent_id": "0000451965",
"lien_status": "Not applicable",
"sequence_number": "0945293",
"applicant_income": 132000,
"preapproval": "Not applicable",
"applicant_race": "White",
"msamd": "Phoenix, Mesa, Scottsdale - AZ",
….
29. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Using Kibana to search and visualize your data
PUTTING IT ALL TOGETHER
30. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Using Elastic Search to optimize your user experience
PUTTING IT ALL TOGETHER
cluster
Single Page
Application
(SPA)
Angular /
React / etc.
JWT
Query
Request
REST API
.NET Core /
Java / etc.
Add security,
logging, etc.
31. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Custom application using Angular, PrimeNG, .NET Core, and Elastic Search
PUTTING IT ALL TOGETHER