[DevDay2018] PLAY & LEARN with Elasticsearch Workshop - By: Thang Luong and Trung Do, Developer at Axon Active Viet Nam

DEVDAY 2018
Da Nang - Apr 2018

www.axon.vnfb.com/AxonActiveVietNam
Who we are?
Lương Văn Thắng

Who we are?
Đỗ Việt Trung

1. LEARN
a. OVERVIEW
b. Searching
c. SCALE
d. SECURITY
e. MIGRATION
2. PLAY - workshop with exercises
Agenda

OVERVIEW

What is Elasticsearch?
❖ Search engine based on Lucene
❖ Real-time distributed, full-text search engine
❖ RESTful API
❖ Schema-free
❖ First public release in Feb 2010
Wikipedia

WHO are using ELASTICSEARCH?

WhY Elasticsearch?
● NoSQL DB for indexing JSON contents
● Schema-free
● Distributed
● High performance
● REST semantics
● Graph capabilities
● Great documentation
● Open source!

WhY Elasticsearch?

WORK WITH Big data

Searching – faster and smarter

Searching – poor performance

Schema changes frequently

Scaling database

Our solution

Search In Action

Get started

❖Node – a started instance of Elasticsearch
❖Cluster - collection of connected nodes of
Elasticsearch
Cluster and Nodes

Cluster and Nodes

❖Similar to database
❖A collection of types
Indexes

❖Similar to table
❖Collection of documents
❖Has schema (implicit or explicit)
Types

❖Similar to record
❖Self-contained data
❖Have id
❖Have schema
Documents

❖Similar to column
❖Documents are structured in fields
❖Special fields: _id, _uid, _index, _type, _all, _source, …
Fields

❖ text, keyword
❖ long, integer, short, double, float, byte
❖ date
❖ boolean
❖ binary
❖ geo
❖ object
❖ nested
❖ ip
❖ arrays
❖ …
Data types

Mappings and settings

Mappings and settings
Mappings – schema of documents
+ dynamic mappings
+ explicit mappings
Settings – configured for each index
+ good enough when using default settings
+ index analysis – expert users

Download and setup
https://www.elastic.co/guide/en/elasticsearch/reference/5.4/setup.html

Config/elasticsearch.yml

Elasticsearch

{
"from": 0,
"size": 20,
"min_score": 5,
"query": {
"bool": {
"should": [],
"must": [],
"must_not": [],
"filter": {}
}
},
"sort": [],
"aggs": {}
}
Query structure

{
"from": 0,
"size": 20,
"min_score": 5,
"query": {
"bool": {
"should": [],
"must": [],
"must_not":
[],
"filter": {}
}
},
"sort": [],
"aggs": {}
}
Query structure
// do pagination

{
"from": 0,
"size": 20,
"min_score": 5,
"query": {
"bool": {
"should": [],
"must": [],
"must_not": [],
"filter": {}
}
},
"sort": [],
"aggs": {}
}
Query structure
// limit hitlist return

{
"from": 0,
"size": 20,
"min_score": 5,
"query": {
"bool": {
"should": [],
"must": [],
"must_not": [],
"filter": {}
}
},
"sort": [],
"aggs": {}
}
Query structure
// statistic

{
"from": 0,
"size": 20,
"min_score": 5,
"query": {
"bool": {
"should": [{..}, {..},..],
"must": [{..}, {..},..],
"must_not": [{..}, {..},..],
"filter": {} // not impact to total score
}
},
"sort": [],
"aggs": {}
}
Query structure
// and
// or
// not

{
"bool": {
"must": [
{“strong”},
{“is man”}
]
}
}
Exercise 1

{
"bool": {
"must": [
{“strong”},
{“is man”}]
}
}
Exercise 1

{
"bool": {
"must":
[{“has red on suite”}],
“should“:
[{“strong”},{“is man”}]
}
}
Exercise 2

{
"bool": {
"must": [{“has red on suite”}],
"should":[{“strong”},{“is man”}]
}
}
Exercise 2

Exercise 3
{
"bool": {
"must":
[{“strong”},{“has cape”}],
“must_not“:
[{“be seen the face”} ]
}
}

{
"bool": {
"must":
[{“strong”},
{“has cape”}],
“must_not“:
[{“be seen the face”}]
}
}
Exercise 3

building a query

Synonym

Synonym token filter

SCALing

What – Why?

How to Scale

SECURITY

WHY
Elasticsearch has no concept of a user.
Essentially, anyone that can send arbitrary requests to your
cluster is a “super user”.

WHAT
❖ Authentication & authorization
❖ Access control
❖ Encryption
❖ Auditing

Authentication
& Authorization
Role-based
access control
Encryption
Auditing
License PaidPaid/UnpaidPaid/Unpaid Paid/Unpaid

Prevent unauthorized access
Keep Data Integrity
what happen on system

Installing….

User Authentication

Identify who send the request

built-in users
elastic kibana logstash_system

ADD More User
PUT localhost:9200/_xpack/security/user/trungdo
{
"full_name" : "Trung Do",
"email" : "trung.do@axonactive.com",
"password" : "axonvn",
"roles" : [ "admin", "superior", "kibana_user" ],
"metadata" : {
"workingYear" : 5
},
"enabled": true
}

How authentication works
Native
(Basic Auth)
LDAP
file
(Basic Auth)
Active
Directory
PKI Custom
Realms

Realms chain
Native file LDAP1 LDAP2

xpack.security.authc:
realms:
file: //id of realm
type: file //type of realm
order: 0 //order in chain
native:
type: native
order: 1
ldap1:
type: ldap
order: 2
enabled: false
url: 'url_to_ldap1'
…
ldap2:
type: ldap
order: 3
enabled: false
url: 'url_to_ldap2'
Configure realms
chain
elasticsearch.yml

Authorization

❖ Identify permission to execute request
❖ Support by Role Based Access Control (RBAC)

Secured Resource
RBAC
Role
Privilege
Permissions
User

Secured Resource
Object need to be restricted accessity
Indices Document Field
User Cluster

Privilege
❖ One or group of actions user can execute
❖ Two types: Cluster and Indice
all read create index delete
See full privileges: https://www.elastic.co/guide/en/x-pack/5.4/security-privileges.html
all monitor manage_watcher manage_security
Indice privilege
Cluster privilege

Permission
Set of privileges on a secured resource
read on kibana
indices
monitor on cluster
write on kibana
indices
kibana system permission

Built-in Role
superuser kibana_user kibana_system logstash_system
monitoring_user reporting_user watcher_admin watcher_user machine_learning_user
See more at: https://www.elastic.co/guide/en/x-pack/5.4/built-in-roles.html

ADD ROLE
POST localhost:9200/_xpack/security/role/scrum_admin
{
"run_as": [ ... ],
"cluster": [ ... ],
"indices": [ ... ]
}
//submit request on behalf of other user
//privilege on cluster
//permission entry on indices

ADD ROLE
POST localhost:9200/_xpack/security/role/scrum_admin
{
"run_as": [ "khangvu" ],
"cluster": [ "monitor" ],
"indices": [
{
"names": [ "scrum*" ],
"privileges": [ "read" ],
"field_security" : {
"grant" : [ "category", "@timestamp", "message" ]
},
"query": "{"match": {"event.type": "technical"}}"
}
]
//scrum_admin can submit request for khangvu user
//can monitor cluster
//name of targeted-indices: all indexes started with scrum
//privilege on targeted-indices
//allowed field to access
//list of targeted-document

Auditing
❖ Audit activities/events that occur in the system
❖ Output to logfile or index
❖ Enable this feature by
xpack.security.audit.enabled to true in elasticsearch.yml.
See more at https://www.elastic.co/guide/en/x-pack/5.4/auditing.html

migration with logstash

Logstash is data collector pipeline

How ?
❖ Event processing pipeline
❖ Has three stages:
➢ Input
➢ Filter
➢ Output
❖ Has bunch of plugins for you to play with

usecase - Collect rdbms to es server
Database Logstash filter ElasticSearch
Store Read
Kibana

Exercise 1 (3 points):
- Install ElasticSearch, Kibana
- Import ~5000 films to Elastic
- Install & configure User and Role with X-Pack
- Query films by some criterias on Kibana
- Install and configure Logstash
- Filter films by Logstash
- Import CSV File to ElasticSearch
- Query new films by some criteria

How to run the exercises
● Register with speaker
● Divide by group of 4
● Time for each group: 45 minutes
● Reach the checkpoint to get score
● Gifts:
○ First finish/highest score team - T-shirt
○ Runner-up - Bag
○ Rest - Paper & Pen

[DevDay2018] PLAY & LEARN with Elasticsearch Workshop - By: Thang Luong and Trung Do, Developer at Axon Active Viet Nam

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a [DevDay2018] PLAY & LEARN with Elasticsearch Workshop - By: Thang Luong and Trung Do, Developer at Axon Active Viet Nam

Similar a [DevDay2018] PLAY & LEARN with Elasticsearch Workshop - By: Thang Luong and Trung Do, Developer at Axon Active Viet Nam (20)

Más de DevDay.org

Más de DevDay.org (20)

Último

Último (20)

[DevDay2018] PLAY & LEARN with Elasticsearch Workshop - By: Thang Luong and Trung Do, Developer at Axon Active Viet Nam