Einführung in Elasticsearch auf der JAX 2015

1. Anwendungsfälle für
Florian Hopf - @fhopf

2. Agenda

3. Vorbereitung

4. Installation
# download archive
wget https://download.elasticsearch.org/
elasticsearch/elasticsearch/elasticsearch-1.4.4.zip
# zip is for windows and linux
unzip elasticsearch-1.4.4.zip
# on windows: elasticsearch.bat
elasticsearch-1.4.4/bin/elasticsearch

5. Zugriff
curl -XGET http://localhost:9200
{
"status" : 200,
"name" : "Anomalito",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "1.4.4",
"build_hash" :
"c88f77ffc81301dfa9dfd81ca2232f09588bd512",
"build_timestamp" : "2015-02-19T13:05:36Z",
"build_snapshot" : false,
"lucene_version" : "4.10.3"
},
"tagline" : "You Know, for Search"
}

6. Document Store

7. Document
{
"title" : "Anwendungsfälle für Elasticsearch",
"speaker" : "Florian Hopf",
"date" : "2015-04-22T10:30:00.000Z",
"tags" : ["Java", "Lucene"],
"conference" : {
"name" : "JAX",
"city" : "Mainz"
}
}

8. Speichern
curl -XPOST http://localhost:9200/conferences/talk/
--data-binary @talk-example.json
{
"_index":"conferences",
"_type":"talk",
"_id":"GqjY7l8sTxa3jLaFx67_aw",
"_version":1,
"created":true
}

9. Speichern
curl -XPOST http://localhost:9200/conferences/talk/
--data-binary @talk-example.json
{
"_index":"conferences",
"_type":"talk",
"_id":"GqjY7l8sTxa3jLaFx67_aw",
"_version":1,
"created":true
}
Index

10. Speichern
curl -XPOST http://localhost:9200/conferences/talk/
--data-binary @talk-example.json
{
"_index":"conferences",
"_type":"talk",
"_id":"GqjY7l8sTxa3jLaFx67_aw",
"_version":1,
"created":true
}
Type

11. Lesen
curl -XGET http://localhost:9200/conferences/talk/
GqjY7l8sTxa3jLaFx67_aw?pretty=true
{
"_index" : "conferences",
[...]
"_source":{
"title" : "Anwendungsfälle für Elasticsearch",
"speaker" : "Florian Hopf",
"date" : "2015-04-22T10:30:00.000Z",
"tags" : ["Java", "Lucene"],
"conference" : {
"name" : "JAX",
"city" : "Mainz"
}
}
}

12. Sharding
●
Aufteilen eines Index in mehrere Teile
●
Default: 5 Shards pro Elasticsearch-Index
●
Mehrere ES-Instanzen bilden einen Cluster
●
Automatische Verteilung auf die Knoten

13. Sharding

14. Sharding

15. Sharding

16. Recap
●
Einfache Speicherung von JSON-Dokumenten
●
Index und Type
●
Sharding für große Datenmengen
●
Verteilung ist First-Class-Citizen

17. Nutzer
●
Atlassian HipChat
●
http://highscalability.com/blog/2014/1/6/how-hipchat-
●
OpenCrowd
●
https://www.elastic.co/use-cases/open-crowd

18. Volltextsuche

21. Suche per Parameter
curl -XGET "http://localhost:9200/conferences/talk/_search
?q=elasticsearch&pretty=true"
{"took" : 73,
[…]
"hits" : {
[…]
"hits" : [ {
[…]
"_score" : 0.076713204,
"_source":{
"title" : "Anwendungsfälle für Elasticsearch",
"tags" : ["Java", "Lucene"],
[…]
} } ]
}
}

22. Query-DSL
curl -XGET "http://localhost:9200/conference/talk/_search" -d'
{
"query": {
"filtered": {
"query": {
"match": {
"title": "elasticsearch"
}
},
"filter": {
"term": {
"tags": "java"
}
}
}
}
}'

23. Query-DSL
client.prepareSearch("conferences")
.setQuery(filteredQuery(
matchQuery("title", "elasticsearch"),
termFilter("tag", "java")))
.execute()
.actionGet();

24. Sprache
curl -XGET "http://localhost:9200/conferences/talk/_search
?q=title:anwendungsfall&pretty=true"
{
[...]
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}

25. Mapping
curl -XDELETE "http://localhost:9200/conferences/"
curl -XPUT "http://localhost:9200/conferences/“
curl -XPUT "http://localhost:9200/conferences/talk/_mapping" -d'
{
"properties": {
"title": {
"type": "string",
"analyzer": "german"
}
}
}'

26. Sprache
curl -XGET
"http://localhost:9200/conferences/talk/_search
?q=title:anwendungsfall&pretty=true"
{
[...]
},
"hits" : {
"total" : 1,
[...]
}
}

27. Was noch?
●
Faceting/Aggregations
●
Suggestions
●
Highlighting
●
Sortierung
●
Pagination
●
...

28. Recap
●
Ausdrucksstarke Suchen über Query DSL
●
Analyzing und Mapping
●
Lucene bietet viele Search-Features

29. Users
●
GitHub
●
http://exploringelasticsearch.com/github_interview.html
●
http://www.elasticsearch.org/case-study/github/
●
StackOverflow
●
http://meta.stackexchange.com/questions/160100/a-new-
search-engine-for-stack-exchange
●
http://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-
stack-overflow/
●
XING
●
http://www.elasticsearch.org/case-study/xing/

30. Flexibler Cache

31. Setup Suche
Anwendung
DB

32. Nur Suche?

33. Queries
Anwendung
DB

34. Ergebnislisting
curl -XPOST "http://localhost:9200/conferences/_search
" -d'
{
"filter": {
"term": {
"tags": "java"
}
}
}'

35. Nutzer
●
Microsoft MSN
●
https://www.elastic.co/elasticon/2015/sf/powerin
g-real-time-search-at-microsoft

36. Geo-SucheGeo-Suche

37. Anwendungen
●
Zeige nächste Filiale
●
Filialsuche
●
Sortierung Kleinanzeigen
●
Filterung auf Nähe
●
Social Media-Analysen

38. Document
{
"title" : "Anwendungsfälle für Elasticsearch",
"speaker" : "Florian Hopf",
"date" : "2014-07-17T15:35:00.000Z",
"tags" : ["Java", "Lucene"],
"conference" : {
"name" : "JAX",
"city" : "Mainz",
"coordinates": {
"lon": "8.275626",
"lat": "50.002090"
}
}
}

39. Mapping
curl -XPUT "http://localhost:9200/conferences/talk/_mapping" -d'
{
"properties": {
[…],
"conference": {
"type": "object",
"properties": {
"coordinates": {
"type": "geo_point"
}
}
}
}
}'

40. Sortierung
curl -XPOST "http://localhost:9200/conferences/_search " -d'
{
"sort" : [
{
"_geo_distance" : {
"conference.coordinates" : {
"lon": 8.403697,
"lat": 49.006616
},
"order" : "asc",
"unit" : "km"
}
}
]
}'

41. Filterung
curl -XPOST "http://localhost:9200/conferences/_search" -d'
{
"filter": {
"geo_distance": {
"conference.coordinates": {
"lon": 8.403697,
"lat": 49.006616
},
"distance": "200km",
"distance_type": "arc"
}
}
}'

42. Recap
●
Elasticsearch kann mehr als Volltext
●
Ausgefeilte Geo-Algorithmen
●
Sortierung nach Distanz
●
Filterung nach Distanz oder Bereich
●
Berechnung von Distanz

43. User
●
FourSquare
●
http://engineering.foursquare.com/2012/08/09/fo
ursquare-now-uses-elastic-search-and-on-a-
related-note-slashem-also-works-with-elastic-
search/
●
Gild
●
http://www.elasticsearch.org/case-study/gild/

44. Logfile-AnalyseLog-Analyse

45. Logfile-Analyse
●
Zentralisierung Logs aus Anwendungen
●
Zentralisierung Logs über Maschinen
●
Auch ohne Zugriff
●
Leichte Durchsuchbarkeit
●
Real-Time-Analysis / Visualisierung
●
Daten für alle!

46. Logfile-Analyse
●
Einlesen
●
Logstash
●
Speicherung
●
Elasticsearch
●
Auswertung
●
Kibana

47. Logfile-Analyse

48. Logstash-Config
input {
file {
path => "/var/log/apache2/access.log"
}
}
filter {
grok {
match => { message => "%{COMBINEDAPACHELOG}" }
}
}
output {
elasticsearch_http {
host => "localhost"
}
}

49. Kibana

50. Recap
●
Einlesen, Anreichern, Speichern von Logevents
●
Zahlreiche Inputs in Logstash
●
Konsolidierung
●
Zentralisierung
●
Auswertung

51. User
●
CERN
●
https://medium.com/@ghoranyi/needle-in-a-
haystack-873c97a99983
●
Bloomberg
●
http://www.elasticsearch.org/videos/using-
elasticsearch-logstash-kibana-techologies-
centralized-viewing-logs-bloomberg/

52. AnalyticsAnalytics

53. Analytics
●
Auswertung auch großer Datenmengen
●
Social Media
●
Data Warehouse
●
Aggregationen auf Feldern
●
Datenkonsolidierung aus unterschiedlichen
Quellen
●
Visualisierung

54. Aggregations
curl -XGET "http://localhost:9200/devoxx/tweet/_search" -d'
{
"aggs" : {
"hashtags" : {
"terms" : {
"field" : "hashtag.text"
}
}
}
}'

55. Aggregations
"aggregations": {
"hashtags": {
"buckets": [
{
"key": "dartlang",
"doc_count": 229
},
{
"key": "java",
"doc_count": 216
},
[...]

56. Aggregations
"aggs" : {
"hashtags" : {
"terms" : {
"field" : "hashtag.text"
},
"aggs" : {
"hashtagusers" : {
"terms" : {
"field" : "user.screen_name"
}
}
}
}
}

57. Aggregations
"key": "scala",
"doc_count": 130,
"hashtagusers": {
"buckets": [
{
"key": "jaceklaskowski",
"doc_count": 74
},
{
"key": "ManningBooks",
"doc_count": 3
},
[...]

58. Visualisierung

59. Recap
●
Auswertung großer Datenmengen
●
Visualisierung
●
Zahlreiche Aggregationen
●
Bucket-Aggregations
●
Metric-Aggregations

60. User
●
Cogenta
●
https://www.elastic.co/use-cases/cogenta
●
The Guardian
●
http://www.elasticsearch.org/blog/using-
elasticsearch-and-logstash-to-serve-billions-of-
searchable-events-for-customers/
●
http://www.infoq.com/presentations/elasticsearch
-guardian

61. Agenda

62. Vielen
Dank!
@fhopf
mail@florian-hopf.de
http://blog.florian-hopf.de

63. Images
●
http://www.morguefile.com/archive/display/685952
●
http://www.morguefile.com/archive/display/2359
●
http://www.morguefile.com/archive/display/615356
●
http://www.morguefile.com/archive/display/914733
●
http://www.morguefile.com/archive/display/826258
●
http://www.morguefile.com/archive/display/170605
●
http://www.morguefile.com/archive/display/181488