Open Distro for ElasticSearch and how Grimoire is using it. Madrid DevOps October 2019

1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Javier Ramirez y Jose Manrique
@supercoco9 @jsmanrique
Getting started with Open Distro for Elasticsearch
And how Bitergia is using it
Madrid DevOps

2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
My personal journey into full-text search (and log processing)
Problem
Solution
SELECT LIKE %
and SOUNDEX are
not good enough
Before 2001
The fuzzy years
Say bye to
transactions and hi
to data corruption
with MyISAM tables
I need to support multi
language, and speed,
and to search by full text
and facets, and rankings,
column weights and
excerpts. I want to be
like Altavista Google
2001-2005
The Lucene epiphany
Embrace JAVA
Hello Lucene (and
Nutch)
I want full text
search for
dynamic
languages
Ruby FTW
2006-2009
The anything but
Java golden age
Sphinx Search
* Spoke about
Sphinx at several
events
My data-sets start growing to
multiple servers
I want to index anything, not just
content from my database
I would like to have the power of
Lucene, without the pain of JAVA
(python and JavaScript are cool), and
the excess of configuration
2010-2013
The sol4r vs elastic civil wars
Sol4r and Elastic are excellent choices
to tame the raw power of Lucene and
make It more approachable
PostgreSQL good enough for many
things. Great if you need GIS
* Spoke about PostgreSQL full- text
search at PGConf
As a DevOps convert, apart
from full-text search, I want
operational search for the
myriad logs of my
microservices.
I want to have pretty and
powerful real-time
dashboards, but either they
are very SQL/Business-
oriented or very low-level
monitoring oriented.
2014-2018
The ELK take over
With LogStash and Kibana
Elasticsearch becomes not
just my search engine, but
also my log analytics engine.
* Spoke about Logstash, then
about ELK or ExK for log and
big data analytics at several
events

3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Source: TechCrunch survey of popular open source software from April’17
• Sometimes referred to as the “ELK
Stack” – Elasticsearch, Logstash &
Kibana
• Distributed search and analytics
engine built on Apache Lucene
• Easy ingestion and visualization
• Developed in Java
What is Elasticsearch?

4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Machine data driving Elasticsearch growth
IT & DevOps:
Databases, Servers,
Storage, Networking
Increase in IoT and Mobile
Devices: Gaming, Sensors,
Web Content
Cloud-based
architectures
Machine-generated data is growing than business data… Logs, logs, and more logs
Source: insideBigData—The Exponential Growth of Data, February 16, 2017

5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How we think about OSS licensing
and distribution?
Base OSS
free of
proprietary code
Keep commercial
software on top
of OSS separate
Allow anyone
to innovate
on OSS
Don’t change
licensing or
distribution
midstream

6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
An Apache 2.0-licensed
distribution of Elasticsearch
enhanced with enterprise-grade
security, alerting, SQL, and more

7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Enterprise-grade
Delivering security
and advanced capabilities
such as alerting, SQL,
and cluster diagnostics
100% open source
Providing you the
freedoms, so you can
freely view, use, change,
and distribute the code
Community-driven
Providing individuals
and organizations the
freedom to easily contribute
changes to the distro
Benefits of Open Distro for Elasticsearch

8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Flexible deployment options
• Docker
• RPM
• Debian

9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Simple to get started
Visit the
website
Download the
Elasticsearch
and Kibana
packages
Load and
query data
1 2 3

10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Open Distro for Elasticsearch - Features
Security Alerting SQL Performance Analyzer
Achieve encryption in-
flight, fine-grained access
control, audit logging,
and compliance
Monitor your data and
send automatic alerts on
any changes in your data
Easily interact with your
Elasticsearch cluster and
extract insights using the
familiar SQL query syntax
Get deep visibility into
system bottlenecks even
when your Elasticsearch
cluster is under duress.

11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security
Keep your data secure
Encryption
Keep your data secure when in transit
Authentication
Leverage your existing authentication infrastructure
RBAC
Granular access control over user actions on your
cluster
Audit logging
Track and record all user actions and meet HIPAA, and
PCI compliance

12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Request with
credentials
Access control flow for RBAC
Authc — via basic HTTP auth, LDAP, AD, SAML, web tokens, SSL
Authz — Backend identities mapped to Open Distro roles
Permissions — allow a role to perform an action against a
cluster/index/document/field
Action groups — Groups of permissions
Authc
provider
Authc
Request with
user/backend
roles
Roles and
permissions
Authz
Response

13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kibana multi-tenancy
Group A Group B
Group B permissionsGroup A permissions
Index
1
Index
2

14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Alerting
Receive alerts on your data
Create monitors
Query the data you want to and receive alerts on it
Customize alert conditions
Define alerting threshold and severity for multiple
trigger conditions
Get notifications
Built-in integrations for webhook and Slack to get
notified on the channels you use
View alerts
All alert executions are indexed for easy tracking
and visualization

15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SQL Support
Query data with SQL
Comprehensive SQL support
Supports over 40 functions, data types, and
commands including join support
Translate SQL to JSON
Create JSON using SQL to configure
sophisticated access control policies
Use existing tools
Provides a JDBC driver so you can use a variety of
business intelligence, analytics, and ETL tools

16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Performance Analyzer
Get deep diagnostic insights into your cluster
Identify bottlenecks across the stack
Provides a powerful REST API for querying
Elasticsearch metrics to diagnose issues across stack
Runs independent of your cluster
Perform diagnostics even if the cluster is under
duress
Analyze hundreds of data points
Supports over 60 metrics across 10 dimensions for
instrumentation of your cluster health

17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
PerfTop CLI
• Provides pre-configured
dashboards for analyzing
cluster, node, and shard
performance
• Custom JSON templates
to create the dashboards
to diagnose your cluster
performance

18. Bitergia helps companies and
organizations with understanding and
improving software development
projects that matter to them

19. Bitergia Analytics

20. ● What is being done in the analyzed projects?
● How many active projects do I contribute to?
● What’s developers engagement level?
● What is being modified and what’s left untouched for too long?
Activity
(what?)
● Who are the contributors to the analyzed projects?
● Where are my developers? Where do they come from?
● Who are my core, regular and casual developers?
● What’s the talent rotation and retention level?
Community
(who?)
● How fast are projects analyzed performing?
● How are we dealing with issues and merge requests?
● Where are the bottlenecks?
● How are we dealing with the backlog?
Performance
(how?)

21. ● What is being done in the analyzed projects?
● How many active projects do I contribute to?
● What’s developers engagement level?
● What is being modified and what’s left untouched for too long?
Activity
(what?)
● Who are the contributors to the analyzed projects?
● Where are my developers? Where do they come from?
● Who are my core, regular and casual developers?
● What’s the talent rotation and retention level?
Community
(who?)
● How fast are projects analyzed performing?
● How are we dealing with issues and merge requests?
● Where are the bottlenecks?
● How are we dealing with the backlog?
Performance
(how?)

22. uber.biterg.io

23. 100% free, open
source software

24. GrimoireLab

25. github.com/chaoss/grimoirelab

26. Because Kibana
Why Elasticsearch?

27. But we ended (soft) forking it ...

29. Alerts
Multitenancy
Roles based access to data
and ...
Customers requests ...

30. 100% free, open
source software

33. Alerts
Multitenancy
Roles based access to data
and ...
Includes ...

34. 100% free, open
source software

36. gitlab.com/jsmanrique/awsome-grimoirelab
For you, to play ...

37. Integrate (latest) Open Distro for
Elasticsearch into GrimoireLab
github.com/chaoss/grimoirelab/issues/219
What’s next?

38. … and one more thing!

39. alpha.cauldron.io

40. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Mooooooarrrr plugins
• Index management (RFC open now)
• jobs scheduler
• What will you contribute? J

41. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Community and Contributions
Open Distro for Elasticsearch’s success is driven by the community’s
participation, contributions and innovation to the project.
You can follow project discussions, engage with fellow community
members, contribute PRs, file bugs or request a feature at:
Discussion Forums
https://discuss.opendistrocommunity.dev/
Community
https://github.com/opendistro-for-elasticsearch/community/issues

42. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Useful Links
Project Website and Technical Documentation
https://opendistro.github.io/for-elasticsearch/
Source Code
https://github.com/opendistro-for-elasticsearch

43. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
Javier Ramirez y Jose Manrique
@supercoco9 @jsmanrique