Geospatial data structures in Elasticsearch and Apache Lucene have been evolving and improving at a rapid pace. Learn how Elastic has simplified the geospatial indexing, search, and analytics use case by advancing the state of geo in Apache Lucene and Elasticsearch. From new spatial indexing strategies and performance improvements in Lucene to changes in geospatial field mappings in Elasticsearch, take a journey through the world of spatial and multi-dimensional indexing and search in the Elastic Stack.

1. 1
Nicholas Knize, Ph.D.
Elasticsearch & Apache Lucene PMC Member / Committer
@nknize
February 28, 2019
Geospatial Advancements in
Elasticsearch & Apache Lucene

2. Geo Search
Topics
2
Geospatial Indexing, Search, and Data Structures
1 Geo Field Mappings
2 Geo Indexing
3

3. Mappings
Geo Field Types

4. 4
PUT crime/incidents/_mapping
{
“properties” : {
“location” : {
“type” : “geo_point”,
“ignore_malformed” : true,
}
}
}
define
geo_point mapping

5. POST crime/incidents
{
“location” : { “lat” : 41.12, “lon” : -71.34 }
}
5
insert
geo_point mapping
POST crime/incidents
{
“location” : “41.12, -71.34”
}
POST crime/incidents
{
“location” : [[-71.34, 41.12], [-71.32, 41.21]]
}

6. 6
define
geo_shape mapping
PUT police/precincts/_mapping
{
“properties” : {
“coverage” : {
“type” : “geo_shape”,
“ignore_malformed” : false,
“tree” : ”quadtree”,
“precision” : “5m”,
“distance_error_pct“ : 0.025,
“orientation” : “ccw”,
“points_only” : false
}
}
}

7. 7
insert
geo_shape mapping
POST police/precincts/
{
“coverage” : {
“type” : “polygon”,
“coordinates” : [[
[-73.9762134, 40.7538588],
[-73.9742356, 40.7526327],
[-73.9656733, 40.7516774],
[-73.9763236, 40.7521246],
[-73.9723788, 40.7516733],
[-73.9732423, 40.7523556],
[-73.9762134, 40.7538588]
]]
}
}

8. • Shapes are parsed using OGC and ISO standards definitions
• OGC Simple Feature Access
• ISO Geographic information — Spatial Schema (19107:2003)
• Supports the following geo_shape types
• Point, MultiPoint
• LineString, MultiLineString
• Polygon (with holes), MultiPolygon (with holes)
• Envelope (bbox)
geo_shape mapping
8
insert

9. 9
geo_point mapping
Pre 5.0

10. 10
geo_point mapping
5.0+

11. 11
geo_shape mapping
Pre 6.6

12. 12
geo_shape mapping
6.6+

13. ‹#›
Geo Indexing
13

14. 14
geo_point indexing
2.x term/postings encoding
term
postings (doc
ids)
1 1, 2, 3, 4, 5
10 1, 2, 4
11 3, 5
100 1
101 2, 4
111 3, 5
1000 2
1010 4
1011 3
1110 3
1111 5

15. 15
geo_point indexing
5.0 - “points” data structure - (Bkd-tree)

16. 16
geo_point indexing
5.0 - “points” data structure - (Bkd-tree)

17. 17
geo_point indexing
performance improvements

18. 18
geo_shape indexing
pre 6.6 - terms/postings encoding
• Max tree_levels == 32 (2 bits / cell)
• distance_error_pct
• “slop” factor to manage transient
memory usage
• % of the diagonal distance
(degrees) of the shape
• Default == 0 if precision set (2.0)
• points_only
• optimization for points only shape
index
• short-circuits recursion

19. 19
geo_shape indexing
6.6+ - “triangle” encoding (Bkd-tree)
• LatLonShape - new Lucene Field Type (Lucene 7.6)
‒ Lat/Lon geometry represented using Triangular Irregular Network (TIN)
Quad Tree decomposition
1,105,889 quad terms at
3 meter spatial resolution
TIN decomposition
8 triangle terms at
~1.11 centimeter spatial resolution

20. 20
geo_shape indexing
6.6+ - “triangle” encoding (Bkd-tree)
• LatLonShape - new Lucene Field Type (Lucene 7.6)
‒ Lat/Lon geometry represented using Triangular Irregular Network (TIN)

21. 21
geo_shape indexing
Performance - Nightly Benchmarks

22. ‹#›
Geo Search
22

23. 23
geo_point search
Pre 5.0 - terms/postings encoding
• Spatial Queries
• BoundingBox, Distance,
DistanceRange, Polygon
• PRECISION_STEP controls number
of query terms (must match with
index)
• TwoPhaseIterator
• Delays boundary confirmation so
other query (filters, conjunctions)
can pre-filter

24. 24
geo_point search
5.0+ - “points” encoding (Bkd-tree)
Leaf cell is fully within polygon
(salmon) - return all docs
Leaf cell crosses the boundary
(gray) - two-phase check
1
2

25. 25
geo_point search
5.0+ - performance improvements

26. 26
geo_shape search
capabilities
• Supports the following geo_shape types
‒ Point, MultiPoint
‒ LineString, MultiLineString
‒ Polygon (with holes), MultiPolygon (with holes)
‒ Envelope (bbox)
• Supports relational queries
‒ INTERSECTS, DISJOINT, WITHIN, CONTAINS (PrefixTrees only)

27. 27
geo_shape search
Pre-6.6 - terms/postings encoding
Recursively Traverse
Query terms
1 2
Collect DocIDs from
Postings based on
requested relation

28. 28
geo_shape search
6.6+ - LatLonShape (B-kd Tree)

29. 29
geo_shape search
6.6+ - LatLonShape (B-kd Tree)

30. 30
geo_shape search
6.6+ Performance - Nightly Benchmarks

31. 31
geo_shape search
6.6+ Performance - Nightly Benchmarks

32. 32
geo_shape search
6.6+ - Performance

33. Thank You
● Web : www.elastic.co
● Products : https://www.elastic.co/products
● Forums : https://discuss.elastic.co/
● Community : https://www.elastic.co/community/meetups
● Twitter : @elastic