Presented by Ishan Chattopadhyaya, LucidWorks This talk is on the technical aspects of a new OpenStreetMap geocoder based on Apache Solr & Lucene. Recent changes to Apache Lucene and Apache Solr (4.0 and onwards) have seen a marked improvement in the spatial search capabilities. Also, its improved support for distributed storage and search, via the SolrCloud mode, makes applications using Solr scale easily. OpenStreetMap's current geocoder, Nomainatim, is based on Postgresql/PostGis. Some benefits of using Solr (as compared to a database system like Postgres) for building a geocoder, is robust partial text search, analysis in various languages (stemming, tokenization, stop words etc.), spell check, faceting, highlighting etc. Through this presentation, the author intends to bring out an appreciation for a Solr based geocoder.

2. Ishan Chattopadhyaya
LucidWorks
OpenStreetMap Foundation
Twitter: @ichattopadhyaya, OSM: chatman

3. What is OpenStreetMap?
●
●
Wikipedia of GeoData
OpenStreetMap is a project aimed squarely
at creating and providing free geographic
data such as street maps to anyone who
wants them.

4. State of OSM
●
Commercial competitors
Google Maps
– Bing Maps
http://tools.geofabrik.de/mc/
–
●

5. The OpenStreetMap Software Stack

6. What is a Geocoder?
●
●
Input: raw query
Output: geocoordinates

7. Nominatim
●
http://nominatim.openstreetmap.org/

8. Goals for the new Geocoder
●
Search for:
Cities and towns
– Streets
– Address points
– Places of Interest, Businesses, Amenities, Attractions etc.
Reverse geocoding
–
●
●
Support for fuzzy queries

9. Good changes in Lucene/Solr 4.x
●
Support for indexing polygons
–
●
RecursivePrefixTree indexing
Special spatial search predicates
Contains
– IsWithin
– Intersects
– Etc.
Reference: David Smiley's LuceneRevolution presentation
–
●
●
SolrCloud mode for distributed indexing/searching

10. Architecture
www.
Geocoder.
in
Indexer
Planet dumps
Solr
API Layer

11. Indexing: OSM Data format
●
Node
“A node defines a single geospatial point using a latitude
and longitude.”
Way
–
●
“A way is an ordered list of between 2 and 2,000 nodes.
Ways are used to represent linear features (vectors), such
as rivers or roads.”
Relation
–
●
–
“A Relation is an all-purpose data structure that documents
a relationship between two or more other objects.”

12. Indexing: Facts and figures
●
Number of OSM Nodes in the database = 2071039612
●
Number of OSM Ways in the database = 202570637
●
Number of OSM Relations in the database = 2217240

13. Indexing: Schema
name
Ireland
geo
Landsdowne Street
admin2
level
s
<shape>
admi
n3
admin admin5 admin6
4
Dublin
County
Dublin
popularity
admin7
street
st_type
Ballsbridge
Lansdo
wne
Street

14. Indexing: Schema
name
Ireland
geo
popularity
Dublin
admin2
level
6
<shape>
1
admi
n3
admin admin5 admin6
4
Dublin
County
Dublin
admin7
street
st_type

15. Indexing: Schema (POIs)
name
Ireland
geo
Ballsbridge Hotel
admin2
category
hotel
<shape>
admi
n3
admin admin5 admin6
4
Dublin
County
Dublin
admin7
Ballsbridge
street
st_type

16. Searching
Raw query
Classifier
Classifications
Validator
Valid classifications
Structured location + geocodes
Geocoder (lookup)

17. Searching: Classification
Query
Tokenizer
Shingles
Bloom Filters
Classifications

18. Searching: Classification
●
●
●
Query
Tokenizer
Shingles
Bloom Filters
Classifications
Query= “hotels near lansdowne rd dublin”
Shingles: hotels, near, lansdowne, rd, dublin,
hotels near, near lansdowne, lansdowne rd,
rd dublin, .., hotels near lansdowne rd dublin

19. Searching: Classification
●
●
Query
Tokenizer
Shingles
Bloom Filters
Classifications
hotels, near, lansdowne, rd, dublin, hotels near, near lansdowne, ..
hotels
Cat
Match
A2
A4
A5
Streets

20. Searching: Classification
●
●
Query
Tokenizer
Shingles
Bloom Filters
Classifications
hotels, near, lansdowne, rd, dublin, hotels near, near lansdowne, ..
dublin
Cat
A2
A4
A5
Streets
Match
Match

21. Searching: Classification
●
●
Query
Tokenizer
Shingles
Bloom Filters
Classifications
hotels, near, lansdowne, rd, dublin, hotels near, near lansdowne, ..
lansdowne
Cat
A2
A4
A5
Streets
Match
Match

22. Searching: Classifications
●
●
Query = “hotels near lansdowne rd dublin”
Classifications:
hotels = category
lansdowne = admin5
lansdowne = street
dublin = admin5
dublin = street

23. Searching: Classifications
●
●
●
Query = “hotels near lansdowne rd dublin”
Classifications:
hotels = category
lansdowne = admin5
lansdowne = street
dublin = admin5
dublin = street
Possible permutations:
C.5.5
C.S.5
C.5.S
C...5
C.5..
etc.

24. Searching: Solr Query
●
●
Query = “hotels near lansdowne rd dublin”
Possible permutations:
C.5.5: +level:5 +admin5:lansdowne +admin5:dublin
C.S.5: +level:s +street:lansdowne +admin5:dublin
C.5.S: +level:s +street:dublin +admin5:lansdowne
C...5: +level:5 +admin5:dublin
C.5..: +level:5 +admin5:lansdowne
etc.

25. Searching: Solr Query
●
●
Query = “hotels near lansdowne rd dublin”
Possible permutations:
C.5.5: +level:5 +admin5:lansdowne +admin5:dublin
C.S.5: +level:s +street:lansdowne +admin5:dublin
C.5.S: +level:s +street:dublin +admin5:lansdowne
C...5: +level:5 +admin5:dublin
C.5..: +level:5 +admin5:lansdowne
etc.

26. Searching: Solr Query
●
●
Query = “hotels near lansdowne rd dublin”
Possible permutations:
C.5.5: +level:5 +admin5:lansdowne +admin5:dublin
C.S.5: +level:s +street:lansdowne +admin5:dublin
C.5.S: +level:s +street:dublin +admin5:lansdowne
C...5: +level:5 +admin5:dublin
C.5..: +level:5 +admin5:lansdowne
etc.
"POINT (-6.232063,53.333833)"

27. Searching: Searching for POIs
●
●
●
Query = “hotels near lansdowne rd dublin”
Query = “hotels near” near "POINT (-6.232063,53.333833)"
Solr query:
fl=*,score
sort=score asc
q={!geofilt score=distance filter=false
sfield=geo pt= 53.333833,-6.232063 d=10}
fq=+category:hotel

28. Searching: Searching for POIs

29. Challenges: Indexing
●
Street Associativity
●
Incomplete polygons

30. Challenges
●
Handling Updates
●
Data validation

31. Distributed Search
●
Need for distributed search?
●
Geographical partitioning

32. Conclusion
●
http://www.geocoder.in/
●
Twitter: @ichattopadhyaya