Sotm Birmingham Osmose quality assurance.

1. SoTM 2013
Osmose
quality assurance tool
Birmingham 2013-09-06
Frédéric Rodrigo <frederic@carte-libre.fr>
(c)left 2013 - CC-BY-SA v3.0
http://osmose.openstreetmap.fr

2. Context
QA tool
– Since 2008 by Etienne Chové
– Maintained by Jocelyn Jaubert and Frédéric Rodrigo
Sources available on gitorious
– https://gitorious.org/osmose/

3. Ecosystem
No duplicate analysis from KeepRight and Osm
Inspector
But integrate good idea from “minor” other tools
Idea from Osmose integrated into JOSM validator
by don_vip (building intersection, powerline...)

4. Support
France mainland and oversea
HOT activated countries
All full French speaking countries and Quebec
Switzerland, Belgium, Luxembourg and Iceland
Hiring for community to support local server for
new countries

5. The Osmose Stack
Frontend BackendsAPI

6. Backends Input
XML or PBF export
Not all supported country are available at
Geofabrik
OSM-FR generate also export
– http://download.openstreetmap.fr/extracts/
– Diff available
– Need for Osmose

7. Analyser - SAX
Process one primitive at time
Each analyser is a plugin
– Check node, way and relation
– Can report errors
Use external data
– Wikipedia tree species
Running filter on language and country

8. Analyser - SAX
Format: number, date, postcode, wikipedia...
Naming: abbreviation, too many space, uppercase...
Bag tag: ref in name, typo, deprecated...
Ensure multiple tags coherence from wiki rules:
multipolygon, parking lanes...
Inconsistency between tags: toll and fee, opposite
cycleway without oneway...

9. class Structural_UnclosedArea(Plugin):
def init(self, logger):
self.errors[1100] = { "item": 1100, "level": 3,
"tag": ["geom", "fix:imagery"], "desc": T_(u"Unclosed
area") }
def way(self, data, tags, nds):
if "area" not in tags or tags["area"] == "no":
return
if nds[0] != nds[-1]:
return [(1100, 0, {})]

10. Analyser - Osmosis
Osmosis, a lossless schema
Multiple objects checking at once
Geospatial analysis
DB
– Lot of indices
– create/drop DB: at each Osmose run, lake of disk sppace
– Keep DB: Quebec (diff mode)

11. Analyser – Osmosis – Non spatial
Done by topology
Roundabout analyser
– Wrong highway level on roundabout
– Missing oneway on link highway
– Roundabout shortcut
– Roundabout crossing

12. Analyser – Osmosis - Geospatial
Roundabout
– Missing junction=roundabout
– Reverse roundabout
Deadend by highway level
– Small gap in highway level continuity
Power line
– Isolated tower
– Unconnected powerline
– Missing tag power=tower
– Predictive position of unmapped power=tower

13. Analyser – Osmosis - Convergent
Soundex
– Phonetic index
– Index all name, way kind + way name
– Compare numerous name again similar with low
frequency
Automatic typo on tags detection
– Not work

14. Analyser - Merge
OpenData Integration
– Integration vs Importation
– Help contributor to add external data
Row external data set (csv) + Osmosis DB
Mapping
– Column to tag
– Complex mapping
Analyser code is just configuration
– Easy to add new data set

15. Analyser - Merge
Conflation by ref
Conflation by distance
Detect
– OSM object without ref
– OSM object not found in OpenData set
– OpenData object not found in OSM
“OpenData by OSM” as sub product

16. Analyser – Merge
French school
Wikipedia
Level crossing

17. self.csv_encoding = "ISO-8859-15"
self.osmTags = {
"amenity": "post_office" }
self.osmRef = "ref:FR:LaPoste"
sself.osmTypes = ["nodes", "ways"]
self.sourceSRID = "4326"
self.defaultTag = {
"amenity": "post_office",
"operator": "La Poste",
"source": "data.gouv.fr:LaPoste – 01/2013" }
self.defaultTagMapping = {
"ref:FR:LaPoste": "identifiant",
"name": "nom",
"wheelchair": lambda res:
"yes" if self.bool[res["autonome_fauteuil_roulant"]] else
"limited" if self.bool[res["pas_ressaut"]] else
"no" }
self.conflationDistance = 1000

18. Analyser – OSMBin
OSMBin: Osmose proprietary DB format
– Flat storage, access in O(1)
Multipolygon / boundary closed fast checking

19. Analyser – External tools
OpenStreetBug
– Close OSB entries when close it on Osmose
Need support OSM Notes

20. Configuration
Configuration by extract (country and smaller part)
List of analysers
– Language: mainly french specific
– Local way of mapping
– Source available: Bing hires, French cadastre...
Options
– Projection
– Driving side
– Language

21. Boundary clipping
Extract have buffer on boundary
– Duplicate error
– False error (e.g. unconnected)
Clip on exact boundary polygon after analyse

22. About Time
Full run every 2 days
Take 80% of availible CPU
2
50
35
43
67
ïle-de-France - Total time 3h17
Download
Osmosis Import
Indexing
SAX Analyser
Osmosis Analyser

23. Diff mode
Speedup by running analyser only on change
Sax
Osmosis
– “changed” tables : edited nodes, ways, relations and transitive
dependencies
– Keep DB between runs
Not all can be run in diff mode
– Error based on other OSM data (clustering, connection...)
Full run every month to keep safe

24. The Osmose Stack
Frontend BackendsAPI

25. Errors
Backends send errors to centralized Frontend
Errors report are by Analyser and Country (aka
source)
Error file replace all errors of his kind on frontend
– Except on diff mode

26. Error
Source (country, analyser)
Identity
– Class
– Subclass
Metainfo
– Level: 1, 2, 3 as severity
– Item number: display category and subcategory
Tags
– Other classificatio:, by nature of OSM object, highway,
waterway, name...
– What is require for fix: chair, imagery, survey

27. Error fixes
Analyser can propose some error fixes
Add, delete or modify many tag at once
Create a new node with tags

28. The Osmose Stack
Frontend BackendsAPI

29. Frontend - Map
Selector
– Levels of severity
– Items
– Other options available on URL (see API)
Markers
– Status: open, close, false-positive
– JOSM remote
– JOSM fix
– Wiki help
Layers
– Heatmap
– Debug map

32. Stats
Errors listing
History graph
By user
– RSS
Update Matrix

33. France - Level 1
Add new analyser
Natural fix over the time
Heavy fix

34. Haiti – All errors
Correct error with locals

35. France – Cadastre building
overlap
New gap
detection
Heavy fdix

36. French OpenData School
Bugs
OpenData
Update

37. The Osmose Stack
Frontend BackendsAPI

38. API
External fixing tools
Stats
JOSM
– confidential plugin
– “Quality Assurance Tools script”
API doc on Wiki
– URL parameters apply any where in the frontend

39. Hardware
Backends
– osm3.openstreetmap.fr HOT and Quebec
– osm5.openstreetmap.fr France
– osm6.openstreetmap.fr France
– Iceland server: Iceland
Frontend
– osm103.openstreetmap.fr ← osmose.openstreetmap.fr

40. Conclusion
osmose.openstreetmap.fr → o.osmfr.org
https://gitorious.org/osmose/
We need you
– Translation: frontend UI and backend errors
– Run your own Osmose local backend server
– Errors correction!
Long list of analysers request on french Trac
Localise language or country specific analyser
– Main st → Main street