The Beauty of Mapping Big Data
•Descargar como PPTX, PDF•
0 recomendaciones•248 vistas
Data Lens is a cloud-based API toolkit for developing web-based geographical data visualizations. Raw data is queried via a REST API from the Data Lens cloud, and transformed and aggregated to provide input to the JavaScript API, which can render map objects, vector shapes and heat maps. This talk is about the beauty behind big data represented by Data Lens heat maps, the beauty behind its accuracy and its engineering.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Similar a The Beauty of Mapping Big Data
Similar a The Beauty of Mapping Big Data (20)
Último
Último (20)
The Beauty of Mapping Big Data
- 1. The Beauty of Mapping Big Data Stoimen Popov R&D Lead, Product Innovation Team, HERE IoT| Dec 05, 2016
- 2. HERE is the Open Location Platform company • Provides mapping services and location intelligence across the automotive, enterprise and internet industries • Employs 7,000 people in 56 countries • Produces maps for every country on earth • Enables four out of five in-car navigation systems in North America and Europe • Enables mobile, web and enterprise solutions for global industry leaders
- 3. Agenda 01. Data Lens 01.1 Cloud Storage 01.2 Rest API & Query Language 01.3 JS API 02. Server-Side Clustering 03. Data Lens Heat Maps 03.1 Averaged Value 03.2 Alpha Mask by Density 03.3 Value Based Heat Map
- 4. 01 Data Lens
- 5. Data Lens is a cloud-based API toolkit for developing sophisticated visualizations of geographically referenced data, accessible in a web browser. Data is queried via a REST API from the Data Lens cloud, and transformed and aggregated to provide input to the JavaScript API, which renders the visualization.
- 6. Cloud Storage • Data storage • Integration with HERE account • Data enrichment
- 7. Data Lens REST API • Authentication • Data Upload • Datasets • Queries • Query Language • Access Management • Data Reprojection • ...
- 8. Data Lens Query Language • JSON formatted queries • Similar to prepared statements in SQL for later execution • Only dataset owners can create queries for a dataset • Protect sensitive datasets
- 9. Presentation title | Month 00, 20169 © 2016 HERE | HERE Internal Use Only Data Lens Query Language
- 10. Data Lens JS API • Data Lens JavaScript API is a module of HERE Maps API for JavaScript and connects it to the Data Lens REST API • Provides data-driven styling of data on a map • Solves non-trivial tasks like tiling, caching and rasterizing data
- 11. Presentation title | Month 00, 201611 © 2016 HERE | HERE Internal Use Only Data Lens Heat Map Dataset with 11M points of data about taxi rides in NY City
- 12. Presentation title | Month 00, 201612 © 2016 HERE | HERE Internal Use Only Data Lens Street Shapes The data is geo-enriched to street geometry in NY City
- 13. Presentation title | Month 00, 201613 © 2016 HERE | HERE Internal Use Only Data Lens ZIP Shapes The data is geo-enriched to ZIP code boundaries in NY City
- 14. Presentation title | Month 00, 201614 © 2016 HERE | HERE Internal Use Only Data Lens Buildings The data is geo-enriched to building geometries in NY City
- 15. 02 Server-Side Clustering How to deal with BIG DATA on the server?
- 16. Data Tiling & Grouping in Pixel Space • Data tiling reduces the amount of data received by the client • Data Lens groups the data points per tile pixel
- 17. 03 Data Lens Heat Maps • Averaged value • Alpha mask by density • Value-based heat map
- 18. Presentation title | Month 00, 201618 © 2016 HERE | HERE Internal Use Only Averaged heat map
- 19. Presentation title | Month 00, 201619 © 2016 HERE | HERE Internal Use Only Averaged heat map alone, and with an applied alpha mask
- 20. Big Data can be visualized in many ways … Heat Maps Server-Side Clustering Hybrid Clustering (Server-side and client-side)
- 21. https://developer.here.com Docs & API Reference Tech Examples Industry Examples Develop Code Examples Data Lens APIs • Detailed Story • JS/HTML Code • Query Definitions • Styles & UI Develop Code Examples Data Lens APIs • Data-Driven Styling • Server-Side Clustering • Hybrid Clustering • …and more! Develop Data Lens • Getting Started • Tutorial • Developer Guide • API Reference
Notas del editor
- HERE is the Open Location Platform company, providing mapping, services and location intelligence across the automotive, enterprise and internet industries + who am I. Few slides in the beginning and at the end (perhaps merge the last slides)
- Few words on what I’m going to talk … and the structure of the talk itself.
- HERE is the Open Location Platform company, providing mapping, services and location intelligence across the automotive, enterprise and internet industries + who am I. Few slides in the beginning and at the end (perhaps merge the last slides)
- We allow our users to upload their own big data as a CSV. Data is protected (public/private) Explain data enrichment
- REST API allows us to perform various operations on our data. Upload (CSV) Manage datasets. Create, list, delete. Get schema, manage schema, upload files, etc. Manage queries (come again on this topic later on in the slides). Managing queries, etc. Publishing (protection of sensitive information) Re-project Data (Lat & Lon to UTM) Enriching data (Anchoring, enrich data of HERE platform to users data)
- The query is not a one-off action like a query in SQL. It corresponds more to the concept of prepared statements in the SQL world: You create a query for later execution; separating sending the query's source code and the actual retrieval. Only dataset owners can create queries for a dataset. They can then decide whether to make that query public or whether to keep it private, which means that only the query owner can send the query in question. This privacy option allows you to upload a sensitive dataset (for example, records with user information), make a less sensitive query (for example, adding the number of users by country in the result) and then only make the less sensitive data more widely available by only publishing that specific query.
- Query language example just to give a notion what it is and what to expect working with it. An example of a query. How data is aggregated and fetched is described with JSON as a query. Query is saved to the backend and executed (/data) every time the user wants to retrieve data.
- The main features of the Data Lens JavaScript API: Data Lens REST API connector ( Service ) for HERE Maps API Markers, clusters and primitives with styling parameterized by data and zoom level (data-driven styling) Value-based heat map with density alpha mask
- Before continuing with the heat map example, few works of what the JS API is capable of doing, not to leave the impression we’re doing only heat maps. Give a notion of what is it in the picture 11M points of Taxi data over NY
- Same data but anchored against HERE street shapes
- Same data anchored against ZIP code boundaries in NY
- Building shapes (note the number of the shapes) - bit of a note about vector tiles (shapes), protobuf and so on.
- What is pixel space and geographical/cartographical space. Grouping geo points into one pixel and we serve x,y, value and count Some words about valid use cases
- What are Data Lens heat maps so special …. Just few words on our different heat maps techniques since later on in the talk will come more detailed info Averaged (weighted, and few words about what weighted average is) and when it is better vs. sum Alpha mask by density – what it is and when it is to be used Value based heat map (averaged with applied alpha mask)
- A value-based heat map is created with a density alpha mask using the KDE method, but displayed as a density map with an applied color scale on its own. The bandwith parameter influences the perceived smoothness of the heat map surface. For smooth transition onto the base map or background, the density's colorScale can either have transparency on the low end on its own, or be applied as an alphaScale . In any case, the resulting output color scale is univariate. The Data Lens query language allows you to group data rows by buckets (in most cases a bucket is a 1x1 pixel) and aggregate row values for each bucket. Normally this type of query is used to draw a heat map. The simplest heat map can be instantiated as follows:
- A value-based heat map is created with a density alpha mask using the KDE method, but displayed as a density map with an applied color scale on its own. The bandwith parameter influences the perceived smoothness of the heat map surface. For smooth transition onto the base map or background, the density's colorScale can either have transparency on the low end on its own, or be applied as an alphaScale . In any case, the resulting output color scale is univariate. The Data Lens query language allows you to group data rows by buckets (in most cases a bucket is a 1x1 pixel) and aggregate row values for each bucket. Normally this type of query is used to draw a heat map. The simplest heat map can be instantiated as follows:
- constantly updated examples