Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Embulk makes Japan visible

4.074 visualizaciones

Publicado el

Open data visalization with Embulk

Publicado en: Datos y análisis
  • Inicia sesión para ver los comentarios

Embulk makes Japan visible

  1. 1. Embulk makes Japan visible Kai Sasaki Treasure Data Inc.
  2. 2. Who am I? • Kai Sasaki (@Lewuathe) • Treasure Data Inc • Maintaining and improving
 Hadoop infrastructure • Hadoop, Spark contributor
  3. 3. Topic • What is Embulk? • Embulk ☓ GeoJSON • DATA.GO.JP (http://www.data.go.jp/) • DEMO • Conclusion
  4. 4. What is Embulk? • Parallel bulk data loader • using plugins • to make data integration relaxed http://www.embulk.org/docs/
  5. 5. http://www.slideshare.net/frsyuki/embulk-56197273/4
  6. 6. Plugins http://www.slideshare.net/frsyuki/embuk-making-data-integration-works-relaxed/12
  7. 7. Plugins http://www.embulk.org/plugins/
  8. 8. Embulk ☓ GeoJSON • GeoJSON is a format for encoding geographic data structures { “type”: “FeatureCollection”, “features”: [ { “type”: “Feature”, “geometry”: { “type”: “Point”, “coordinates”: [37.0, 128.4] }, “properties”: { “name”: “Point A” } } ] }
  9. 9. Embulk ☓ GeoJSON https://github.com/benbalter/dc-wifi-social/blob/master/bars.geojson
  10. 10. Embulk ☓ GeoJSON • embulk-formatter-geojson
 https://rubygems.org/gems/embulk-formatter-geojson • Convert any type of source data (csv, tsv, json msgpack etc) supported by input plugin into GeoJSON format. $ embulk new ruby-formatter …
  11. 11. Embulk ☓ GeoJSON id,name,population,… 1,Tokyo,1000,… 2,Osaka,800,… template.geojson { “id”: 1, “properties”: { “name”: “Tokyo”, “population”: 1000 }, “geometry”: 
 <From template.geojson> }
  12. 12. embulk-formatter-geojson $ embulk gem install embulk-formatter-geojson $ cat config.yml … out: type: file formatter: type: geojson template_file: /path/to/template.geojson identifier: “id" … $ embulk run config.yml
  13. 13. DATA.GO.JP http://www.data.go.jp/
  14. 14. DEMO http://www.lewuathe.com/opendata/
  15. 15. d3.json(url, function(error, geoJp) { svg.selectAll("path") .data(geoJp.features) .enter().append(“path") .on("mouseover", function(d) { $("#description").text(d.properties["name"]); }) .attr("class", function(d) { return d.id; }) .attr("d", geopath) .attr("fill", function(d) { var prop = d.properties[“population”]; return colors[prop]; }); }); • d3.js (https://d3js.org/)
  16. 16. d3.json(url, function(error, geoJp) { svg.selectAll("path") .data(geoJp.features) .enter().append(“path") .on("mouseover", function(d) { $("#description").text(d.properties["name"]); }) .attr("class", function(d) { return d.id; }) .attr("d", geopath) .attr("fill", function(d) { var prop = d.properties[“population”]; return colors[prop]; }); }); • d3.js (https://d3js.org/) Embedded Properties
  17. 17. Conclusion • Embulk can be yet another format converter • GeoJSON as a container including 
 data and topology • DATA.GO.JP provides various type of open data
  18. 18. Thank you!

×