"Attend this session to learn how the City of Vancouver uses FME to: • Provide data for both their Open Data and Road Ahead initiatives. • Improve the City's Road Ahead map with more current information and open formats. (Learn how FME is being used to retrieve the source data from SQL Server and Oracle Spatial, and to generate SDF3/SDF2 files and a GeoRSS/RSS feed integrated with Google Street View, which is accessible from the Road Ahead home page. Plus, see how the Road Ahead FME workspace is also published to FME Server where the relevant data format files can be generated on the fly via data streaming services.) • Prepare datasets for the City's OpenData website on a weekly basis. "

1. Vancouver Uses FME to Open
Data to the World
Meng Li, GIS Systems Analyst, City of Vancouver
Jonathan Mark, GIS Manager, City of Vancouver
March 29, 2010

2. We’ll Talk About
 Overview
 The Open Data Initiative and ICIS support
 The Road Ahead Website Upgrade
 Safe software in use at the City
 Open Data Infrastructure Platform
 Workflow, ETL scripts, and deployment
 The Road Ahead Website Upgrade Project
 ETL, Atom, GeoRSS, data streaming experience

3. The Open Data Initiative
 The Open3 Council motion of May 2009
 Open data
 Open standards
 Open source software
 More sharing with ICIS
 Directive to make City’s data freely accessible
subject to
 Security concerns
 Privacy concerns
 Terms of use

4. Open Data Website
 September 2009 launch
 About 50 layers, many hard to find
 Mostly static datasets, limited formats
 FME used for most of initial creation
 January 2010 launch
 About 100 layers, all easy to find
 Most in DWG, KML, SHP with VanMap groupings
 Weekly updating (thanks to FME software)
 Can display in Google Maps or Bing Map easily
 ICIS-specific data groups for GIS & AddressBC
 Demo

5. The Road Ahead Website
 Map in AJAX Viewer of MGE using SDF3 files
 Atom and GeoRSS feed data generated
 Source data: SQL Server and Oracle Spatial
 FME ETL process to update SDF3, SDF2 &
feed data hourly 7am-7pm Mon to Fri
 The Road Ahead feed data being widely used
 Feed data with Street View via data
streaming

6. Safe Software in Use
 FME Desktop
 The City has gradually increased its pool of
licenses to 10
 Being used for IMS, Open Data, Road Ahead,
and a variety of day-to-day data ETL needs
 FME Server
 Acquired in November 2009
 Important foundation component of our GIS
software infrastructure
 Moving Open Data and Road Ahead to Server

7. Open Data Infrastructure
 Requirements of source and target data:
 Public VanMap consistency
 layers/groups specifications (filtering, grouping, naming)
 source formats: MDB, DWG, SDF, SQL Server, Oracle
 Target Data Catalogue in OpenData Website
 formats: DWG, KML/KMZ, SHAPE, CSV, XLS
 naming convention: low casing and no spaces
 file structure: data folders for FTP and Web servers
 flexible for data packaging: via published parameters
 ICIS specific data specifications
 Automated Execution of the Data Extraction
Scripts via Windows scheduler tasks

8. Open Data—Processes/workflow
outputs
VM source Windows
Scheduler
Task
ETL Workspace and scripts
loaded with Python pre-
processing, filtering, routing
& post-processing logic

9. Open Data: one Workbench script
shape
KML
DWG

10. Open Data—Automation
 Scheduler Tasks for ETL Scripts (run weekly)

11. Open Data - Deployment
 All files in KML/KMZ format uploaded weekly
to the Open Data web server
 All files in other formats uploaded weekly to
the public FTP server
 FME ETL Scripts (now Desktop) Updates for
Flexible Deployment

12. Open Data- Challenges
 Data Catalogue Design: groupings and classification
 Required Data Groupings and File Structure
 Naming Conventions: machine and user friendly
 KML Specification Compliance/Validation Issue
 Data fitness of use: attribute requests & selection

13. The Road Ahead Website

14. The Road Ahead—New Features
 Hourly update replacing nightly update:
 FME ETL job runs hourly from 7am to 7pm
weekdays
 update SDF3/SDF2 (for VanMap) and feeds
data
 MapGuide Enterprise AJAX Viewer uses SDF3
 Atom and Geo RSS feed data generated
 The Road Ahead feed data replaced an email
list and is being widely used (including EOC)
 Feed data with Street View via data
streaming

15. The Road Ahead—Data Extract
 Source Data
 SQL Server--point & street-based projects
 Oracle Spatial--street network reference data
 Output files:
 SDF3 file of point & line features with 3 Road Ahead categories
 Atom xml file, and a GeoRSS xml file
 SDF2 files for existing VanMap Road Ahead layers
 Feed xml file with Google Street View (not in production)
 FME script data extract job automation

16. Road Ahead— Workbench file

17. The Road Ahead-FME Server Deployment
 The Road Ahead FME script workspace
published to our FME Server (2009)
 The same ETL script job is run hourly against
the FME Server:
 "D:FMEServerClientsFMEServerConsolefmeserverconsole.exe"
RUN_WORKSPACE Roadahead_dataextract/
uat_roadahead_sdf3_georss_sdf2_lp.fmw
 Feed data (Atom and GeoRSS) integrated with
Google Street View is accessible live via data
streaming services from FME Server

18. The Road Ahead-FME Server
Interface
 http://gisweb9/fmedatastreaming/
Roadahead_dataextract/
prd_roadahead_georss_lp_sv.fmw
Feed data:

19. The Road Ahead- Challenges
 FME Date Formatting: customized transformer
 Feed format (RSS or Atom) and specification
issues:
 unique ID for feed data,
 duplicate feed entries on disjoin hundred block streets
 FME Server user account: ID required cross
domain access
 Authentication limitation: use FME 2010!

20. Summary
 FME is very good solution for both Open Data
and Road Ahead (and lots of other things!)
 FME Desktop and Server have greatly
improved the openness and interoperability of
City’s GIS infrastructure platform
 FME Server 2010 will expand our abilities in
supporting GIS, Open3, Road Ahead and
other initiatives going forward

21. Thank You!
 Questions?
 For more information:
 meng.li@vancouver.ca
 jonathan.mark@vancouver.ca
 IT Department
 City of Vancouver