The Citywide Street Centerline Database (CSCL) is the authoritative source for location data used by New York City's public safety agencies (NYPD, FDNY) for 911 call-taking and dispatch. It is an ESRI enterprise geodatabase that is jointly maintained by 2 city agencies- the Department of Information Technology and Telecommunications and the Department of City Planning. Designed more for data maintenance, CSCL is not easily accessible by the day-to-day GIS user. This presentation will show how we have applied FME to create a version of CSCL that is better suited for general GIS tasks and made it available to all city agencies, to the public via NYC Open Data, and to Batman.
4. My IT diagram
Developed after 9/11 to serve as an authoritative source of addresses and location info
- Consolidated data: streets, addresses, common places, etc.
- 5-day update cycle (Mayor’s Executive Order #39)
- Data distribution system
- CSCL is Hosted by DoITT
- Maintained jointly by DoITT and Dept. of City Planning (DCP) since 2009
- Governance by 7 City Agencies (DoITT, DCP, NYPD, FDNY, NYC EM, DOT, and DEP)
Citywide Street
Centerline Geodatabase
(CSCL)
5. CSCL is really, really, really designed
7 City Agencies + 2 Contractors + ESRI Advisors =
68 feature classes
40 tables
47 Relationship Classes
2 Feature Datasets
8 Topology rules
93 Domains
2 Subtypes….. plus Custom ArcMap Editing tools!
7. CSCL Data Distribution System
CSCL Database
ESRI GDB Replication
This is the FME part…
NYC EM
DOT
9-1-1 Call-taking
NYPD Dispatch
Transportation
planning and analysis
Fire/EMS Dispatch
NYPD
FDNY
Disaster planning,
response, and recovery
“Published” CSCL data for City Agencies and NYC
OpenData
8. CSCL was designed for maintenance, so...
NYPD
FDNY
CSCL Replicas Custom coded ETLs
ArcObjects (Pokemons)
NYPD
FDNY
Export GDBs E911 and Dispatch
CAD Systems
CSCL Published
CSCL
This part
broke.
9. A Stab from the Past!
FME is in the
ESRI Data Interop
Extension…
10. Spatial ETL Tool Version
“Direct Copy” Data
“Named Intersection” Data
Writes simultaneously to FGDB and SDE. Nice!
15. Standalone FME
I got my agency to buy something!
1 floating license and one FME upgrade from ESRI Data Interop vs
$50,000 (or worse) for software consulting services
Provisioned VM exclusively to run FME processes
Easier to run FMWs as scheduled tasks
Can open FMW files created by others
Updated software (v2016 vs v2013) e.g. Socrata Writer
FME v2016 is 64bit vs 32bit version in Data Interop
Best of all- We have full control of our ETL
16. Converting Spatial ETL tool to FMW
a) Saving the ESRI-created FMW files to new *.FMW
names
b) Copy/pasting the Readers, Writers, Transformers
from Spatial ETL tools to FME 2016 Workbench
c) Recreating the old tools in FME 2016 Workbench-
this worked the best.
17. Loose ends…
How does FME use the machine resources?
Local run = 2 hours (Xeon 3.2GHz/8GB/Win 7 64)
Server run = 4 hours (Quad 2.6GHz /8GB/ WinServer R2 64)
“Memory allocation” freezing for converted tools
Recreated tools in FME 2016 had same problem
Safe Software Support case C111319
Moved to 64 bit FME
Changed out some Feature Holders for Cloners
18. Other FME uses
Direct updates to SDE from anything: xlsx, WFS, Access….
QC Tools for CSCL data maintenance
Previously Model Builder, Python (ArcPy), Python Toolbox
Metadata handling (once we figure it out)
FME Documentation for the XML stuff is not awesome
Linear Referencing applications, a hard sell…
Replacement of other ETL custom software
Once they taste our cooking….