This project aims to create an FME Workspace Template Collection for reading the complex formats of fundamental geographic data that is available from the Japanese government via the web for public use.
FME Workspace Template Collection for Reading Fundamental Geographic Data in Japan
1. CONNECT. TRANSFORM. AUTOMATE.
FME Workspace Template Collection for Reading
Fundamental Geographic Data in Japan
Takashi Iijima
FME Certified Professional, pragmatica inc.
2. Background: KSJ and FGD
Major Fundamental Geographic Datasets
provided by Japanese Governmental Organizations
KSJ: 国⼟土数値情報
(Kokudo Suuchi Jouho)
National Land Numerical Information
FGD: 基盤地図情報
(Kiban Chizu Jouho)
Fundamental Geospatial Data
CONNECT. TRANSFORM. AUTOMATE.
3. Background: KSJ Overview
! Provider: 国⼟土交通省省
-- Ministry of Land,
Infrastructure, Transport and Tourism (MLIT)
! http://nlftp.mlit.go.jp/ksj/
! Feature Types: 60+; Geographic Features, Land
Use, Political Designated Zones, Administrative
Areas, Public Facilities, Transportation Facilities
etc. (Point, Line or Area)
! Format: XML conforming to GML
! GML reader cannot read the data.
CONNECT. TRANSFORM. AUTOMATE.
4. Background: FGD Overview
! Provider: 国⼟土地理理院
-- Geospatial Information
Authority of Japan (GSI)
! http://www.gsi.go.jp/kiban/etsuran.html
! Feature Types: Vectorized Topographical Map
Elements (Point, Line or Area), Survey Reference
Point, Digital Elevation Model (DEM)
! Format: XML conforming to GML
! GML reader can read the data except DEM.
CONNECT. TRANSFORM. AUTOMATE.
5. Project Overview
! Purpose: To provide easy and flexible way to read
KSJ and DEM in FGD with FME to increase their
usability.
! Approach: Create FME workspaces to read the
datasets with the XML reader and create features
having appropriate geometry and attributes.
! Goal: FME Template Collection
CONNECT. TRANSFORM. AUTOMATE.
6. Part 1. Reading KSJ
! General Data Structure
! Framework of Workspaces
! Geometry Creation – Point
! Geometry Creation – Line
! Geometry Creation – Area
! Resolving Incorrect Data with xfMap
CONNECT. TRANSFORM. AUTOMATE.
7. <?xml version="1.0" encoding="UTF-8"?>
<ksj:Dataset>
<gml:boundedBy>
<gml:EnvelopeWithTimePeriod srsName="JGD2000 / (B, L)" frame="GC / JST">
<gml:lowerCorner>20.0 123.0</gml:lowerCorner>
<gml:upperCorner>46.0 154.0</gml:upperCorner>
<gml:beginPosition calendarEraName="西暦">1900</gml:beginPosition>
<gml:endPosition indeterminatePosition="unknown"/>
</gml:EnvelopeWithTimePeriod>
</gml:boundedBy>
<gml:Point gml:id="n1">
<gml:pos>34.98509412 139.86044116</gml:pos>
</gml:Point>
<ksj:BusStop gml:id="ED01_1">
<ksj:position xlink:href="#n1"/>
<ksj:busStopName>西下町</ksj:busStopName>
<ksj:busRouteInformation>
<ksj:BusRouteInformation>
<ksj:busType>1</ksj:busType>
<ksj:busOperationCompany>JRバス関東(株)</ksj:busOperationCompany>
<ksj:busLineName>フラワー号</ksj:busLineName>
</ksj:BusRouteInformation>
</ksj:busRouteInformation>
</ksj:BusStop>
</ksj:Dataset>
KSJ: General Data Structure
CONNECT. TRANSFORM. AUTOMATE.
Envelope
Geometries
Features
GML: Point, Curve,
OrientableCurve, Surface
A feature links to a geometry
via "href" attribute.
Contains SRS name:
JGD2000 or Tokyo Datum
8. KSJ: Framework of Workspaces
CONNECT. TRANSFORM. AUTOMATE.
Merge
Features
Geometries
Envelope (SRS)
Merge
Set coordinate system
Modify attribute names
XML Reader feature type
with Feature Type Specific xfMap
XML Reader feature type with Common xfMap
Creates geometries (Line, Area) with Custom Transformer
XML Reader feature type created
with Common xfMap
Add geometry to feature
Group By: fme_basename
Join On: geometry ID
Add SRS name to feature
Group By: fme_basename
Unconditional merging
9. KSJ: Geometry Creation - Point
Mapping rule for <Point>
<mapping match="Point">
<feature-type>
<literal expr="Point" />
</feature-type>
<attributes>
<attribute>
<name><literal expr="Point.id" /></name>
<value><extract expr="@gml:id" /></value>
</attribute>
</attributes>
<geometry activate="xml-point">
<data name="data-string">
<extract expr="./pos"/>
</data>
<data name="axis-order">
<literal expr="2,1" />
</data>
</geometry>
</mapping>
CONNECT. TRANSFORM. AUTOMATE.
<Point> element example
<gml:Point gml:id="p1">
<gml:pos>36.02232800 139.00306800</gml:pos>
</gml:Point>
<gml:Point gml:id="p2">
<gml:pos>36.01240300 139.02192100</gml:pos>
</gml:Point>
<gml:Point gml:id="p3">
<gml:pos>36.04452300 139.05435900</gml:pos>
</gml:Point>
<gml:Point gml:id="p4">
<gml:pos>35.98977700 139.07710600</gml:pos>
</gml:Point>
<gml:Point gml:id="p5">
<gml:pos>35.99583200 139.07745600</gml:pos>
</gml:Point>
Points can be created simply with xfMap.
10. KSJ: Geometry Creation - Line
CONNECT. TRANSFORM. AUTOMATE.
Read <LineStringSegment> elements as XML fragments.
Create Line geometries with a Custom Transformer.
Mapping rule for <Curve>
<mapping match="Curve">
<feature-type>
<literal expr="Curve" />
</feature-type>
<attributes>
<attribute>
<name><literal expr="Curve.id" /></name>
<value><extract expr="@gml:id" /></value>
</attribute>
<attribute type="list">
<name><literal expr="LineStringSegment" /></name>
<value>
<extract expr="./segments/LineStringSegment" as-xml="true" />
</value>
</attribute>
</attributes>
</mapping>
<Curve> element example
<gml:Curve gml:id="c-85">
<gml:segments>
<gml:LineStringSegment>
<gml:posList>
35.91887573 139.92888998
35.91885861 139.92893361
35.91885472 139.92894500
35.91885472 139.92895778
35.91887972 139.92899667
35.91925667 139.92939778
35.91927722 139.92945889
</gml:posList>
</gml:LineStringSegment>
</gml:segments>
</gml:Curve>
11. KSJ: Geometry Creation - Line
CONNECT. TRANSFORM. AUTOMATE.
KsjLineBuilder (Custom Transformer) creates Line geometries.
12. Surface element example
<gml:Surface gml:id="a00001">
<gml:patches>
<gml:PolygonPatch>
<gml:exterior>
<gml:Ring>
<gml:curveMember xlink:href="#_c00003"/>
<gml:curveMember xlink:href="#_c00002"/>
<gml:curveMember xlink:href="#_c00001"/>
</gml:Ring>
</gml:exterior>
<gml:interior>
<gml:Ring>
<gml:curveMember xlink:href="#_c00004"/>
</gml:Ring>
</gml:interior>
<gml:interior>
<gml:Ring>
<gml:curveMember xlink:href="#_c00005"/>
</gml:Ring>
</gml:interior>
</gml:PolygonPatch>
</gml:patches>
</gml:Surface>
KSJ: Geometry Creation - Area
CONNECT. TRANSFORM. AUTOMATE.
Mapping rule for Surface
<mapping match="Surface">
<feature-type>
<literal expr="Surface" />
</feature-type>
<attributes>
<attribute>
<name><literal expr="Surface.id" /></name>
<value><extract expr="@gml:id" /></value>
</attribute>
</attributes>
<structure
matched-attributes="yes"
matched-prefix="attributes"
cardinality="*/patches/PolygonPatch{}/exterior/Ring/curveMember{}
*/patches/PolygonPatch{}/interior{}/Ring/curveMember{}" />
</mapping>
Complicated hierarchy XML structure can be
mapped to nested-structured list attribute.
Create Area geometries with a Custom
Transformer.
13. KSJ: Geometry Creation - Area
CONNECT. TRANSFORM. AUTOMATE.
OrientableCurve element example
<gml:OrientableCurve gml:id="_c00100002" orientation="-">
<gml:baseCurve xlink:href="#c00100002"/>
</gml:OrientableCurve>
An OrientableCurve represents a reversed Curve,
links to a Curve by its ID.
Mapping Rule for OrientableCurve
<mapping match="OrientableCurve">
<feature-type>
<literal expr="OrientableCurve" />
</feature-type>
<structure
matched-attributes="yes"
matched-prefix="attributes"
cardinality="*/+" />
</mapping>
14. KSJ: Geometry Creation - Area
CONNECT. TRANSFORM. AUTOMATE.
KsjAreaBuilder (Custom Transformer) creates Area geometries.
18. KSJ: Resolving Incorrect Data with xfMap
Case 1
CONNECT. TRANSFORM. AUTOMATE.
Incorrect - wrong feature type/attribute names
<ksj:AdministrativeArea gml:id="gy2">
<ksj:are xlink:href="#sf2"/>
<ksj:prn>千葉県</ksj:prn>
<ksj:cn2>柏市</ksj:cn2>
…
Two Mapping rules for the same feature type
<!-- For correct schema -->
<mapping match="AdministrativeBoundary">
<feature-type>
<literal expr="AdministrativeBoundary" />
</feature-type>
<structure
matched-attributes="yes"
matched-prefix="attributes"
cardinality="*/+" />
</mapping>
<!-- For wrong schema -->
<mapping match="AdministrativeArea">
<feature-type>
<literal expr="AdministrativeBoundary" />
</feature-type>
<attributes>
<attribute>
<name><literal expr="AdministrativeBoundary.id" /></name>
<value><extract expr="@gml:id" /></value>
</attribute>
<attribute>
<name><literal expr="bounds.href" /></name>
<value><extract expr="./are[@xlink:href]" /></value>
</attribute>
<attribute>
<name><literal expr="prefectureName" /></name>
<value><extract expr="./prn" /></value>
</attribute>
<attribute>
<name><literal expr="cityName" /></name>
<value><extract expr="./cn2" /></value>
</attribute>
…
Multiple mapping rules for the same feature type are allowed
if “match” attributes (XML element names) are different.
Correct
<ksj:AdministrativeBoundary gml:id="ec00100004">
<ksj:bounds xlink:href="#a00100004"/>
<ksj:prefectureName>千葉県</ksj:prefectureName>
<ksj:cityName>柏市</ksj:cityName>
…
19. KSJ: Resolving Incorrect Data with xfMap
Case 2
CONNECT. TRANSFORM. AUTOMATE.
Correct
<gml:Point gml:id="p1">
<gml:pos>36.02232800 139.00306800</gml:pos>
</gml:Point>
Incorrect - wrong GML element name
<gml:Point gml:id="pt0">
<gml:position>36.08985800 139.78870900</gml:position>
</gml:Point>
Mapping rule modified for incorrect data
<mapping match="Point">
<feature-type>
<literal expr="Point" />
</feature-type>
<attributes>
<attribute>
<name><literal expr="Point.id" /></name>
<value><extract expr="@gml:id" /></value>
</attribute>
</attributes>
<geometry activate="xml-point">
<data name="data-string">
<extract expr="./pos"/>
<extract expr="./position" />
</data>
<data name="axis-order">
<literal expr="2,1" />
</data>
</geometry>
</mapping>
<extract> element in xfMap generates just an
empty string if the XML path specified by “expr”
attribute doesn’t exist. This mapping rule works
for both correct data and incorrect data.
20. Part 2. Reading DEM in FGD
! DEM Types
! Data Structure
! Sequence of Elevation Values
! Skeleton of Workspace
! Process of Creating Raster
! Implementation and Performance
CONNECT. TRANSFORM. AUTOMATE.
21. DEM: DEM Types
Type Name
Cell Sizes
width x height
Columns x Rows
per a Dataset
Extent Sizes
width x height
5m Mesh
0.2 x 0.2 sec.
225 x 150
45 x 30 sec.
10m Mesh
0.4 x 0.4 sec.
1125 x 750
450 x 300 sec.
250m Mesh
11.25 x 7.5 sec.
320 x 320
3600 x 2400 sec.
CONNECT. TRANSFORM. AUTOMATE.
Both 5m Mesh DEM and 10m Mesh DEM are covering all of Japanese
land area; 250m Mesh DEM is covering a special area only.
Ideally the workspace should be applicable to any type of
DEM.
22. DEM: Data Structure
CONNECT. TRANSFORM. AUTOMATE.
<?xml version="1.0" encoding="Shift_JIS"?>
<Dataset>
<DEM><coverage>
<gml:boundedBy>
<gml:Envelope srsName="fguuid:jgd2011.bl">
<gml:lowerCorner>38.916666667 139.8</gml:lowerCorner>
<gml:upperCorner>38.925 139.8125</gml:upperCorner>
</gml:Envelope>
</gml:boundedBy>
<gml:gridDomain>
<gml:Grid>
<gml:limits>
<gml:GridEnvelope>
<gml:low>0 0</gml:low>
<gml:high>224 149</gml:high>
</gml:GridEnvelope>
</gml:limits>
</gml:Grid>
</gml:gridDomain>
<gml:rangeSet>
<gml:DataBlock>
<gml:tupleList>
海水面,-9999.
海水面,-9999.
海水面,-9999.
地表面,0.63
…
地表面,6.83
地表面,6.67
地表面,6.57
</gml:tupleList>
</gml:DataBlock>
</gml:rangeSet>
<gml:coverageFunction>
<gml:GridFunction>
<gml:startPoint>108 0</gml:startPoint>
</gml:GridFunction>
</gml:coverageFunction>
</coverage></DEM>
</Dataset>
Number of columns and rows:
num_columns = high{0} + 1 – low{0}
num_rows = high{1} + 1 – low{1}
List of (Type, Elevation)
SRS name: “jgd2000.bl” or “jgd2011.bl”
Extents: (ymin, xmin), (ymax, xmax)
Index of start point in the grid matrix:
start_col = startPoint{0}
start_row = startPoint{1}
23. DEM: Sequence of Elevation Values
CONNECT. TRANSFORM. AUTOMATE.
0 1 2 3 4 5 6 7 …
0
1
2
3
4
5
6
Start Column Index = 6, Start Row Index = 2
Elevation values are put in order in the turn of going to Right-Lower
from Left-Upper.
Location of the starting cell is specified by start column, row index
described in the <GridFunction> element.
<gml:rangeSet>
<gml:DataBlock>
<gml:tupleList>
海水面,-9999.
海水面,-9999.
海水面,-9999.
地表面,0.63
…
地表面,6.83
地表面,6.67
地表面,6.57
</gml:tupleList>
</gml:DataBlock>
</gml:rangeSet>>
24. DEM: Skeleton of Workspace
CONNECT. TRANSFORM. AUTOMATE.
Dataset
Extract Number of
Columns / Rows
Extract Coordinates of
Lower / Upper Corner
Create Raster
Origin = (0, 0)
Cell Sizes = 1 x 1
Width = Number of Columns
Height = Number of Rows
Resize, Move,
Set CoordSys
XML Reader
with Feature Paths
Scaler
Offsetter
CoordinateSystemSetter
AttributeSplitter
AttributeCreator
AttributeSplitter
NumericRasterizer parameter settings
Size Specification: RowsColumns
Number of Columns (cells): <num_columns>
Number of Rows (cells): <num_rows>
Ground Extents
Minimum X: 0
Minimum Y: 0
Maximum X: <num_columns>
Maximum Y: <num_rows>
Scaler
Offsetter
(0, 0)
(xmin, ymin)
25. DEM: Process of Creating Raster
CONNECT. TRANSFORM. AUTOMATE.
Extract Start Column / Row Index
Serialize Elevation Values
Create 3D Grid Points
Transform 3D Points into a Raster
3D Point Coordinate Calculation
Index = (0-based Index in the Sequence)
+ (Start Row Index) * (Number of Columns)
+ (Start Column Index)
Column Index = Index % (Number of Columns)
Row Index = INT(Index / (Number of Columns))
X = (Column Index) + 0.5
Y = (Number of Rows) – 0.5 – (Row Index)
Z = Elevation
NumericRasterizer parameter settings
Size Specification: RowsColumns
Number of Columns (cells): <num_columns>
Number of Rows (cells): <num_rows>
Ground Extents
Minimum X: 0
Minimum Y: 0
Maximum X: <num_columns>
Maximum Y: <num_rows>
Extract Number of Columns / Rows
26. DEM: Implementation and Performance
CONNECT. TRANSFORM. AUTOMATE.
1st Implementation: uses existing Transformers only.
About 110 sec. for rasterizing a 10m Mesh DEM (843,750 cells).
The ListExploder seems to consume long time.
27. DEM: Implementation and Performance
CONNECT. TRANSFORM. AUTOMATE.
2nd Implementation: uses a Python script to extract Elevation
values and create 3D Grid Points.
About 25 sec.: Performance has been improved considerably.
But there should be more efficient way.
import fmeobjects
class GridPointCreator(object):
def input(self, feature):
count = feature.getAttribute('_count')
numCols = int(feature.getAttribute('_num_columns'))
numRows = int(feature.getAttribute('_num_rows'))
startCol = int(feature.getAttribute('_start{0}'))
startRow = int(feature.getAttribute('_start{1}'))
tuples = feature.getAttribute('DEM.coverage.rangeSet.DataBlock.tupleList').strip()
index = startRow * numCols + startCol
for z in [float(t.split(',')[1]) for t in tuples.split()]:
x = index % numCols + 0.5
y = numRows - 0.5 - index / numCols
point = fmeobjects.FMEFeature()
point.setAttribute('_count', count)
point.setAttribute('_num_columns', numCols)
point.setAttribute('_num_rows', numRows)
point.setGeometry(fmeobjects.FMEPoint(x, y, z))
self.pyoutput(point)
index += 1
28. DEM: Implementation and Performance
CONNECT. TRANSFORM. AUTOMATE.
3rd Implementation: inserted a PointCloudCombiner to
transform 3D Points into a Point Cloud before rasterizing.
About 12 sec.: Performance has been improved further more.
Is it the best performance?
Special Thanks: The Point Cloud Technology was suggested by
Dale and Chris@Safe.
29. DEM: Implementation and Performance
CONNECT. TRANSFORM. AUTOMATE.
Final Implementation: updated the Python script so that it will
create a 3D Line having all the 3D Grid Points as vertices.
About 2.5 sec.: Quite faster than creating many point
features.
import fmeobjects
def create3DLine(feature):
numCols = int(feature.getAttribute('_num_columns'))
numRows = int(feature.getAttribute('_num_rows'))
startCol = int(feature.getAttribute('_start{0}'))
startRow = int(feature.getAttribute('_start{1}'))
tuples = feature.getAttribute('DEM.coverage.rangeSet.DataBlock.tupleList').strip()
index = startRow * numCols + startCol
coords = []
for z in [float(t.split(',')[1]) for t in tuples.split()]:
x = index % numCols + 0.5
y = numRows - 0.5 - index / numCols
coords.append((x, y, z))
index += 1
feature.setGeometry(fmeobjects.FMELine(coords))
30. DEM: Implementation and Performance
Level
Processing Time
Comments
1st
110 sec. (100%)
Existing Transformers Only
2nd
25 sec. (23%)
Create 3D Grid Points with Python
3rd
12 sec. (11%)
Introduced the Point Cloud Technology
Final
2.5 sec. (2.3%)
Create 3D Line with Python
CONNECT. TRANSFORM. AUTOMATE.
Tested with FGD 10m Mesh DEM: FG-GML-5338-41-dem10b-20090201.xml
FME 2014 SP2 WIN32, Windows 7 64-bit
Conclusion:
• Python scripting is worth to consider for improving performance, especially
if the ListExploder looks inefficient.
• Using a Point Cloud as input for the NumericRasterizer is very efficient.
Summary
31. DEM: Implementation and Performance
CONNECT. TRANSFORM. AUTOMATE.
Rasterizing Result:
Final Workspace
32. Thank You!
! Questions?
! For more information:
! Takashi Iijima iijima@pragmatica.jp
! Pragmatica Inc.
! http://fme-memorandum-takashi.blogspot.jp/
CONNECT. TRANSFORM. AUTOMATE.