Cultivating Open Geo-Data in the Real-World: A Guided Tour of Three Portland-based Projects

1. Cultivating Open Geo-Data
in the Real-World:
A Guided Tour of Three Portland-based Projects
Matt Blair
OSCON, Portland, OR
July 29, 2011

2. Who’s working with
Open Data?

3. What’s the Scope?
• National
• State/Province/Canton/Prefecture
• City
• Neighborhood

4. What’s been
challenging?

5. What’s been easy?

6. Outline
• Common project goals
• The three projects, and their challenges
• Solutions
• Summary/Q&A

7. Project Goals

8. Free iOS Apps
For Education and Easy Participation

9. Open Datasets

10. Open Datasets
• real-time geo-queryable
• living/dynamic/well-maintained
• available to other projects/platforms
• relevant beyond lifecycle of initial apps

11. Broader Goals
To Promote:
• Learning and exploring
• Appreciation of place (Portland)
• Participation in community projects
• Walking and health

12. The Projects

16. Poetry Posts

18. What’s a Poetry Box?

19. “A poetry post...is a wooden pole, usually,
mounted on private property, so that it
faces pedestrians. On top of the pole is a
box, with a glass or clear face and a lid.
Inside the box is a sheet of paper containing
a poem...”
– Laura O. Foster

21. http://poetrybox.info

22. Challenges

23. Where are they?

24. Where are they?
No comprehensive list.

25. Wait – what are they called?

26. A Poetry Box

27. Poetry Posts

28. A Poetry Pole

29. A Poetry “Tree”

30. ?!?

31. Divergent Requests

32. PDX Trees

33. (a side project)

34. Civic Apps
Lots of great ideas – most without data

35. Civic Apps Data

36. Civic Apps Data

39. What are Heritage Trees?
"...a list of trees that – because of their age,
size, type, historical association or
horticultural value – are of special
importance to the City. Upon recommendation of
the Urban Forestry Commission, the City Council may
designate a tree as a Heritage Tree provided the tree's
health, aerial space, and open ground area for the root
system have been certified as sufficient."

40. 283 Heritage Trees

41. Challenges

42. Thin Metadata

43. A Verbose Specimen
{
"address": "2403 WI/ SW JEFFERSON ST",
"circumfere": "12.300000000000001",
"common_nam": "Deodar cedar",
"diameter": "47",
"geometry": {
"coordinates": [
-122.70463884770101,
45.521710633334202
],
"type": "Point"
},
"gid": "103",
"height": "73",
"notes": "between SW Marconi Ave and SW Tichner Dr",
"objectid": "103",
"owner": "Right Of Way",
"scientific": "Cedrus deodara",
"spread": "73",
"stateid": "1N1E32 100",
"status": "Heritage",
"treeid": "113",
"year": "1996"
}

45. Visual Appeal?

46. PDFs and Photos?

47. PDFs and Photos
(© All Rights Reserved)

48. “...like PDX Trees, but
with Public Art...

49. Public Art PDX

50. Rewind to May 2010...

51. Before PDX Trees...

52. Before Poetry Posts...

53. Green Line (2009)

54. Green Line Art on the Web

55. Green Line Art Podcast

56. Permission for re-use?

57. Fast Forward to
November 2010

58. No longer an outsider?

62. Challenges

63. Multiple Organizations

64. Multiple Data Formats

65. Will they share data?

66. Do they have data?!

67. Inaccurate Geo-coding

68. Intellectual Property

69. Refactored List of
Challenges
• Building Data • Creating Visual Appeal
• Lack of Consensus • Data Accuracy
• Multiple Sources • IP Uncertainty
• Multiple Data Formats • Teams and Roles

70. Solutions

71. Building Datasets

72. Data
Ideas
The Open Data Universe

73. Data
Ideas
Work here.
Option #1

74. Option #2:
Assemble your own data

75. If it’s already digitized:
• Screen-scrape
• Repurpose feeds or reporting systems
• Google Refine
• Convince governments/stakeholders to
release it

76. If it’s not digitized?

77. Web Forms?

78. Most data arrives like this:

79. Tough to Automate

80. Poetry Posts
Dozens and dozens of emails
in and outside of a Google Group

81. Don’t underestimate
data collection.

82. Lack of Consensus

83. What’s this called?

84. “You should add social
features...”

85. “You should add social
features...”
Comments, Photo Uploads, Likes, Friends, Flags

86. “You know, make it like
Facebook!”

87. Scope Creep

88. Who’s the client?

89. Who decides? How?

90. Build the least
functional version

91. Rapid Feedback Cycles

92. Just Pick Something.

93. Just Pick Something.
(Benevolent Dictator For This Release?)

94. Multiple Sources

95. • Who gathered it?
• To what end?
• Are they maintaining it?
• Do they want to share?

96. PDX Trees
One list, managed by one department

97. Public Art Data
in Portland

98. Data Sources (Nov 2010)
• PDX API: 279 works of art
• Civic Apps CSV file: 366 works of art
• BTS: 300ish works of art (no direct access)
• RACC.org: 1800+ works or art

99. PDXAPI
RACC.org
Civic Apps
BTS?
How do they fit together?

100. RACC.org
Civic Apps
PDXAPI
Nested subsets?

101. But they didn’t match...

102. RACC.org
Permanent
Collection

103. Text = Available Data

104. Text = Available Data
As of November 2010:
~ 370 works of art

105. RACC.org

106. RACC.org
(City of Portland)
(Multnomah County)

107. Done?

108. Not quite...

109. (Caveat: Not To Scale)
Accuracy Not Guaranteed

110. RACC.org
Public Art

111. RACC.org Murals
Public Art

112. RACC.org Murals
TriMet
Public Art

113. RACC.org Murals
TriMet
Convention
Center?
Public Art

114. RACC.org Murals
TriMet
Metro?
Convention
Center?
Public Art

115. RACC.org Murals
Port of
TriMet
Portland
Metro
Convention
Center
Public Art

116. Fountains
RACC.org Murals
Port of
TriMet
Portland
Metro
Convention
Center
Public Art

117. Fountains
RACC.org Murals
Parks
& Rec
Port of
TriMet
Portland
Metro
Convention
Center
Public Art

118. Fountains
RACC.org Murals
Parks
& Rec
Port of
TriMet
Portland
Beaverton?
Metro
Convention
Center
Public Art

119. Fountains
RACC.org Murals
Parks
& Rec
Port of
TriMet
Portland
Beaverton?
Hillsboro? Metro
Convention
Center
Public Art

120. Fountains
RACC.org Murals
Parks
& Rec
Port of
TriMet
Portland
Beaverton?
Hillsboro? Metro
Clark Convention
County? Center
Public Art

121. Fountains
RACC.org Murals
Parks
& Rec
Port of
TriMet
Portland
Performance
Beaverton?
Hillsboro? Metro
Clark Convention
County? Center
Public Art

122. Fountains
RACC.org Murals
Parks
& Rec
Port of
TriMet
Portland
Performance Temporary
Installations
Beaverton?
Hillsboro? Metro
Clark Convention
County? Center
Public Art

123. Fountains Street
RACC.org Murals Art?
Parks
& Rec
Port of
TriMet
Portland
Performance Temporary
Installations
Beaverton?
Hillsboro? Metro
Clark Convention
County? Center
Public Art

124. (aka Graffiti)

125. “Of course not!”

126. “But what if it’s Banksy?”

127. Fountains Street
RACC.org Murals Art?
Parks
& Rec
Port of
TriMet
Portland
Performance Temporary
Installations
Beaverton?
Hillsboro? Metro
Clark Convention
County? Center
Community?
Public Art

128. Community Collection

129. Where’s Paul?

130. photo by Cacophony (via Wikipedia)

131. Intersection
Repair
photo by City Repair (via Flickr)

132. Julian Voss-
Andrae’s
Alpha Helix
(at the Linus Pauling House)
photo via julianvossandrae.com

133. Fountains Street
RACC.org Murals Art?
Parks
& Rec
Port of
TriMet
Portland
Performance Temporary
Installations
Beaverton?
Hillsboro? Metro
Clark Convention
County? Center
Community?
Public Art

134. Fountains Street
RACC.org Murals Art?
Parks
& Rec
Available
Port of
Data
TriMet
Portland
Performance Temporary
Installations
Beaverton?
Hillsboro? Metro
Clark Convention
County? Center
Community?
Public Art

135. A Community-wide
Database
Build an inter-agency superset of public art

136. Yes, it’s complicated.

137. Yes, it’s complicated.
Yet Portland has some of the best
public art data in the country.

138. Data Source Tips
• Start small
• Find allies and implement an example
• Don’t wait for the perfect dataset
• Plan for chaos
• But be ambitious in the long-term!

139. Multiple Data Formats

140. Each source has its own:
• metadata
• schema
• volatility
• level of accuracy and currency
• messes to clean up

141. A dynamic mix of...
• noisy data
• shifting standards
• unexpected restrictions
• adapting to community requests
• addressing data provider concerns
• ambiguities of project 'ownership'

142. Less about architecture
than improvisation...

143. Metadata Flexibility
Let early adopters set the standards

144. NoSQL

145. Document Databases

146. CouchDB

147. Eventual Schema

148. “There is always
schema somewhere.”

149. “There is always
schema somewhere.”
If it’s not in your software, you’re forcing
your audience to build it in their heads.

150. Delay Schema Decisions
Push them up to presentation/client layer

151. But what about MVC?

152. But what about MVC?
My model is in my view?!

153. More nimble than
relational database
Especially during development

154. YKmMV

155. Creating Visual Appeal

156. Heritage Trees
{
"address": "2403 WI/ SW JEFFERSON ST",
"circumfere": "12.300000000000001",
"common_nam": "Deodar cedar",
"diameter": "47",
"geometry": {
"coordinates": [
-122.70463884770101,
45.521710633334202
],
"type": "Point"
},
"gid": "103",
"height": "73",
"notes": "between SW Marconi Ave and SW Tichner Dr",
"objectid": "103",
"owner": "Right Of Way",
"scientific": "Cedrus deodara",
"spread": "73",
"stateid": "1N1E32 100",
"status": "Heritage",
"treeid": "113",
"year": "1996"
}

157. Public Art
{ "docs": [{
"addrCity": "",
"addrState": " ",
"addrStreet": "",
"addrZip": "",
"artists": "Dan Corson",
"date": "2009",
"dateModified": "2011-04-18 00:00:00",
"description": "Mercurial Sky is an ever-changing array of light played on LED
tubes integrated into the Director Park Canopy. The digital video only emits from the
lighted bars, and provides a sense of movement through an abstract tapestry of light
and color. If you stand farther away, or look in nearby reflections, the images are
compressed and give a clearer view of the video. nn"I filmed images and patterns of
natural phenomena like waves, clouds, fire, earthworms, and jellyfish to bring the
movement and randomness of nature into this mostly hardscaped park."",
"detailPageURL": "http://racc.org/public-art/search/?recid=2909.101",
"dimensions": "duration: 1:23:10",
"discipline": "video",
"fundingSource": "Percent for Art - City of Portland",
"thumbnailURL": "http://data.racc.org/pa_inventory/1844/1844thumb.jpg",
"location": "Director Park Canopy",
"mappableDiscipline": "other",
"medium": "Digital video on DVD",
"recordID": "2909",
"title": "Mercurial Sky",
"dataSource": "RACC",
"collection": "None",
"photoCredit": "RACC",
"artCopyright": "TBD",
"locationVerified": "YES",
"geometry": {
"coordinates": [
-122.681124,
45.518759
],
"type": "Point"
}
}

158. Public Art
{ "docs": [{
"addrCity": "",
"addrState": " ",
"addrStreet": "",
"addrZip": "",
"artists": "Dan Corson",
"date": "2009",
"dateModified": "2011-04-18 00:00:00",
"description": "Mercurial Sky is an ever-changing array of light played on LED
tubes integrated into the Director Park Canopy. The digital video only emits from the
lighted bars, and provides a sense of movement through an abstract tapestry of light
and color. If you stand farther away, or look in nearby reflections, the images are
compressed and give a clearer view of the video. nn"I filmed images and patterns of
natural phenomena like waves, clouds, fire, earthworms, and jellyfish to bring the
movement and randomness of nature into this mostly hardscaped park."",
"detailPageURL": "http://racc.org/public-art/search/?recid=2909.101",
"dimensions": "duration: 1:23:10",
"discipline": "video",
"fundingSource": "Percent for Art - City of Portland",
"thumbnailURL": "http://data.racc.org/pa_inventory/1844/1844thumb.jpg",
"location": "Director Park Canopy",
"mappableDiscipline": "other",
"medium": "Digital video on DVD",
"recordID": "2909",
"title": "Mercurial Sky",
"dataSource": "RACC",
"collection": "None",
"photoCredit": "RACC",
"artCopyright": "TBD",
"locationVerified": "YES",
"geometry": {
"coordinates": [
-122.681124,
45.518759
],
"type": "Point"
}
}

159. Public Art
{ "docs": [{
"addrCity": "",
"addrState": " ",
"addrStreet": "",
"addrZip": "",
"artists": "Dan Corson",
"date": "2009",
"dateModified": "2011-04-18 00:00:00",
"description": "Mercurial Sky is an ever-changing array of light played on LED
tubes integrated into the Director Park Canopy. The digital video only emits from the
lighted bars, and provides a sense of movement through an abstract tapestry of light
and color. If you stand farther away, or look in nearby reflections, the images are
compressed and give a clearer view of the video. nn"I filmed images and patterns of
natural phenomena like waves, clouds, fire, earthworms, and jellyfish to bring the
movement and randomness of nature into this mostly hardscaped park."",
"detailPageURL": "http://racc.org/public-art/search/?recid=2909.101",
"dimensions": "duration: 1:23:10",
"discipline": "video",
"fundingSource": "Percent for Art - City of Portland",
"thumbnailURL": "http://data.racc.org/pa_inventory/1844/1844thumb.jpg",
"location": "Director Park Canopy",
"mappableDiscipline": "other",
"medium": "Digital video on DVD",
"recordID": "2909",
"title": "Mercurial Sky",
"dataSource": "RACC",
"collection": "None",
"photoCredit": "RACC",
"artCopyright": "TBD",
"locationVerified": "YES",
"geometry": {
"coordinates": [
-122.681124,
45.518759
],
"type": "Point"
}
}

160. PDX Trees
{
"address": "2403 WI/ SW JEFFERSON ST",
"circumfere": "12.300000000000001",
"common_nam": "Deodar cedar",
"diameter": "47",
"geometry": {
+ ?
"coordinates": [
-122.70463884770101,
45.521710633334202
],
"type": "Point"
},
"gid": "103",
"height": "73",
"notes": "between Marconi Ave and SW Tichner Dr",
"objectid": "103",
"owner": "Right Of Way",
"scientific": "Cedrus deodara",
"spread": "73",
"stateid": "1N1E32 100",
"status": "Heritage",
"treeid": "113",
"year": "1996"
}

161. PDX Trees
{
"address": "2403 WI/ SW JEFFERSON ST",
"circumfere": "12.300000000000001",
"common_nam": "Deodar cedar",
"diameter": "47",
"geometry": {
+
"coordinates": [
-122.70463884770101,
45.521710633334202
],
"type": "Point"
},
"gid": "103",
"height": "73",
"notes": "between Marconi...",
"objectid": "103",
"owner": "Right Of Way",
"scientific": "Cedrus deodara",
"spread": "73",
"stateid": "1N1E32 100",
"status": "Heritage",
"treeid": "113",
"year": "1996"
}

162. PDX Trees
{
"address": "2403 WI/ SW JEFFERSON ST",
"circumfere": "12.300000000000001",
"common_nam": "Deodar cedar",
"diameter": "47",
"geometry": {
+ ??
"coordinates": [
-122.70463884770101,
45.521710633334202
],
"type": "Point"
},
"gid": "103",
"height": "73",
"notes": "between Marconi Ave and SW Tichner Dr",
"objectid": "103",
"owner": "Right Of Way",
"scientific": "Cedrus deodara",
"spread": "73",
"stateid": "1N1E32 100",
"status": "Heritage",
"treeid": "113",
"year": "1996"
}

163. Poetry Posts
1991 SW Mill St Terrace

164. Poetry Posts
1991 SW Mill St Terrace

165. PDX Trees
{
"address": "2403 WI/ SW JEFFERSON ST",
"circumfere": "12.300000000000001",
"common_nam": "Deodar cedar",
"diameter": "47",
"geometry": {
+
"coordinates": [
-122.70463884770101,
45.521710633334202
],
"type": "Point"
},
"gid": "103",
"height": "73",
"notes": "between Marconi...",
"objectid": "103",
"owner": "Right Of Way",
"scientific": "Cedrus deodara",
"spread": "73",
"stateid": "1N1E32 100",
"status": "Heritage",
"treeid": "113",
"year": "1996"
}

166. Sunny October Day?

167. Sunny October Day?
No one will believe it.

168. More Realistic

169. In Rarer Weather, too...

171. But there are 283 Trees!

172. But there are 283 Trees!
I need help.

174. Create a
crowd-
sourced,
longitudinal,
season-
sortable
collection of
tree photos...

175. Will anyone send photos?

176. by Brad B

177. by kateinoregon

178. “Keep
Portland
Green!”
by Dan Flynn

179. (and I still add some...)

180. 350+ Photos Sent In

181. Submitted under
Creative Commons

182. Photos (and data)
available for re-use

183. Data Accuracy

184. What’s Missing
or Incorrect?

185. What’s Missing
or Incorrect?
Errors and omissions are
an opportunity for participation.

188. > 1 mile

189. Artifact of Geo-Coding
Address != Location

190. Capacity
Who has the time to manually verify all these locations?

191. Crowd-Correction

194. IP Uncertainty

195. I thought this was
“open” data?

196. Restrictions in
Terms of Use

197. Linked Media

198. Public Art
{ "docs": [{
"addrCity": "",
"addrState": " ",
"addrStreet": "",
"addrZip": "",
"artists": "Dan Corson",
"date": "2009",
"dateModified": "2011-04-18 00:00:00",
"description": "Mercurial Sky is an ever-changing array of light played on LED
tubes integrated into the Director Park Canopy. The digital video only emits from the
lighted bars, and provides a sense of movement through an abstract tapestry of light
and color. If you stand farther away, or look in nearby reflections, the images are
compressed and give a clearer view of the video. nn"I filmed images and patterns of
natural phenomena like waves, clouds, fire, earthworms, and jellyfish to bring the
movement and randomness of nature into this mostly hardscaped park."",
"detailPageURL": "http://racc.org/public-art/search/?recid=2909.101",
"dimensions": "duration: 1:23:10",
"discipline": "video",
"fundingSource": "Percent for Art - City of Portland",
"thumbnailURL": "http://data.racc.org/pa_inventory/1844/1844thumb.jpg",
"location": "Director Park Canopy",
"mappableDiscipline": "other",
"medium": "Digital video on DVD",
"recordID": "2909",
"title": "Mercurial Sky",
"dataSource": "RACC",
"collection": "None",
"photoCredit": "RACC",
"artCopyright": "TBD",
"locationVerified": "YES",
"geometry": {
"coordinates": [
-122.681124,
45.518759
],
"type": "Point"
}
}

199. Public Art
{ "docs": [{
"addrCity": "",
"addrState": " ",
"addrStreet": "",
"addrZip": "",
"artists": "Dan Corson",
"date": "2009",
"dateModified": "2011-04-18 00:00:00",
"description": "Mercurial Sky is an ever-changing array of light played on LED
tubes integrated into the Director Park Canopy. The digital video only emits from the
lighted bars, and provides a sense of movement through an abstract tapestry of light
and color. If you stand farther away, or look in nearby reflections, the images are
compressed and give a clearer view of the video. nn"I filmed images and patterns of
natural phenomena like waves, clouds, fire, earthworms, and jellyfish to bring the
movement and randomness of nature into this mostly hardscaped park."",
"detailPageURL": "http://racc.org/public-art/search/?recid=2909.101",
"dimensions": "duration: 1:23:10",
"discipline": "video",
"fundingSource": "Percent for Art - City of Portland",
"thumbnailURL": "http://data.racc.org/pa_inventory/1844/1844thumb.jpg",
"location": "Director Park Canopy",
"mappableDiscipline": "other",
"medium": "Digital video on DVD",
"recordID": "2909",
"title": "Mercurial Sky",
"dataSource": "RACC",
"collection": "None",
"photoCredit": "RACC",
"artCopyright": "TBD",
"locationVerified": "YES",
"geometry": {
"coordinates": [
-122.681124,
45.518759
],
"type": "Point"
}
}

200. Linked Media
Who took that photo?
Who owns it?
Who can license it?

201. Teams and Roles

202. Who do you need?

203. Standard Roles
• Project Management • Software Engineering
• Graphic Design • Testing
• Information Architecture • Maintenance
• Metadata Design • Coordination
• Data Entry/Import • Marketing
• UI Design • Communication

204. Roles That Deserve
Special Attention

205. Data Collection

206. Fountains Street
RACC.org Murals Art?
Parks
& Rec
Available
Port of
Data
TriMet
Portland
Performance Temporary
Installations
Beaverton?
Hillsboro? Metro
Clark Convention
County? Center
Community?
Public Art

207. Data Scrubbing
Probably Need Tech Skills

208. Research & Verification
Not necessarily a technical person

209. Curation
What’s in, what’s out and why

210. Authority
(not required, but it helps)

211. Advocacy
Why are we doing this?
Why should you release this data?

212. PDX Trees

213. PDX Trees
‘Mostly’ a solo project

214. There are no
‘solo’ projects.

215. PDX Trees Collaborators
• Urban Forestry Commission
• City Council
• City of Portland’s Bureau of Technology
Services
• Max Ogden’s de-shapefile-ification
• Everyone who has contributed photos

216. Public Art PDX 1.0
Essential Collaborators
• Regional Arts & Culture Council (RACC)
• The Office of Mayor Sam Adams
• City of Portland’s Bureau of Technology
Services
• City Attorney (and pro bono attorneys)
• Former Mayor Bud Clark! (see video)

217. Public Art PDX 1.0
Collaborators Provided:
• Graphic Design • Communication
• Metadata Design • Data Collection
• Data Entry/Import • Data Scrubbing
• Testing • Authority
• Marketing • Advocacy

218. Beyond 1.x:
Community Collection
Different Phases, Different Roles, New Participants

219. Expanding Roles for the
Community Collection
• Metadata Design • Curation
• Data Entry/Import • Photography
• Research and Verification • Communication

220. Two More Things

221. Data Has Stories

223. “While the map makes class and race
differences all the more evident, it's great to
learn about the few murals where I live and
I look forward to using this app on a walking
tour downtown soon.”
– Marshal Kirkpatrick, Read Write Web

226. Why?

227. Incomplete Data

228. No Recent Public
Building Projects

229. Less Economic Activity

230. Voice and
Representation

231. Share Open Data:
Interpretations Emerge

232. A Tool for Advocacy?

233. The Paces of Change

234. – From How Buildings Learn: What Happens After They're Built by
Stewart Brand, p13
Scan source: CC by-nc-sa via http://www.flickr.com/photos/andrewcoulterenright/15663020/

235. The Pace of Each Layer
http://www.ecobuildtrends.com/2011/02/designing-in-layers.html

236. "A design imperative emerges: An adaptive building has
to allow slippage between the differently-paced systems of
Site, Structure, Skin, Services, Space plan, and Stuff.
Otherwise the slow systems block the flow of the
quick ones, and the quick ones tear up the slow ones
with their constant change. Embedding the systems
together may look efficient at first, but over time it is
the opposite and destructive as well."
– From How Buildings Learn: What Happens After They're Built by
Stewart Brand, p20

237. Websites =
Presentation

238. Apps = Presentation

239. “Can I get that icon in
cornflower blue?”

240. Presentation Layers are
Ephemeral

241. Presentation Layers are
Ephemeral
They come and go, like fashion

242. This year’s attire

243. What costumes will
your data wear in
five years’ time?

244. Who knows.

245. Data is the Substrate

246. Data is the
Site and Structure

247. Design for Different Paces

248. A particular app
is a short-term goal

249. Like Furniture

250. A Shearing Layers
Model of Open Data?

251. Hmm...

252. Hmm...
(Need More Input)

253. De-couple Data and
Presentation

254. A Design Philosophy:
Data is the generative
source of the
entire ecosystem.

255. Project Websites
• http://poetrybox.info
• http://pdxtrees.org
• http://publicartpdx.com

256. Thank You
• http://mattblair.net
• Email: elsewisemedia@gmail.com
• Blog: http://elsewisestrategic.com
• github.com/mattblair