Presentation given to OKCON 2010 (Open Knowledge Foundation Conference), held at ULU, London, April 24. Note this is a slightly updated and reworked version of the presentation given to the Manchester Social Media Cafe on April 6
7. But what about local data?
• Frankly it’s a mess
• Sporadically published by central government
8. But what about local data?
• Frankly it’s a mess
• Sporadically published by central government
• Inaccessible & impenetrable council websites
9. But what about local data?
• Frankly it’s a mess
• Sporadically published by central government
• Inaccessible & impenetrable council websites
• Opaque local public bodies and NDPBs (quangos)
10. But what about local data?
• Frankly it’s a mess
• Sporadically published by central government
• Inaccessible & impenetrable council websites
• Opaque local public bodies and NDPBs (quangos)
• At best unclear & at worst unusable legal situation
11. But what about local data?
• Frankly it’s a mess
• Sporadically published by central government
• Inaccessible & impenetrable council websites
• Opaque local public bodies and NDPBs (quangos)
• At best unclear & at worst unusable legal situation
• Start with the basics. Who are the councillors, where do they
represent and what committees do they sit on?
12. But what about local data?
• Frankly it’s a mess
• Sporadically published by central government
• Inaccessible & impenetrable council websites
• Opaque local public bodies and NDPBs (quangos)
• At best unclear & at worst unusable legal situation
• Start with the basics. Who are the councillors, where do they
represent and what committees do they sit on?
• How easy it that information to find & reuse?
16. Enter OpenlyLocal...
• Inspiration was a Manchester project, MCC Work For You. Doing
something concrete to solve a problem. First code written 1 year ago
17. Enter OpenlyLocal...
• Inspiration was a Manchester project, MCC Work For You. Doing
something concrete to solve a problem. First code written 1 year ago
• Screen-scrapes council websites. Now over 140 councils done, with
basic information on all 434 in the UK
18. Enter OpenlyLocal...
• Inspiration was a Manchester project, MCC Work For You. Doing
something concrete to solve a problem. First code written 1 year ago
• Screen-scrapes council websites. Now over 140 councils done, with
basic information on all 434 in the UK
• Pulls info from about 150 local councils, plus ONS, OS, NPIA, CLG...
19. Enter OpenlyLocal...
• Inspiration was a Manchester project, MCC Work For You. Doing
something concrete to solve a problem. First code written 1 year ago
• Screen-scrapes council websites. Now over 140 councils done, with
basic information on all 434 in the UK
• Pulls info from about 150 local councils, plus ONS, OS, NPIA, CLG...
• Building a graph of connections – councillors, committees, social
networking, police forces, pension funds
20. Enter OpenlyLocal...
• Inspiration was a Manchester project, MCC Work For You. Doing
something concrete to solve a problem. First code written 1 year ago
• Screen-scrapes council websites. Now over 140 councils done, with
basic information on all 434 in the UK
• Pulls info from about 150 local councils, plus ONS, OS, NPIA, CLG...
• Building a graph of connections – councillors, committees, social
networking, police forces, pension funds
• All open data – free for reuse, including commercial reuse
21. Enter OpenlyLocal...
• Inspiration was a Manchester project, MCC Work For You. Doing
something concrete to solve a problem. First code written 1 year ago
• Screen-scrapes council websites. Now over 140 councils done, with
basic information on all 434 in the UK
• Pulls info from about 150 local councils, plus ONS, OS, NPIA, CLG...
• Building a graph of connections – councillors, committees, social
networking, police forces, pension funds
• All open data – free for reuse, including commercial reuse
• 100% accessible. Data first; bells & whistles later
31. OpenlyLocal: where next?
• More data. More councils. More connections.
• Election data, cabinet members, mayors, police authorities, health authorities,
pension funds, etc... it’s all about the network
32. OpenlyLocal: where next?
• More data. More councils. More connections.
• Election data, cabinet members, mayors, police authorities, health authorities,
pension funds, etc... it’s all about the network
• Data used by Hyperlocal sites (inc Guardian). Ning app. Google
Gadget. Widget. And now starting to get info in the other direction
(i.e. connecting blog posts to councillors)
33. OpenlyLocal: where next?
• More data. More councils. More connections.
• Election data, cabinet members, mayors, police authorities, health authorities,
pension funds, etc... it’s all about the network
• Data used by Hyperlocal sites (inc Guardian). Ning app. Google
Gadget. Widget. And now starting to get info in the other direction
(i.e. connecting blog posts to councillors)
• Some things can’t be done programmatically. Need crowdsourcing
tools & help with tying info to Wikipedia, other sources
34. OpenlyLocal: where next?
• More data. More councils. More connections.
• Election data, cabinet members, mayors, police authorities, health authorities,
pension funds, etc... it’s all about the network
• Data used by Hyperlocal sites (inc Guardian). Ning app. Google
Gadget. Widget. And now starting to get info in the other direction
(i.e. connecting blog posts to councillors)
• Some things can’t be done programmatically. Need crowdsourcing
tools & help with tying info to Wikipedia, other sources
• Some good visualisations done, but lots more potential, particularly
now that we have OS geo data
36. Why does open local data matter?
• Transparency – can we see what’s going on (and understand it too)
37. Why does open local data matter?
• Transparency – can we see what’s going on (and understand it too)
• Engagement – reducing barriers to getting involved
38. Why does open local data matter?
• Transparency – can we see what’s going on (and understand it too)
• Engagement – reducing barriers to getting involved
• Equality of access to information (and thus power). Much of this data
is already available... for a price
39. Why does open local data matter?
• Transparency – can we see what’s going on (and understand it too)
• Engagement – reducing barriers to getting involved
• Equality of access to information (and thus power). Much of this data
is already available... for a price
• Efficiency – Don’t let five different branches of government needlessly
do the same thing (aka keeping it DRY)
41. Simple questions, hard to answer
• How does the budget of my council compare with similar ones?
42. Simple questions, hard to answer
• How does the budget of my council compare with similar ones?
• What’s the background of the chief executive of my council?
43. Simple questions, hard to answer
• How does the budget of my council compare with similar ones?
• What’s the background of the chief executive of my council?
• Is my route to work going to affected by roadworks next week?
44. Simple questions, hard to answer
• How does the budget of my council compare with similar ones?
• What’s the background of the chief executive of my council?
• Is my route to work going to affected by roadworks next week?
• What are the connections between the companies awarded contracts
and the councillors & senior management?
45. Simple questions, hard to answer
• How does the budget of my council compare with similar ones?
• What’s the background of the chief executive of my council?
• Is my route to work going to affected by roadworks next week?
• What are the connections between the companies awarded contracts
and the councillors & senior management?
• How will cuts in services be decided and what can I do to influence
the decisions?
50. Transparency
A brief example.
• A story in Private Eye
• An investigation by the District
Auditor
• A report buried in the nether
recesses of the website...
51. Transparency
A brief example.
• A story in Private Eye
• An investigation by the District
Auditor
• A report buried in the nether
recesses of the website...
52. Transparency
A brief example.
• A story in Private Eye
• An investigation by the District
Auditor
• A report buried in the nether
recesses of the website...
• ...with no relevant heading
53. Transparency
A brief example.
• A story in Private Eye
• An investigation by the District
Auditor
• A report buried in the nether
recesses of the website...
• ...with no relevant heading
54. Transparency
A brief example.
• A story in Private Eye
• An investigation by the District
Auditor
• A report buried in the nether
recesses of the website...
• ...with no relevant heading
• Accessible only as a PDF of a
scan of a document
55. Transparency
A brief example.
• A story in Private Eye
• An investigation by the District
Auditor
• A report buried in the nether
recesses of the website...
• ...with no relevant heading
• Accessible only as a PDF of a
scan of a document
56. Transparency
A brief example.
• A story in Private Eye
• An investigation by the District
Auditor
• A report buried in the nether
recesses of the website...
• ...with no relevant heading
• Accessible only as a PDF of a
scan of a document
• Is it any wonder suspicions are
raised?
58. Engagement
• Local Authorities now have a
duty to engage. This means
• Reducing barriers to understanding
and involvement
59. Engagement
• Local Authorities now have a
duty to engage. This means
• Reducing barriers to understanding
and involvement
• Accessible for all
60. Engagement
• Local Authorities now have a
duty to engage. This means
• Reducing barriers to understanding
and involvement
• Accessible for all
• Available to use when, where
& how we want to
61. Engagement
• Local Authorities now have a
duty to engage. This means
• Reducing barriers to understanding
and involvement
• Accessible for all
• Available to use when, where
& how we want to
• Publishing as data means easy
to be repurposed – for mobile,
for mashups, for offline use
(e.g. Postcode Paper)
63. Equality
• At the moment, all
this information
is available...
at a cost
64. Equality ££
£
• At the moment, all
this information
is available...
at a cost
65. Equality ££
£
• At the moment, all
this information
is available...
at a cost
• Huge asymmetry
of information
(and thus power)
66. Equality ££
£
• At the moment, all
this information
is available...
at a cost
• Huge asymmetry
of information
(and thus power)
• Raises the barriers
to involvement,
and, critically, to
challenge
70. Efficiency
Now: cumbersome, wasteful, opaque, error prone
Typical local data flows
Govt Dept A public (maybe)
email
web
Council Govt Dept B
upload Govt Dept C
73. Efficiency
A better way?
<tr rel='openelection:candidacy'>
<!-- Here we are saying the candidate has given names of
Keith William and a familyName Cockroft -->
<th scope="row" rel="openelection:candidate"><span
property="foaf:givenName">Keith William</span> <span
property="foaf:familyName">Cockroft</span></th>
<td rel="openelection:party" resource="http://
openelectiondata.org/id/parties/6"><span
property="rdfs:label">Labour</span></td>
<!-- The candidacy got 330 votes -->
<td property="openelection:candidateVoteCount"
datatype="xsd:integer">330</td>
<td>16.6%</td>
open data
Council <td property="openelection:elected"
datatype="xsd:boolean" content="false">No</td>
</tr>
<tr rel='openelection:candidacy'>
<th scope="row" rel="openelection:candidate">
<span typeof="openelection:Candidate"
property="foaf:name">Brenda Lilian Constable</span>
<span rel="openelection:address">
<span property="v:street-address">37 Morley Road</
span>,
<span property="v:locality">Burntwood</span>,
<span property="v:region">Staffordshire</span>
<span property="v:postal-code">WS7 2DE</span>
</span>
</th>
74. Efficiency
A better way?
<tr rel='openelection:candidacy'>
<!-- Here we are saying the candidate has given names of
Keith William and a familyName Cockroft -->
<th scope="row" rel="openelection:candidate"><span
public
property="foaf:givenName">Keith William</span> <span
property="foaf:familyName">Cockroft</span></th>
<td rel="openelection:party" resource="http://
openelectiondata.org/id/parties/6"><span
property="rdfs:label">Labour</span></td>
<!-- The candidacy got 330 votes -->
<td property="openelection:candidateVoteCount"
open data
datatype="xsd:integer">330</td>
<td>16.6%</td> Govt Dept A
Council <td property="openelection:elected"
datatype="xsd:boolean" content="false">No</td>
</tr>
<tr rel='openelection:candidacy'>
<th scope="row" rel="openelection:candidate">
Govt Dept B
<span typeof="openelection:Candidate"
property="foaf:name">Brenda Lilian Constable</span>
<span rel="openelection:address">
<span property="v:street-address">37 Morley Road</
span>,
Govt Dept C
<span property="v:locality">Burntwood</span>,
<span property="v:region">Staffordshire</span>
<span property="v:postal-code">WS7 2DE</span>
</span>
</th>
75. Efficiency
A better way?
<tr rel='openelection:candidacy'>
<!-- Here we are saying the candidate has given names of
Keith William and a familyName Cockroft -->
<th scope="row" rel="openelection:candidate"><span
public
property="foaf:givenName">Keith William</span> <span
property="foaf:familyName">Cockroft</span></th>
<td rel="openelection:party" resource="http://
openelectiondata.org/id/parties/6"><span
property="rdfs:label">Labour</span></td>
<!-- The candidacy got 330 votes -->
<td property="openelection:candidateVoteCount"
open data
datatype="xsd:integer">330</td>
<td>16.6%</td> Govt Dept A
Council <td property="openelection:elected"
datatype="xsd:boolean" content="false">No</td>
</tr>
<tr rel='openelection:candidacy'>
<th scope="row" rel="openelection:candidate">
Govt Dept B
<span typeof="openelection:Candidate"
property="foaf:name">Brenda Lilian Constable</span>
<span rel="openelection:address">
<span property="v:street-address">37 Morley Road</
span>,
Govt Dept C
<span property="v:locality">Burntwood</span>,
<span property="v:region">Staffordshire</span>
<span property="v:postal-code">WS7 2DE</span>
</span>
</th>
Publish once, consume many times
77. Problems
• IDs – councils don’t generally use them; central government uses
many... inconsistently
78. Problems
• IDs – councils don’t generally use them; central government uses
many... inconsistently
• Data tied up in PDFs (will we ever get this back?)
79. Problems
• IDs – councils don’t generally use them; central government uses
many... inconsistently
• Data tied up in PDFs (will we ever get this back?)
• Legacy systems. That nobody now understands
80. Problems
• IDs – councils don’t generally use them; central government uses
many... inconsistently
• Data tied up in PDFs (will we ever get this back?)
• Legacy systems. That nobody now understands
• Private companies/JVs. No FoI. No access to data
81. Problems
• IDs – councils don’t generally use them; central government uses
many... inconsistently
• Data tied up in PDFs (will we ever get this back?)
• Legacy systems. That nobody now understands
• Private companies/JVs. No FoI. No access to data
• Outsourcing (councils have outsourced the skills & knowledge and
now are often dumb consumers)
82. Problems
• IDs – councils don’t generally use them; central government uses
many... inconsistently
• Data tied up in PDFs (will we ever get this back?)
• Legacy systems. That nobody now understands
• Private companies/JVs. No FoI. No access to data
• Outsourcing (councils have outsourced the skills & knowledge and
now are often dumb consumers)
• Silos, especially of budgets. How do you get money from one budget
to another (even if the whole benefits)
85. The OpenElectionData project
• Tackling the open local data problem, one set at a time & learning
lessons on the way
• Succeed, or fail forward
86. The OpenElectionData project
• Tackling the open local data problem, one set at a time & learning
lessons on the way
• Succeed, or fail forward
• No public database of local election results, only a commercial one
(subsidized by the Electoral Commission)
87. The OpenElectionData project
• Tackling the open local data problem, one set at a time & learning
lessons on the way
• Succeed, or fail forward
• No public database of local election results, only a commercial one
(subsidized by the Electoral Commission)
• Allows even those with no prior knowledge of linked data/RDF/
semantic web(choose fave buzzword here) to take part
88. The OpenElectionData project
• Tackling the open local data problem, one set at a time & learning
lessons on the way
• Succeed, or fail forward
• No public database of local election results, only a commercial one
(subsidized by the Electoral Commission)
• Allows even those with no prior knowledge of linked data/RDF/
semantic web(choose fave buzzword here) to take part
• Just need HTML competence
89.
90. Open Election Data project
How it works
• Instead of publishing their election results as arbitrary HTML they
publish it as HTML that has been semantically marked up.
• e.g.
<tr rel='openelection:candidacy'>
<th scope="row" rel="openelection:candidate"><span typeof="openelection:Candidate"
property="foaf:name">Helen Elizabeth Fisher</span></th>
<td rel="openelection:party" resource="http://openelectiondata.org/id/parties/25"><span
property="rdfs:label">Conservative</span></td>
<td property="openelection:candidateVoteCount" datatype="xsd:integer">655</td>
<td>33.0%</td>
<td property="openelection:elected" datatype="xsd:boolean" content="true">Yes</td>
</tr>
instead of
<tr class='zebraBandOdd'>
<th scope="row" class='darkGreyCell'><span class='boldEntry'>Helen Elizabeth Fisher</span></th>
<td class='lightCell'>Conservative</td>
<td class='lightCell'>655</td>
<td class='lightCell'>33.0%</td>
<td class='lightCell'>Yes</td>
</tr>
93. Open Election Data project
How it works
• Allows:
• data to be pulled from the page, and consumed by RDF reader
94. Open Election Data project
How it works
• Allows:
• data to be pulled from the page, and consumed by RDF reader
• Local Authorities to develop an understanding of the key issues of
open linked data - e.g. URIs as identifiers
95. Open Election Data project
How it works
• Allows:
• data to be pulled from the page, and consumed by RDF reader
• Local Authorities to develop an understanding of the key issues of
open linked data - e.g. URIs as identifiers
• the beginnings of a complete database of local election results
96. Open Election Data project
How it works
• Allows:
• data to be pulled from the page, and consumed by RDF reader
• Local Authorities to develop an understanding of the key issues of
open linked data - e.g. URIs as identifiers
• the beginnings of a complete database of local election results
• a concrete way of identifying the blockers & pain points when
moving towards open local data
98. How’s it going? Too early to say but
• After the May 6 election (local elections at 166 councils) will be
somewhere between 12 and 40 councils publishing their data this way
(too early to give more accurate figure yet, lots joining at last minute)
99. How’s it going? Too early to say but
• After the May 6 election (local elections at 166 councils) will be
somewhere between 12 and 40 councils publishing their data this way
(too early to give more accurate figure yet, lots joining at last minute)
• Already we have increased knowledge and publishing of open data
by councils by an order of magnitude, and have established informal
network of people who get it
100. How’s it going? Too early to say but
• After the May 6 election (local elections at 166 councils) will be
somewhere between 12 and 40 councils publishing their data this way
(too early to give more accurate figure yet, lots joining at last minute)
• Already we have increased knowledge and publishing of open data
by councils by an order of magnitude, and have established informal
network of people who get it
• Council people have begun to help each other, solving problems,
writing scripts for internal CMSs, and it now has its own momentum
101. How’s it going? Too early to say but
• After the May 6 election (local elections at 166 councils) will be
somewhere between 12 and 40 councils publishing their data this way
(too early to give more accurate figure yet, lots joining at last minute)
• Already we have increased knowledge and publishing of open data
by councils by an order of magnitude, and have established informal
network of people who get it
• Council people have begun to help each other, solving problems,
writing scripts for internal CMSs, and it now has its own momentum
• Core blockers already becoming clear: lack of basic skills (even
HTML); intransigent middle managers; inflexible IT systems
103. Where next?
• A Freedom of Data act – too many different bodies
104. Where next?
• A Freedom of Data act – too many different bodies
• A recasting of how Government (central & local) does IT. Avoiding
the big #fail
105. Where next?
• A Freedom of Data act – too many different bodies
• A recasting of how Government (central & local) does IT. Avoiding
the big #fail
• A change in relationship between central & local government
106. Where next?
• A Freedom of Data act – too many different bodies
• A recasting of how Government (central & local) does IT. Avoiding
the big #fail
• A change in relationship between central & local government
• A change between between government and citizen
107. Where next?
• A Freedom of Data act – too many different bodies
• A recasting of how Government (central & local) does IT. Avoiding
the big #fail
• A change in relationship between central & local government
• A change between between government and citizen
• Some new business models – to support hyperlocal sites, innovative
and supportive suppliers, public-interest groups
108. Where next?
• A Freedom of Data act – too many different bodies
• A recasting of how Government (central & local) does IT. Avoiding
the big #fail
• A change in relationship between central & local government
• A change between between government and citizen
• Some new business models – to support hyperlocal sites, innovative
and supportive suppliers, public-interest groups
• Develop the meme: enabler or blocker