Over the last two days (5th and 6th of November 2015) I was very happy to present to a range of Victorian Government agencies and give them some context on what data management can do for their organisations. From first principles we went through why data was important and what infrastructure was already in place via data.vic.gov.au for them to leverage. We covered examples of how other agencies, such as the Office of Environment and Heritage in NSW, are rebuilding their data management system to provide a more efficient pipeline for publishing internal and public data. As always, I could not help highlighting the awesome leadership of WA Parks and Wildlife and the work done by Florian Mayer as the best case example for reducing the costs and friction often involved with publishing data as contextually marked up knowledge. We covered a number of scenarios where the concept of resource containers for data were considered. This created valuable feedback which has further galvanized my thoughts about how to further extend CKAN to meet the needs of both private and open data portals, and other forms of realtime or unstructured data.

2. 2Link Digital’s Network Map

3. CKAN portals managed by Link Digital 3
Data.Gov.au
data.vic.gov.au
WoVG IAR
VicRoads Demo IAR
data.nsw.gov.au
data.nsw.gov.au
data.nsw.gov.au/iar
openOEH
OEH Internal CKAN
data.sa.gov.au
data.wa.gov.au
data.qld.gov.au
data.Brisbane.qld.gov.au
CKANau.org
WeAreCBR.com.au
AWS Marketplace (15 sites worldwide)

5. Data portal software:
1. Open Source
2. Large and expanding installation base
within Government worldwide
3. Expanding use cases in the wider data
ecosystem
4. Python web app, PostgreSQL DB
5. Built for machines,
custodians and end users
WHAT IS CKAN?

6. LEARN MORE AT CKAN.org

7. DOWNLOAD FROM github.com/ckan/ckan

8. 1. >> Organisations (optionally with sub-organisations)
2. >> >> Datasets
3. >> >> >> Resources
4. >> Platform Custodian
5. >> >> Organisation Custodian, Editor or Member
6. >> Published or Private datasets
CKAN STRUCTURE

9. 1. >> Constitution
2. >> >> Parliamentary Legislation and Acts (Jurisdiction = Platform)
3. >> >> >> Ministries (Organisation)
4. >> >> >> >> Programs (Sub-Organisations)
5. >> >> >> >> >> Projects (Datasets)
6. >> >> >> >> >> >> Outcomes (Resources)
CKAN USE CASE PARADIGM

10. 1. User registration
2. User management
3. Custodian workflows (manage datasets and data resources)
4. Directory Browse by organisation or group
5. Faceted search for multiple fields (supporting end user discovery)
6. Resource views to preview data (a recently improved feature)
7. Metadata view
CKAN UI

11. 1. Create an organistation (usually done by platform owner)
2. Login as member of organisation
3. Click ‘add dataset’
4. Step 1: Add a title, description and other metadata
5. Step 2: Add resources (links to data or upload data files for hosting)
6. Step 3: Add any additional info
CKAN CUSTODIAN WORKFLOW

12. 1. Join the dev mailing list (monitored by tech team):
https://lists.okfn.org/mailman/listinfo/ckan-dev
2. Search Stack overflow under CKAN:
http://stackoverflow.com/search?q=CKAN
3. Check the roadmap on waffle.io:
https://waffle.io/ckan/ideas-and-roadmap
4. Join a tech team meeting:
http://ckan.org/about/technical-team/
WHERE TO GET HELP

13. CKAN API DOCUMENTATION

14. 1. Get JSON-formatted lists of a site’s datasets, groups or other CKAN objects
2. Get a full JSON representation of a dataset, resource or other object
3. Search for packages or resources matching a query
4. Create, update and delete datasets, resources and other objects
5. Get an activity stream of recently changed datasets on a site
CKAN API

15. 1. CKAN as an Information Asset Register
2. FileStore – For hosting of data and resources
3. DataStore - provides a database for structured storage of data together with a powerful Web-
accessible Data API
4. License Selection (machine ready?)
5. Harvesting
A FEW MORE POINTS

16. Delivering the world’s best open
data management system
The purpose of the CKAN Association
is to support sustainable growth and
development of CKAN while also
protecting the interests of the CKAN
community.
The Association values a healthy and
thriving community which continues
to deliver the best open data
management system in the world.
CKAN ASSOCIATION: STATEMENT OF PURPOSE

17. Community interests, or needs, can be generalised as:
1. Users need an enterprise level open data management system tailored to meet their needs
now and into the future.
2. Individual contributors need a project that is rewarding to work for, inclusive and active.
COMMUNITY INTERESTS

18. Who owns or directly manages the CKAN project?
The project, its releases and future direction are cooperatively managed by its community of
users and contributors. Association delegates, staff and office holders may be active within the
project but will exercise no more or less influence than any other contributor or user.
More information about CKAN and how to contribute can be found at CKAN.org.
DELEGATION TO SERVE AND PROTECT

19. What is the current structure of
the CKAN Association?
See http://ckan.org
The steering group carries on a
number of business activities.
This includes raising revenue,
managing resources and directing
projects or programs of activity
relevant to the CKAN Association’s
statement of purpose.
COMPLETENESS OF PURPOSE

20. Foundations of Open Government 20
My Administration is committed to creating an
unprecedented level of openness in Government.
We will work together to ensure the public trust
and establish a system of transparency, public
participation, and collaboration. Openness will
strengthen our democracy and promote efficiency
and effectiveness in Government.
Sources:
http://www.whitehouse.gov/the_press_office/TransparencyandOpenGovernment
Hope Poster by Shepard Fairey: http://en.wikipedia.org/wiki/File:Barack_Obama_Hope_poster.jpg

21. The role of Government in Australia 21
It should support civil society and its multiplicity of voices and activities.
It should provide the economic framework and the essential infrastructure for
public and private enterprise.
Source: http://www.australiancollaboration.com.au/pdf/FactSheets/Role-government-FactSheet.pdf

22. Business Cases: Theory of the firm & Transaction costs 22

23. Where is there work to do with Open Data? 23

24. Published Data Handbook 24

25. Published Data Handbook 25
Recipe Name What you’ll need Ingredients Method Perfect for…

26. 26Establishing Data Publishing Policy

27. 27Establishing Data Management Systems

28. 28Educating Data Publishers

29. 29Facilitation within tech communities

30. Establishing enterprise solution architecture 30

31. 31Extract, Transform and Load Connectors

32. 32Cleaning of Messy Data

33. 33Geospatial Integration

34. 34Resource View Integration

35. 35Enriched Data and Analytics

36. 36The perfect storm

37. 37Drupal interface

38. 38CKAN interface

39. 39What the DFMP does

40. 40Data.vic.gov.au on CKAN, Drupal and AWS

41. Classifying data 41
The three tiers are:
internal
whole of Government
open

42. 42The ‘go. no go’ gates for going open

43. 43Data.vic.gov.au data classification

44. Classifying data 44
The three tiers are:
internal
whole of Government
open

45. OEH Data Management System 45

46. Take Note: What is NOT good 46

47. Take Note: What is best… 47