1.
Tom Johnson Managing Director Inst. for Analytic Journalism Santa Fe, New Mexico USA t o m @ j t j o h n s o n . c o m It’s not the documents; it’s the DATA!

2.
It’s not the documents, it’s the DATA! <ul><li>Presentation at </li></ul><ul><li>“ 2011 Open Government Academy” March 26, 2011 </li></ul><ul><li>Presented by the New Mexico Foundation for Open Government , </li></ul><ul><li>New Mexico Press Association and New Mexico Broadcasters Association </li></ul>This PowerPoint deck and Tipsheet posted at: http:// j o h n s o n – f o g . n o t l o n g . c o m Licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License .

3.
<ul><li>Important point </li></ul>1 Nothing is as important – and valuable – as a good theory!

4.
Theory of Journalistic Process <ul><li>Data In  Analysis  Info Out </li></ul><ul><li>Data = that which, upon Analysis, yields Information. “Data” has many forms. </li></ul><ul><li>Analysis = Examination of data and facts to uncover and understand cause-effect and contextual relationships and patterns , thus providing basis for problem solving and decision making . </li></ul><ul><li>Information = that which aids in making decisions </li></ul>

5.
<ul><li>Important point </li></ul><ul><li>The document is not the data. </li></ul>2

6.
Bertillon system: Public Records DB <ul><li>Early public records </li></ul><ul><li>Intricate data collection </li></ul><ul><li>Potential for error in data entry </li></ul><ul><li>Potential for error in filing </li></ul><ul><li>No machine retrieval or analysis </li></ul><ul><li>Even today, OCR would be impossible </li></ul>

7.
Bertillon system: Public Records DB <ul><li>By 1910… </li></ul><ul><li>Indexing system has improved </li></ul><ul><li>Typewriters instead of pen </li></ul><ul><li>Better haircuts </li></ul><ul><li>But still … </li></ul><ul><li>Null fields </li></ul><ul><li>Subject to data entry errors; lost or misfiled cards/data </li></ul><ul><li>Limited large-scale analysis resources </li></ul>

8.
Bertillon system: Public Records DB <ul><li>Early public records </li></ul><ul><li>Intricate data collection </li></ul><ul><li>Data entry potential for error </li></ul><ul><li>Filing potential for error </li></ul><ul><li>No machine retrieval or analysis </li></ul><ul><li>Even today, no OCR </li></ul><ul><li>By 1910… </li></ul><ul><li>Indexing system has improved </li></ul><ul><li>Typewriters instead of pen </li></ul><ul><li>Better haircuts </li></ul><ul><li>But still … </li></ul><ul><li>Null fields </li></ul><ul><li>Subject to data entry errors; lost or misfiled cards/data </li></ul><ul><li>Limited large-scale analysis resources </li></ul>Early “hard drives,” data retrieval and data analysis of public records

9.
Bertillon system: Public Records DB <ul><li>Early public records </li></ul><ul><li>Intricate data collection </li></ul><ul><li>Data entry potential for error </li></ul><ul><li>Filing potential for error </li></ul><ul><li>No machine retrieval or analysis </li></ul><ul><li>Even today, no OCR </li></ul><ul><li>By 1910… </li></ul><ul><li>Indexing system has improved </li></ul><ul><li>Typewriters instead of pen </li></ul><ul><li>Better haircuts </li></ul><ul><li>But still … </li></ul><ul><li>Null fields </li></ul><ul><li>Subject to data entry errors; lost or misfiled cards/data </li></ul><ul><li>Limited large-scale analysis resources </li></ul><ul><li>A public record, but one of limited usage </li></ul><ul><li>A DOCUMENT , but no efficient, productive, insightful way to FIND the data </li></ul><ul><li>A DOCUMENT , but no efficient, productive, insightful way to EXTRACT the data </li></ul><ul><li>Sorta like a PDF </li></ul>Early “hard drives,” data retrieval and data analysis of public records

10.
Traditional Data In  Analysis  Info Out Data In  Analysis  Info Out <ul><li>Notes </li></ul><ul><li>Text </li></ul><ul><li>Numeric </li></ul><ul><li>Images </li></ul><ul><li>Maps </li></ul><ul><li>How? Who? </li></ul>

11.
Digital Age Data In  Analysis  Info Out <ul><li>Notes </li></ul><ul><li>Text </li></ul><ul><li>Numeric </li></ul><ul><li>Images </li></ul><ul><li>Charts/Graphs </li></ul><ul><li>Maps </li></ul><ul><li>Audio </li></ul><ul><li>Video </li></ul><ul><li>Atoms  Bits </li></ul><ul><li>How? Who? </li></ul><ul><li>New data is ubiquitous, shareable, scaleable. </li></ul><ul><li>Retrieval, copying and storage costs trivial </li></ul><ul><li>Can be validated and explored by individuals and applications </li></ul>

12.
Digital Age Data In  Analysis  Info Out <ul><li>Notes </li></ul><ul><li>Text </li></ul><ul><li>Numeric </li></ul><ul><li>Images </li></ul><ul><li>Charts/Graphs </li></ul><ul><li>Maps </li></ul><ul><li>Audio </li></ul><ul><li>Video </li></ul><ul><li>Atoms  Bits </li></ul><ul><li>How? Who? </li></ul>} <ul><li>All data today requires NEW tools for ANALYSIS and STORY-TELLING </li></ul><ul><li>Statutes are usually adequate; the CULTURES are the challenge. </li></ul>

13.
<ul><li>Important point </li></ul><ul><li>The document is not the data . Without analysis, the data are not the story. </li></ul>3

14.
Four stories <ul><li>Doig: Hurricane Andrew, Data (from documents) = Pulitizer Prize & bldg. inspectors in jail </li></ul><ul><li>Craig Harris: “ Arizona pension systems a soaring burden ” </li></ul><ul><li>Waite: water, developers, land use = disappearing wet lands </li></ul><ul><li>UK: Investigate Your MPs Expenses “We have 458,832 pages of documents. 27,731 of you have reviewed 223,475 of them. Only 235,357 to go” MP’s expense claims on Google spreadsheet </li></ul>

15.
Journalism and GIS <ul><li>Steve Doig [Miami Herald] </li></ul><ul><li>1992 </li></ul>Hurricane Andrew + damage reports + building inspection = jail terms

16.
Doig: Hurricane Andrew

17.
Four stories <ul><li>Doig: Hurricane Andrew, Data (from documents) = Pulitizer Prize & bldg. inspectors in jail </li></ul><ul><li>Craig Harris: “ Arizona pension systems a soaring burden ” </li></ul>

18.
Analysis with real data Search Sort DB info

19.
Four stories <ul><li>Doig: Hurricane Andrew, Data (from documents) = Pulitizer Prize & bldg. inspectors in jail </li></ul><ul><li>Craig Harris: “ Arizona pension systems a soaring burden” </li></ul><ul><li>Waite: water, developers, land use = “ Vanishing Wetlands ” </li></ul>

20.
Vanishing Wetlands

21.
Four stories <ul><li>Doig: Hurricane Andrew, Data (from documents) = Pulitizer Prize & bldg. inspectors in jail </li></ul><ul><li>Craig Harris: “ Arizona pension systems a soaring burden” </li></ul><ul><li>Waite: water, developers, land use = disappearing wet lands </li></ul><ul><li>UK: Investigate Your MPs Expenses “We have 458,832 pages of documents. 27,731 of you have reviewed 223,475 of them. Only 235,357 to go” MP’s expense claims on Google spreadsheet </li></ul><ul><ul><li>EFF Seeks Cooperating FOIA Reviewers </li></ul></ul>

22.
UK MP’s expenses Solid search tools These are PDFs, POST -search

23.
Major questions? <ul><li>As participants in a liberal democracy… </li></ul><ul><li>How do we get the necessary data? </li></ul><ul><li>And from where? </li></ul><ul><li>And in appropriate forms? </li></ul>

24.
Files, Transparency, Ease of Analysis Easier Challenging

25.
Files, Transparency, Ease of Analysis

26.
Data In: Objectives/Requirements <ul><li>Move data from “out there” to analytic site/tools </li></ul><ul><li>Looking for connections; patterns </li></ul>

27.
Data In: Objectives/Requirements <ul><li>Seeking fine-grained data, NOT aggregations </li></ul><ul><ul><li>Seek data in original form (i.e. NO PDFs) </li></ul></ul><ul><ul><li>Get data in lowest common denominator format: - Comma-delimited files in ASCII or Text </li></ul></ul><ul><ul><li>Who collected the data? Why? How? </li></ul></ul><ul><ul><li>Who proofed/edited the data? Why? How? </li></ul></ul><ul><ul><li>If from data base, first ask for “record layout” or “code sheet” or “schema” </li></ul></ul><ul><ul><li>Definitions of variables or fields. Constant or ??? </li></ul></ul>

28.
Data In: “Typical” problems with gov sites <ul><ul><li>Barriers data = barriers to analysis </li></ul></ul><ul><ul><ul><li>NO site search capability; no site map </li></ul></ul></ul><ul><ul><ul><li>Failure to use open-standard HTML; using closed-standard Adobe Flash/Shockwave environment. </li></ul></ul></ul><ul><ul><ul><li>Page formats/layouts not consistent; too many drill-downs instead of search-driven generators </li></ul></ul></ul><ul><ul><ul><li>Jiggly roll-overs; too much effort spent on bling </li></ul></ul></ul><ul><ul><ul><li>Impossible to download or scrape data for analysis </li></ul></ul></ul><ul><ul><ul><li>Information available only in Adobe PDF files; notoriously unfriendly to data analysis. </li></ul></ul></ul>

29.
Good NM sites Search! Español Feedback!

30.
NM Legis. Bill Finder Could be better: no way to find what bills were introduced by X legislator Download bill in TWO formats

31.
Data In: Challenges <ul><ul><li>New site in New Mexico: www.sunshineportalnm.com </li></ul></ul><ul><ul><li>“ Beta ,” but facade for taxpayers; a secondary tax bcs of minimal utility; torture for journos </li></ul></ul>

32.
Data In: Challenges in SunshinePort <ul><ul><li>Comprehensive Annual Financial Reports </li></ul></ul><ul><ul><ul><li>Possible to machine download, but laborious to format for analysis </li></ul></ul></ul><ul><ul><li>Investment Holdings reports are far worse </li></ul></ul><ul><ul><ul><li>They are poor-quality static image files, not machine-readable. </li></ul></ul></ul><ul><ul><ul><li>Tabular data roughly formatted; makes conversion for analysis an arduous, if not impossible task. </li></ul></ul></ul>

33.
Bottom line on SunshinePortalNM.com <ul><li>“ If the State of New Mexico takes the position that through this site it is discharging all of its disclosure obligations with respect to these particular records, open government is in trouble there. ” </li></ul>“ This is not even a web page, it’s a Flash application, so there’s not going to be much sunlight escaping from this portal. “

34.
Bottom line on SunshinePortalNM.com <ul><li>“ If the State of New Mexico takes the position that through this site it is discharging all of its disclosure obligations with respect to these particular records, open government is in trouble there. ” </li></ul>“ This is not even a web page, it’s a Flash application, so there’s not going to be much sunlight escaping from this portal. “ “ A perfect example of creating the appearance of transparency without actually being transparent.”

35.
Good data sites – Gov and NGO <ul><li>Data.gov [A beta site] www.data.gov/ </li></ul><ul><ul><li>Metrics www.data.gov/metric </li></ul></ul><ul><li>DataSF - http://datasf.org/ a clearinghouse of datasets available from the City & County of San Francisco </li></ul><ul><li>San Francisco Enterprise GIS Program - http:// gispub02.sfgov.org/data.asp </li></ul><ul><li>Maplight.com – an example of how citizens can use data Nonprofit, nonpartisan research organization, provides citizens and journalists the transparency tools to shine a light on the influence of money on politics. </li></ul><ul><li>Prize-winning gov’t agency web sites: http:// www.centerdigitalgov.com/survey/88/2010 </li></ul>

36.
Common aspects? <ul><li>All have up-front search capabilities </li></ul><ul><li>All are written in “data-accessible” code </li></ul><ul><li>All data can be downloaded with “relative” ease </li></ul><ul><li>Some have various languages available </li></ul><ul><li>ALL are run by GOVERNMENT; no commercial sites </li></ul>

37.
Challenge for Watchdogs? <ul><li>Failure on the part of planners/bureaucrats to simply… </li></ul><ul><li>Give The People THEIR Data… </li></ul><ul><li>In The Most Basic, Original, Straightforward Form… </li></ul><ul><li>And Let Them Figure Out What Should Be Done With It! </li></ul><ul><li>The governor agrees </li></ul>

38.
Tomorrow? Public Access to Original Data Impact Why not?

39.
It’s not the documents, it’s the DATA! Tom Johnson Managing Director Inst. for Analytic Journalism Santa Fe, New Mexico USA t o m @ j t j o h n s o n . c o m Gracias a todos

40.
It’s not the documents, it’s the DATA! <ul><li>Presentation at </li></ul><ul><li>“ 2011 Open Government Academy” March 26, 2011 </li></ul><ul><li>Presented by the New Mexico Foundation for Open Government , </li></ul><ul><li>New Mexico Press Association and New Mexico Broadcasters Association </li></ul>This PowerPoint deck and Tipsheet posted at: http://johnson-fog.notlong.com

41.
FOI history <ul><li>The world’s rst reedom o inormation legislation was adopted by the Swedish parliament in 1766. This publication includes the English translation o this ordinance on reedom o writing and the press. The enlightenment thinker and politicianAnders </li></ul><ul><li>Chydenius (1729-180), rom the Finnish city o Kokkola, played a crucial </li></ul><ul><li>role in creating the new law. As Proessor Juha Manninen describes in his article, the key achievements o the 1766 Act were the abolishment o political censorship and the gaining o public access to government documents. Although the innovation was suspended rom 1772-1809, the principle o publicity has since remained central in the Nordic countries. </li></ul><ul><li>http://www.scribd.com/doc/5885744/The-Worlds-First-Freedom-of-Information-Act-SwedenFinland-1766 </li></ul>

42.
Early police data base: incomplete data Source: Jay, Ricky. “Grifters, Bunco Artists & Flimflammen.” Wired, Feb. 2011, p.88. http://rickyjay.com/

43.
NM HB 406 <ul><li>“… information contained in information systems databases created or maintained by or on behalf of a public body … shall be subject to disclosure to any person requesting the information in the format requested. </li></ul><ul><li>“ The information shall be provided in the most effective and efficient manner available to the custodian , as defined in the Inspection of Public Records Act. </li></ul><ul><li> B. The custodian may charge a reasonable fee for production of the information requested . The fee shall not exceed the cost of the materials and reasonable charges for the personnel required to retrieve and provide the information. </li></ul>But what if it wasn’t New Mexico state employees directly at fault?

44.
Analytic Tools <ul><li>Text </li></ul><ul><ul><li>ThemeRiver - http://infoviz.pnl.gov/research_themeriver.stm </li></ul></ul>

45.
“ Analytic tools” also for story-telling <ul><li>Spreadsheets: </li></ul><ul><ul><li>Tables, charts, infographics </li></ul></ul><ul><li>Data base programs </li></ul><ul><ul><li>Charts, graphs, data tables </li></ul></ul><ul><li>Stats programs ( SPSS or SAS or R ) </li></ul><ul><ul><li>Generate graphics </li></ul></ul><ul><li>Social network analytic graphics </li></ul><ul><li>GIS </li></ul>

46.
FOIA b(3) Exemptions Original: http://www.propublica.org/article/foia-exemptions-sunshine-law

47.
Content Analysis

48.
Content analysis of legis party text

49.
Positive example of gov’t data <ul><li>Positive example: NM Leg Bill Locator </li></ul><ul><li>http://www.nmlegis.gov/lcs/_session.aspx?chamber=H&legtype=B&legno=%20406&year=11 </li></ul>Same data available in two formats!

50.
NM HB 406 <ul><li>Senate approved 39-0 on Feb. 9 http:// www.nmlegis.gov/Sessions/11%20Regular/bills/house/HB0406.html </li></ul><ul><li>“ An Act RELATING TO PUBLIC RECORDS; PROVIDING FOR THE INSPECTION OF ELECTRONIC RECORDS.” </li></ul>

51.
“ Data In” questions Data In  Analysis  Info Out <ul><li>Notes </li></ul><ul><li>Text </li></ul><ul><li>Numeric </li></ul><ul><li>Images </li></ul><ul><li>Charts/Graphs </li></ul><ul><li>Maps </li></ul><ul><li>Audio </li></ul><ul><li>Video </li></ul><ul><li>#1 – Keep a logbook (Try using Notesync.com) </li></ul><ul><li>Qualitative and/or Quantitative? </li></ul><ul><li>Objective: strive to get the data in the most fine-grained and original form. </li></ul><ul><ul><li>Online data is rarely complete nor totally accurate </li></ul></ul><ul><li>Where is the data? In what format? I-o-P? Original digital file type(s)? </li></ul>

52.
“ Data In” questions Data In  Analysis  Info Out <ul><li>Notes </li></ul><ul><li>Text </li></ul><ul><li>Numeric </li></ul><ul><li>Images </li></ul><ul><li>Charts/Graphs </li></ul><ul><li>Maps </li></ul><ul><li>Audio </li></ul><ul><li>Video </li></ul><ul><li>#1 – Keep a logbook (Try using Notesync.com) </li></ul><ul><li>Who created the data? Why? How? Legal catalysts for creation? If so, what do they say? </li></ul><ul><li>Have definitions and collection process changed? </li></ul><ul><li>Who could review and edit the data? What was/is the vetting process to insure accuracy? </li></ul><ul><li>Who has analyzed the data? For what purpose and with what methods? </li></ul>

53.
<ul><li>Data In </li></ul><ul><li> </li></ul><ul><li>Analysis </li></ul><ul><li> </li></ul><ul><li>Info Out </li></ul>

54.
“ Analysis” phase Data In  Analysis  Info Out <ul><li>Notes </li></ul><ul><li>Text </li></ul><ul><li>Numeric </li></ul><ul><li>Images </li></ul><ul><li>Charts/Graphs </li></ul><ul><li>Maps </li></ul><ul><li>Audio </li></ul><ul><li>Video </li></ul><ul><li>Atoms  Bits How? Who? </li></ul><ul><li>What are we looking for? How can we be surprised? </li></ul><ul><li>Previous/parallel investigations? (Start with IRE site stories and tipsheets) </li></ul><ul><li>Context, i.e. past environment(s) and changes? Trends past and future? </li></ul><ul><li>Quantitative and Qualitative methods? </li></ul><ul><li>Data cleaning tools? </li></ul>

55.
“ Analysis” phase Data In  Analysis  Info Out <ul><li>Notes </li></ul><ul><li>Text </li></ul><ul><li>Numeric </li></ul><ul><li>Images </li></ul><ul><li>Charts/Graphs </li></ul><ul><li>Maps </li></ul><ul><li>Audio </li></ul><ul><li>Video </li></ul><ul><li>Atoms  Bits How? Who? </li></ul><ul><li>Measurement of phenomena </li></ul><ul><ul><li>Strength of relationships </li></ul></ul><ul><ul><li>Change </li></ul></ul><ul><li>Estimating </li></ul><ul><li>Counting </li></ul><ul><li>Statistical </li></ul><ul><li>Geostatistical </li></ul><ul><li>Social Network Analysis </li></ul><ul><li>Forensic accounting </li></ul><ul><li>Who’s your rabbi? </li></ul>

56.
Data In  Analysis  Info Out Data In  Analysis  Info Out <ul><li>Notes </li></ul><ul><li>Text </li></ul><ul><li>Numeric </li></ul><ul><li>Images </li></ul><ul><li>Charts/Graphs </li></ul><ul><li>Maps </li></ul><ul><li>Audio </li></ul><ul><li>Video </li></ul><ul><li>Atoms  Bits How? Who? </li></ul><ul><li>What are we looking for? How can we be surprised? </li></ul><ul><li>Source </li></ul><ul><li>Definition </li></ul><ul><li>Context </li></ul><ul><li>Estimating </li></ul><ul><li>Counting </li></ul><ul><li>Statistical </li></ul><ul><li>Geostatistical </li></ul><ul><li>Social Network Analysis </li></ul><ul><li>Forensic accounting </li></ul>

57.
Data In  Analysis  Info Out Data In  Analysis  Info Out <ul><li>Notes </li></ul><ul><li>Text </li></ul><ul><li>Numeric </li></ul><ul><li>Images </li></ul><ul><li>Charts/Graphs </li></ul><ul><li>Maps </li></ul><ul><li>Audio </li></ul><ul><li>Video </li></ul><ul><li>Atoms  Bits How? </li></ul><ul><li>What are we looking for? How can we be surprised? </li></ul><ul><li>Source </li></ul><ul><li>Definition </li></ul><ul><li>Context </li></ul><ul><li>Estimating </li></ul><ul><li>Counting </li></ul><ul><li>Statistical </li></ul><ul><li>Geostatistical </li></ul><ul><li>Social Network Analysis </li></ul><ul><li>Forensic accounting </li></ul><ul><li>Broadcast </li></ul><ul><li>Web </li></ul><ul><li>Audio </li></ul><ul><li>Video </li></ul><ul><li>Text </li></ul><ul><li>Data visualization </li></ul><ul><li>Maps </li></ul><ul><li>Dynamic databases </li></ul><ul><li>Archives </li></ul>

58.
Theory of Journalistic Process Copyright © J. T. Johnson Data In <ul><li>Interviews </li></ul><ul><li>Text docs </li></ul><ul><li>Clips </li></ul><ul><li>Pictures </li></ul><ul><li>Infographics </li></ul>This is a headline DATELINE -- And the traditional text story starts here and goes on and on and on. Info Out Analysis