Knowledge for Everyone
•Download as PPTX, PDF•
9 likes•1,850 views
We are living through a Data Reformation, with information increasingly available to us when we need it, and how we need it. But what are the implications for the non-data-savvy? Does sharing data just lead to greater knowledge inequalities? In this talk, Jeni will discuss how the choices that we make about sharing data can change the society we live in.
Recommended
Recommended
More Related Content
What's hot
What's hot (9)
Viewers also liked
Similar to Knowledge for Everyone
Similar to Knowledge for Everyone (20)
More from Jeni Tennison
More from Jeni Tennison (6)
Recently uploaded
Recently uploaded (20)
Knowledge for Everyone
- 1. Knowledge For Everyone Jeni Tennison – @JeniT – Technical Director, ODI
- 2. almost 500 years ago… … an open Bible
- 5. The laity, now able to read and examine traditional creedal content, was encouraged to test its fidelity to Scriptures; the Bible began to take on the character of an ur-text for faith; and a new emphasis on personal piety resulted. This required a different kind of internal balance between the new, wider accessibility of texts, and the need for informed interpretation of the Scriptures: attendance at public preaching and lecturing events grew. It also allowed individual ownership of a previously more contained theological process, so that individuals found themselves more invested in understanding and living out their faith. http://en.wikipedia.org/wiki/The_Ninety-Five_Theses
- 9. "encouraged to test its fidelity" http://prescribinganalytics.com
- 10. "a different kind of balance" http://www.opencorporates.com
- 11. "more invested individuals" http://london-fire.labs.theodi.org
- 12. people sustainable planet profit
- 13. "Paradiso Canto 31" by Gustave Doré
- 14. What is wrong with this picture?
- 18. Wealth Inequality in America – politzone
- 19. What data can you get? What can you do with it?
- 20. What data can you get? What can you do with it?
- 21. Open data Data anyone can access, use and share
- 24. private sector data http://smtm.labs.theodi.org
- 25. data created in partnership http://openstreetmap.org
- 26. Midata Data only you can access, use and share
- 27. your data (just for you) https://www.mint.com
- 28. your data (just for you) http://pancreas-api.herokuapp.com/
- 29. What data can you get? What can you do with it?
- 37. What data can you get? What can you do with it?
- 38. What data can you provide? What can you help people do with it?
- 39. 1. Give people access to their (personal) data - as spreadsheets - through OAuth'd APIs
- 40. 2. Publish open data - statistical summaries - reference data - accessible, usable, sharable
- 41. 3. Make the tools people need - visualisation & analysis - combining data
- 42. Luther Before the Diet of Worms by Anton von Werner
- 44. Jeni Tennison – @JeniT – Technical Director, ODI
- 45. Acknowledgements Wealth Inequality in America – politzone https://www.youtube.com/watch?v=QPKKQnijnsM
Editor's Notes
- Let's first rewind about 500 years. This picture is of the German Language Bible, translated by Martin Luther. The fact that this Bible was written in German was literally revolutionary. Before Martin Luther, Bibles were written in Latin; they could only be read by scholarly monks. They were closed, available only to restricted sets of people. Martin Luther made his Bible open. He translated the Bible into German, the language that people actually spoke.
- The German Language Bible was a crucial part of a massive social change that we now term The Reformation.
- Before The Reformation, the Bible was walled off behind doctrine: essentially what monks and priests said the Bible said. Everything that normal people knew about what the Bible said was received second hand. Which of course meant that the monks and priests had tremendous power. They could ignore some bits of the Bible and emphasise others, and no one would ever know. The opening of the Bible, through its translation into the language that they spoke, enabled lay people to directly engage with the word of God.
- And the effects of that were enormous. Wikipedia identifies three particular effects that I wanted to highlight. The first is that people were able to test the fidelity of what monks and priests told them against what the Bible actually said. And naturally that led to changes in the power dynamic between the people and the church. The second point is that there was still the need for informed interpretation. It wasn't the case that everyone sat and studied the Bible (although they could). There were lots of new interpretations. Some of them were slightly nuts. But the important thing was that because the Bible was open for everyone to read, there could be educated and informed arguments about these interpretations. And the third thing was that the availability of the Bible to individuals made them want to know more. They engaged more with what the Bible said, because they could read it themselves. And they changed their behaviour as a result, becoming more invested in and interested in what religion was to them.
- Obviously you're not here to listen to a lecture about religious events from 500 years ago. You're here to learn about data! So what's all that got to do with us, here and now? Well, the same changes that arose 500 years ago because of the publication of an open Bible are happening now because of the increased availability of data.
- There is the same dynamic at play. Currently, most data is locked behind a wall of analysis. We are reliant on statisticians and analysts to perform analyses, creating reports that tell us what the data says, in just the same way as lay people were reliant on monks and priests. We are reliant on them asking the questions that we are interested in and providing us with the summaries and visualisations that make sense of it for us. But in some places, people are beginning to have direct access to data, cutting through the wall of analysis. Making it possible to create new interpretations. I think that this trend will have dramatic effects on our society, just as the Reformation did. Which is why I call this…
- The Data Reformation. Now there are all sorts of things going on with data right now: commodity data services, internet of things, NoSQL, all the things that you're going to be hearing about today. There were all sorts of changes going on through the Protestant Reformation too. But to me it was the opening up of the Bible that was the most subtly revolutionary part. It was that crucial step that turned the academic arguments within the church into a global movement, into a revolution that affected people's lives. And I think the same is true in the Data Reformation. We need data to be available to enable everyone to benefit from the changes that the Data Reformation will bring. I've been quite abstract so far, so let's have a look at what that means.
- This slide shows the results of a study that was done around the prescribing of statins. Statins have fairly recently come out of patent, which means that as a GP you have a choice: you can prescribe a cheap, generic statin, or you can prescribe an expensive proprietary statin. And all the tests show that these will have exactly the same effects. So you would have thought that all GPs would prescribe generic statins and save the NHS money, right? Wrong. The NHS has known for some time that GPs continue to prescribe expensive proprietary statins, usually just because they know the brand name. So there have been various efforts to try to change GP's behaviour. But all of this has happened behind closed doors and with no real public engagement. The study shown above, Prescribing Analytics, took open data about GP prescribing behaviour and analysed the percentage of statin prescriptions that were for proprietary statins, testing what academics were saying about the kinds of prescribing behaviours that were going on. They found that £200m could be saved each year if GPs were to switch their prescribing habits. But they didn't just give that top-line figure, they also provided the tools to enable anyone to dig into the data. So you can see where you have really successful drug reps, like in the East Midlands or the South East. And you can see what your GPs prescribing behaviours are like, and maybe if you are being prescribed statins yourself have a conversation with them about switching. By making data available, there is an opportunity for those who are interested to test hypotheses, to provide new interpretations, new visualisations, bring understanding to the public and maybe even change behaviour.
- Having data available can challenge existing power relationships. This visualisation uses OpenCorporates data. OpenCorporates aims to be an open database of all the companies in the world. They take publicly available data, usually published by different governments, and bring it together in ways that individual countries can't or don't do. The particular data shown in this visualisation is about the companies that are directly and indirectly owned by Goldman Sachs. Each dot is a different company. The lines show, for the highlighted company (in this case Shire UK Limited), what companies it's owned by and what companies it owns. So in this case you can see that Shire UK Limited is owned by Goldman Sachs Plc which is registered in the US, and owns a whole bunch of companies which are registered in this massively outsized country here: the Cayman Islands. You can view this as a map of the world as seen by Goldman Sachs. You can see that the tax havens like the Cayman Islands, Ireland and Luxembourg are very important to them. Now, do you trust this data and this interpretation that I've just told you? Does the fact that this analysis is coming from OpenCorporates make it any more trustworthy than an analysis done by the government, or by the press? All data should be questioned. We should have debates about whether it's fair or accurate to visualise data in these ways. What OpenCorporates does which is great in this regard is that it captures and makes available the full provenance trail for all the data that it holds. So you can find out, for each fact, where it came from, and you can decide whether you trust it or not.
- This third example highlights the third change that came out of the Protestant Reformation: more invested individuals. A little while ago, London needed to close some fire stations. They did the usual thing that's done in this case, which is to commission a company who is good at doing this kind of thing to perform an analysis to recommend which fire stations should be closed. So they did that and they produced a report which said what to do but didn't really say why. And of course there was then lots of debate and campaigns to keep local fire stations, as there should be, but none of the response to those campaigns could point to the data, because it was all locked away behind a report. The visualisation here uses open data about the fires in London and the fire engines and other appliances that were sent to each fire to produce a map that shows the impact of closing particular fire stations. It turns out, as you'd hope, that the ones that they chose to close were those that had least impact on fire response times. But having access to the data helps to ground the debate. Different people will interpret the figures differently, or argue that response times aren't the thing that you should be measuring, or that fire station X should remain open because of the other benefits that it brings to the local area. But these are the right kind of debate to have: informed debate that is rooted in facts and values. I think that all consultations that involve data analysis should be opening up that data for others to look at, to use to support their case, to make for a more constructive dialog and for more engaged citizens.
- These examples demonstrate the kind of effect that data can have, because it enables better, more engaging, more informed decision making. We can measure the benefits against a triple bottom line, of the effect of data on people, on the planet, and on profit. Take transport data as an example. I use citymapper a lot in London. It helps me to find my way to wherever I need to go using a route and a transport method that suits me. It encourages me to walk or cycle, or to use public transport, so it makes me healthier and the environment healthier too. And they make money from people buying the app; I save money by not being late for meetings. The availability of data leads to social, environmental and economic benefit.
- I've painted a heavenly picture here, of empowered people using data to make the very best decisions, and ultimately saving the world. But this is not something that will happen naturally. It is something we have to work towards.
- What is wrong with this idealistic picture?
- What's wrong with this picture is inequality. And it's the same as the problem with the Reformation. Let's not pretend that the effect of the open, translated Bible was evenly spread. It's not like everyone was educated enough to read it, or that everyone could afford to even have a copy of the Bible they could pretend to read. The Reformation was a long term trend and it wasn't a smooth process. There were lots of ignorant peasants. We will face similar problems with the Data Reformation.
- Data is linked to other data. The more we know, the more we can know: the more we know, the more sense other data makes, the more context we have. On a global level, data is a network where the parts of the network that you can see are the parts that enable you to make decisions, the parts whose fidelity you can test, that you can use to find a different kind of balance, and that will make you invested. Different organisations and different individuals can see and make sense of different portions of the data that's available.
- It's fairly predictable what happens whenever you have a network, particularly networks of people and organisations. These are a couple of books that describe the patterns that arise: Positive Linking by Paul Ormerod and Everything is Obvious (once you know the answer) by Duncan Watts. The tldr; is that networks result in power laws. A few things win big, and there's a long tail of things that don't. So for example, if you look at paintings there are a few incredibly famous paintings, like the Mona Lisa, and a large number of paintings that practically no one has ever heard of, like the one my gran painted for me of her garden. This pattern just arises because of the network, not because of anything particularly special about the painting (although I will admit my gran's painting isn't as good as the Mona Lisa). You can't predict what will win in a network. The Mona Lisa didn't become the most famous painting in the world because it was the best, it became the most famous because it was slightly more well known than some other paintings, and then enough people talked about how well known it was, which made it even more well known, and so on. The effect just escalates. I have reinforced the effect right now by saying that this is most famous painting in the world. Networks mean power laws, and power laws mean inequality. They mean a big difference between the few and the many.
- Network effects apply to wealth, and they result in huge inequalities. This graph shows wealth inequality in America. It's a screenshot from a fantastic animation which you should all watch if you haven't seen that goes through the wealth distribution that people think America should have, compared to what they think it has (which is a little more unequal), compared to what it actually has (which is so massively more unequal that it's literally off the chart: you have to chop off the top of the bars in the distribution and stack them up at the side here to fit it in). Wealth inequality is an illustration of another power law as a result of a network. But this kind of inequality is really important because it has real effects on real people, not just whether or not a particular portrait gets hung in the Louvre. In this case, it has a real effect on the 15% of people in America below the poverty line. Now what worries me is that information inequalities are like wealth inequalities. They happen because of network effects, because data attracts data, knowledge enables more knowledge. And because knowledge enables people to make good decisions about the world and the way they live their lives, inequalities in knowledge have real impacts on real people. We're not going to be able to escape this. It's a natural law. So what interests me is what we can do about it. How can we make the distribution of information fairer than the distribution of wealth? How can we enable everyone to benefit from the Data Reformation, and not let the benefits to the few be so massively much more than the benefits to the many?
- I think it comes down to two questions: what data can people get, and what can they do with it. It's like the poor peasants in the Protestant Reformation who wanted to benefit from the Bible. To benefit they needed to get hold of a copy, and they needed to be able to read. For the many to be able to benefit from the Data Reformation, they need to be able to get hold of data, and they need to be able to do something with it. Another way of thinking about it is the way in which we deal with income inequalities. We deal with income inequalities through taxation. We get people who earn a lot of money to give a bit of it to be used to benefit everyone. Not all of it, not even a big part of it, but some of it. If we translate taxation as a method of addressing inequality into data, then we should be asking the organisations that are data rich to give some of that data to be used to benefit everyone. Again, not all of it - I'm not saying "throw open your databases for everyone to access!" – but some of it. But it's not enough to just make data available to people. That would be like giving every ignorant peasant a copy of the Bible. Unlike money, which everyone can spend, people need to be equipped with tools and understanding to be able to use data. So these are the two questions that I'm going to address now.
- First, what kind of data should people be able to get?
- The first big category of data that people can easily get is open data. Open data is data that anyone can access, use and share with other people. To be open data, people need to be able to find it and download it or access it via an API. And anyone needs to be able to do this. So not just those who have paid a subscription or only researchers, or only non-commercial organisations: everyone. They need to be able to use it, so it needs to be in a machine-readable format and preferably one that's not too obscure. Ideally it needs to be easy to use with other data as well: it needs to be easy to combine with other data to create greater insights. They also need to be able to share it with other people. That means they need to be able to republish the data if they want to, or to combine it with other data and then share that. The barriers to that are usually legal ones: the terms and conditions that say that you can download one copy for personal use, or specifically forbid redistribution. There's a lot of open data around, but it won't surprise you, given I work at the Open Data Institute, if I say that there should be more. Let's look at some examples.
- When we say "open data" we tend to think of government data, because it's really been governments that have led the way on open data. This map shows two datasets from government for example: areas that are prone to flooding from the Environment Agency, in purple, and areas where there's lots of new development, from Land Registry data, in blue. Where they overlap is where there are new builds of flood plains, which isn't really a good idea. The state has been very entrepreneurial with respect to open data. It has been leading the way. In some ways it's easy for the state to open up data because the data is collected for public good, using public funds, so should be public. But government data isn't the only kind of open data.
- Philanthropic organisations are also opening up data. This visualisation shows which donors are opening up data about their international aid programmes. There is huge scope for the philanthropic sector to open up data to enable more efficient working between philanthropic organisations, not least knowing which projects are already being funded, and for due diligence to understand which recipients make best use of their money.
- The private sector's use of open data is the least evolved, but there are some private sector organisations that are opening up data. This visualisation brings together data from several peer-to-peer lenders to look at how money is being flowing from lenders to borrowers through the peer-to-peer market. They are interesting in opening up this data because it helps the regulators to understand what they're doing, and lowers their reporting requirements. Plus being open is a marked differentiator between peer-to-peer lenders and the big banks. So what's interesting about this is that it's an example of commercial organisations opening up data because it benefits them. Not just because it's a moral thing to do. Not because they're being forced to do it. But because they benefit themselves. And there are lots of examples where the private sector can actually benefit from opening up data. They can benefit from having easier flows of information between them and their customers, suppliers and partners, and even internally. And they can benefit from encouraging innovative uses of their data, and adopting those innovations. I like to use the example of supermarkets. One piece of data that supermarkets could open up that everyone would benefit from would be simply where their stores are. Or even where the stores are and what kind of store they are, what kind of services they offer. We would benefit from that because we could find supermarkets that provided what we needed more easily. They would benefit because we'd find it easier to find their stores and shop in them, which is ultimately what earns them money. So making data open can be beneficial to the organisation that is making the data open as well as to anyone else. Businesses don't just have to do it because it's the right thing to do, to redistribute information for everyone, but because it has business benefit.
- The final kind of open data that I'll talk about is the data that we create in partnership with each other. We can do more together and this is fundamentally enabled through the data being open. This screenshot is from Open Street Map, which is like a Wikipedia for maps. Open Street Map is a crowd-sourced map, and the community around Open Street Map does amazing things together, like mapping Haiti after the earthquake there, and the Philippines in the aftermath of Typhoon Haiyan, to aid the relief efforts there. Like open source, the real power of open data is in the collaboration that it enables. When we work together, we can create open data for ourselves, from our observations of the world around us, not just passed down from information-rich organisations. This is another way in which data isn't like money: you can't just create money out of thin air, but you can create data.
- I've talked about open data a lot, but that's not the only kind of data that's important. Whereas open data is data that anyone can access, use and share, midata is data that only you – the person the data is about – can access, use and share.
- One example of this is financial data. Financial planning services like mint rely on being able to get your bank transactions and other data from your bank. In the UK there's legislation that says that certain kinds of companies, particularly banks and energy companies, must make data available to their customers. For banks that's transactions, for energy companies it's energy usage. Midata is important because getting people to engage with data also means making data engaging. Personal data - about your health, your weight, your energy usage – is engaging because it's about you. The previous examples that I had, of open data, were big, large-scale things that perhaps only data nerds would find interesting. But we all find ourselves fascinating, so access to personal data is very important.
- The other kind of personal data I wanted to highlight was personal data that we gather about ourselves, perhaps through monitors like the step tracker I have on my wrist. My devops, Sam, is diabetic so he constantly monitors his blood sugar. This screenshot is of a nice RESTful API that he made, over which he's created dashboards and so on that only he can view. There are a growing number of devices that monitor us. We should be able to get hold of the data from them so that we can set it in context, understand it, and make decisions based on it.
- So we've looked at the kinds of data that people need to be able to get hold of: open data and midata. The second question is how to enable people to do something with that data. A lot of this comes down to education and training: equipping people not just with data skills but with critical thinking skills that enable them to ask the right questions. I'm not going to focus on capability building here because it's not our day jobs. Instead I'm going to focus on the tools that make it easier for people to work with data.
- Lots of applications that monitor you will provide you with visualisations that help you to gain insights from the data that they have. Lots of organisations that have large amounts of data will provide analyses and reports that summarise that data. But these are limited and limiting. It is the pre-data-reformation approach: they provide the interface, and we, the passive observers, accept the analysis they provide. If the Data Reformation is going to work, people don't just need visualisations, they need the underlying data, so that they can perform their own analyses, ask their own questions, combine it with other data that they (and perhaps only they) have.
- If we want to make data usable by everyone we need to start simple and accessible. I know not everyone can use Excel (and many of us in this room wouldn't want to use Excel) but it's the closest we have to a universally available data analysis tool. I believe it should be the starting point for all the data that we make available, at the moment, if we want it to be accessible to people other than geeks. Sorting, filtering, charting, pivot tables: Excel, and similar spreadsheet software, can perform pretty much all the data analysis that most people need to do. There's a big difference in usability between providing a spreadsheet and providing a JSON API. They are both useful, and I'll come on to look at why APIs are necessary in a moment. But downloadable spreadsheets are the easiest way of making data accessible to the majority of people.
- People can use spreadsheets on their desktops, but there are also a growing number of online tools that provide easy-to-use interfaces for creating charts and graphs, such as Datawrapper shown here, and which are usually driven by upload of a CSV file.
- There are tools that make it easy to create beautiful, useful, insightful maps, such as cartodb. Again, the easiest way of using these tools is to upload a CSV file. We need easy to use tools like this to enable everyone to get insights into the data that they have.
- Going beyond single datasets, there are tools available to create dashboards, such as Leftronic or freeboard. Which means that people can start to bring together multiple sources of information into their personalised, up to date, view of the world. All through a web interface that you don't need to be a geek to use. This is where APIs do start to become important, because most of these dashboards can be most easily driven through APIs. So while downloads of spreadsheets addresses a lot of what people need to do, the provision of APIs unlocks a whole new range of tooling.
- Another example of API use are tools that people can use to link data-based events together, such as ifttt or zapier. So they can set things up to get messages when their step count for the day is low, reminding them to walk an extra stop.
- And as these tools get more sophisticated, they might embed logic, enable joining of data sources, and most importantly put this within the grasp of ordinary people. This is Yahoo! Pipes. And it's not within the grasp of many people, but it's the kind of tool that you could imagine making more engaging, more approachable, and then enabling people to do more things with the data that they have.
- We've looked at how data will have a revolutionary effect on society. We've looked at how the benefits won't be evenly distributed, because of the power law relationships that arise in networks. We've looked at how we might be able to tilt the balance just a little more in favour of the many rather than the few by enabling them to get more data and do more with it.
- Now let's turn it round and look at you, as technologists. The questions that you need to answer are 'what data can you provide?' and 'what can you help people do with it?'
- These are the three things you should be doing. Give people access to their data. And give it to them both as spreadsheets and through OAuth'd APIs, so that they can use all the tooling that I've been talking about to analyse and understand their data. Don't just assume that only developers would want to access this data. Think about how you can help other people take control of it. Think about not just providing them with dashboards yourself, but showing them how they can use other tools to look at the data if they want to. Show them how to integrate their data into dashboards that you don't control. Show them what other data, from other sources, they might use with their data. The more ways in which people can use the data that your tools and platforms are collecting about them, the more useful your platform is to people.
- The second thing is to publish open data for everyone to use. That might be statistical summaries of the data that you collect, like in the peer-to-peer lending example I showed which broke down lending and borrowing by region. Or it might be reference data, like the example of the locations of supermarkets that I gave. It's important that it's open data: accessible, usable and sharable by anyone, so that everyone can benefit from it. Not just graphs and analyses, the data itself, so that it can be combined with other data in ways that you haven't thought of. No one is asking for you to throw open your databases for everyone: only provide what it makes sense to provide. We need to get into having a culture of sharing what we can share if everyone is going to benefit from the Data Reformation.
- Finally, make the tools that people need, to visualise and analyse data, and to combine it with other data. Make the tools that ordinary people can understand, not just the power tools for other developers.
- This picture is of Martin Luther before the Diet of Worms (which has to be the craziest name for a court ever), making his Protestant case before kings and bishops. During the Reformation, as I've said, it was monks, priests and bishops who formed the gateway to the Bible. They held the keys that the people needed to unlock the word of God. But Martin Luther was a monk too. He unlocked the Bible and put that knowledge and power, in the hands of the people. In the Data Reformation, it is us, the technologists, who are the monks. We can choose whether to keep things closed or to open them up. It is up to us whether the Data Reformation happens at all. It is up to us whether it is for the many or for the few.
- This final picture is from the London Olympics, where Tim Berners-Lee sent the message shown here about the web: "This is For Everyone". That was his original vision. It is up to us to make the web for everyone, to make data for everyone, to provide knowledge for everyone. Thank you.