14. THE DATA A FRAMEWORK EXTRACTING VALUE
Asking Questions of Big Data
ANALYTICS &
VISUALIZATION
ALGORITHMSFACTS
Facts are complicated and
rarely “just the facts”.
Questions include:
Where do the facts come from?
How were they collected?
By whom?
Under what circumstances?
Rendering facts visible
shape how we see things.
Questions include:
What is being assumed?
What is visible/invisible?
Does it scale?
Who is the audience?
Is there local knowledge?
Are only as good what we
base them on. Questions
include:
What is being assumed?
What is being erased?
What is being analysed?
What are the “models”?
What is the desired outcome?
Notas del editor
And ask the question: Does data have a secret life? Of course it does …
This looks like graffiti but to the right trained eye it is data.
This graffiti on the walls is actually the traces of the last 100 years of shearers. Each leaving their mark at the end of a shearing season … you can trace familes, boom & bust years, and changes in stenciling technology if you know what you are looking at, and looking for.
It made me think about data, and the stories it can tell … I want to reframe how we think about big data … I want to uncover its secret
Sometimes data is in the eye of the beholder
So it is great to be here to tell help unpack the stories of data
Let me back up just a little bit … and tell you about me and why you might want to listen
Grew up in Australia
Daughter of an anthropologist
Learned how to get water out of frogs
Lets start with a story … a story about big data but … perhaps not the one you are anticipating. …
Let me tell you about this man – William of Normandy … born around 1028AD in what is now France. The descendent of Viking raiders, he inherited the Duchy of Normandy before he was 10, and would spend decades successfully defending his duchy against various threats. He was also a distant contender for the English crown, through his father’s family
In 1051, King Edward makes William his likely successor but local politics intervenes. England’s11century – lots of fighting, invasions, lots of monarchs but also a strong Anglo-Saxon culture.
Edward dies in Jan 1066, and his brother-in-law Harold becomes King (SUCH COMPLICTED GENEALOGIES).
In October 1066, William of Normandy crosses the English Channel to claim the throne, in a decisive battle on the 14th of October 1066 at Hastings he kills Harold, and becomes the first Norman King of England. Ends Anglo-Saxon control of England
The first 20 years are busy. Norman rule is very different… lots of churn & thrash & radically cultural transformation – English gives way again to Latin/French … changes in ruling class, land-ownership patterns, role of the church etc.
In 1085 England faces another invasion threat. William needed to raise an army
At Christmas in Gloucester with his Great Council – he decided to conduct a full “survey” on his land – how it was occupied and by what sort of men
a written questionnaire was sent out to all the Lords: it asked about Size of land, types of occupants, stock, types of land,
Ultimately, Records from 13,418 settlements were taken in less than 8 months
The survey did NOT include Winchester or London or County Durham and Northumberland. Also didn’t include women, children …
The King’s officials then sent to verify info & collect more. Also established a circuit jury, public hearings. These were NOT popular – fights, etc …
Findings then sent to advisors & shown to King and finalized! – hence judgement day -- Domesday
Ultimately 2 volumes of the Winchester Roll were created – new categories of ownership.etc were created. (data organized by manors & landholders rather than geographically)
2 books producted – 900 sheepskins. It was of course out of date before it was even finished, but it was built to last forever.
It followed the King around England a big lock box for 200+ years before settling in Westminster Abbey.
Shaped English life (and ultimately American and Australian too – helped form the idea of a census): Taxation, Land ownership, Relations btw monarch, and everyone else; hierarchies, power
Last consulted 1960s
Nothing of its scale was attempted again until the 1800s in the UK
Still exists. Intact, and accessible: Put online in 2006
So why am I telling you about this what can we learn from the Domesday book? Well I would argue – it is a source of big data before computers!
The Domesday book is really about three quite distinct things
The survey – which collected information, facts
The Winchester Roll that was created out of the facts
The “day of judgement “when taxes/ownership & military obligations were determined for hundreds of years to come
Turns out “big data” is also made up of the same three kinds of things .. .just on awe inspiring scale … but it turns on the same things
Facts/data
Framework = visualizations & analytics
Extracting value = algorithms
and all are worthy of dissection
And just like the Domesday book, today’s big data is messy… because ultimatley Big data starts and ends with us. Big data is actually intensely human … we create it, we understand it, we use it. Big data doesn’t exist without us. WHICH MEANS Data, visualizations & analytics, and algorithms MUST all more complicated than they first appear.