2. Overview
What is text mining?
Text Mining in digital history
Trading Consequences
“Big data”
Visualisation
Challenge of noisy data
Collaborating with historians
Digital scholarship: day of ideas 2, Edinburgh, 02/05/2013
3. Text Mining
Describes a set of linguistic, statistical and
machine learning techniques that model and
structure the information content of textual
resources.
Turns unstructured text into structured data (e.g.
relational database or linked data).
Is very useful for analysing large text collections
automatically.
Digital scholarship: day of ideas 2, Edinburgh, 02/05/2013
4. Text Mining
TM methods often rely on a set of linguistic pre-
processing steps such as tokenisation, sentence
detection, part-of-speech tagging, lemmatisation,
syntactic parsing (chunking).
Currently our focus is on named entity
recognition, entity grounding and relation
extraction.
Digital scholarship: day of ideas 2, Edinburgh, 02/05/2013
5. TM in Digital History
Goal: By analysing large amounts of digitised
data, help historians to discover novel patterns
and explore hypothesis.
Methods: linguistic text analysis, named entity
recognition, geo-grounding and relation extraction
to transform the text into structured data.
Sea-change to methods used in ‘traditional’
history.
Digital scholarship: day of ideas 2, Edinburgh, 02/05/2013
7. Trading Consequences
Digging into Data II project (till Dec. 2013)
Edinburgh Team: Prof. Ewan Klein, Dr. Beatrice Alex,
Dr. Claire Grover, Clare Llewellyn, Richard Tobin,
James Reid, Nicola Osborne, Ian Fieldhouse
Digital scholarship: day of ideas 2, Edinburgh, 02/05/2013
9. Trading Consequences
What does archival text say about the economic
and environmental consequences of global
commodity trading during the nineteenth century?
Scope: global, but with focus on Canadian natural
resources.
Example questions:
‣ What were the routes and volumes of international trade in
resource commodities in the nineteenth century?
‣ What were the local environmental consequences of this
demand for these resources?
Digital scholarship: day of ideas 2, Edinburgh, 02/05/2013
12. Mined Information
Example sentence:
Extracted entities:
commodity: cassia bark
date: 1871
location: Padang
location: America
quantity + unit: 6,127 piculs
Digital scholarship: day of ideas 2, Edinburgh, 02/05/2013
13. Mined Information
Example sentence:
Normalised and grounded entities:
commodity: cassia bark
date: 1871 (year=1871)
location: Padang (lat=-0.94924;long=100.35427;country=ID)
location: America (lat=39.76;long=-98.50;country=n/a)
quantity + unit: 6,127 piculs
Digital scholarship: day of ideas 2, Edinburgh, 02/05/2013
14. Mined Information
Example sentence:
Extracted entity attributes and relations:
origin location: Padang
destination location: America
commodity–date relation: cassia bark – 1871
commodity–location relation: cassia bark – Padang
commodity–location relation: cassia bark – America
Digital scholarship: day of ideas 2, Edinburgh, 02/05/2013
19. Noisy Data
Optical character recognition contains many errors
and often the structure of the page layout is lost.
Sophistication of the OCR engine and scanning equipment.
Quality of the original print and paper.
Use of historical language.
Information in page margins (header, page numbers, etc.).
Information in tables.
Language of the text.
Digital scholarship: day of ideas 2, Edinburgh, 02/05/2013
20. Fixing Noisy Data
Text normalisation and correction:
End-of-line soft hyphen removal
Dehyphen all token-splitting hyphens using a dictionary-based
approach.
“False f”-to-s conversion
Convert all false f characters to s using a corpus.
Example: reduced number of words unrecognised by
spell checker from 61 to 21 -> 67%, on average 12%
reduction in word error rate in a random sample (Alex et
al, 2012).
Digital scholarship: day of ideas 2, Edinburgh, 02/05/2013
23. Extract from document
10.2307/60238580 in FCOC.
How Noisy Is Too Noisy?
qBiu si }S3A:req s,uauuaqsu aq} }Bq} uirepo.ifT'papua}X3
sSuiav }qSuq Jiaq} qiiM jib ui snnS bbs aqx 'a"3(s aq} tnojj
ssfitns q}TM Sni5[ooi si jb}s }S.ii; aqx'papnaoSB q}Bq naABSjj
qS;H °1 ssbui s.uauuaqsu aqx
Digital scholarship: day of ideas 2, Edinburgh, 02/05/2013
24. The Users (Historians)
Involvement of historians:
Everything is based on the use cases and build on users’
hypotheses/research questions.
They are responsible for identification of relevant collections
and are involved in the ontology development.
They provide feedback for us to improve technology iteratively:
Partners at York use of the prototype for their research and
track errors; Workshop at CHESS 2013 with a group of
independent historians
Clarity on the text mining accuracy is IMPORTANT.
Digital scholarship: day of ideas 2, Edinburgh, 02/05/2013
25. Summary
Text mining historic documents in Trading
Consequences.
Processing “big data”.
Power of visualising structured data.
Fixing noisy data.
Importance of two-way collaboration between
technology experts and users in digital history.
Digital scholarship: day of ideas 2, Edinburgh, 02/05/2013
26. Thank you
Questions? Fire away or contact me at:
balex@inf.ed.ac.uk
Digital scholarship: day of ideas 2, Edinburgh, 02/05/2013