Presentation from the official launch event for Pulse Lab Jakarta, held in Indonesia on 1 October 2012. Presentation includes a background on "Big Data for Development," a showcase of Pulse Lab Jakarta's initial social media analysis research results, and roadmap for the Lab. http://www.unglobalpulse.org/PLJLaunch
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Pulse Lab Jakarta Launch Presentation
1. BAPPENAS
Launch Event
Peluncuran
1 October 2012 / 1 Oktober 2012
Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
2. Ir. Wismana Adi
Suryabrata
Deputy Minister for Development Funding,
Bappenas
Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
3. Prof. Dr. Armida S.
Alisjahbana
Minister of National Development
Planning Agency / Head of Bappenas
Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
16. PRIVATE SECTOR
Monitor operations…in real time
Track market trends…in real time
Get customer feedback…in real time
GLOBAL DEVELOPMENT
Unemployment?
Food Security?
Public Health?
Education?
Migration?
Disaster Relief?
17. BIG DATA IN REAL TIME: OPPORTUNITIES
1. Better early warning: Faster detection of
anomalies in at the onset of a crisis allows rapid,
effective, efficient response.
2. Real-time awareness: A more accurate and up-to-
date picture of population needs and concerns
informs better design of programs and policies;
3. Real-time feedback: Understand sooner where
programs and policies are NOT working and
make rapid, adaptive adjustments until progress
is on track.
18. BIG DATA, PRIVACY, AND HUMAN RIGHTS
Global Pulse privacy guidelines:
• Never analyze personally identifiable data
Never analyze confidential data
• Never seek to re-identify individuals
22. MEN AND WOMEN USE THEIR PHONES DIFFERENTLY
Men: Women:
• Fewer calls • More calls
• Shorter calls • Longer calls
• Smaller social network • Larger social network
• More work-related calls • More personal calls
44. PULSE LABS
R&D INNOVATION STRATEGY
1. Partner with governments to establish Pulse Labs
around the world.
2. Build world-class teams of data scientists,
engineers, and policy experts
3. Partner with private sector for access to real-time
data and cutting edge technology
4. Design and conduct exploratory research
5. Build an open source toolkit for real-time
monitoring
6. Support broad adoption of useful innovations
7. Share everything we learn and build
46. Dr. Lukita
Dinarsyah Tuwo
Vice Minister of National Development
Planning Agency / Bappenas
Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
48. Indonesia & Social Media
Indonesia 5th in worldwide Twitter users & volume
90% of Indonesian Internet users are on Facebook
10-25% of Indonesian regularly access Internet,
including via Internet cafes and mobile devices
Language barriers, local slang and Indonesia-specific
sites present new challenges for social media
analytics.
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
49. Pulse Lab Jakarta
An innovation lab for real-time, Big Data analysis
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
50. Research Agenda
2012-2013
Regions of focus:
• Jakarta
• Jogja
• Makassar (South Sulawesi)
• Medan (North Sumatera)
Topic of focus:
• Changes in social welfare, especailly as related to
food prices, fuel prices (transport and cooking) and
employment
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
51. Research Agenda
2012-2013
Approaches for Exploration:
• Social Media & Twitter Analysis
• Mobile CDR Analysis
• Rapid Mobile Surveys
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
52. Research Questions
2012-2013
How are these topics discussed on Social Media?
• Which social media sources?
• How much relevant conversation is there?
How can we monitor how they affect Indonesians?
• Which analytical approaches?
• Which technological capabilities?
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
54. Proof of Concept Research
Initial Exploration
101001
001001
Data Sources
001110 • Sources: Twitter, Facebook, Blogs, News…
010011
• Real-time data + 2-year historical archive
Analytical Approaches
• Identify trending topics in online conversation
• Assess mood and sentiment for those topics
Results
• Correlate with official data sources (CPI,
unemployment)
• Visualize trends & map the results
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
55. Research Tool 1
Crimson Hexagon: ForSight
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
56. Research Tool 2
SAS Social Media Analytics and SAS Text Miner
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
57. The Signals Are Getting Stronger
è Big increase in volume of relevant conversations over 18 months
40000
35000
minyak
(oil)
ketahanan
pangan
(food
security)
30000
budidaya
(cul>va>on)
25000
telur
(eggs)
20000
15000
10000
5000
0
Indonesians are increasingly using social media to discuss basic needs
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
58. So Are the Temporal Correlations
è Listening to social conversations provides insight on official data
2.5
2
1.5
1
0.5
0
-‐0.5
-‐1
-‐1.5
Social
Media
Food
Index
-‐2
BAPPENAS
Food
Price
Index
-‐2.5
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
59. Overview of Preliminary Findings
Food prices influenced by global events
1 • Global soy bean shortage
• Rising prices in Indonesia
• Online complaints about affording food
• New tool to detect impact faster?
2 Fuel prices affected by government policy
• Cuts in subsidies => price rise
• Social outcry - worry, uncertainty, anger
• New tool to estimate vulnerability?
3 Social media predicts official statistics
• New tool to derive price index?
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
60. The Way Forward
Building up Pulse Lab capabilities:
• Build interdisciplinary R&D team that consists of
Government, private sector, academia, NGOs and
development partners.
• Develop a framework for ensuring data privacy
Developing Partnerships:
• Secure access to new sources of real-time data,
particularly with mobile carriers.
• Secure access to state-of-the-art technology for analysis
• Translate analysis technologies into Bahasa and other local
languages
• Collaborate with BIG and CSIRO on geo-location projects.
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
61. Next Steps
• Through key partnerships and projects, continue to
explore the usefulness of new digital data sources and
new analytical tools.
• After gaining 2 years’ of experience with the Pulse Lab
approach, assess whether this approach can contribute
usefully to improvements in social protection, early
warning, and Government Development Planning
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
64. OUTLINE
1. Mining Social Media: Topics & Research Questions
2. Research Tools & Results
• Crimson Hexagon
• SAS
3. Discussion
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
65. TOPICS FOR EXPLORATION
1. Food Prices
staple food (sembako), rice (beras), sugar (gula),
eggs (telur)
2. Fuel Prices
vehicle fuel (bensin), oil fuel (BBM), LPG (elpiji)
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
66. BIG QUESTIONS
• How are food and fuel prices in Indonesia being
discussed on Social Media?
• How can we monitor how they affect Indonesians?
• Can social media serve as a real-time approximation of
official statistics?
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
67. OUTLINE
1. Mining Social Media: Topics & Research Questions
2. Research Tools & Results
• Crimson Hexagon
• SAS
3. Discussion
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
68. Crimson Hexagon: Methods
Step
1
Human
Training
Select
Choose
Define
Label
Data
Sources
Keywords
Categories
Examples
beras Posi>ve
OR telur
OR gula Neutral
I
love
when
mrice
cooks
If
tofu
p um
eggs
instead
of
tempeh…
goes
higher,
I
can’t
Hope
prices
rmore
ore!
afford
any
ise
m :(
Nega>ve
Irrelevant
èHuman
labels
a
few
examples
to
teach
machine.
69. Crimson Hexagon: Methods
Step
2
Machine
Classifica9on
Consume
Match
Detect
Examine
Data
Sources
Keywords
Categories
Results
beras Posi>ve
+4%
OR telur
OR gula Neutral
-‐1%
Nega>ve
+3%
Irrelevant
-‐1%
èMachine
analyzes
millions
of
tweets
automa>cally.
70. Food Prices: Global Influence
23
July
25
-‐
27
July
Global
soy
bean
price
hike.
Indonesian
DPR
urges
government
produc>on
of
tempe
and
tofu
cut
back.
to
reduce
soy
bean
import
duty
and
give
subsidy.
10-‐22
July:
Not
many
men>ons
of
‘tempe’
and
‘tahu’
10
July
–
2
August
2012
Global
soy
bean
shortage.
23
-‐
28
July
‘tahu’
and
‘tempe’
become
Spikes
of
comments
on
tempe
and
tofu
hot
topics
in
Bahasa
tweets
71. Food Prices: Example Tweets
“nyokap
ngomel-‐ngomel
harga
tempe
tahu
naik
:D”
“Harga
Kedelai
Naik,
Sbecause
the
tdit,
buruan
nonton
sp
:D”
tar
“udeh..
kebanyakan
emes>nya
u
emerintah
Bprice’s
u ono
n
“mom’s
grumbling
ngomong
l P empe
tahu
eri
Subsidi”
“Harga
tempe
hprice
kedelai
nkenapa
k-‐
Nega9ve
subsidy”
ise
(Jul
26,
2012)
“Soy
bean
arga
hise,
Govt.
supposed
to
gmpor?,
rice
r
keburu
naik..,
r mmm,
aik
lagi...”
-‐ edelai
i ive
to
pmemangnya
tanah
Indonesia
ga
isubur
ud-‐-‐
Atnnouncement
with
no
t
en9ment
(Jul
2bean
“disaat
harga
tyou
tntuk
empe
nuch,
alhamdulillah
mama
msoy
8,
2012)
“stop
t..
ahu
alked
edelai?
#curhatpenggemartempe”
an
ktoo
m aik,
go
watch
i s before
the
asih
bikin
price
up
up..,
hmmm,
why
w
t price’s
again...”
“Tempe
ahu
dan
tempe
balado
:D”
e
import
soy
bean?,
is
Indonesian
land
inferCle
to
his
tofu
and
tempe
price
hike,
thanks
God
mom
sCll
can
6,
2012)
“during
t plant
soy?
#tempe-‐lover’s-‐vent-‐out”
no
emo9on
(Jul
2
-‐-‐
Realize
price
rise,
cook
spicy
tahu
and
tempe
:Confused
reac>on
to
price
rise
(Jul
28,
2012)
-‐-‐
D“
-‐-‐
Posi9ve/unaffected
by
price
rise
(Jul
30,
2012)
10
July
–
2
August
2012
Global
soy
bean
shortage.
‘tahu’
and
‘tempe’
become
hot
topics
in
Bahasa
tweets
72. Food Prices: Crisis Passes
23
July
-‐
2
Aug
‘tempe’
and
‘tahu’
hot
debate
14
-‐
21
Aug
Ramadhan
/
Idul
Fitri
10
July
–
25
August
2012
Ramadhan
/
Idul
Fitri
Food
men>ons
not
as
high
as
during
‘tempe’
&
‘tahu’
crisis
73. Food Prices: The Next Topic
“Tempe
tahu
udh
naik
harga,
skrng
daging
menyusul
+_+“
“Tempe
tahu
price
has
hiked,
now
meat
follows
+_+
”
-‐-‐
Aug
16,
2012
“Dedek
jgn
keseringan
minta
dibua>n
mie
telor
ya.
harga
sembako
“Langsung
belanja,
harga
daging
naik
!
mentok
sekilo
30rb.
#lol
“
naik.
mamah
susah.”
“Let’s
go
shopping
now,
meat
price
hike!
It
reached
Rp
30.000
per
“LiFle
kid,
please
don’t
ask
to
cook
noodle
egg
too
oGen.
The
kg.
#lol
”
staple
food
price’s
hiking.
Mom’s
troubled.”
-‐-‐
Aug
14,
2012
-‐-‐
Aug
16,
2012
14
-‐
21
August
‘daging’
(meat)
and
‘sembako’
(staple
food)
become
main
topics
74. Food Prices: Social vs Official
Spikes
in
social
media
conversa>ons
about
food
prices…
2
…
are
reflected
in
1.5
CPI,
%
change
infla>on
sta>s>cs
1
measured
by
govt.
0.5
0
-‐0.5
February
March
April
May
June
July
August
-‐1
75. Fuel Prices: Local Event
“BBM
naik
,
harga
makanan
juga
naik
x_x
“
“Fuel
price
has
hiked,
now
food
follows
x_x
”
-‐-‐
Mar
27,
2012
“Waduh
bbm
naik
harga
makanan
naik
saya
bs
agak
kurusan
ntar”
“Whoa..
Fuel
price
rise,
the
food
price
rise,
then
I
will
become
“Bagi
rakyat
kecil
bkn
masalah
u/
mngurangi
pnggunaan
BBM,
tp
slimmer”
efek
domino
dr
knaikan
BBM
(harga
pangan,
trnsportasi
umum
-‐-‐
Mar
26,
2012
naik)
yg
memberatkan”
“For
poor,
reducing
fuel
usage
is
not
the
problem,
but
the
domino
effects
of
fuel
price
rise
(food,
public
transportaCon
price
rise)
is”
-‐-‐
”
Mar
30,
2012
18
March
–
7
April
Fuel
subsidy
cut
plan
and
protests:
‘bbm’,
‘makanan’
&
‘sembako’
trending
topics
76. Fuel Prices: Timeline
27
Mar:
Massive
protests
against
fuel
price
rise
in
Jakarta.
25
Mar:
Some
DPR
members
agree
to
increase
fuel
price.
Protests
began
31
Mar:
Govt.
several
days
before.
announces
it
22
Feb:
News
of
government
plan
to
will
restore
increase
subsidized
fuel
price
subsidies.
Few
men>ons
of
“Fuel”
on
Twiner
15
Feb
–
15
April
2012
Government
plan
to
reduce
fuel
subsidies
triggered
protests
77. Comparing Crises
Tweets
about
food
18
Mar
-‐
7
Apr
23
July
-‐
2
Aug
Fuel
subsidy
cut
plan
and
‘tempe’
and
‘tahu’
hot
debate
protests
against
it
78. Social Protection Programs
‘Social
Protec>on
Program’
–
men>ons
gepng
higher
over
>me,
esp.
near
Jakarta
Governor
elec>on
11
July
‘12:
20
September
‘12:
Jakarta
governor
elec>on,
Jakarta
governor
elec>on,
1st
round
2nd
round
79. OUTLINE
1. Mining Social Media: Topics & Research Questions
2. Research Tools & Results
• Crimson Hexagon
• SAS
3. Discussion
Source:
80. SAS Analytic Workflow
1)
Over
200,000
new
5)
Explore
results
and
correlate
3)
Capture
sen>ment
with
official
sta>s>cs
to
official
Indonesian
language
and
mood
for
Bahasa
BPS
sta>s>cs
:
Consumer
Price
documents
per
day
Index
(CPI)
for
12
common
foods
Global
Pulse
Sen>ment,
Topic
&
Internet
Relevance
Mood
&
Geography
Interac>ve
Conversa>on
Filter
Influence
Categories
Dashboard
2)
Extract
conversa>ons
4)
Detect
loca>on,
price,
about
rice,
cooking
oil,
availability,
specific
fuel,
employment,
etc.
govt.
programs,
etc.
anxious,
confident,
confused,
hos>le,
sad,
happy
(-”-) ;-) ((+_+)) :@ :( :)
83. The signals are growing stronger
èBig increase in volume of relevant conversations over 18 months
40000
35000
minyak
(oil)
ketahanan
pangan
(food
security)
30000
budidaya
(cul>va>on)
25000
telur
(eggs)
20000
15000
10000
5000
0
Indonesians are increasingly using social media to discuss basic needs
84. B
So are the temporal correlations
èListening to social conversations provides insight on official data
2.5
2
1.5
1
0.5
0
-‐0.5
-‐1
-‐1.5
Social
Media
Food
Index
Social
edia
Food
Index
-‐2
BPS
Food
PFood
ndex
I
ndex
BAPPENAS
rice
I Price
-‐2.5
85. Linear Regression Analysis
Social
Media
Topics
BAPPENAS
Sta>s>cs
0.9%
Agricultural
Cul9va9on
(budidaya
pertanian)
Eggs
-‐0.5%
(telur)
Food
Security
=
0.89
Infla>on
(CPI)
0.1%
(ketahanan
pangan)
Cooking
Oil
-‐0.1%
(minyak
goreng)
Regression
Weights
Correla9on
Coefficient
The
predic>ve
power
of
0
=
no
rela>onship
each
conversa>on
topic
1
=
perfect
correla>on
86. OUTLINE
1. Mining Social Media: Topics & Research Questions
2. Research Tools & Results
• Crimson Hexagon
• SAS
3. Discussion
Source:
87. ADVANTAGES
• Real-time detection of trends to understand community
concerns & coping mechanisms
• Detect quick response to social protection plans and
policies
• Augment and complement official statistics with real
time indicators
Source:
88. CAVEATS
• Current social media users do not represent the entire
population
=> can’t measure direct response to poverty issues
• Twitter content good for immediate, short-term trends
=> does not capture long-term aspirations and intentions
• Machine classification methodologies need human
supervision
=> are not able to deal with sarcasm, and complex human
communications
• Correlation ≠ Causation
• Noise, rumors & reliability
Source:
89. NEXT STEPS
New applications
– health
– natural disasters
– intention to migrate
New datasets
– additional social media sources
– mobile phone call detail records (CDRs)
New research projects ready to start
– UNICEF (child welfare and education)
– ILO (female labor condition and employment access)
Source:
90. Oktorialdi
MA, PhD
Head of PUSDATIN, Bappenas
Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
91. Pulse Lab Operations &
Roadmap
Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
92. Innovation Lifecycle
Phase 1 – Research
PHASE 1 – Research Phase
Carry out a series of “Proof of Concept” projects exploring
the utility of various sources of new digital data:
• Exploratory research on social media, mobile data
(CDRs) and other relevant sources
• Development or adaptation of technology tools for
analysis
• Conduct analysis of historical data, and compare
against official statistics, to look for correlations and
proxy indicators
• Publication and sharing of findings
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
93. Innovation Lifecycle
Phase 2 – Prototype
PHASE 2 – Prototype Phase
If Proof-of-Concept phase proves that there are useful
insights within the new data source, built prototypes for
real-time monitoring:
• Obtain sustainable access to digital data source in real-
time
• Develop technology prototype for ongoing analysis
• Sample from 4 cities: Medan, Jakarta, Jogjakarta,
Makassar
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
94. Innovation Lifecycle
Phase 3 – Implementation
PHASE 3 – Use of Data in Real-Time
Decision Making and Planning
If prototypes are useful to the work of line ministries and
programme planning, mainstream and operationalize the
technique and the tools:
• Mainstreaming use of new indicator/data source
• Expand to larger-scale operational data analysis after a
year
• Operationalize data monitoring software, hardware and
human expertise
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
95. YEAR 1 ROADMAP
(2012 – 2013)
² Move to permanent Lab space
² Series of Proof-of-Concept research projects and prototypes
² Forge new partnerships around data, technology and
expertise
² Development of legal and data privacy frameworks
² Organize workshops & meet-ups on Big Data for
Development
² Knowledge sharing & publication of findings from research
projects
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
96. Work with Pulse Lab Jakarta
Research Project Partners
• Explore and test new digital data sources that could
contribute to real time program monitoring, or
complement an existing monitoring system.
• Test data sources that could serve as proxy
development indicators.
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
97. Partner with Pulse Lab Jakarta
Innovation Partners
• Data Sharing Partners
• Research Partners
• Technology Partners
• Host a Training Session
• Academic Fellowships
• Funding Support
Source: Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
98. Q&A
Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
99. Innovation
Showcase
CSIRO SAS International
Mercy Corps Microsoft LIFE
Nokia Life PoliticaWave
Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
100. BAPPENAS
Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012
101. Thank You!
Pulse Lab Jakarta Launch / Jakarta, Indonesia / 1 October 2012