SlideShare una empresa de Scribd logo
1 de 24
Descargar para leer sin conexión
Recommending Targeted Strangers for
Answering Questions on Social media
Jalal Mahmud, Michelle Zhou, Nimrod Megiddo, Jeffrey Nichols, Clemens Drews
IBM Research – Almaden
San Jose, CA
The Buzz of the Crowd
 Hundreds of millions of people express
themselves on social media daily
– Location-based information
– Status update
– Sentiment about products or services
 The buzz of the crowd creates a unique
opportunity for building a new type of crowd-
powered information collection systems.
 Such systems will actively identify and engage
the right people at the right time on social
media to elicit desired information. E.g.
- current wait time at a restaurant
- airport security wait time
- information in an emergency situation.
 Our initial experiment when we manually
selected strangers based on their ability to
answer questions achieved 42% response rate
[Nichols et el. 2012]
•400+ million tweets daily
•3.2 billion Facebook likes
and comments daily
Our System - qCrowd
Monitors twitter stream to
identify relevant posts
Monitors twitter stream to
identify relevant posts
Generates questions and
sends them to each selected
person
Generates questions and
sends them to each selected
person
Analyzes received responses
and synthesizes the answers
together
Analyzes received responses
and synthesizes the answers
together
Evaluates the authors of
identified posts and
recommends a sub-set of
people to engage
Evaluates the authors of
identified posts and
recommends a sub-set of
people to engage
 How to identify strangers who are willing, able and ready to provide requested
information?
- Ability to provide information is domain dependent such as being at a location or
knowledge about a product/service and hence we use a set of rules to determine ability.
Key Contributions
 Features
- Set of features that are likely to impact one’s willingness and readiness to
respond
 Prediction of Response Likelihood
- A statistical model to infer the contribution of each feature to one’s willingness
and readiness, which are used to predict one’s likelihood to respond.
 Recommendation Algorithm
- A recommendation algorithm that automatically selects a set of targeted
strangers to maximize the overall response rate of an information request.
 Effectiveness
- Demonstrated effectiveness in real world scenarios and insights for building new
class of crowd-powered intelligent information collection systems.
Outline
 Background - Buzz of the Crowd
 Our System - qCrowd
 Key Contributions
 Active Engagement and Data Collection
 Baselines
 Features
 Statistical Model & Recommendation Algorithm
 Evaluation
 Summary and Future Work
Active Engagement and Data Collection
 TSA-tracker Question Datasets:
Our first two data sets were obtained
in the process of collecting location-
based information—airport security
check wait time via Twitter.
- @bbx If you went through security at JFK,
can you reply with your wait time? Info will
be used to help other travelers.
Product Question Dataset:
Collected by asking people on Twitter describing their product/service experience.
- @johnny Trying to learn about tablets...sounds like you have Galaxy Tab 10.1. How fast
is it?
Domain # of
Questions
# of
Responses
Response
Rate
TSA-tracker-1 589 245 42%
TSA-tracker-2 409 134 33%
Product 1540 474 31%
Baseline: Asking Random Strangers
 Sent questions to random people on Twitter.
@needy Doing a research about your local public safety.
Would you be willing to answer a related question?
@john Doing a survey about your local school system.
Would you be willing to answer a related question?
@dolly Collecting local weather data for a research.
Would you tell us what your local weather was last week?
Domain # of
Questions
# of
Responses
Response
Rate
Weather 187 7 3.7%
Public Safety 178 6 3.4%
Education 101 3 3.0%
 It is ineffective to ask random strangers on social media without
considering their willingness, ability, or readiness to answer
Baseline: Crowd As Human Operator
 Crowd-sourcing a human operator’s task to test a crowd’s ability to
identify the right targets.
 We conducted two surveys on CrowdFlower – a crowd-sourcing
platform.
- Willingness Survey: Asked each participant to predict if a displayed Twitter
user would be willing to respond to a given question, assuming that the user
has the ability to answer.
- Readiness Survey: Asked each participant to predict how soon the person
would respond assuming that s/he is willing to respond.
 The participants were also required to provide an explanation of
their predictions.
 We wanted to know what criteria a crowd would use to identify
targeted strangers.
Willingness Survey
 Randomly picked 200 users from each of our datasets:
- 100 participant from CrowdFlower.
- Each participant was given 2 randomly selected users for judgment.
- Participants were asked to predict if the displayed Twitter user would respond.
- Compared with Twitter user’s actual response (responded/not-responded).
 Correctness:
- 29% correct when only tweets of a user was displayed.
- 38% correct when complete twitter profile was displayed.
- The task of selecting users for question asking is also difficult for the crowd.
 Top Predictors:
- Past responsiveness and interaction behavior (57.6%)
"The user seems extremely social, both asking questions and replying to others“.
- Profile information (10.45%)
"Because him being a social media guy and his tagline saying “we should hang out”.
- Personality (7.4%)
"I think he won’t respond. Doesn't seem to be very friendly” .
- Retweeting behavior (6%)
“No. Most of the tweets are retweets instead of anything personal”.
- General tweeting activity (10.45%)
“This user tweets a lot, seems very chatty”.
Readiness Survey
 Participants Judged how soon a person would respond to an information
request, assuming that the person would respond.
- Used a multiple choice question with varied time windows as choices.
- Randomly selected 100 people from our collected datasets.
- Recruited 50 participants on CrowdFlower.
- Each of them was given two randomly chosen people and their
twitter handlers.
 Computed prediction correctness
- Compared with ground truth.
- For example, if a participant predicted that person X will respond within an hour, but the
response was not received in time, the prediction is then incorrect.
 Correctness:
- 58% correct in making prediction.
 Top Predictors:
- Activeness and Steadiness of Twitter usage: 25% .
- Promptness of Response: 30% .
Key Features for Selection of Strangers
Responsiveness Features
Responsiveness Features Computation
Mean Response Time Avg(T), T denote previous response times
Median Response Time Med(T), T denote previous response times
Mode Response Time Mod(T), T denote previous response times
Max Response Time Max(T), T denote previous response times
Min Response Time Min(T), T denote previous response times
Past Response Rate NR/ND, NR is the number of the user’s responses and ND is the
number of direct questions the user was asked in Twitter.
Proactiveness NR/NI, NR is the number of user’s responses and NI is the
number of indirect questions the user was asked in Twitter.
 We hypothesize that one’s willingness to respond to questions is
related to one’s past response behavior.
a
Key Features for Selection of Strangers
Profile Features
We use a profile-based
 CountSocialWords - Count of the following phrases in description field in user profile:
{“social”, “social media”, “social network”, “social networking”, “friend”, “tweet”,
“twitter”, “tweeting”, “tweets”, “tell”, “telling”, “talk”, “talking”, “communication”,
“communicator”}.
- Adopted from LIWC “social process” category and by observing words related to
modern social network activity.
- The intuition is that a user who has such words in her profile will be more active and
engaging than others, hence may likely to respond.
 MsgCount - Number of status messages
 DailyMsgCount - Number of status messages per day
Retweet Features
 RetweetRatio – Ratio of the total number of retweets and total number of tweets
 DailyRetweetCount - Ratio of the total number of retweets and total number of days
since the account is created.
Activity Features
Key Features for Selection of Strangers
Personality Features
 Personality traits such as Friendliness and Extraversion are intuitively related
with one’s willingness to respond to questions.
 Previous researchers have shown that word usage in one’s writings such as blogs
and essays is related with one’s personality.
Personality
Features
Total
Number
Examples Computation
LIWC 68 Comm@Communication
[admit, advice, affair*,
apolog*, …]
Let g be a LIWC category, Ng denotes the
number of occurrences of words in that
category in one’s tweets and N denotes the
total number of words in his/her tweets. A
score for category g is then: Ng/N.
Big Five 5 Extraversion Using correlations with LIWC features as
reported by previous researchers (e.g.,
Yarkoni et al.)
Big Five
Facets
30 Friendliness, Anxiety Using correlations with LIWC features as
reported by previous researchers (e.g.,
Yarkoni et al.)
Key Features for Selection of Strangers
Readiness Features
 Even if a person is willing to respond to questions, he/she may not be ready to
respond at the time of questioning.
 Since one’s readiness is highly context dependent (e.g. mobile device in use to
send answers is running out of battery) and often difficult to capture
computationally, we use several features to approximate one’s readiness:
Readiness Features Computation
Tweeting Likelihood of the Day TD/N, where TD is the number of tweets sent by the user
on day D and N is the total number of tweets.
Tweeting Likelihood of the
Hour
TH/N, where TH is the number of tweets sent by the user
on hour H and N is the total number of tweets.
Tweeting Steadiness 1/σ, where σ is the standard deviation of the elapsed time
between consecutive tweets of users, computed from
users’ most recent K tweets (where K is set, for example,
to 20).
Tweeting Inactivity TQ - TL, where TQ is the time the question was sent and
TL is the time the user last tweeted.
Feature Analysis
Significant Features
- Statistical significance test was done using Chi-square test with Bonferroni correction.
- For TSA-tracker-1 dataset, we found 42 significant features (FDR was 2.8%).
- For Product dataset, we found 31 features as significant (FDR 4.2%)
- For TSA-tracker-2 dataset, we found 11 significant features (FDR 11.2%)
Top-4 Features
Features Feature-type
Communication personality
Past response rate responsiveness
tweeting inactivity readiness
tweeting likelihood of the
day
readiness
- Top-4 features were found using
extensive experiments.
Dataset Top ten statistically significant features
TSA-
tracker-1
Past Response Rate, Tweet Inactivity, Negative
Emotions, Cautiousness, Depression,
Excitement-Seeking, DailyMsgCount, Intellect,
Communication, Immoderation
TSA-
tracker-2
Prepositions, Past, Exclusion, Sensation, Past
Response Rate, Space, Tweeting Steadiness,
Achievement-striving, Agreeableness,
CountSocialWords
Product Mode Response Time, Tweet Inactivity,
Activity Level, Depression, Present,
Cautiousness, Positive Emotion, Excitement-
Seeking, DailyMsgCount, Past Response Rate
Statistical Model & Recommendation
Binary-classification
 Classify a person as responder or non-responder and send questions to
people who are classified as responders.
Top-K Selection
 Rank according to their probabilities computed by the statistical model and
select top-K people to send questions.
Our Recommendation Algorithm
 Automatically selects a subset of people from a set of available people.
 Such subset selection is designed with the goal of maximizing the response rate.
 Once features are computed, we trained statistical model such as Support
Vector Machine and Logistic Regression to predict likelihood of response.
Statistical Model
Recommendation Algorithm
 Ranks people in the training set in the order of non-decreasing
probabilities.
 Finds the best interval that has the maximum response rate among all
interval subsets in the linear order.
 Computes best subinterval in the test set from the best subinterval in
training set using simple linear projection.
 Can apply various constraints on selecting minimum/maximum/exact
number of people.
- select at-least/at-most/exactly K% people
Evaluation
Evaluating Prediction Model
TSA-tracker-1 TSA-tracker-2 Product
SVM Logistic SVM Logistic SVM Logistic
Precision 0.62 0.60 0.52 0.51 0.67 0.654
Recall 0.63 0.61 0.53 0.55 0.71 0.62
F1 0.625 0.606 0.525 0.53 0.689 0.625
AUC 0.657 0.599 0.592 0.514 0.716 0.55
5-fold cross validation experiments
AUC – Area Under ROC Curve
F1 = Harmonic mean of precision and recall
 Our models are 60-70% correct in making a prediction.
Evaluating Recommendation Algorithm
Comparison of Average Response Rates using Different Approaches
TSA-tracker-1 TSA-tracker-2 Product
Baseline 42% 33% 31%
Binary-classification 62% 52% 67%
Top-K-Selection 61% 54% 67%
Our Algorithm 67% 56% 69%
 Baseline is the response rates achieved by a human operator during data collection
 Used “asking at least K% of people from the original set” as a constraint to
search for the interval that maximizes the response rate.
 Computed the response rates on with varied K (e.g., K=5%, …,90%) to find the respective
optimal intervals.
 Computed the response rates achieved using a simple binary classification (response rate
is the precision of the predictive model) and simply selecting the top- K (e.g., K=5%, …,
90%.) people by their computed probabilities.
Recall of Recommendation
Selecting K% people Response Rate Recall
25% 76% 37%
50% 68% 64%
75% 53% 82%
100% 31% 100%
Response Rate and Recall for Our Algorithm with Fixed Size (Product Data, all features, SVM Model)
 Trade-off between response rate and recommendation recall, which
captures the ratio of the actual responders our algorithm identifies for
sending questions.
Use of Different Feature Sets
Feature Set Response Rate
TSA-tracker-1 TSA-tracker-2 Product
All 0.79 0.72 0.78
Significant 0.83 0.75 0.82
Top-10 Significant 0.83 0.74 0.81
Top-4 features 0.82 0.73 0.83
Common Significant features 0.81 0.72 0.82
Selecting at-least 5% people, SVM Model
Comparison of Average Response Rates
Live Experiments
 Used Twitter’s Search API and a set of rules to find 500 users who
mentioned that they were at any US airport in their tweets.
- Randomly asked 100 users for the security wait time.
- Used our algorithm to identify 100 users for questioning from the remaining 400 users.
- Used the SVM-based model with the identified significant features.
- Waited 48 hours for the responses.
 The same process was repeated for sending product questions.
 Large improvement of response rate in a live setting.
Live Experiment Random Selection Our Algorithm
TSA-Tracker-1 29% 66%
Product 26% 60%
Summary & Future Work
 We focused on modeling users willingness and readiness to answer questions.
 We can predict one’s likelihood of response to questions and identified sub-set
of features that have significant prediction power.
 Our experiments including the live one in a real-world setting demonstrated our
approach’s effectiveness in maximizing the response rate.
 Future Work
- Applicability
- Apply to other social media platforms
- Apply to other information collection applications
- Handling Skew in the user base
- Identify inactive users similar to active users in terms of personality
- Modeling the fitness of a stranger to engage
- Develop model to receive high quality response
- Model dutifulness and trustworthiness of users
- Handling complex situations
- Incorporate various costs/benefits which might change over time
- Develop model to maximize expected net benefit
- Handling unexpected answer
- Incorporate voluntary responses from people in social media
- Grow potential targets
- Protecting Privacy
- Tune selection algorithm to exclude people who are concerns about privacy
Questions?

Más contenido relacionado

La actualidad más candente

Finding Pattern in Dynamic Network Analysis
Finding Pattern in Dynamic Network AnalysisFinding Pattern in Dynamic Network Analysis
Finding Pattern in Dynamic Network AnalysisAndry Alamsyah
 
IRJET- Real Time Sentiment Analysis of Political Twitter Data using Machi...
IRJET-  	  Real Time Sentiment Analysis of Political Twitter Data using Machi...IRJET-  	  Real Time Sentiment Analysis of Political Twitter Data using Machi...
IRJET- Real Time Sentiment Analysis of Political Twitter Data using Machi...IRJET Journal
 
Detection of cyber-bullying
Detection of cyber-bullying Detection of cyber-bullying
Detection of cyber-bullying Ziar Khan
 
Automatic Hate Speech Detection: A Literature Review
Automatic Hate Speech Detection: A Literature ReviewAutomatic Hate Speech Detection: A Literature Review
Automatic Hate Speech Detection: A Literature ReviewDr. Amarjeet Singh
 
An exaustive survey of trust models in p2 p network
An exaustive survey of trust models in p2 p networkAn exaustive survey of trust models in p2 p network
An exaustive survey of trust models in p2 p networkijwscjournal
 
IRJET - Fake News Detection using Machine Learning
IRJET -  	  Fake News Detection using Machine LearningIRJET -  	  Fake News Detection using Machine Learning
IRJET - Fake News Detection using Machine LearningIRJET Journal
 
09 Respondent Driven Sampling and Network Sampling with Memory
09 Respondent Driven Sampling and Network Sampling with Memory09 Respondent Driven Sampling and Network Sampling with Memory
09 Respondent Driven Sampling and Network Sampling with Memorydnac
 
06 Network Study Design: Ethical Considerations and Safeguards
06 Network Study Design: Ethical Considerations and Safeguards06 Network Study Design: Ethical Considerations and Safeguards
06 Network Study Design: Ethical Considerations and Safeguardsdnac
 
03 Ego Network Analysis
03 Ego Network Analysis03 Ego Network Analysis
03 Ego Network Analysisdnac
 
A Novel Frame Work System Used In Mobile with Cloud Based Environment
A Novel Frame Work System Used In Mobile with Cloud Based EnvironmentA Novel Frame Work System Used In Mobile with Cloud Based Environment
A Novel Frame Work System Used In Mobile with Cloud Based Environmentpaperpublications3
 
A potency relation for worms and next generation attack tools
A potency relation for worms and next generation attack toolsA potency relation for worms and next generation attack tools
A potency relation for worms and next generation attack toolsUltraUploader
 
Twitter Based Sentiment Analysis of Each Presidential Candidate Using Long Sh...
Twitter Based Sentiment Analysis of Each Presidential Candidate Using Long Sh...Twitter Based Sentiment Analysis of Each Presidential Candidate Using Long Sh...
Twitter Based Sentiment Analysis of Each Presidential Candidate Using Long Sh...CSCJournals
 
IRJET- Tweet Segmentation and its Application to Named Entity Recognition
IRJET- Tweet Segmentation and its Application to Named Entity RecognitionIRJET- Tweet Segmentation and its Application to Named Entity Recognition
IRJET- Tweet Segmentation and its Application to Named Entity RecognitionIRJET Journal
 
12 Network Experiments and Interventions: Studying Information Diffusion and ...
12 Network Experiments and Interventions: Studying Information Diffusion and ...12 Network Experiments and Interventions: Studying Information Diffusion and ...
12 Network Experiments and Interventions: Studying Information Diffusion and ...dnac
 
Exploring the Psychological Mechanisms used in Ransomware Splash Screens
Exploring the Psychological Mechanisms used in Ransomware Splash ScreensExploring the Psychological Mechanisms used in Ransomware Splash Screens
Exploring the Psychological Mechanisms used in Ransomware Splash ScreensJeremiah Grossman
 
2015.07.18 mor the business communicator as presence allocator
2015.07.18 mor the business communicator as presence allocator2015.07.18 mor the business communicator as presence allocator
2015.07.18 mor the business communicator as presence allocatorAdela Rebancos
 
Entity linking with a knowledge base issues techniques and solutions
Entity linking with a knowledge base issues techniques and solutionsEntity linking with a knowledge base issues techniques and solutions
Entity linking with a knowledge base issues techniques and solutionsCloudTechnologies
 

La actualidad más candente (19)

Finding Pattern in Dynamic Network Analysis
Finding Pattern in Dynamic Network AnalysisFinding Pattern in Dynamic Network Analysis
Finding Pattern in Dynamic Network Analysis
 
IRJET- Real Time Sentiment Analysis of Political Twitter Data using Machi...
IRJET-  	  Real Time Sentiment Analysis of Political Twitter Data using Machi...IRJET-  	  Real Time Sentiment Analysis of Political Twitter Data using Machi...
IRJET- Real Time Sentiment Analysis of Political Twitter Data using Machi...
 
Detection of cyber-bullying
Detection of cyber-bullying Detection of cyber-bullying
Detection of cyber-bullying
 
Automatic Hate Speech Detection: A Literature Review
Automatic Hate Speech Detection: A Literature ReviewAutomatic Hate Speech Detection: A Literature Review
Automatic Hate Speech Detection: A Literature Review
 
An exaustive survey of trust models in p2 p network
An exaustive survey of trust models in p2 p networkAn exaustive survey of trust models in p2 p network
An exaustive survey of trust models in p2 p network
 
IRJET - Fake News Detection using Machine Learning
IRJET -  	  Fake News Detection using Machine LearningIRJET -  	  Fake News Detection using Machine Learning
IRJET - Fake News Detection using Machine Learning
 
32 99-1-pb
32 99-1-pb32 99-1-pb
32 99-1-pb
 
09 Respondent Driven Sampling and Network Sampling with Memory
09 Respondent Driven Sampling and Network Sampling with Memory09 Respondent Driven Sampling and Network Sampling with Memory
09 Respondent Driven Sampling and Network Sampling with Memory
 
06 Network Study Design: Ethical Considerations and Safeguards
06 Network Study Design: Ethical Considerations and Safeguards06 Network Study Design: Ethical Considerations and Safeguards
06 Network Study Design: Ethical Considerations and Safeguards
 
03 Ego Network Analysis
03 Ego Network Analysis03 Ego Network Analysis
03 Ego Network Analysis
 
A Novel Frame Work System Used In Mobile with Cloud Based Environment
A Novel Frame Work System Used In Mobile with Cloud Based EnvironmentA Novel Frame Work System Used In Mobile with Cloud Based Environment
A Novel Frame Work System Used In Mobile with Cloud Based Environment
 
A potency relation for worms and next generation attack tools
A potency relation for worms and next generation attack toolsA potency relation for worms and next generation attack tools
A potency relation for worms and next generation attack tools
 
Twitter Based Sentiment Analysis of Each Presidential Candidate Using Long Sh...
Twitter Based Sentiment Analysis of Each Presidential Candidate Using Long Sh...Twitter Based Sentiment Analysis of Each Presidential Candidate Using Long Sh...
Twitter Based Sentiment Analysis of Each Presidential Candidate Using Long Sh...
 
IRJET- Tweet Segmentation and its Application to Named Entity Recognition
IRJET- Tweet Segmentation and its Application to Named Entity RecognitionIRJET- Tweet Segmentation and its Application to Named Entity Recognition
IRJET- Tweet Segmentation and its Application to Named Entity Recognition
 
12 Network Experiments and Interventions: Studying Information Diffusion and ...
12 Network Experiments and Interventions: Studying Information Diffusion and ...12 Network Experiments and Interventions: Studying Information Diffusion and ...
12 Network Experiments and Interventions: Studying Information Diffusion and ...
 
[IJET-V2I2P3] Authors:Vijay Choure, Kavita Mahajan ,Nikhil Patil, Aishwarya N...
[IJET-V2I2P3] Authors:Vijay Choure, Kavita Mahajan ,Nikhil Patil, Aishwarya N...[IJET-V2I2P3] Authors:Vijay Choure, Kavita Mahajan ,Nikhil Patil, Aishwarya N...
[IJET-V2I2P3] Authors:Vijay Choure, Kavita Mahajan ,Nikhil Patil, Aishwarya N...
 
Exploring the Psychological Mechanisms used in Ransomware Splash Screens
Exploring the Psychological Mechanisms used in Ransomware Splash ScreensExploring the Psychological Mechanisms used in Ransomware Splash Screens
Exploring the Psychological Mechanisms used in Ransomware Splash Screens
 
2015.07.18 mor the business communicator as presence allocator
2015.07.18 mor the business communicator as presence allocator2015.07.18 mor the business communicator as presence allocator
2015.07.18 mor the business communicator as presence allocator
 
Entity linking with a knowledge base issues techniques and solutions
Entity linking with a knowledge base issues techniques and solutionsEntity linking with a knowledge base issues techniques and solutions
Entity linking with a knowledge base issues techniques and solutions
 

Destacado

Kerajaan negeri
Kerajaan negeriKerajaan negeri
Kerajaan negerimunnianwar
 
Yang di pertuan agong
Yang di pertuan agongYang di pertuan agong
Yang di pertuan agongmunnianwar
 
Majlis raja raja
Majlis raja rajaMajlis raja raja
Majlis raja rajamunnianwar
 
Ciri ciri Negara Malaysia
Ciri ciri  Negara MalaysiaCiri ciri  Negara Malaysia
Ciri ciri Negara Malaysiamunnianwar
 
Proses pembentukan malaysia
Proses pembentukan malaysiaProses pembentukan malaysia
Proses pembentukan malaysiamunnianwar
 
Badan Perundangan Parlimen
Badan Perundangan ParlimenBadan Perundangan Parlimen
Badan Perundangan Parlimenmunnianwar
 
Ciri – ciri negara
Ciri – ciri negara Ciri – ciri negara
Ciri – ciri negara munnianwar
 

Destacado (12)

Pengajian Am Konsep Negara
Pengajian Am Konsep NegaraPengajian Am Konsep Negara
Pengajian Am Konsep Negara
 
Kerajaan negeri
Kerajaan negeriKerajaan negeri
Kerajaan negeri
 
Yang di pertuan agong
Yang di pertuan agongYang di pertuan agong
Yang di pertuan agong
 
Majlis raja raja
Majlis raja rajaMajlis raja raja
Majlis raja raja
 
Ciri ciri Negara Malaysia
Ciri ciri  Negara MalaysiaCiri ciri  Negara Malaysia
Ciri ciri Negara Malaysia
 
Proses pembentukan malaysia
Proses pembentukan malaysiaProses pembentukan malaysia
Proses pembentukan malaysia
 
Kabinet
KabinetKabinet
Kabinet
 
Kementerian
KementerianKementerian
Kementerian
 
Badan Perundangan Parlimen
Badan Perundangan ParlimenBadan Perundangan Parlimen
Badan Perundangan Parlimen
 
Negara autokrasi modern
Negara autokrasi modernNegara autokrasi modern
Negara autokrasi modern
 
Ciri – ciri negara
Ciri – ciri negara Ciri – ciri negara
Ciri – ciri negara
 
Perlembagaan
PerlembagaanPerlembagaan
Perlembagaan
 

Similar a Response modeling-iui-2013-talk

Social Engineering threats and Countermeasures in SHCT
Social Engineering threats and Countermeasures in SHCTSocial Engineering threats and Countermeasures in SHCT
Social Engineering threats and Countermeasures in SHCTijcnes
 
76201960
7620196076201960
76201960IJRAT
 
Engaging with Users on Public Social Media
Engaging with Users on Public Social MediaEngaging with Users on Public Social Media
Engaging with Users on Public Social MediaJeffrey Nichols
 
IRJET- An Experimental Evaluation of Mechanical Properties of Bamboo Fiber Re...
IRJET- An Experimental Evaluation of Mechanical Properties of Bamboo Fiber Re...IRJET- An Experimental Evaluation of Mechanical Properties of Bamboo Fiber Re...
IRJET- An Experimental Evaluation of Mechanical Properties of Bamboo Fiber Re...IRJET Journal
 
Problem statement-1-friend-affinity-finder
Problem statement-1-friend-affinity-finderProblem statement-1-friend-affinity-finder
Problem statement-1-friend-affinity-finderAmitabhDas22
 
Practical Applications for Social Network Analysis in Public Sector Marketing...
Practical Applications for Social Network Analysis in Public Sector Marketing...Practical Applications for Social Network Analysis in Public Sector Marketing...
Practical Applications for Social Network Analysis in Public Sector Marketing...Mike Kujawski
 
Project for executive summary v2
Project for executive summary v2Project for executive summary v2
Project for executive summary v200000000A1
 
Group Chapter Presentation
Group Chapter PresentationGroup Chapter Presentation
Group Chapter Presentationfeinal
 
A NOVEL APPROACH FOR TWITTER SENTIMENT ANALYSIS USING HYBRID CLASSIFIER
A NOVEL APPROACH FOR TWITTER SENTIMENT ANALYSIS USING HYBRID CLASSIFIERA NOVEL APPROACH FOR TWITTER SENTIMENT ANALYSIS USING HYBRID CLASSIFIER
A NOVEL APPROACH FOR TWITTER SENTIMENT ANALYSIS USING HYBRID CLASSIFIERIRJET Journal
 
Beginner's Guide: Social Listening
Beginner's Guide: Social ListeningBeginner's Guide: Social Listening
Beginner's Guide: Social ListeningJason Cruz
 
Outline Of Explicit Ideas And Information
Outline Of Explicit Ideas And InformationOutline Of Explicit Ideas And Information
Outline Of Explicit Ideas And InformationJennifer Wright
 
Sentiment analysis using machine learning and deep Learning
Sentiment analysis using machine learning and deep LearningSentiment analysis using machine learning and deep Learning
Sentiment analysis using machine learning and deep LearningVenkat Projects
 
IRJET- Personality Recognition using Social Media Data
IRJET- Personality Recognition using Social Media DataIRJET- Personality Recognition using Social Media Data
IRJET- Personality Recognition using Social Media DataIRJET Journal
 

Similar a Response modeling-iui-2013-talk (20)

Social Engineering threats and Countermeasures in SHCT
Social Engineering threats and Countermeasures in SHCTSocial Engineering threats and Countermeasures in SHCT
Social Engineering threats and Countermeasures in SHCT
 
76201960
7620196076201960
76201960
 
F017433947
F017433947F017433947
F017433947
 
Engaging with Users on Public Social Media
Engaging with Users on Public Social MediaEngaging with Users on Public Social Media
Engaging with Users on Public Social Media
 
ACCESS Behaviors
ACCESS BehaviorsACCESS Behaviors
ACCESS Behaviors
 
IRJET- An Experimental Evaluation of Mechanical Properties of Bamboo Fiber Re...
IRJET- An Experimental Evaluation of Mechanical Properties of Bamboo Fiber Re...IRJET- An Experimental Evaluation of Mechanical Properties of Bamboo Fiber Re...
IRJET- An Experimental Evaluation of Mechanical Properties of Bamboo Fiber Re...
 
Problem statement-1-friend-affinity-finder
Problem statement-1-friend-affinity-finderProblem statement-1-friend-affinity-finder
Problem statement-1-friend-affinity-finder
 
Practical Applications for Social Network Analysis in Public Sector Marketing...
Practical Applications for Social Network Analysis in Public Sector Marketing...Practical Applications for Social Network Analysis in Public Sector Marketing...
Practical Applications for Social Network Analysis in Public Sector Marketing...
 
Project for executive summary v2
Project for executive summary v2Project for executive summary v2
Project for executive summary v2
 
E017433538
E017433538E017433538
E017433538
 
Misinformation And False Memory Essay
Misinformation And False Memory EssayMisinformation And False Memory Essay
Misinformation And False Memory Essay
 
Group Chapter Presentation
Group Chapter PresentationGroup Chapter Presentation
Group Chapter Presentation
 
Brm unit iii - cheet sheet
Brm   unit iii - cheet sheetBrm   unit iii - cheet sheet
Brm unit iii - cheet sheet
 
A NOVEL APPROACH FOR TWITTER SENTIMENT ANALYSIS USING HYBRID CLASSIFIER
A NOVEL APPROACH FOR TWITTER SENTIMENT ANALYSIS USING HYBRID CLASSIFIERA NOVEL APPROACH FOR TWITTER SENTIMENT ANALYSIS USING HYBRID CLASSIFIER
A NOVEL APPROACH FOR TWITTER SENTIMENT ANALYSIS USING HYBRID CLASSIFIER
 
Beginner's Guide: Social Listening
Beginner's Guide: Social ListeningBeginner's Guide: Social Listening
Beginner's Guide: Social Listening
 
757
757757
757
 
Outline Of Explicit Ideas And Information
Outline Of Explicit Ideas And InformationOutline Of Explicit Ideas And Information
Outline Of Explicit Ideas And Information
 
Sentiment analysis using machine learning and deep Learning
Sentiment analysis using machine learning and deep LearningSentiment analysis using machine learning and deep Learning
Sentiment analysis using machine learning and deep Learning
 
IRJET- Personality Recognition using Social Media Data
IRJET- Personality Recognition using Social Media DataIRJET- Personality Recognition using Social Media Data
IRJET- Personality Recognition using Social Media Data
 
Managerialstatistics
ManagerialstatisticsManagerialstatistics
Managerialstatistics
 

Último

COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 

Último (20)

COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 

Response modeling-iui-2013-talk

  • 1. Recommending Targeted Strangers for Answering Questions on Social media Jalal Mahmud, Michelle Zhou, Nimrod Megiddo, Jeffrey Nichols, Clemens Drews IBM Research – Almaden San Jose, CA
  • 2. The Buzz of the Crowd  Hundreds of millions of people express themselves on social media daily – Location-based information – Status update – Sentiment about products or services  The buzz of the crowd creates a unique opportunity for building a new type of crowd- powered information collection systems.  Such systems will actively identify and engage the right people at the right time on social media to elicit desired information. E.g. - current wait time at a restaurant - airport security wait time - information in an emergency situation.  Our initial experiment when we manually selected strangers based on their ability to answer questions achieved 42% response rate [Nichols et el. 2012] •400+ million tweets daily •3.2 billion Facebook likes and comments daily
  • 3. Our System - qCrowd Monitors twitter stream to identify relevant posts Monitors twitter stream to identify relevant posts Generates questions and sends them to each selected person Generates questions and sends them to each selected person Analyzes received responses and synthesizes the answers together Analyzes received responses and synthesizes the answers together Evaluates the authors of identified posts and recommends a sub-set of people to engage Evaluates the authors of identified posts and recommends a sub-set of people to engage  How to identify strangers who are willing, able and ready to provide requested information? - Ability to provide information is domain dependent such as being at a location or knowledge about a product/service and hence we use a set of rules to determine ability.
  • 4. Key Contributions  Features - Set of features that are likely to impact one’s willingness and readiness to respond  Prediction of Response Likelihood - A statistical model to infer the contribution of each feature to one’s willingness and readiness, which are used to predict one’s likelihood to respond.  Recommendation Algorithm - A recommendation algorithm that automatically selects a set of targeted strangers to maximize the overall response rate of an information request.  Effectiveness - Demonstrated effectiveness in real world scenarios and insights for building new class of crowd-powered intelligent information collection systems.
  • 5. Outline  Background - Buzz of the Crowd  Our System - qCrowd  Key Contributions  Active Engagement and Data Collection  Baselines  Features  Statistical Model & Recommendation Algorithm  Evaluation  Summary and Future Work
  • 6. Active Engagement and Data Collection  TSA-tracker Question Datasets: Our first two data sets were obtained in the process of collecting location- based information—airport security check wait time via Twitter. - @bbx If you went through security at JFK, can you reply with your wait time? Info will be used to help other travelers. Product Question Dataset: Collected by asking people on Twitter describing their product/service experience. - @johnny Trying to learn about tablets...sounds like you have Galaxy Tab 10.1. How fast is it? Domain # of Questions # of Responses Response Rate TSA-tracker-1 589 245 42% TSA-tracker-2 409 134 33% Product 1540 474 31%
  • 7. Baseline: Asking Random Strangers  Sent questions to random people on Twitter. @needy Doing a research about your local public safety. Would you be willing to answer a related question? @john Doing a survey about your local school system. Would you be willing to answer a related question? @dolly Collecting local weather data for a research. Would you tell us what your local weather was last week? Domain # of Questions # of Responses Response Rate Weather 187 7 3.7% Public Safety 178 6 3.4% Education 101 3 3.0%  It is ineffective to ask random strangers on social media without considering their willingness, ability, or readiness to answer
  • 8. Baseline: Crowd As Human Operator  Crowd-sourcing a human operator’s task to test a crowd’s ability to identify the right targets.  We conducted two surveys on CrowdFlower – a crowd-sourcing platform. - Willingness Survey: Asked each participant to predict if a displayed Twitter user would be willing to respond to a given question, assuming that the user has the ability to answer. - Readiness Survey: Asked each participant to predict how soon the person would respond assuming that s/he is willing to respond.  The participants were also required to provide an explanation of their predictions.  We wanted to know what criteria a crowd would use to identify targeted strangers.
  • 9. Willingness Survey  Randomly picked 200 users from each of our datasets: - 100 participant from CrowdFlower. - Each participant was given 2 randomly selected users for judgment. - Participants were asked to predict if the displayed Twitter user would respond. - Compared with Twitter user’s actual response (responded/not-responded).  Correctness: - 29% correct when only tweets of a user was displayed. - 38% correct when complete twitter profile was displayed. - The task of selecting users for question asking is also difficult for the crowd.  Top Predictors: - Past responsiveness and interaction behavior (57.6%) "The user seems extremely social, both asking questions and replying to others“. - Profile information (10.45%) "Because him being a social media guy and his tagline saying “we should hang out”. - Personality (7.4%) "I think he won’t respond. Doesn't seem to be very friendly” . - Retweeting behavior (6%) “No. Most of the tweets are retweets instead of anything personal”. - General tweeting activity (10.45%) “This user tweets a lot, seems very chatty”.
  • 10. Readiness Survey  Participants Judged how soon a person would respond to an information request, assuming that the person would respond. - Used a multiple choice question with varied time windows as choices. - Randomly selected 100 people from our collected datasets. - Recruited 50 participants on CrowdFlower. - Each of them was given two randomly chosen people and their twitter handlers.  Computed prediction correctness - Compared with ground truth. - For example, if a participant predicted that person X will respond within an hour, but the response was not received in time, the prediction is then incorrect.  Correctness: - 58% correct in making prediction.  Top Predictors: - Activeness and Steadiness of Twitter usage: 25% . - Promptness of Response: 30% .
  • 11. Key Features for Selection of Strangers Responsiveness Features Responsiveness Features Computation Mean Response Time Avg(T), T denote previous response times Median Response Time Med(T), T denote previous response times Mode Response Time Mod(T), T denote previous response times Max Response Time Max(T), T denote previous response times Min Response Time Min(T), T denote previous response times Past Response Rate NR/ND, NR is the number of the user’s responses and ND is the number of direct questions the user was asked in Twitter. Proactiveness NR/NI, NR is the number of user’s responses and NI is the number of indirect questions the user was asked in Twitter.  We hypothesize that one’s willingness to respond to questions is related to one’s past response behavior. a
  • 12. Key Features for Selection of Strangers Profile Features We use a profile-based  CountSocialWords - Count of the following phrases in description field in user profile: {“social”, “social media”, “social network”, “social networking”, “friend”, “tweet”, “twitter”, “tweeting”, “tweets”, “tell”, “telling”, “talk”, “talking”, “communication”, “communicator”}. - Adopted from LIWC “social process” category and by observing words related to modern social network activity. - The intuition is that a user who has such words in her profile will be more active and engaging than others, hence may likely to respond.  MsgCount - Number of status messages  DailyMsgCount - Number of status messages per day Retweet Features  RetweetRatio – Ratio of the total number of retweets and total number of tweets  DailyRetweetCount - Ratio of the total number of retweets and total number of days since the account is created. Activity Features
  • 13. Key Features for Selection of Strangers Personality Features  Personality traits such as Friendliness and Extraversion are intuitively related with one’s willingness to respond to questions.  Previous researchers have shown that word usage in one’s writings such as blogs and essays is related with one’s personality. Personality Features Total Number Examples Computation LIWC 68 Comm@Communication [admit, advice, affair*, apolog*, …] Let g be a LIWC category, Ng denotes the number of occurrences of words in that category in one’s tweets and N denotes the total number of words in his/her tweets. A score for category g is then: Ng/N. Big Five 5 Extraversion Using correlations with LIWC features as reported by previous researchers (e.g., Yarkoni et al.) Big Five Facets 30 Friendliness, Anxiety Using correlations with LIWC features as reported by previous researchers (e.g., Yarkoni et al.)
  • 14. Key Features for Selection of Strangers Readiness Features  Even if a person is willing to respond to questions, he/she may not be ready to respond at the time of questioning.  Since one’s readiness is highly context dependent (e.g. mobile device in use to send answers is running out of battery) and often difficult to capture computationally, we use several features to approximate one’s readiness: Readiness Features Computation Tweeting Likelihood of the Day TD/N, where TD is the number of tweets sent by the user on day D and N is the total number of tweets. Tweeting Likelihood of the Hour TH/N, where TH is the number of tweets sent by the user on hour H and N is the total number of tweets. Tweeting Steadiness 1/σ, where σ is the standard deviation of the elapsed time between consecutive tweets of users, computed from users’ most recent K tweets (where K is set, for example, to 20). Tweeting Inactivity TQ - TL, where TQ is the time the question was sent and TL is the time the user last tweeted.
  • 15. Feature Analysis Significant Features - Statistical significance test was done using Chi-square test with Bonferroni correction. - For TSA-tracker-1 dataset, we found 42 significant features (FDR was 2.8%). - For Product dataset, we found 31 features as significant (FDR 4.2%) - For TSA-tracker-2 dataset, we found 11 significant features (FDR 11.2%) Top-4 Features Features Feature-type Communication personality Past response rate responsiveness tweeting inactivity readiness tweeting likelihood of the day readiness - Top-4 features were found using extensive experiments. Dataset Top ten statistically significant features TSA- tracker-1 Past Response Rate, Tweet Inactivity, Negative Emotions, Cautiousness, Depression, Excitement-Seeking, DailyMsgCount, Intellect, Communication, Immoderation TSA- tracker-2 Prepositions, Past, Exclusion, Sensation, Past Response Rate, Space, Tweeting Steadiness, Achievement-striving, Agreeableness, CountSocialWords Product Mode Response Time, Tweet Inactivity, Activity Level, Depression, Present, Cautiousness, Positive Emotion, Excitement- Seeking, DailyMsgCount, Past Response Rate
  • 16. Statistical Model & Recommendation Binary-classification  Classify a person as responder or non-responder and send questions to people who are classified as responders. Top-K Selection  Rank according to their probabilities computed by the statistical model and select top-K people to send questions. Our Recommendation Algorithm  Automatically selects a subset of people from a set of available people.  Such subset selection is designed with the goal of maximizing the response rate.  Once features are computed, we trained statistical model such as Support Vector Machine and Logistic Regression to predict likelihood of response. Statistical Model
  • 17. Recommendation Algorithm  Ranks people in the training set in the order of non-decreasing probabilities.  Finds the best interval that has the maximum response rate among all interval subsets in the linear order.  Computes best subinterval in the test set from the best subinterval in training set using simple linear projection.  Can apply various constraints on selecting minimum/maximum/exact number of people. - select at-least/at-most/exactly K% people
  • 18. Evaluation Evaluating Prediction Model TSA-tracker-1 TSA-tracker-2 Product SVM Logistic SVM Logistic SVM Logistic Precision 0.62 0.60 0.52 0.51 0.67 0.654 Recall 0.63 0.61 0.53 0.55 0.71 0.62 F1 0.625 0.606 0.525 0.53 0.689 0.625 AUC 0.657 0.599 0.592 0.514 0.716 0.55 5-fold cross validation experiments AUC – Area Under ROC Curve F1 = Harmonic mean of precision and recall  Our models are 60-70% correct in making a prediction.
  • 19. Evaluating Recommendation Algorithm Comparison of Average Response Rates using Different Approaches TSA-tracker-1 TSA-tracker-2 Product Baseline 42% 33% 31% Binary-classification 62% 52% 67% Top-K-Selection 61% 54% 67% Our Algorithm 67% 56% 69%  Baseline is the response rates achieved by a human operator during data collection  Used “asking at least K% of people from the original set” as a constraint to search for the interval that maximizes the response rate.  Computed the response rates on with varied K (e.g., K=5%, …,90%) to find the respective optimal intervals.  Computed the response rates achieved using a simple binary classification (response rate is the precision of the predictive model) and simply selecting the top- K (e.g., K=5%, …, 90%.) people by their computed probabilities.
  • 20. Recall of Recommendation Selecting K% people Response Rate Recall 25% 76% 37% 50% 68% 64% 75% 53% 82% 100% 31% 100% Response Rate and Recall for Our Algorithm with Fixed Size (Product Data, all features, SVM Model)  Trade-off between response rate and recommendation recall, which captures the ratio of the actual responders our algorithm identifies for sending questions.
  • 21. Use of Different Feature Sets Feature Set Response Rate TSA-tracker-1 TSA-tracker-2 Product All 0.79 0.72 0.78 Significant 0.83 0.75 0.82 Top-10 Significant 0.83 0.74 0.81 Top-4 features 0.82 0.73 0.83 Common Significant features 0.81 0.72 0.82 Selecting at-least 5% people, SVM Model Comparison of Average Response Rates
  • 22. Live Experiments  Used Twitter’s Search API and a set of rules to find 500 users who mentioned that they were at any US airport in their tweets. - Randomly asked 100 users for the security wait time. - Used our algorithm to identify 100 users for questioning from the remaining 400 users. - Used the SVM-based model with the identified significant features. - Waited 48 hours for the responses.  The same process was repeated for sending product questions.  Large improvement of response rate in a live setting. Live Experiment Random Selection Our Algorithm TSA-Tracker-1 29% 66% Product 26% 60%
  • 23. Summary & Future Work  We focused on modeling users willingness and readiness to answer questions.  We can predict one’s likelihood of response to questions and identified sub-set of features that have significant prediction power.  Our experiments including the live one in a real-world setting demonstrated our approach’s effectiveness in maximizing the response rate.  Future Work - Applicability - Apply to other social media platforms - Apply to other information collection applications - Handling Skew in the user base - Identify inactive users similar to active users in terms of personality - Modeling the fitness of a stranger to engage - Develop model to receive high quality response - Model dutifulness and trustworthiness of users - Handling complex situations - Incorporate various costs/benefits which might change over time - Develop model to maximize expected net benefit - Handling unexpected answer - Incorporate voluntary responses from people in social media - Grow potential targets - Protecting Privacy - Tune selection algorithm to exclude people who are concerns about privacy

Notas del editor

  1. 30 sec.
  2. Such as celebrating birthdays, attending weddings, and becoming parents. Just by the sheer volume of activities on social media, we can certainly feel the heat and power of social media. 1 min.
  3. 1 min.
  4. 1 min.
  5. 30 sec. So far: 4 min.
  6. 1 min. So far: 5 min.
  7. 30 sec.
  8. 30 sec.
  9. 1 min.
  10. 30 sec. So far: 8 min.
  11. 30 sec.
  12. 30 sec.
  13. 1 min.
  14. 1 min.
  15. 30 sec. From survey to feature analysis: 6 min. So far 5 + 6 = 11 min/12 min.
  16. 30 sec
  17. 30 sec Statistical model + recommendation = 1 min. So far 12 + 1 = 13 min => 14 min.
  18. 30 sec.
  19. 1 min.
  20. 30 sec.
  21. 30 sec.
  22. 1 min. From Slide 18-21 = 2 min Total = 14 + 2 + 1 = 17 min.
  23. 1 min. From Slide 17-21 = 3.5 min -> 4 min.