People are interested in predicting the future. For example, which films will bomb or who will win the upcoming Grammy awards? Making predictions about the future is not only fun matters but can bring real value to those who correctly predict the course of world events, such as which stocks are the best purchases for short-term gains. Predictive analytics is thus a field that has attracted major attention in both academia and the industry. As social media has become an inseparable part of modern life, there has been increasing interest in research of leveraging and exploiting social media as an information source for inferring rich social facts and knowledge. In this talk, we will address an interesting and challenging problem in social media research, i.e., predicting social media popularity. We aim to discover which image posts on social media are the “stars of tomorrow”, those will be the most engaging for social media audiences, e.g., receiving the most likes. Sociological finding and our novel solutions to effectively develop a structured modeling for popularity dynamics will be presented.
Exploring the Future Potential of AI-Enabled Smartphone Processors
Predicting the “Stars of Tomorrow” on Social Media
1. Predicting the “Stars of Tomorrow”
on Social Media
Wen-Huang Cheng (鄭文皇)
Multimedia Computing Lab (MCLab)
Research Center for Information Technology Innovation (CITI),
Academia Sinica, Taipei, Taiwan
whcheng@citi.sinica.edu.tw
Presented at on 10 May 2017
3. Academia Sinica (中央研究院)
• The highest national research institute in Taiwan
– with about 1,000 professors (60 in EE/CS)
3
located in Nangang, Taipei
15. Sociology and Human Interaction
• With the huge number of people who are involved nowadays with
social networks, it is very interesting to note how they are influenced
by each other in many different ways.
– e.g., identity in the age of social media
15
[Ref] http://edition.cnn.com/2015/10/05/health/being-13-teens-social-media-study/index.html
17. Social Popularity Prediction
• General Popularity Prediction: Predicting the popularity
score of a new social media post by combining post
content (photo, text or video) and user cues
17
Score: 4.9
?
Model
A new post
Predicted Popularity
Training Images
5.6 2.3
5.1 2.8
7.8 3.1
History data
18. Why is it important?
• wide applications and high business value
– e.g., predicting the “Stars of Tomorrow” (top popular
models) within the fashion Industry using social media
18
[Ref] “Style in the Age of Instagram: Predicting Success within the Fashion Industry using Social Media,” CSCW 2016.
Fashion Model Directory (FMD) profile page
Can you tell who will be the “top”?
19. People are desired for knowing the future…
19
[Ref] https://www.oreilly.com/ideas/inside-the-washington-posts-popularity-prediction-experiment
24. Our Related Publications
• “Sequential Prediction of Social Media Popularity with
Deep Temporal Context Networks,” IJCAI 2017.
• “Time Matters: Multi-scale Temporalization of Social
Media Popularity,” ACM Multimedia 2016 (full paper).
• “Unfolding Temporal Dynamics: Predicting Social Media
Popularity Using Multi-scale Temporal Decomposition,”
AAAI 2016.
• “SocialCRC: Enabling Socially-Consensual Rendezvous
Coordination by Mobile Phones,” Pervasive and Mobile
Computing, 2016.
24
25. What Makes A Post Popular?
25
[Ref]“What Makes an Image Popular?” WWW, 2014.
26. What Makes A Post Popular?
• Features for prediction
– Post content
• e.g., visual sentiment features (color and texture)
26
[Ref]“Analyzing and predicting sentiment of images on the social web,” ACM Multimedia 2010.
27. What Makes A Post Popular?
• Features for prediction
– User cues
• e.g., followers (a user’s follower count), friends (how many
users a user follows), statuses (a user’s current total post
count), user time (a user’s account creation time), etc.
27
A friend graph:
6
1
28. What Makes A Post Popular?
• Features for prediction
– User cues (topological features)
• e.g., closeness centrality, the average length of the shortest
path between the node and all other nodes in the graph
28
1
2
3
4
5
6
7
30. Latent Factor Models
• The popularity prediction task is formulated as a
matrix completion problem of filling in the
missing entries of a partially observed matrix.
30
known popularity
to be estimated
31. Our Observations: Time Matters
31
[Ref] http://www.adweek.com/socialtimes/best-time-to-post-social-media/504222
32. Temporal Modeling for Popularity
• To incorporate the temporal evolving structures
in popularity prediction
32
33. • The popularity evolving at multi-granularities with
different patterns
33
Challenge 1: Temporal Evolving
Multi-granularities Characteristics of Popularity Dynamics
34. Challenge 2: Data Noise
• Popularity patterns are covered in very noisy behavior
data or information
Popularity distribution on time series
41. Our Solution#3 [IJCAI’17]:
Deep Temporal Context Networks (DTCN)
• We address the problem as a sequential prediction task, where the input is
a user-photo sequence (with time order) while the output is the popularity of
a “future” photo (a photo before its publication on social media)
41
43. More Influential Factors: Cultures
• A voting survey of the 2014 TripAdvisor's Top 10 Attractions in Japan by visitors from
different countries shows how much the favorites for attractions can vary among
people from different regions, i.e., different cultures.
43
[Ref] 2014 TripAdvisor’s Top 20 Attractions in Japan: http://www.tripadvisor.com/pages/- HotSpotsJapan.html.
48. Learning Relevance by Neighbor Voting
48
[Ref] X. Li, C.G.M. Snoek, M. Worring, “Learning tag relevance by neighbor voting for social
image retrieval,” Proc. ACM Intl. Conf. Multimedia Information Retrieval (MIR), 2008.
49. More Influential Factors:
Personal Fashion Flavor
49
[Ref] “Fashion Analysis: Current Techniques and Future Directions,” IEEE Multimedia, 2014.
50. Urban Tribes: Analyzing Group Photos from
a Social Perspective [CVPR’12]
50
Urban tribe: the term to describe subcultures of people who share common
interests and tend to have similar styles of dress, to behave similarly, and to
congregate together. (coined by French sociologist Michel Maffesoli in 1985)
Which groups of people would more likely choose to interact socially? (a) and (b) or (a) and (c)?
51. Clothing Fashion Analysis
51
"i-Stylist: Finding the Right Dress Through Your Social Networks,"
MMM 2017.
"A Framework of Enlarging Face Datasets Used for Makeup
Face Analysis," BigMM 2016.
"What are the Fashion Trends in New York?" MM 2014. (Grand
Challenge Prize)
"Clothing Genre Classification by Exploiting the Style Elements,"
MM 2012.
52. Clothing fashion is a reflection of
the society of a period
• The global fashion apparel market today has surpassed
1 trillion US dollars since 2013, and accounts for nearly
2 percent of the world's Gross Domestic Product (GDP)
52
53. Trend Analysis for the Clothing Fashion
Our work received “Multimedia Grand Challenge Award” in 2014 ACM Multimedia Conference.
54. Applications: “Fashion is becoming mobile first with apps that help track down
must-have clothes, accessories and shoes” - theguardian.com
LIKEtoKNOW.it The Netbook
Snap Fashion
The Hunt
56. Color Cut
Pattern Head decoration
major elements for
fashion style investigation
key factors for discovering fashion trends:
coherence (frequently occur within a fashion week)
uniqueness (occur much more often in a fashion week than in other fashion weeks)
59. Collect full-body image of catwalk models
Catwalk Models
e.g. NYFW Autumn/Winter 2014
Positive set Negative set
e.g. all catwalk models at NYFW
except for Autumn/Winter 2014
Divide the collection of full-body images into two sets
Distributional clustering technique
W.‐H. Cheng et al., "Learning and Recognition of On‐Premise Signs (OPSs) from Weakly
Labeled Street View Images," IEEE Tran. on Image Processing (TIP), 2014.
60. Query Image
Query Image
Color Analysis Texture Analysis Color + Texture Analysis
Query ImageQuery Image
Color Analysis Texture Analysis Color + Texture Analysis
Query Image Query Image Query Image
Color Analysis Texture Analysis Color + Texture Analysis
Query Image Query Image Query Image
Color Analysis Texture Analysis Color + Texture Analysis
Query Image Query Image Query Image
Spring/Summer
2011
Spring/Summer
2013
Spring/Summer
2013
Spring/Summer 2011 Spring/Summer 2013 Spring/Summer 2013
61. Predicting Occupation via Human
Clothing and Contexts [ICCV’11]
• Diving into the recognition of high-level semantic
categories of human such as occupations
61
62. Recognizing City Identity via Attribute
Analysis of Geo-tagged Images [ECCV’14]
• A set of 7 high-level attributes is used to describe the spatial
form of a city (amount of vertical buildings, type of
architecture, water coverage, and green space coverage)
and its social functionality (transportation network, athletic
activity, and social activity).
62
63. From Scene Attributes to City Attributes
• 102 scene attributes are defined.
• Each of the city attribute classifier is modeled as an ensemble of SVMs.
63
64. Spatial Analysis of City Attributes
• The city perception map visualizes the spatial distribution of the 7 city
attributes in different colors and exhibits the visitors’ and inhabitants’ own
experience and perception of the cities, while it reflects the spatial
popularity of places in the city across attributes.
64
67. 67
ACM Multimedia 2017
http://www.acmmm.org/2017/
Grand Challenge
Social Media Prediction (SMP):
Predicting the “Stars of Tomorrow” on Social Media
https://social-media-prediction.github.io/MM17PredictionChallenge/
Organizers
Wen-Huang Cheng
Academia Sinica
Bo Wu
Chinese Academy of
Sciences
Yongdong Zhang
Chinese Academy of
Sciences
Tao Mei
Microsoft Research
Asis
70. YFCC100M
• This dataset contains 100 million media objects and
explain the rationale behind its creation. This list is
compiled from data available on Yahoo! Flickr.
70
Two photos of real world scenes from photographers in the YFCC100M dataset.
76. 76
General Chairs Program Chairs
Wan-Chi Siu
Hong Kong Polytechnic University
Chia-Wen Lin
National Tsinghua University
Wen-Huang Cheng
Academia Sinica
Gene Cheung
National Institute of Informatics
vcip2018.org