[2024]Digital Global Overview Report 2024 Meltwater.pdf
AI Tools for Creative Work
1. AI Tools for Creative Work
Lydia B. Chilton
DUB Seminar
2 October 2019
2.
3. AI can do things it couldn’t do before…
But it still struggles at ill-defined tasks.
4. It looks like you’re trying to
solve world peace.
Do you need assistance?
AI can do things it couldn’t do before…
But it still struggles at ill-defined tasks.
5. Long history of creativity support tools in HCI
Jonas Frich, Lindsay MacDonald Vermeulen, Christian Remy, Michael Mose Biskjaer, and Peter Dalsgaard. 2019.
Mapping the Landscape of Creativity Support Tools in HCI. CHI 2019
6. How does AI change the future of creative work?
Gatys, Ecker, Bethge. A Neural Algorithm of Artistic Style. http://arxiv.org/abs/1508.06576 2015
7. Outline: AI Tools for Creative Work
• Introduce A Creative Problem: Visual Metaphors
• Can AI do it?
• VisiBlends: A Workflow for Prototyping Visual Metaphors
• Symbol Finder: Adding AI to the workflow
• VisiFit: AI Tools for Iteratively Improving Visual Blends
• The Future of AI and creative work
8. AdvertisementsNews Public Service Announcements
Earth + MeltBrazil + Takes Off Tabasco + Hot
Visual Metaphors: image that convey a message symbolically
10. Outline: AI Tools for Creative Work
• Introduce A Creative Problem: Visual Metaphors
• Can AI do it?
• VisiBlends: A Workflow for Prototyping Visual Metaphors
• Symbol Finder: Adding AI to the workflow
• VisiFit: AI Tools for Iteratively Improving Visual Blends
• The Future of AI and creative work
11. With successes in deep learning,
what is the future of creative work?
15. Visual Metaphor:
Burgers are Healthy, like apples
Artist:AI:
Blending objects requires more than just blending “style.”
You need more context about the objects - how to take them apart, blend the
pieces, think about how they’ll be perceived.
16. Outline: AI Tools for Creative Work
• Introduce A Creative Problem: Visual Metaphors
• Can AI do it?
• VisiBlends: A Workflow for Prototyping Visual Metaphors
• Symbol Finder: Adding AI to the workflow
• VisiFit: AI Tools for Iteratively Improving Visual Blends
• The Future of AI and creative work
17. But there is no simple pattern for visual blends
We need to break up the problem into tasks.
18. 1. Two objects are integrated into one object
2. Both objects are individually identifiable
Visual Blending Structure:
Whole-to-Part Shape Matching
Abstract problem structure:
Single Shape Mapping
19. Why blend based on shape?
Shape is the primary feature the human visual system uses to recognize objects.
Colors, silhouette and details and secondary.
20. 1. Two objects are integrated into one object
2. Both objects are individually identifiable
Visual Blending Structure:
Whole-to-Part Shape Matchingby using a design pattern to make it a search problem
We can solve a creative design problem
Earth Melt
Visual Blending Structure:
Whole-to-Part Shape Matching
26. A) Brainstorm
for summer
B) Find and annotate images
for summer
C) Matching algorithm
Starbucks + summer
D) Automatic blends + human evaluation
Starbucks + summer
matches = []
for a in summer_symbols:
for b in starbucks_symbols:
a_ratio = a.height / a.width
b_ratio = b.height / b.width
ratios = sort(a_ratio, b_ratio)
if (a.shape == b.shape) and
(a.coverage != coverage) and
(ratios[0] >= 0.5*ratios[1]):
matches.push([a, b])
return matches
VisiBlends:
A web interface to collaboratively make blends
31. Forget the knives and forks: the humble hamburger is in
the midst of a renaissance. From Melbourne to Milan,
restaurateurs have helped elevate this classic piece of
American folklore into an iconic dish. Every day, more
and more enthusiasts join the search for the perfect
34. Does VisiBlends make people better?
• 13 undergraduates (novice designers)
• Each made 6 blends (3 with VisiBlends, 3 without)
• 5 minutes for each blend
• Metric: How many successful blends can 5 minutes?
35. Novices are 10x better with VisiBlends
Control-first
Avg. Number of
blends made
VisiBlends-first
Without VisiBlends With VisiBlends
Without VisiBlendsWith VisiBlends
Avg. Number of
blends made
0.56 5.56 (t(18)=4.88, p<0.001)
0.675.67 (t(21)=5.84, p<0.001)
36. Bicycle + Smart
VisiBlendsUsers Unassisted
VisiBlends helps them have
many ideas, and pick the best.
Novices get stuck “improving”
their first idea.
37. New York City + Night
Users Unassisted
Novices pick “perfect” symbols,
but sacrifice shape-fit.
VisiBlends helps novices meet all
the criteria for a visual blend
(especially shape-fit).
VisiBlends
38. Do we want “perfect” symbols for Visual Blends?
By meeting more constraints,
Visual Metaphors are more
novel and interesting
39. Novices are 10x better with VisiBlends
10x improvement in novices ability
VisiBlendsUnassisted
Why?
41. Football Dangerous
Brainstorm
associations
Find Images
of objects
Annotate
shapes
Annotate
shape coverage
Are both objects identifiable?
Are two objects integrated into one object?
Evaluate
Blend
Shape covers
All of object
Shape covers
Part of object
42. Football Dangerous
Brainstorm
associations
Find Images
of objects
Annotate
shapes
Annotate
shape coverage
Are both objects identifiable?
Are two objects integrated into one object?
Evaluate
Blend
Shape covers
All of object
Shape covers
Part of object
50. Orange Healthy
Brainstorm
associations
Find Images
of objects
Annotate
shapes
Annotate
shape coverage
Shape covers
Part of object
Shape covers
All of object
Are both objects identifiable?
Are two objects integrated into one object?
Evaluate
Blend
52. Orange Healthy
Brainstorm
associations
Find Images
of objects
Annotate
shapes
Annotate
shape coverage
Shape covers
All of object
Shape covers
Part of object
Are both objects identifiable?
Are two objects integrated into one object?
Evaluate
Blend
Exercise equipment
53. Iteration requires fluidity –
putting together pieces in new ways.
No matches
Improve object fit
Objects are
not identifiable + =
+ =
Within same search space,
meet other constraints.
Find versions of an object
with different
color, style, aspect ratio
Focus on meeting a
specific constraint:
Find symbols with
a different shape
Search in a new
subspace
Find symbols with
a different shape
55. Outline: AI Tools for Creative Work
• Introduce A Creative Problem: Visual Metaphors
• Can AI do it?
• VisiBlends: A Workflow for Prototyping Visual Metaphors
• Symbol Finder: AI tools for finding symbols
• VisiFit: AI Tools for Iteratively Improving Visual Blends
• The Future of AI and creative work
58. Simply searching for abstract words doesn’t return
simple, iconic symbols that we can blend.
59. SymbolFinder uses AI + KBs
to find concrete images for an abstract term
Savvas Petridis
PhD Student
word2vec, synonyms, ConceptNet, concreteness classifiers
61. Outline: AI Tools for Creative Work
• Introduce A Creative Problem: Visual Metaphors
• Can AI do it?
• VisiBlends: A Workflow for Prototyping Visual Metaphors
• Symbol Finder: AI tools for finding symbols
• VisiFit: AI Tools for Iteratively Improving Visual Blends
• The Future of AI and creative work
62. How often do VisiBlend prototypes need editing?
Output:
A visual blend
object
4 of 27 were ok to print on social media
(according to 2 evaluators who are graphic designers)
64. Fully automatic approaches fail
Artist:AI:
Design Principle #1 :
Instead of pursuing fully automatic approaches,
break up the objects into visual components that can each be blended.
65. Design Principle #2:
When improving a blend,
Recombine secondary visual aspects: color, silhouette, details.
Co-Design with 2 Graphic Designers
66. Formative study with novice designer
Design Principle #3:
Translating intent into action is a barrier for novices.
Use AI tools to automate actions or
provide more direct mapping between intent and action
“I want to make the metro card the
same shape as the purse.”
67. VisiFit: AI Tools to Iteratively Improve Visual Blends
Each visual aspect has it’s own AI tool for novices to apply
68. AI tools to improve each visual dimension
Color blending
Crop to extracted
silhouette
Deep Object
Segmentation
Interactive
Object
Segmentation
Interactive
Object
Segmentation
69. Evaluation
Improved:
97.5% of cases
11 Novice designers (8 female)
1 hr study
11 blends to extract and improve
Publishable quality:
65.3% of the time.
Experts Publishable:
87.0% of the time.
121 total blends produced
Outputs rated by 2 designers:
Improved, publishable.
71. Response of Professional Visual Blend Designers
1. Amazed by the AI tools
They did try things no novice did
(even if it didn’t work).
2. Much easier than “pixel pushing”.
Enabled them to explore more designs quickly and
easily.
But did suggest more PhotoShop features to all,
like Context Aware Fill
Wanted it as a plugin to sketch.
Everyone else Designer
72. Outline: AI Tools for Creative Work
• Introduce A Creative Problem: Visual Metaphors
• Can AI do it?
• VisiBlends: A Workflow for Prototyping Visual Metaphors
• Symbol Finder: AI tools for finding symbols
• VisiFit: AI Tools for Iteratively Improving Visual Blends
• The Future of AI and creative work
74. If we identify the key aspects of the problem
VisiBlends collects and combines the
shape of blended objects
VisiFit extracts and combines the
color, silhouette, details of the objects
AI tools can help collect and combine elements for each aspect
75. AI Tools for writers:
Key aspects of writing:
Substance, structure, and style.
Metaphoria (CHI 2019):
Within the metaphor structure,
helps generate substance.
Style Thesaurus (C&C 2019):
Same substance, new style
(word level)
Language Style Transfer (INLG 2019):
Same substance, new style
(sentence level)
Katy Gero
PhD Candidate
76. Characteristic motion
Reference video
Static image
Future Work: Animated Blends
Use AI to extract key aspects of motion (speed),
and reapply them to the blend (use a different path).
77. Ecenaz Jen Ozmen
Columbia CC ‘21
Claire Chen
Columbia CC ‘21
Maria Kogan
Columbia SEAS ‘21
Sam Ross
Barnard ‘21
Annie Sui
Columbia CC ‘22
Maneesh Agrawala
Stanford
Katy Gero
PhD Candidate
Savvas Petridis
PhD Candidate
Collaborators
79. Future of Creative Work:
AI should augment fluid intelligence
AI shouldn’t try to understand complexities of culture and working context.
AI can provide people the tools and resources
to more easily turn users’ intent into action
AI Artist Novice with AI Tools
80. VisiBlends: AI Tools for Prototyping Visual Blends
VisiBlendsUnassisted
10x improvement in novices’ ability
81. VisiFit: AI Tools to Iteratively Improve Visual Blends
97.5% of cases improved, 65% to publishable quality
84. Iteratively Improving: one key element at a time
Prototyping: collect and recombined parts Brainstorming: easily collect parts
AI Tools for Creative Work
Repurposing: Elements of a Reference Object
Notas del editor
Visual blends are a way to convey a message visually.
If you have a message you want to convey, like “Washing your hands is smart”
We can make a image that blends together two objects related to that message.
This technical is used in new, advertising, and public service announcements to
Visual depict symbols of
Brazil + taking off
Tabasco + hot
Earth + melt
There are a wide range of ads out there -
So I wondered, what the considered the most creative ads.
“Creative ads”
Many of the results are visual metaphors.
Too hard for a computer.
It would be really nice if we could break up the problem, or find a pattern
No surface level pattern:
Different colors, different objects, different meanings
But there is a more abstract pattern.
So how could
How do we get two objects to appear blended, but still individually identifiable?
We find objects that share a shape. Like a circle. When we blend on the circle, we get visual cues of both objects.
Let's go through the steps of the pipeline to create a blend that associates:
Starbucks and Summer.
And finally, if they are unhappy with the prototype, they can iterate.
And finally, if they are unhappy with the prototype, they can iterate.
And finally, if they are unhappy with the prototype, they can iterate.
This is a pipeline,
But it’s also fairly open.
People can move between step as they see fit.
Anytime you put in a new image, the matching algorithm will run, and it will look through all pairs for new matches.
This blend was made for the news headline: “Football linked to lasting brain damage”
This blend was made for the advertisement: “Joe’s Coffee: Open Late”
This blend was made for the advertisement:
“Join the Philosophy Dept’s Holiday Celebration”
Football + Dangerous: We know which constraint is tight and look for that constrant.
Oragne + healthy: We need to search in a new subspace.
One thing I’ve glossed over is that the rules are complex..
Here are the rules for finding images..
Visual blends are a way to convey a message visually.
If you have a message you want to convey, like “Washing your hands is smart”
We can make a image that blends together two objects related to that message.
To make the motion characteristic of the image, we used a reference video of a floating astronaut. Q: What properties did we add from the video? What motion did we NOT add?
Visual blends are a way to convey a message visually.
If you have a message you want to convey, like “Washing your hands is smart”
We can make a image that blends together two objects related to that message.
To make the motion characteristic of the image, we used a reference video of a floating astronaut. Q: What properties did we add from the video? What motion did we NOT add?