7. “Deep learning is a set of
algorithms in machine learning
that attempt to learn in multiple
levels, corresponding to
different levels of abstraction.”
8. AI > today’s focus
use of several modes (media) to
create a single artifact.
Multimodality
“Mode”
Socially and culturally shaped
resource for making meaning.
— Gunther Kress
11. Creativity
1. Making unfamiliar combinations of familiar ideas.
2. Explore a structured conceptual space
3. (Radically) transforming ones structured conceptual space
“Exploration”
“Remix”
“The Creative Mind”
— Margaret Boden
“Transformation”
12. • Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality.
Creativity > “Traits” software has to exhibit in order to
avoid easy criticism of being “non-creative”.
(Simon Colton)
22. Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets
23. Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets
24. Creative AI > Current possibilities > Appropriating “standard” nets for creative use Deep Dream
see also: www.csc.kth.se/~roelof/deepdream/
25. Creative AI > Current possibilities > Appropriating “standard” nets for creative use Deep Dream
see also: www.csc.kth.se/~roelof/deepdream/ codeyoutubeRoelof Pieters 2015
26. Creative AI > Current possibilities > Appropriating “standard” nets for creative use Deep Dream
see also: www.csc.kth.se/~roelof/deepdream/
C.M.Kosemen &
Roelof Pieters (2015)
Gizmodo
27. Creative AI > Current possibilities > Appropriating “standard” nets for creative use
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge , 2015.
A Neural Algorithm of Artistic Style (GitXiv)
Style Net
30. Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets
31.
32. Creative AI > Current possibilities > Reinforcement Learning
• AMN: Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov 2015, Actor-Mimic:
Deep Multitask and Transfer Reinforcement Learning (arxiv)
• DQN: Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A., Veness,
Joel, Bellemare, Marc G., Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K.,
Ostrovski, Georg, Petersen, Stig, Beattie, Charles, Sadik, Amir, Antonoglou, Ioannis,
King, Helen, Kumaran, Dharshan, Wierstra, Daan, Legg, Shane, and Hassabis,
Demis. Human-level control through deep reinforcement learning. Nature, 518(7540):
529–533, 2015.
33. Creative AI > Current possibilities > Reinforcement Learning
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, Raul Vicente, 2015
Multiagent Cooperation and Competition with Deep Reinforcement Learning (GitXiv)
(YouTube)
34. Reinforcement Learning
Ning Xie, Hirotaka Hachiya, Masashi Sugiyama, 2013 ,
Artist Agent: A Reinforcement Learning Approach to Automatic
Stroke Generation in Oriental Ink Painting (Paper, Lecture,
YouTube)
36. Ning Xie, Hirotaka Hachiya, Masashi Sugiyama, 2013
Artist Agent: A Reinforcement Learning Approach to Automatic Stroke Generation
in Oriental Ink Painting (Paper, Lecture, YouTube)
37. Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets
38. Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets
39. Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-encoders
• Attention-based Models
• Generative Adversarial Nets
40. Creative AI > Current possibilities
• Standard (“denoising”) Autoencoders
• Variational Autoencoder (VAE) / Stochastic Gradient VB
• Deep Convolutional Inverse Graphics Network
• Variational RNN (VRNN)
Vincent et al, 2010. Stacked Denoising Autoencoders: Learning Useful Representations in
a Deep Network with a Local Denoising Criterion (paper) (code)
41. Creative AI > Current possibilities
• Standard “denoising” Autoencoders
• Variational Autoencoder (VAE) / Stochastic Gradient VB
• Deep Convolutional Inverse Graphics Network
• Variational RNN (VRNN)
• Diederik P Kingma, Max Welling, 2013.
Auto-Encoding Variational Bayes (GitXiv)
42. Creative AI > Current possibilities
• Standard “denoising” Autoencoders
• Variational Autoencoder (VAE)
• Deep Convolutional Inverse Graphics Network (modified VAE)
• Variational RNN (VRNN)
Tejas D. Kulkarni, Will Whitney, Pushmeet Kohli, Joshua B. Tenenbaum, 2015
Deep Convolutional Inverse Graphics Network (GitXiv)
43. Creative AI > Current possibilities
• Standard “denoising” Autoencoders
• Variational Autoencoder (VAE)
• Deep Convolutional Inverse Graphics Network
• Variational RNN (VRNN) (VAE at every time step)
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, Yoshua Bengio, 2015
A Recurrent Latent Variable Model for Sequential Data (GitXiv)
VAEVAEVAE
44. Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, Yoshua Bengio , 2015.
A Recurrent Latent Variable Model for Sequential Data (GitXiv) (Audio Samples)
45. Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets
46. Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, Daan Wierstra, 2015
DRAW: A Recurrent Neural Network For Image Generation (GitXiv)
Variational Auto-Encoder
Deep Recurrent Attentive Writer
(DRAW) Network
48. Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adverserial Nets
49. Emily Denton, Soumith Chintala, Arthur Szlam, Rob Fergus, 2015.
Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks (GitXiv)
50. Alec Radford, Luke Metz, Soumith Chintala , 2015.
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)
51. Alec Radford, Luke Metz, Soumith Chintala , 2015.
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)
52. ”turn” vector created from four averaged samples of faces looking
left vs looking right.
Alec Radford, Luke Metz, Soumith Chintala , 2015.
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)
57. Creative AI > Needs as I see it
Creative AI as a
“tool”
or “brush” to paint
with
58. A system which marries the need for a creative
process with the need for a creative output
• with as less human input as possible (data)
• with its own style
• with the possibility for human level supervision
for rapid experimentation
Creative AI > a “brush”
59. A system which marries the need for a creative
process with the need for a creative output
• with as less human input as possible ( )
• with its own style
• with the possibility for human level supervision
for rapid experimentation
Creative AI > a “brush”
data
60. Creative AI > a “brush” > data
• reuse nets as much as possible
• combining unsupervised & supervised
• multiple modalities
• plug in external knowledge bases
61. Creative AI > a “brush” > data input
• unlabeled & labeled data
• external knowledge bases (dbpedia, wikipedia)
• one-shot learning
• zero-shot learning
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng, 2013
Zero-Shot Learning Through Cross-Modal Transfer
a zero-shot model that can predict both seen and unseen classes
62. Creative AI > a “brush” > data input
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng, 2013
Zero-Shot Learning Through Cross-Modal Transfer
(slides)
63. Creative AI > a “brush” > data input
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng, 2013
Zero-Shot Learning Through Cross-Modal Transfer
(slides)
64. Creative AI > a “brush” > data input
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng, 2013
Zero-Shot Learning Through Cross-Modal Transfer
(slides)
65. A system which marries the need for a creative
process with the need for a creative output
• with as less human input as possible (data)
• with its own style
• with the possibility for human level
for rapid experimentation
Creative AI > a “brush”
supervision
66. Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space
67. Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space
68. Creative AI > a “brush” > data
Deep Dream
Alexander Mordvintsev, Christopher Olah, Mike Tyka, 2015.
Inceptionism: Going Deeper into Neural Networks
Google Research Blog
69. Creative AI > a “brush” > data
Deep Dream
Roelof Pieters, 2015 DeepDream - Class visualization Experiment (link)
71. Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space
72. Creative AI > a “brush” > data
Deep Dream
Roelof Pieters, 2015 DeepDream - Overview of standard bvlc googlenet (inception) layers (link)
Constrain Layers
73. Creative AI > a “brush” > data
Deep Dream
Roelof Pieters, 2015 Single Unit Activations (early layer) (Flickr Album)
Constrain Units
74. Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space
75. Creative AI > a “brush” > data
Deep Dream
Roelof Pieters, 2015 DeepDream Video (GitHub)
76. Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space
77. Creative AI > a “brush” > data
Style Net
Roelof Pieters (graphific) (tweet) Roelof Pieters (graphific) (tweet)
78. Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space
81. Image -> Text
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron
Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio,
Show, Attend and Tell: Neural Image Caption Generation with
Visual Attention (arxiv) (info) (code)
Andrej Karpathy Li Fei-Fei , 2015.
Deep Visual-Semantic Alignments for Generating Image Descriptions (pdf) (info) (code)
Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan ,
2015. Show and Tell: A Neural Image Caption Generator (arxiv)
82. Text -> Image “A stop sign is flying in blue skies.”
“A herd of elephants flying in the blue skies.”
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov, 2015.
Generating Images from Captions with Attention (arxiv) (examples)
83. Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov, 2015.
Generating Images from Captions with Attention (arxiv) (examples)
Text -> Image
84. Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond Mooney,
Trevor Darrell, Kate Saenko , 2015. Sequence to Sequence -- Video to Text (GitXiv)
Video -> Text
85. A system which marries the need for a creative
process with the need for a creative output
• with as less human input as possible (data)
• with its own style
• with the possibility for human level supervision
for
Creative AI > a “brush”
rapid experimentation
87. Widening
Deepening
Tianqi Chen, Ian Goodfellow, Jonathon Shlens, 2015. Net2Net: Accelerating Learning via
Knowledge Transfer (arxiv) / code (torch)
Reusing Nets:
Bigger Net
88. Teacher and Student net Hint training
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta,
Yoshua Bengio, 2014. FitNets: Hints for Thin Deep Nets (arxiv)
Knowledge distillation
SVHN Error
MNIST Error
Reusing Nets:
Smaller Net
89. Hashed Net
Wenlin Chen, James T. Wilson, Stephen Tyree, Kilian Q. Weinberger, Yixin Chen, 2015.
Compressing Neural Networks with the Hashing Trick (arxiv)
Shrinking Nets:
Hashing
90. Song Han, Huizi Mao, William J. Dally, 2015. Deep Compression: Compressing Deep Neural
Networks with Pruning, Trained Quantization and Huffman Coding (arxiv)
Shrinking Nets:
Pruning,
Quantization &
Huffman coding
91. Creative AI > a “brush” > rapid experimentation
• experiments need “tooling”, specialised design
software to
• try things
• explore latent spaces (z-space)
• push the AI in the right direction
• be surprised by AI
92. Creative AI > a “brush” > rapid experimentation
human-machine collaboration
93. Creative AI > a “brush” > rapid experimentation
(YouTube, Paper)
94. Creative AI > a “brush” > rapid experimentation
(YouTube, Paper)
95. Creative AI > a “brush” > rapid experimentation
(Vimeo, Paper)
96. Creative AI > a “brush” > rapid experimentation
• Advertising and marketing
• Architecture
• Crafts
• Design: product, graphic and fashion design
• Film, TV, video, radio and photography
• IT, software and computer services
• Publishing
• Museums, galleries and libraries
• Music, performing and visual arts