SlideShare una empresa de Scribd logo
1 de 26
Descargar para leer sin conexión
Lifelong Learning with
Dynamically Expandable Networks
𝑱𝑱𝑱𝑱𝑱𝑱𝑱𝑱𝑱𝑱𝑱𝑱𝑱𝑱 𝒀𝒀𝒀𝒀𝒀𝒀𝒀𝒀𝟏𝟏,𝟑𝟑∗
, 𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸 𝑌𝑌𝑌𝑌𝑌𝑌𝑔𝑔1,3
, 𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽 𝐿𝐿𝐿𝐿𝑒𝑒2
, 𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆 𝐽𝐽𝐽𝐽 𝐻𝐻𝐻𝐻𝐻𝐻𝐻𝐻𝑔𝑔1,3
𝐾𝐾𝐾𝐾𝐾𝐾𝐾𝐾 𝑇𝑇1, 𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑇𝑇2, 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 𝑠𝑠3
*𝑊𝑊𝑊𝑊𝑊𝑊𝑊𝑊 𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑 𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤 𝑎𝑎𝑎𝑎 𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈
Introduction
Korea Advanced Institute of Science and Technology (KAIST)
• Ph.D. in School of Computing (Aug. 2018. – )
• Advisor: Prof. Sung Ju Hwang
Ulsan National Institute of Science and Technology (UNIST)
• M. S. in Computer Engineering (Aug. 2016 – Feb. 2018)
• Advisor: Prof. Sung Ju Hwang
• B. S. in Computer Science Engineering (Mar. 2012 – Aug. 2016)
• Biological Science Minor
Jaehong Yoon
- Education
Introduction
Juho Lee, S. Kim, J. Yoon, H. B. Lee, E. Yang, S. J. Hwang, “Adaptive Network Sparsification via Dependent
Variational Beta-Bernoulli Dropout”, arXiv preprint arXiv:1805.10896 (2018).
Jaehong Yoon, E. Yang, J. Lee, and S. J. Hwang, “Lifelong Learning with Dynamically Expandable Networks”,
International Conference on Learning Representation (ICLR), 2018
Jaehong Yoon, and S. J. Hwang, “Combined Group and Exclusive Sparsity for Deep Neural Networks”,
International Conference on Machine Learning (ICML), 2017
- Experience
- Publications
Korea Advanced Institute of Science and Technology (KAIST)
• Contract Research Scientist (Feb. 2018 ~ Aug. 2018)
AItrics
• Research Intern (Mar. 2018 ~ May 2018)
Challenge: Incomplete, Growing Dataset
In many large-scale learning scenarios, not all training data might be available when
we want to begin training the network.
Car
Convertible Sports car
ImageNet
22,000 classes
Sedan
Roadster
Challenge: Incomplete, Growing Dataset
In many large-scale learning scenarios, not all training data might be available when
we want to begin training the network.
Car
Sports car
Sedan
Roadster
1M classes
BMW Z4
Ferrari 458 spider
Convertible Ferrari 458 Italia
Porsche 911
Turbo
Hyundai Sonata
BMW 3 series
Challenge: Incomplete, Growing Dataset
Even worse, the set of tasks may dynamically grow as new tasks are introduced.
Car
Sports car
Sedan
Roadster
BMW Z4
Ferrari 458 spider
Convertible
2015 Mustang
Convertible
Ferrari 458 Italia
Porsche 911
Turbo
Hyundai Sonata
Tesla Model SBMW 3 series
1M classes
Solution: Lifelong Learning
Humans learn forever throughout their lives - couldn’t we build a similar system
that basically learns forever while becoming increasingly smarter over time?
We integrate our model into a lifelong learning framework, that continuously learns by
actively discovering new categories and learning them in the context of known ones.
t-2 t-1 t
Learning
Model
t+1
Learned knowledge
3) New knowledge is
stored for
future use
2) Knowledge is
transferred from
previously
Learned tasks
1) Tasks are received in
a sequential order
4) Refine existing
knowledge
Humans learn forever throughout their lives
Lifelong Learning of a Deep Neural Network
However, if the classes we had in the early stages of learning significantly differs from
the new class, utilization of prior knowledge may degenerate performance.
𝑾𝑾 1
𝑾𝑾 2
t-2 t-1 t t+1
New class
+
𝑾𝑾 2
Semantic Drift
Introduction of new units can also result in semantic drift or catastrophic forgetting,
where original meaning of the features change as they fit to later tasks.
𝑾𝑾 1
𝑾𝑾 2
New class
+
Network Expansion
To learn new tasks which are relatively different from early stages of learning, model
may need to expand network capacity.
𝑾𝑾 1
𝑾𝑾 2
+
New k hidden units (fixed)
New class
t-2 t-1 t t+1
+
…
Dynamically Expandable Network (DEN)
To prevent this, we propose a novel deep network that can selectively utilize prior
knowledge for each task while dynamically expanding its capacity when necessary.
𝑾𝑾 1
𝑾𝑾 2
+
New hidden units
New class
t-2 t-1 t t+1
+
+
Dynamically Expandable Network (DEN)
Existing models simply retrain the network for the new task, or expand the network
with fixed number of neurons without retraining.
Elastic Weight Consolidation
[Kirkpatrick et al. 16]
Progressive Network
[Rusu et al. 16]
Dynamically Expandable Network
[Ours]
Our dynamically expandable network, on the other hand, partially retrain the existing
network and add in only the necessary number of neurons.
Incremental Training of a DEN
We further prevent semantic drift by splitting/duplicating units that have significantly
changed in their meanings after learning for each task 𝑡𝑡, and timestamping units.
Selective retraining Dynamic network
expansion
Network split /
duplication
For all hidden unit i,
We first identify and retrain only the relevant parameters for task 𝑡𝑡. If the loss is still
high, we expand each layer by 𝑘𝑘 neurons with group sparsity to drop unnecessary ones.
Incremental Training of a DEN
minimize
𝑾𝑾𝐿𝐿,𝑡𝑡
𝑡𝑡
𝓛𝓛 𝑾𝑾𝐿𝐿,𝑡𝑡
𝑡𝑡
; 𝑾𝑾1:𝐿𝐿–1
𝑡𝑡–1
, 𝒟𝒟𝑡𝑡 + μ 𝑾𝑾𝐿𝐿,𝑡𝑡
𝑡𝑡
1
minimize
𝑾𝑾𝑆𝑆
𝑡𝑡
𝓛𝓛 𝑾𝑾𝑆𝑆
𝑡𝑡
; 𝑾𝑾𝑆𝑆 𝑐𝑐
𝑡𝑡–1
, 𝒟𝒟𝑡𝑡 + μ 𝑾𝑾𝑆𝑆
𝑡𝑡
2
1. Selective Retraining
• Initially, train the network with ℓ𝟏𝟏-regularization
to promote sparsity in the weights.
• Fit a sparse linear model to predict task 𝑡𝑡 using
topmost hidden units of the neural network.
• Perform breadth-first search on the network
starting from selected nodes.
When the model learns new tasks, the network finds relevant neurons, and retrains
only them.
t-1 t
𝒙𝒙𝟐𝟐 𝒙𝒙𝒊𝒊. . .𝒙𝒙𝟏𝟏
Incremental Training of a DEN
minimize
𝑾𝑾𝑙𝑙
𝑁𝑁
𝓛𝓛 𝑾𝑾𝑙𝑙
𝑁𝑁
; 𝑾𝑾𝑙𝑙
𝑡𝑡–1
, 𝒟𝒟𝑡𝑡 + λ∑𝑔𝑔 𝑾𝑾𝑙𝑙,𝑔𝑔
𝑁𝑁
2
When loss is higher than threshold 𝝉𝝉, expand constant k neurons at each layer,
and remove useless ones among them.
t-1 t
𝒙𝒙𝟐𝟐 𝒙𝒙𝒊𝒊. . .𝒙𝒙𝟏𝟏
+
+
2. Dynamically Network Expansion
• Perform group sparsity regularization on the
added parameters.
where 𝑔𝑔 ∈ 𝐺𝐺 is a group defined on the incoming weights
for each neuron.
• The model captures new features that were not
previously represented by 𝑾𝑾𝑙𝑙
𝑡𝑡−1
.
Group Sparsity Regularization
Ω 𝑾𝑾 𝑙𝑙
= �
𝑔𝑔
𝑾𝑾𝑔𝑔
𝑙𝑙
2
Group sparsity
Layer 𝒍𝒍 − 𝟏𝟏
Layer 𝒍𝒍
[Wen16] Wen, Wei, et al. "Learning structured sparsity in deep neural networks." Advances in Neural Information Processing Systems. 2016.
Layer 𝒍𝒍 − 𝟏𝟏
Layer 𝒍𝒍
Grouping !
(2,1)-norm, which is the 1-norm over 2-norm groups, promotes feature sharing
and results in complete elimination of the features that are not shared.
Incremental Training of a DEN
minimize
𝑾𝑾𝑡𝑡
𝓛𝓛 𝑾𝑾𝑡𝑡
; 𝒟𝒟𝑡𝑡 + λ 𝑾𝑾𝑡𝑡
– 𝑾𝑾𝑡𝑡−1
2
2
After 2., if the similarity with neurons of previous step is larger than the threshold σ,
we split & duplicate those neurons and restore them to previous step.
t-1 t
+
+
Copy
3. Network Split / Duplication
• Measure the amount of semantic drift 𝜌𝜌𝑖𝑖
𝑡𝑡
for each
hidden unit 𝑖𝑖, if 𝜌𝜌𝑖𝑖
𝑡𝑡
> 𝜎𝜎, copy it.
• After the duplication, retrain the network since
split changes the overall structure.
Incremental Training of a DEN
We timestamp each newly added units to record the stage 𝑡𝑡 when it is added to the
network, to further prevent drift by the introduction of new hidden units.
t-2 tt-1
Datasets and Networks
We validate our method on four public datasets for classification, with various
networks.
CIFAR-100
• 100 animal and
vehicle classes
• Used modified
version of AlexNet
MNIST-variation
• Modified MNIST
dataset including
perturbation
• Used LeNet-4
(2 of conv., 2 of fc.)
Permuted-MNIST
• Different random
permutation of the
input pixels
• Used LeNet-4
AwA
• 50 animal classes
• Used feedforward
network
Baselines
We compare our networks against relevant baselines.
D D D
M1 M2 M3
D D D
M1 M3 M3
STL
MTL
minimize
𝑾𝑾𝑡𝑡
𝓛𝓛 𝑾𝑾𝑡𝑡; 𝒟𝒟𝑡𝑡 +
+ λ 𝑾𝑾𝑡𝑡– 𝑾𝑾𝑡𝑡−1
2
2
L2
EWC
Progressive Networks
Rusu, Andrei A., et al. "Progressive neural networks." arXiv preprint arXiv:1606.04671 (2016).
Kirkpatrick, James, et al. "Overcoming catastrophic forgetting in neural networks." Proceedings of the National Academy of Sciences 114.13 (2017): 3521-3526.
DEN
Results
Incremental training with DEN results in obtaining a much smaller network that
performs almost the same as the networks that are trained in batch.
Further fine-tuning of DEN on all tasks obtains the best performance, which shows
that DEN is also useful for network capacity estimation.
Results
DEN maintains the performance obtained on the previous tasks and allows for
higher performance improvements for later tasks.
Also, timestamped inference is highly effective in preventing semantic drift.
Results
Selective retraining takes significantly less time than the full retraining of the
network, even shows much higher AUROC.
DNN-selective mostly selects less portion of upper level units which are more task-
specific, while selecting larger portion of more generic lower layer units.
Results
There are the models with a variant of our model that does selective retraining and
layer expansion, but without network split on MNIST-Variation dataset.
DEN-Dynamic even outperforms DEN-Constant with similar capacity, since the model
can dynamically adjust the number of neurons at each layers.
Results
In the permuted MNIST, our DEN outperforms all lifelong learning baselines while
using only 1.39 times of base network capacity.
Further, DEN-Finetune achieves the best AUROC among all models, including DNN-STL
and DNN-MTL.
Conclusion
• We proposed a novel deep neural network for lifelong learning, Dynamically
Expandable Network (DEN).
• DEN performs partial retraining of the network trained on old while
increasing its capacity when necessary.
• DEN significantly outperforms the existing lifelong learning methods,
achieving almost the same performance as the network trained in batch.
• Further fine-tuning of the models on all tasks results in obtaining models that
outperform the batch models, which shows that DEN is useful for network
structure estimation as well.

Más contenido relacionado

La actualidad más candente

【DL輪読会】Scaling Laws for Neural Language Models
【DL輪読会】Scaling Laws for Neural Language Models【DL輪読会】Scaling Laws for Neural Language Models
【DL輪読会】Scaling Laws for Neural Language ModelsDeep Learning JP
 
Continual Learning with Deep Architectures - Tutorial ICML 2021
Continual Learning with Deep Architectures - Tutorial ICML 2021Continual Learning with Deep Architectures - Tutorial ICML 2021
Continual Learning with Deep Architectures - Tutorial ICML 2021Vincenzo Lomonaco
 
[DL輪読会]Model soups: averaging weights of multiple fine-tuned models improves ...
[DL輪読会]Model soups: averaging weights of multiple fine-tuned models improves ...[DL輪読会]Model soups: averaging weights of multiple fine-tuned models improves ...
[DL輪読会]Model soups: averaging weights of multiple fine-tuned models improves ...Deep Learning JP
 
AlphaGo Zero 解説
AlphaGo Zero 解説AlphaGo Zero 解説
AlphaGo Zero 解説suckgeun lee
 
【DL輪読会】Free Lunch for Few-shot Learning: Distribution Calibration
【DL輪読会】Free Lunch for Few-shot Learning: Distribution Calibration【DL輪読会】Free Lunch for Few-shot Learning: Distribution Calibration
【DL輪読会】Free Lunch for Few-shot Learning: Distribution CalibrationDeep Learning JP
 
Recurrent Neural Networks
Recurrent Neural NetworksRecurrent Neural Networks
Recurrent Neural NetworksSeiya Tokui
 
【DL輪読会】"Language Instructed Reinforcement Learning for Human-AI Coordination "
【DL輪読会】"Language Instructed Reinforcement Learning  for Human-AI Coordination "【DL輪読会】"Language Instructed Reinforcement Learning  for Human-AI Coordination "
【DL輪読会】"Language Instructed Reinforcement Learning for Human-AI Coordination "Deep Learning JP
 
[DL輪読会]MetaFormer is Actually What You Need for Vision
[DL輪読会]MetaFormer is Actually What You Need for Vision[DL輪読会]MetaFormer is Actually What You Need for Vision
[DL輪読会]MetaFormer is Actually What You Need for VisionDeep Learning JP
 
[DL輪読会]相互情報量最大化による表現学習
[DL輪読会]相互情報量最大化による表現学習[DL輪読会]相互情報量最大化による表現学習
[DL輪読会]相互情報量最大化による表現学習Deep Learning JP
 
Learning to forget continual prediction with lstm
Learning to forget continual prediction with lstmLearning to forget continual prediction with lstm
Learning to forget continual prediction with lstmFujimoto Keisuke
 
[DL輪読会]ドメイン転移と不変表現に関するサーベイ
[DL輪読会]ドメイン転移と不変表現に関するサーベイ[DL輪読会]ドメイン転移と不変表現に関するサーベイ
[DL輪読会]ドメイン転移と不変表現に関するサーベイDeep Learning JP
 
最新の多様な深層強化学習モデルとその応用(第40回強化学習アーキテクチャ講演資料)
最新の多様な深層強化学習モデルとその応用(第40回強化学習アーキテクチャ講演資料)最新の多様な深層強化学習モデルとその応用(第40回強化学習アーキテクチャ講演資料)
最新の多様な深層強化学習モデルとその応用(第40回強化学習アーキテクチャ講演資料)Shota Imai
 
深層学習の非常に簡単な説明
深層学習の非常に簡単な説明深層学習の非常に簡単な説明
深層学習の非常に簡単な説明Seiichi Uchida
 
【DL輪読会】Is Conditional Generative Modeling All You Need For Decision-Making?
【DL輪読会】Is Conditional Generative Modeling All You Need For Decision-Making?【DL輪読会】Is Conditional Generative Modeling All You Need For Decision-Making?
【DL輪読会】Is Conditional Generative Modeling All You Need For Decision-Making?Deep Learning JP
 
[DL輪読会]One Model To Learn Them All
[DL輪読会]One Model To Learn Them All[DL輪読会]One Model To Learn Them All
[DL輪読会]One Model To Learn Them AllDeep Learning JP
 
[DL輪読会]Temporal DifferenceVariationalAuto-Encoder
[DL輪読会]Temporal DifferenceVariationalAuto-Encoder[DL輪読会]Temporal DifferenceVariationalAuto-Encoder
[DL輪読会]Temporal DifferenceVariationalAuto-EncoderDeep Learning JP
 
[DL輪読会]SafePicking: Learning Safe Object Extraction via Object-Level Mapping ...
[DL輪読会]SafePicking: Learning Safe Object Extraction via Object-Level Mapping ...[DL輪読会]SafePicking: Learning Safe Object Extraction via Object-Level Mapping ...
[DL輪読会]SafePicking: Learning Safe Object Extraction via Object-Level Mapping ...Deep Learning JP
 
Efficient Lifelong Learning with A-GEM ( ICLR 2019 読み会 in 京都 20190602)
Efficient Lifelong Learning with A-GEM ( ICLR 2019 読み会 in 京都 20190602)Efficient Lifelong Learning with A-GEM ( ICLR 2019 読み会 in 京都 20190602)
Efficient Lifelong Learning with A-GEM ( ICLR 2019 読み会 in 京都 20190602)YuMaruyama
 
[DL輪読会]A System for General In-Hand Object Re-Orientation
[DL輪読会]A System for General In-Hand Object Re-Orientation[DL輪読会]A System for General In-Hand Object Re-Orientation
[DL輪読会]A System for General In-Hand Object Re-OrientationDeep Learning JP
 

La actualidad más candente (20)

【DL輪読会】Scaling Laws for Neural Language Models
【DL輪読会】Scaling Laws for Neural Language Models【DL輪読会】Scaling Laws for Neural Language Models
【DL輪読会】Scaling Laws for Neural Language Models
 
Continual Learning with Deep Architectures - Tutorial ICML 2021
Continual Learning with Deep Architectures - Tutorial ICML 2021Continual Learning with Deep Architectures - Tutorial ICML 2021
Continual Learning with Deep Architectures - Tutorial ICML 2021
 
[DL輪読会]Model soups: averaging weights of multiple fine-tuned models improves ...
[DL輪読会]Model soups: averaging weights of multiple fine-tuned models improves ...[DL輪読会]Model soups: averaging weights of multiple fine-tuned models improves ...
[DL輪読会]Model soups: averaging weights of multiple fine-tuned models improves ...
 
AlphaGo Zero 解説
AlphaGo Zero 解説AlphaGo Zero 解説
AlphaGo Zero 解説
 
【DL輪読会】Free Lunch for Few-shot Learning: Distribution Calibration
【DL輪読会】Free Lunch for Few-shot Learning: Distribution Calibration【DL輪読会】Free Lunch for Few-shot Learning: Distribution Calibration
【DL輪読会】Free Lunch for Few-shot Learning: Distribution Calibration
 
Recurrent Neural Networks
Recurrent Neural NetworksRecurrent Neural Networks
Recurrent Neural Networks
 
【DL輪読会】"Language Instructed Reinforcement Learning for Human-AI Coordination "
【DL輪読会】"Language Instructed Reinforcement Learning  for Human-AI Coordination "【DL輪読会】"Language Instructed Reinforcement Learning  for Human-AI Coordination "
【DL輪読会】"Language Instructed Reinforcement Learning for Human-AI Coordination "
 
[DL輪読会]MetaFormer is Actually What You Need for Vision
[DL輪読会]MetaFormer is Actually What You Need for Vision[DL輪読会]MetaFormer is Actually What You Need for Vision
[DL輪読会]MetaFormer is Actually What You Need for Vision
 
[DL輪読会]相互情報量最大化による表現学習
[DL輪読会]相互情報量最大化による表現学習[DL輪読会]相互情報量最大化による表現学習
[DL輪読会]相互情報量最大化による表現学習
 
[DL輪読会]World Models
[DL輪読会]World Models[DL輪読会]World Models
[DL輪読会]World Models
 
Learning to forget continual prediction with lstm
Learning to forget continual prediction with lstmLearning to forget continual prediction with lstm
Learning to forget continual prediction with lstm
 
[DL輪読会]ドメイン転移と不変表現に関するサーベイ
[DL輪読会]ドメイン転移と不変表現に関するサーベイ[DL輪読会]ドメイン転移と不変表現に関するサーベイ
[DL輪読会]ドメイン転移と不変表現に関するサーベイ
 
最新の多様な深層強化学習モデルとその応用(第40回強化学習アーキテクチャ講演資料)
最新の多様な深層強化学習モデルとその応用(第40回強化学習アーキテクチャ講演資料)最新の多様な深層強化学習モデルとその応用(第40回強化学習アーキテクチャ講演資料)
最新の多様な深層強化学習モデルとその応用(第40回強化学習アーキテクチャ講演資料)
 
深層学習の非常に簡単な説明
深層学習の非常に簡単な説明深層学習の非常に簡単な説明
深層学習の非常に簡単な説明
 
【DL輪読会】Is Conditional Generative Modeling All You Need For Decision-Making?
【DL輪読会】Is Conditional Generative Modeling All You Need For Decision-Making?【DL輪読会】Is Conditional Generative Modeling All You Need For Decision-Making?
【DL輪読会】Is Conditional Generative Modeling All You Need For Decision-Making?
 
[DL輪読会]One Model To Learn Them All
[DL輪読会]One Model To Learn Them All[DL輪読会]One Model To Learn Them All
[DL輪読会]One Model To Learn Them All
 
[DL輪読会]Temporal DifferenceVariationalAuto-Encoder
[DL輪読会]Temporal DifferenceVariationalAuto-Encoder[DL輪読会]Temporal DifferenceVariationalAuto-Encoder
[DL輪読会]Temporal DifferenceVariationalAuto-Encoder
 
[DL輪読会]SafePicking: Learning Safe Object Extraction via Object-Level Mapping ...
[DL輪読会]SafePicking: Learning Safe Object Extraction via Object-Level Mapping ...[DL輪読会]SafePicking: Learning Safe Object Extraction via Object-Level Mapping ...
[DL輪読会]SafePicking: Learning Safe Object Extraction via Object-Level Mapping ...
 
Efficient Lifelong Learning with A-GEM ( ICLR 2019 読み会 in 京都 20190602)
Efficient Lifelong Learning with A-GEM ( ICLR 2019 読み会 in 京都 20190602)Efficient Lifelong Learning with A-GEM ( ICLR 2019 読み会 in 京都 20190602)
Efficient Lifelong Learning with A-GEM ( ICLR 2019 読み会 in 京都 20190602)
 
[DL輪読会]A System for General In-Hand Object Re-Orientation
[DL輪読会]A System for General In-Hand Object Re-Orientation[DL輪読会]A System for General In-Hand Object Re-Orientation
[DL輪読会]A System for General In-Hand Object Re-Orientation
 

Similar a Lifelong Learning for Dynamically Expandable Networks

Deep learning notes.pptx
Deep learning notes.pptxDeep learning notes.pptx
Deep learning notes.pptxPandi Gingee
 
deepnet-lourentzou.ppt
deepnet-lourentzou.pptdeepnet-lourentzou.ppt
deepnet-lourentzou.pptyang947066
 
AI Class Topic 6: Easy Way to Learn Deep Learning AI Technologies
AI Class Topic 6: Easy Way to Learn Deep Learning AI TechnologiesAI Class Topic 6: Easy Way to Learn Deep Learning AI Technologies
AI Class Topic 6: Easy Way to Learn Deep Learning AI TechnologiesValue Amplify Consulting
 
Designing your neural networks – a step by step walkthrough
Designing your neural networks – a step by step walkthroughDesigning your neural networks – a step by step walkthrough
Designing your neural networks – a step by step walkthroughLavanya Shukla
 
deeplearning
deeplearningdeeplearning
deeplearninghuda2018
 
ML Module 3 Non Linear Learning.pptx
ML Module 3 Non Linear Learning.pptxML Module 3 Non Linear Learning.pptx
ML Module 3 Non Linear Learning.pptxDebabrataPain1
 
Machine Learning
Machine LearningMachine Learning
Machine Learningbutest
 
Introduction to deep learning
Introduction to deep learningIntroduction to deep learning
Introduction to deep learningJunaid Bhat
 
Introduction to Neural networks (under graduate course) Lecture 9 of 9
Introduction to Neural networks (under graduate course) Lecture 9 of 9Introduction to Neural networks (under graduate course) Lecture 9 of 9
Introduction to Neural networks (under graduate course) Lecture 9 of 9Randa Elanwar
 
Objective Evaluation of a Deep Neural Network Approach for Single-Channel Spe...
Objective Evaluation of a Deep Neural Network Approach for Single-Channel Spe...Objective Evaluation of a Deep Neural Network Approach for Single-Channel Spe...
Objective Evaluation of a Deep Neural Network Approach for Single-Channel Spe...csandit
 
Facial Emotion Detection on Children's Emotional Face
Facial Emotion Detection on Children's Emotional FaceFacial Emotion Detection on Children's Emotional Face
Facial Emotion Detection on Children's Emotional FaceTakrim Ul Islam Laskar
 
IRJET-Breast Cancer Detection using Convolution Neural Network
IRJET-Breast Cancer Detection using Convolution Neural NetworkIRJET-Breast Cancer Detection using Convolution Neural Network
IRJET-Breast Cancer Detection using Convolution Neural NetworkIRJET Journal
 
Neuralnetwork 101222074552-phpapp02
Neuralnetwork 101222074552-phpapp02Neuralnetwork 101222074552-phpapp02
Neuralnetwork 101222074552-phpapp02Deepu Gupta
 
Classification by backpropacation
Classification by backpropacationClassification by backpropacation
Classification by backpropacationSiva Priya
 
Muhammad Usman Akhtar | Ph.D Scholar | Wuhan University | School of Co...
Muhammad Usman Akhtar  |  Ph.D Scholar  |  Wuhan  University  |  School of Co...Muhammad Usman Akhtar  |  Ph.D Scholar  |  Wuhan  University  |  School of Co...
Muhammad Usman Akhtar | Ph.D Scholar | Wuhan University | School of Co...Wuhan University
 
Deep Recurrent Neural Networks for Sequence Learning in Spark by Yves Mabiala
Deep Recurrent Neural Networks for Sequence Learning in Spark by Yves MabialaDeep Recurrent Neural Networks for Sequence Learning in Spark by Yves Mabiala
Deep Recurrent Neural Networks for Sequence Learning in Spark by Yves MabialaSpark Summit
 

Similar a Lifelong Learning for Dynamically Expandable Networks (20)

Deep learning notes.pptx
Deep learning notes.pptxDeep learning notes.pptx
Deep learning notes.pptx
 
deepnet-lourentzou.ppt
deepnet-lourentzou.pptdeepnet-lourentzou.ppt
deepnet-lourentzou.ppt
 
AI Class Topic 6: Easy Way to Learn Deep Learning AI Technologies
AI Class Topic 6: Easy Way to Learn Deep Learning AI TechnologiesAI Class Topic 6: Easy Way to Learn Deep Learning AI Technologies
AI Class Topic 6: Easy Way to Learn Deep Learning AI Technologies
 
Designing your neural networks – a step by step walkthrough
Designing your neural networks – a step by step walkthroughDesigning your neural networks – a step by step walkthrough
Designing your neural networks – a step by step walkthrough
 
MaLAI_Hyderabad presentation
MaLAI_Hyderabad presentationMaLAI_Hyderabad presentation
MaLAI_Hyderabad presentation
 
deeplearning
deeplearningdeeplearning
deeplearning
 
ML Module 3 Non Linear Learning.pptx
ML Module 3 Non Linear Learning.pptxML Module 3 Non Linear Learning.pptx
ML Module 3 Non Linear Learning.pptx
 
Machine Learning
Machine LearningMachine Learning
Machine Learning
 
Introduction to deep learning
Introduction to deep learningIntroduction to deep learning
Introduction to deep learning
 
Introduction to Neural networks (under graduate course) Lecture 9 of 9
Introduction to Neural networks (under graduate course) Lecture 9 of 9Introduction to Neural networks (under graduate course) Lecture 9 of 9
Introduction to Neural networks (under graduate course) Lecture 9 of 9
 
Objective Evaluation of a Deep Neural Network Approach for Single-Channel Spe...
Objective Evaluation of a Deep Neural Network Approach for Single-Channel Spe...Objective Evaluation of a Deep Neural Network Approach for Single-Channel Spe...
Objective Evaluation of a Deep Neural Network Approach for Single-Channel Spe...
 
Facial Emotion Detection on Children's Emotional Face
Facial Emotion Detection on Children's Emotional FaceFacial Emotion Detection on Children's Emotional Face
Facial Emotion Detection on Children's Emotional Face
 
02 Fundamental Concepts of ANN
02 Fundamental Concepts of ANN02 Fundamental Concepts of ANN
02 Fundamental Concepts of ANN
 
IRJET-Breast Cancer Detection using Convolution Neural Network
IRJET-Breast Cancer Detection using Convolution Neural NetworkIRJET-Breast Cancer Detection using Convolution Neural Network
IRJET-Breast Cancer Detection using Convolution Neural Network
 
Neuralnetwork 101222074552-phpapp02
Neuralnetwork 101222074552-phpapp02Neuralnetwork 101222074552-phpapp02
Neuralnetwork 101222074552-phpapp02
 
Classification by backpropacation
Classification by backpropacationClassification by backpropacation
Classification by backpropacation
 
Muhammad Usman Akhtar | Ph.D Scholar | Wuhan University | School of Co...
Muhammad Usman Akhtar  |  Ph.D Scholar  |  Wuhan  University  |  School of Co...Muhammad Usman Akhtar  |  Ph.D Scholar  |  Wuhan  University  |  School of Co...
Muhammad Usman Akhtar | Ph.D Scholar | Wuhan University | School of Co...
 
SoftComputing6
SoftComputing6SoftComputing6
SoftComputing6
 
Ffnn
FfnnFfnn
Ffnn
 
Deep Recurrent Neural Networks for Sequence Learning in Spark by Yves Mabiala
Deep Recurrent Neural Networks for Sequence Learning in Spark by Yves MabialaDeep Recurrent Neural Networks for Sequence Learning in Spark by Yves Mabiala
Deep Recurrent Neural Networks for Sequence Learning in Spark by Yves Mabiala
 

Más de NAVER Engineering

디자인 시스템에 직방 ZUIX
디자인 시스템에 직방 ZUIX디자인 시스템에 직방 ZUIX
디자인 시스템에 직방 ZUIXNAVER Engineering
 
진화하는 디자인 시스템(걸음마 편)
진화하는 디자인 시스템(걸음마 편)진화하는 디자인 시스템(걸음마 편)
진화하는 디자인 시스템(걸음마 편)NAVER Engineering
 
서비스 운영을 위한 디자인시스템 프로젝트
서비스 운영을 위한 디자인시스템 프로젝트서비스 운영을 위한 디자인시스템 프로젝트
서비스 운영을 위한 디자인시스템 프로젝트NAVER Engineering
 
BPL(Banksalad Product Language) 무야호
BPL(Banksalad Product Language) 무야호BPL(Banksalad Product Language) 무야호
BPL(Banksalad Product Language) 무야호NAVER Engineering
 
이번 생에 디자인 시스템은 처음이라
이번 생에 디자인 시스템은 처음이라이번 생에 디자인 시스템은 처음이라
이번 생에 디자인 시스템은 처음이라NAVER Engineering
 
날고 있는 여러 비행기 넘나 들며 정비하기
날고 있는 여러 비행기 넘나 들며 정비하기날고 있는 여러 비행기 넘나 들며 정비하기
날고 있는 여러 비행기 넘나 들며 정비하기NAVER Engineering
 
쏘카프레임 구축 배경과 과정
 쏘카프레임 구축 배경과 과정 쏘카프레임 구축 배경과 과정
쏘카프레임 구축 배경과 과정NAVER Engineering
 
플랫폼 디자이너 없이 디자인 시스템을 구축하는 프로덕트 디자이너의 우당탕탕 고통 연대기
플랫폼 디자이너 없이 디자인 시스템을 구축하는 프로덕트 디자이너의 우당탕탕 고통 연대기플랫폼 디자이너 없이 디자인 시스템을 구축하는 프로덕트 디자이너의 우당탕탕 고통 연대기
플랫폼 디자이너 없이 디자인 시스템을 구축하는 프로덕트 디자이너의 우당탕탕 고통 연대기NAVER Engineering
 
200820 NAVER TECH CONCERT 15_Code Review is Horse(코드리뷰는 말이야)(feat.Latte)
200820 NAVER TECH CONCERT 15_Code Review is Horse(코드리뷰는 말이야)(feat.Latte)200820 NAVER TECH CONCERT 15_Code Review is Horse(코드리뷰는 말이야)(feat.Latte)
200820 NAVER TECH CONCERT 15_Code Review is Horse(코드리뷰는 말이야)(feat.Latte)NAVER Engineering
 
200819 NAVER TECH CONCERT 03_화려한 코루틴이 내 앱을 감싸네! 코루틴으로 작성해보는 깔끔한 비동기 코드
200819 NAVER TECH CONCERT 03_화려한 코루틴이 내 앱을 감싸네! 코루틴으로 작성해보는 깔끔한 비동기 코드200819 NAVER TECH CONCERT 03_화려한 코루틴이 내 앱을 감싸네! 코루틴으로 작성해보는 깔끔한 비동기 코드
200819 NAVER TECH CONCERT 03_화려한 코루틴이 내 앱을 감싸네! 코루틴으로 작성해보는 깔끔한 비동기 코드NAVER Engineering
 
200819 NAVER TECH CONCERT 10_맥북에서도 아이맥프로에서 빌드하는 것처럼 빌드 속도 빠르게 하기
200819 NAVER TECH CONCERT 10_맥북에서도 아이맥프로에서 빌드하는 것처럼 빌드 속도 빠르게 하기200819 NAVER TECH CONCERT 10_맥북에서도 아이맥프로에서 빌드하는 것처럼 빌드 속도 빠르게 하기
200819 NAVER TECH CONCERT 10_맥북에서도 아이맥프로에서 빌드하는 것처럼 빌드 속도 빠르게 하기NAVER Engineering
 
200819 NAVER TECH CONCERT 08_성능을 고민하는 슬기로운 개발자 생활
200819 NAVER TECH CONCERT 08_성능을 고민하는 슬기로운 개발자 생활200819 NAVER TECH CONCERT 08_성능을 고민하는 슬기로운 개발자 생활
200819 NAVER TECH CONCERT 08_성능을 고민하는 슬기로운 개발자 생활NAVER Engineering
 
200819 NAVER TECH CONCERT 05_모르면 손해보는 Android 디버깅/분석 꿀팁 대방출
200819 NAVER TECH CONCERT 05_모르면 손해보는 Android 디버깅/분석 꿀팁 대방출200819 NAVER TECH CONCERT 05_모르면 손해보는 Android 디버깅/분석 꿀팁 대방출
200819 NAVER TECH CONCERT 05_모르면 손해보는 Android 디버깅/분석 꿀팁 대방출NAVER Engineering
 
200819 NAVER TECH CONCERT 09_Case.xcodeproj - 좋은 동료로 거듭나기 위한 노하우
200819 NAVER TECH CONCERT 09_Case.xcodeproj - 좋은 동료로 거듭나기 위한 노하우200819 NAVER TECH CONCERT 09_Case.xcodeproj - 좋은 동료로 거듭나기 위한 노하우
200819 NAVER TECH CONCERT 09_Case.xcodeproj - 좋은 동료로 거듭나기 위한 노하우NAVER Engineering
 
200820 NAVER TECH CONCERT 14_야 너두 할 수 있어. 비전공자, COBOL 개발자를 거쳐 네이버에서 FE 개발하게 된...
200820 NAVER TECH CONCERT 14_야 너두 할 수 있어. 비전공자, COBOL 개발자를 거쳐 네이버에서 FE 개발하게 된...200820 NAVER TECH CONCERT 14_야 너두 할 수 있어. 비전공자, COBOL 개발자를 거쳐 네이버에서 FE 개발하게 된...
200820 NAVER TECH CONCERT 14_야 너두 할 수 있어. 비전공자, COBOL 개발자를 거쳐 네이버에서 FE 개발하게 된...NAVER Engineering
 
200820 NAVER TECH CONCERT 13_네이버에서 오픈 소스 개발을 통해 성장하는 방법
200820 NAVER TECH CONCERT 13_네이버에서 오픈 소스 개발을 통해 성장하는 방법200820 NAVER TECH CONCERT 13_네이버에서 오픈 소스 개발을 통해 성장하는 방법
200820 NAVER TECH CONCERT 13_네이버에서 오픈 소스 개발을 통해 성장하는 방법NAVER Engineering
 
200820 NAVER TECH CONCERT 12_상반기 네이버 인턴을 돌아보며
200820 NAVER TECH CONCERT 12_상반기 네이버 인턴을 돌아보며200820 NAVER TECH CONCERT 12_상반기 네이버 인턴을 돌아보며
200820 NAVER TECH CONCERT 12_상반기 네이버 인턴을 돌아보며NAVER Engineering
 
200820 NAVER TECH CONCERT 11_빠르게 성장하는 슈퍼루키로 거듭나기
200820 NAVER TECH CONCERT 11_빠르게 성장하는 슈퍼루키로 거듭나기200820 NAVER TECH CONCERT 11_빠르게 성장하는 슈퍼루키로 거듭나기
200820 NAVER TECH CONCERT 11_빠르게 성장하는 슈퍼루키로 거듭나기NAVER Engineering
 
200819 NAVER TECH CONCERT 07_신입 iOS 개발자 개발업무 적응기
200819 NAVER TECH CONCERT 07_신입 iOS 개발자 개발업무 적응기200819 NAVER TECH CONCERT 07_신입 iOS 개발자 개발업무 적응기
200819 NAVER TECH CONCERT 07_신입 iOS 개발자 개발업무 적응기NAVER Engineering
 

Más de NAVER Engineering (20)

React vac pattern
React vac patternReact vac pattern
React vac pattern
 
디자인 시스템에 직방 ZUIX
디자인 시스템에 직방 ZUIX디자인 시스템에 직방 ZUIX
디자인 시스템에 직방 ZUIX
 
진화하는 디자인 시스템(걸음마 편)
진화하는 디자인 시스템(걸음마 편)진화하는 디자인 시스템(걸음마 편)
진화하는 디자인 시스템(걸음마 편)
 
서비스 운영을 위한 디자인시스템 프로젝트
서비스 운영을 위한 디자인시스템 프로젝트서비스 운영을 위한 디자인시스템 프로젝트
서비스 운영을 위한 디자인시스템 프로젝트
 
BPL(Banksalad Product Language) 무야호
BPL(Banksalad Product Language) 무야호BPL(Banksalad Product Language) 무야호
BPL(Banksalad Product Language) 무야호
 
이번 생에 디자인 시스템은 처음이라
이번 생에 디자인 시스템은 처음이라이번 생에 디자인 시스템은 처음이라
이번 생에 디자인 시스템은 처음이라
 
날고 있는 여러 비행기 넘나 들며 정비하기
날고 있는 여러 비행기 넘나 들며 정비하기날고 있는 여러 비행기 넘나 들며 정비하기
날고 있는 여러 비행기 넘나 들며 정비하기
 
쏘카프레임 구축 배경과 과정
 쏘카프레임 구축 배경과 과정 쏘카프레임 구축 배경과 과정
쏘카프레임 구축 배경과 과정
 
플랫폼 디자이너 없이 디자인 시스템을 구축하는 프로덕트 디자이너의 우당탕탕 고통 연대기
플랫폼 디자이너 없이 디자인 시스템을 구축하는 프로덕트 디자이너의 우당탕탕 고통 연대기플랫폼 디자이너 없이 디자인 시스템을 구축하는 프로덕트 디자이너의 우당탕탕 고통 연대기
플랫폼 디자이너 없이 디자인 시스템을 구축하는 프로덕트 디자이너의 우당탕탕 고통 연대기
 
200820 NAVER TECH CONCERT 15_Code Review is Horse(코드리뷰는 말이야)(feat.Latte)
200820 NAVER TECH CONCERT 15_Code Review is Horse(코드리뷰는 말이야)(feat.Latte)200820 NAVER TECH CONCERT 15_Code Review is Horse(코드리뷰는 말이야)(feat.Latte)
200820 NAVER TECH CONCERT 15_Code Review is Horse(코드리뷰는 말이야)(feat.Latte)
 
200819 NAVER TECH CONCERT 03_화려한 코루틴이 내 앱을 감싸네! 코루틴으로 작성해보는 깔끔한 비동기 코드
200819 NAVER TECH CONCERT 03_화려한 코루틴이 내 앱을 감싸네! 코루틴으로 작성해보는 깔끔한 비동기 코드200819 NAVER TECH CONCERT 03_화려한 코루틴이 내 앱을 감싸네! 코루틴으로 작성해보는 깔끔한 비동기 코드
200819 NAVER TECH CONCERT 03_화려한 코루틴이 내 앱을 감싸네! 코루틴으로 작성해보는 깔끔한 비동기 코드
 
200819 NAVER TECH CONCERT 10_맥북에서도 아이맥프로에서 빌드하는 것처럼 빌드 속도 빠르게 하기
200819 NAVER TECH CONCERT 10_맥북에서도 아이맥프로에서 빌드하는 것처럼 빌드 속도 빠르게 하기200819 NAVER TECH CONCERT 10_맥북에서도 아이맥프로에서 빌드하는 것처럼 빌드 속도 빠르게 하기
200819 NAVER TECH CONCERT 10_맥북에서도 아이맥프로에서 빌드하는 것처럼 빌드 속도 빠르게 하기
 
200819 NAVER TECH CONCERT 08_성능을 고민하는 슬기로운 개발자 생활
200819 NAVER TECH CONCERT 08_성능을 고민하는 슬기로운 개발자 생활200819 NAVER TECH CONCERT 08_성능을 고민하는 슬기로운 개발자 생활
200819 NAVER TECH CONCERT 08_성능을 고민하는 슬기로운 개발자 생활
 
200819 NAVER TECH CONCERT 05_모르면 손해보는 Android 디버깅/분석 꿀팁 대방출
200819 NAVER TECH CONCERT 05_모르면 손해보는 Android 디버깅/분석 꿀팁 대방출200819 NAVER TECH CONCERT 05_모르면 손해보는 Android 디버깅/분석 꿀팁 대방출
200819 NAVER TECH CONCERT 05_모르면 손해보는 Android 디버깅/분석 꿀팁 대방출
 
200819 NAVER TECH CONCERT 09_Case.xcodeproj - 좋은 동료로 거듭나기 위한 노하우
200819 NAVER TECH CONCERT 09_Case.xcodeproj - 좋은 동료로 거듭나기 위한 노하우200819 NAVER TECH CONCERT 09_Case.xcodeproj - 좋은 동료로 거듭나기 위한 노하우
200819 NAVER TECH CONCERT 09_Case.xcodeproj - 좋은 동료로 거듭나기 위한 노하우
 
200820 NAVER TECH CONCERT 14_야 너두 할 수 있어. 비전공자, COBOL 개발자를 거쳐 네이버에서 FE 개발하게 된...
200820 NAVER TECH CONCERT 14_야 너두 할 수 있어. 비전공자, COBOL 개발자를 거쳐 네이버에서 FE 개발하게 된...200820 NAVER TECH CONCERT 14_야 너두 할 수 있어. 비전공자, COBOL 개발자를 거쳐 네이버에서 FE 개발하게 된...
200820 NAVER TECH CONCERT 14_야 너두 할 수 있어. 비전공자, COBOL 개발자를 거쳐 네이버에서 FE 개발하게 된...
 
200820 NAVER TECH CONCERT 13_네이버에서 오픈 소스 개발을 통해 성장하는 방법
200820 NAVER TECH CONCERT 13_네이버에서 오픈 소스 개발을 통해 성장하는 방법200820 NAVER TECH CONCERT 13_네이버에서 오픈 소스 개발을 통해 성장하는 방법
200820 NAVER TECH CONCERT 13_네이버에서 오픈 소스 개발을 통해 성장하는 방법
 
200820 NAVER TECH CONCERT 12_상반기 네이버 인턴을 돌아보며
200820 NAVER TECH CONCERT 12_상반기 네이버 인턴을 돌아보며200820 NAVER TECH CONCERT 12_상반기 네이버 인턴을 돌아보며
200820 NAVER TECH CONCERT 12_상반기 네이버 인턴을 돌아보며
 
200820 NAVER TECH CONCERT 11_빠르게 성장하는 슈퍼루키로 거듭나기
200820 NAVER TECH CONCERT 11_빠르게 성장하는 슈퍼루키로 거듭나기200820 NAVER TECH CONCERT 11_빠르게 성장하는 슈퍼루키로 거듭나기
200820 NAVER TECH CONCERT 11_빠르게 성장하는 슈퍼루키로 거듭나기
 
200819 NAVER TECH CONCERT 07_신입 iOS 개발자 개발업무 적응기
200819 NAVER TECH CONCERT 07_신입 iOS 개발자 개발업무 적응기200819 NAVER TECH CONCERT 07_신입 iOS 개발자 개발업무 적응기
200819 NAVER TECH CONCERT 07_신입 iOS 개발자 개발업무 적응기
 

Último

Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 

Último (20)

Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 

Lifelong Learning for Dynamically Expandable Networks

  • 1. Lifelong Learning with Dynamically Expandable Networks 𝑱𝑱𝑱𝑱𝑱𝑱𝑱𝑱𝑱𝑱𝑱𝑱𝑱𝑱 𝒀𝒀𝒀𝒀𝒀𝒀𝒀𝒀𝟏𝟏,𝟑𝟑∗ , 𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸 𝑌𝑌𝑌𝑌𝑌𝑌𝑔𝑔1,3 , 𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽 𝐿𝐿𝐿𝐿𝑒𝑒2 , 𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆 𝐽𝐽𝐽𝐽 𝐻𝐻𝐻𝐻𝐻𝐻𝐻𝐻𝑔𝑔1,3 𝐾𝐾𝐾𝐾𝐾𝐾𝐾𝐾 𝑇𝑇1, 𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑇𝑇2, 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 𝑠𝑠3 *𝑊𝑊𝑊𝑊𝑊𝑊𝑊𝑊 𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑 𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤 𝑎𝑎𝑎𝑎 𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈
  • 2. Introduction Korea Advanced Institute of Science and Technology (KAIST) • Ph.D. in School of Computing (Aug. 2018. – ) • Advisor: Prof. Sung Ju Hwang Ulsan National Institute of Science and Technology (UNIST) • M. S. in Computer Engineering (Aug. 2016 – Feb. 2018) • Advisor: Prof. Sung Ju Hwang • B. S. in Computer Science Engineering (Mar. 2012 – Aug. 2016) • Biological Science Minor Jaehong Yoon - Education
  • 3. Introduction Juho Lee, S. Kim, J. Yoon, H. B. Lee, E. Yang, S. J. Hwang, “Adaptive Network Sparsification via Dependent Variational Beta-Bernoulli Dropout”, arXiv preprint arXiv:1805.10896 (2018). Jaehong Yoon, E. Yang, J. Lee, and S. J. Hwang, “Lifelong Learning with Dynamically Expandable Networks”, International Conference on Learning Representation (ICLR), 2018 Jaehong Yoon, and S. J. Hwang, “Combined Group and Exclusive Sparsity for Deep Neural Networks”, International Conference on Machine Learning (ICML), 2017 - Experience - Publications Korea Advanced Institute of Science and Technology (KAIST) • Contract Research Scientist (Feb. 2018 ~ Aug. 2018) AItrics • Research Intern (Mar. 2018 ~ May 2018)
  • 4. Challenge: Incomplete, Growing Dataset In many large-scale learning scenarios, not all training data might be available when we want to begin training the network. Car Convertible Sports car ImageNet 22,000 classes Sedan Roadster
  • 5. Challenge: Incomplete, Growing Dataset In many large-scale learning scenarios, not all training data might be available when we want to begin training the network. Car Sports car Sedan Roadster 1M classes BMW Z4 Ferrari 458 spider Convertible Ferrari 458 Italia Porsche 911 Turbo Hyundai Sonata BMW 3 series
  • 6. Challenge: Incomplete, Growing Dataset Even worse, the set of tasks may dynamically grow as new tasks are introduced. Car Sports car Sedan Roadster BMW Z4 Ferrari 458 spider Convertible 2015 Mustang Convertible Ferrari 458 Italia Porsche 911 Turbo Hyundai Sonata Tesla Model SBMW 3 series 1M classes
  • 7. Solution: Lifelong Learning Humans learn forever throughout their lives - couldn’t we build a similar system that basically learns forever while becoming increasingly smarter over time? We integrate our model into a lifelong learning framework, that continuously learns by actively discovering new categories and learning them in the context of known ones. t-2 t-1 t Learning Model t+1 Learned knowledge 3) New knowledge is stored for future use 2) Knowledge is transferred from previously Learned tasks 1) Tasks are received in a sequential order 4) Refine existing knowledge Humans learn forever throughout their lives
  • 8. Lifelong Learning of a Deep Neural Network However, if the classes we had in the early stages of learning significantly differs from the new class, utilization of prior knowledge may degenerate performance. 𝑾𝑾 1 𝑾𝑾 2 t-2 t-1 t t+1 New class + 𝑾𝑾 2
  • 9. Semantic Drift Introduction of new units can also result in semantic drift or catastrophic forgetting, where original meaning of the features change as they fit to later tasks. 𝑾𝑾 1 𝑾𝑾 2 New class +
  • 10. Network Expansion To learn new tasks which are relatively different from early stages of learning, model may need to expand network capacity. 𝑾𝑾 1 𝑾𝑾 2 + New k hidden units (fixed) New class t-2 t-1 t t+1 + …
  • 11. Dynamically Expandable Network (DEN) To prevent this, we propose a novel deep network that can selectively utilize prior knowledge for each task while dynamically expanding its capacity when necessary. 𝑾𝑾 1 𝑾𝑾 2 + New hidden units New class t-2 t-1 t t+1 + +
  • 12. Dynamically Expandable Network (DEN) Existing models simply retrain the network for the new task, or expand the network with fixed number of neurons without retraining. Elastic Weight Consolidation [Kirkpatrick et al. 16] Progressive Network [Rusu et al. 16] Dynamically Expandable Network [Ours] Our dynamically expandable network, on the other hand, partially retrain the existing network and add in only the necessary number of neurons.
  • 13. Incremental Training of a DEN We further prevent semantic drift by splitting/duplicating units that have significantly changed in their meanings after learning for each task 𝑡𝑡, and timestamping units. Selective retraining Dynamic network expansion Network split / duplication For all hidden unit i, We first identify and retrain only the relevant parameters for task 𝑡𝑡. If the loss is still high, we expand each layer by 𝑘𝑘 neurons with group sparsity to drop unnecessary ones.
  • 14. Incremental Training of a DEN minimize 𝑾𝑾𝐿𝐿,𝑡𝑡 𝑡𝑡 𝓛𝓛 𝑾𝑾𝐿𝐿,𝑡𝑡 𝑡𝑡 ; 𝑾𝑾1:𝐿𝐿–1 𝑡𝑡–1 , 𝒟𝒟𝑡𝑡 + μ 𝑾𝑾𝐿𝐿,𝑡𝑡 𝑡𝑡 1 minimize 𝑾𝑾𝑆𝑆 𝑡𝑡 𝓛𝓛 𝑾𝑾𝑆𝑆 𝑡𝑡 ; 𝑾𝑾𝑆𝑆 𝑐𝑐 𝑡𝑡–1 , 𝒟𝒟𝑡𝑡 + μ 𝑾𝑾𝑆𝑆 𝑡𝑡 2 1. Selective Retraining • Initially, train the network with ℓ𝟏𝟏-regularization to promote sparsity in the weights. • Fit a sparse linear model to predict task 𝑡𝑡 using topmost hidden units of the neural network. • Perform breadth-first search on the network starting from selected nodes. When the model learns new tasks, the network finds relevant neurons, and retrains only them. t-1 t 𝒙𝒙𝟐𝟐 𝒙𝒙𝒊𝒊. . .𝒙𝒙𝟏𝟏
  • 15. Incremental Training of a DEN minimize 𝑾𝑾𝑙𝑙 𝑁𝑁 𝓛𝓛 𝑾𝑾𝑙𝑙 𝑁𝑁 ; 𝑾𝑾𝑙𝑙 𝑡𝑡–1 , 𝒟𝒟𝑡𝑡 + λ∑𝑔𝑔 𝑾𝑾𝑙𝑙,𝑔𝑔 𝑁𝑁 2 When loss is higher than threshold 𝝉𝝉, expand constant k neurons at each layer, and remove useless ones among them. t-1 t 𝒙𝒙𝟐𝟐 𝒙𝒙𝒊𝒊. . .𝒙𝒙𝟏𝟏 + + 2. Dynamically Network Expansion • Perform group sparsity regularization on the added parameters. where 𝑔𝑔 ∈ 𝐺𝐺 is a group defined on the incoming weights for each neuron. • The model captures new features that were not previously represented by 𝑾𝑾𝑙𝑙 𝑡𝑡−1 .
  • 16. Group Sparsity Regularization Ω 𝑾𝑾 𝑙𝑙 = � 𝑔𝑔 𝑾𝑾𝑔𝑔 𝑙𝑙 2 Group sparsity Layer 𝒍𝒍 − 𝟏𝟏 Layer 𝒍𝒍 [Wen16] Wen, Wei, et al. "Learning structured sparsity in deep neural networks." Advances in Neural Information Processing Systems. 2016. Layer 𝒍𝒍 − 𝟏𝟏 Layer 𝒍𝒍 Grouping ! (2,1)-norm, which is the 1-norm over 2-norm groups, promotes feature sharing and results in complete elimination of the features that are not shared.
  • 17. Incremental Training of a DEN minimize 𝑾𝑾𝑡𝑡 𝓛𝓛 𝑾𝑾𝑡𝑡 ; 𝒟𝒟𝑡𝑡 + λ 𝑾𝑾𝑡𝑡 – 𝑾𝑾𝑡𝑡−1 2 2 After 2., if the similarity with neurons of previous step is larger than the threshold σ, we split & duplicate those neurons and restore them to previous step. t-1 t + + Copy 3. Network Split / Duplication • Measure the amount of semantic drift 𝜌𝜌𝑖𝑖 𝑡𝑡 for each hidden unit 𝑖𝑖, if 𝜌𝜌𝑖𝑖 𝑡𝑡 > 𝜎𝜎, copy it. • After the duplication, retrain the network since split changes the overall structure.
  • 18. Incremental Training of a DEN We timestamp each newly added units to record the stage 𝑡𝑡 when it is added to the network, to further prevent drift by the introduction of new hidden units. t-2 tt-1
  • 19. Datasets and Networks We validate our method on four public datasets for classification, with various networks. CIFAR-100 • 100 animal and vehicle classes • Used modified version of AlexNet MNIST-variation • Modified MNIST dataset including perturbation • Used LeNet-4 (2 of conv., 2 of fc.) Permuted-MNIST • Different random permutation of the input pixels • Used LeNet-4 AwA • 50 animal classes • Used feedforward network
  • 20. Baselines We compare our networks against relevant baselines. D D D M1 M2 M3 D D D M1 M3 M3 STL MTL minimize 𝑾𝑾𝑡𝑡 𝓛𝓛 𝑾𝑾𝑡𝑡; 𝒟𝒟𝑡𝑡 + + λ 𝑾𝑾𝑡𝑡– 𝑾𝑾𝑡𝑡−1 2 2 L2 EWC Progressive Networks Rusu, Andrei A., et al. "Progressive neural networks." arXiv preprint arXiv:1606.04671 (2016). Kirkpatrick, James, et al. "Overcoming catastrophic forgetting in neural networks." Proceedings of the National Academy of Sciences 114.13 (2017): 3521-3526. DEN
  • 21. Results Incremental training with DEN results in obtaining a much smaller network that performs almost the same as the networks that are trained in batch. Further fine-tuning of DEN on all tasks obtains the best performance, which shows that DEN is also useful for network capacity estimation.
  • 22. Results DEN maintains the performance obtained on the previous tasks and allows for higher performance improvements for later tasks. Also, timestamped inference is highly effective in preventing semantic drift.
  • 23. Results Selective retraining takes significantly less time than the full retraining of the network, even shows much higher AUROC. DNN-selective mostly selects less portion of upper level units which are more task- specific, while selecting larger portion of more generic lower layer units.
  • 24. Results There are the models with a variant of our model that does selective retraining and layer expansion, but without network split on MNIST-Variation dataset. DEN-Dynamic even outperforms DEN-Constant with similar capacity, since the model can dynamically adjust the number of neurons at each layers.
  • 25. Results In the permuted MNIST, our DEN outperforms all lifelong learning baselines while using only 1.39 times of base network capacity. Further, DEN-Finetune achieves the best AUROC among all models, including DNN-STL and DNN-MTL.
  • 26. Conclusion • We proposed a novel deep neural network for lifelong learning, Dynamically Expandable Network (DEN). • DEN performs partial retraining of the network trained on old while increasing its capacity when necessary. • DEN significantly outperforms the existing lifelong learning methods, achieving almost the same performance as the network trained in batch. • Further fine-tuning of the models on all tasks results in obtaining models that outperform the batch models, which shows that DEN is useful for network structure estimation as well.