Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Introduction to Spiking Neural Networks: From a Computational Neuroscience perspective
1. Jason Tsai (蔡志順)
Oct. 19, 2019 @Mozilla Community Space Taipei
*Picture adopted from
https://bit.ly/2ts8xCk
Introduction to Spiking Neural Networks
2. *Copyright Notice:
All figures in this presentation are taken from
the quoted sources as mentioned in the
respective slides and their copyright belongs
to the owners. This presentation itself adopts
Creative Commons license.
3. Neural Networks 3D Simulation
(Video demo)
*Video from https://youtu.be/3JQ3hYko51Y
4. Questions
What are the advantages of spiking
neural networks and neuromorphic
computing?
What are current challenges of spiking
neural networks (SNNs)?
10. Neuron’s Spike: Action Potential
*Figure adopted from https://en.wikipedia.org/wiki/Action_potential &
The front cover of “Spikes: Exploring the Neural Code (1999)”
12. The Effect of Presynaptic Spikes on
Postsynaptic Neuron
*Figure adopted from Wulfram Gerstner & Werner M. Kistler. Spiking Neuron Models:
Single Neurons, Populations, Plasticity. Cambridge University Press. 2002. Page 5.
13. Hebb’s Learning Postulate
"When an axon of cell A is near enough to excite a cell B and
repeatedly or persistently takes part in firing it, some growth
process or metabolic change takes place in one or both cells such
that A's efficiency, as one of the cells firing B, is increased.“*
* Refer to Donald O. Hebb, The Organization of Behavior: A Neuropsychological Theory. 1949 & 2002. Page 62.
Causality
Repetition
14. Long-Term Potentiation (LTP) / Long-
Term Depression (LTD)
LTP is a long-lasting, activity-dependent increase in synaptic
strength that is a leading candidate as a cellular mechanism
contributing to memory formation in mammals in a very
broadly applicable sense.*
* Refer to J. David Sweatt. Mechanisms of Memory, Second Edition. Academic Press. 2010. Page 112.
15. Synaptic Plasticity
*Figure adopted from Wulfram Gerstner & Werner M. Kistler. Spiking Neuron Models:
Single Neurons, Populations, Plasticity. Cambridge University Press. 2002. Page 353.
16. Back-propagating Action Potential (bAP)
*Further reading: https://en.wikipedia.org/wiki/Neural_backpropagation
Induction of tLTP requires activation of the presynaptic
input milliseconds before the bAP in the postsynaptic
dendrite.
17. *Figure adopted from https://doi.org/10.3389/fnsyn.2011.00004
Spike-Timing-Dependent Plasticity
(STDP)
18. Experiment Evidence of STDP
From Wikipedia:
“Henry Markram, when he was in Bert Sakmann's lab and published their
work in 1997, used dual patch clamping techniques to repetitively
activate pre-synaptic neurons 10 milliseconds before activating the post-
synaptic target neurons, and found the strength of the synapse
increased. When the activation order was reversed so that the pre-
synaptic neuron was activated 10 milliseconds after its post-synaptic
target neuron, the strength of the pre-to-post synaptic connection
decreased.
Further work, by Guoqiang Bi, Li Zhang, and Huizhong Tao in Mu-Ming
Poo's lab in 1998, continued the mapping of the entire time course
relating pre- and post-synaptic activity and synaptic change, to show that
in their preparation synapses that are activated within 5-20 ms before a
postsynaptic spike are strengthened, and those that are activated within a
similar time window after the spike are weakened.”
*Further reading: https://en.wikipedia.org/wiki/Spike-timing-dependent_plasticity
20. Lateral Inhibition
Lateral inhibition is a Central Nervous System process whereby
application of a stimulus to the center of the receptive field excites a
neuron, but a stimulus applied near the edge inhibits it.
*Figure adopted from https://bit.ly/2yaat37
23. Dopamine: Essential for Reward
Processing in Mammalian Brain
*Figure adopted from http://www.jneurosci.org/content/29/2/444
Dopamine neurons form huge synaptic contacts to target!
25. Two Hot Approaches
Supervised: Stochastic Gradient Descent
based Backpropagation learning rule
(Treat the membrane potentials of spiking neurons as
differentiable signals, where discontinuities at spike
times are considered as noise.*)
Unsupervised: STDP (Spike-Timing-
Dependent Plasticity) based learning rule
*Refer to Jun Haeng Lee, et al., Training Deep Spiking Neural Networks Using Backpropagation.
Frontiers in Neuroscience, 08 November 2016. https://doi.org/10.3389/fnins.2016.00508
26. *Refer to Yu, Q., Tang, H., Hu, J., Tan, K.C., Neuromorphic Cognitive Systems: A Learning and Memory
Centered Approach. Springer International Publishing. 2017. Page 9.
STDP Learning Rule
27. STDP Learning Rule (1-to-1)
*Figure adopted from http://dx.doi.org/10.7551/978-0-262-33027-5-ch037
28. STDP Learning Rule (2-to-1)
N0 is stimulated until N1 fires, then e0 is stopped for 30 ms.
N2 is stimulated by e2 during those 30 ms.
*Figure adopted from http://dx.doi.org/10.7551/978-0-262-33027-5-ch037
29. STDP Finds Spike Patterns
*Figure adopted from https://doi.org/10.1371/journal.pone.0001377
34. 1st Generation of Neuron Models
(McCulloch–Pitts Neuron Model)
*Figure adopted from http://wwwold.ece.utep.edu/research/webfuzzy/docs/kk-thesis/kk-thesis-html/node12.html
35. 2nd Generation of Neuron Models
*Figure adopted from http://cs231n.github.io/neural-networks-1/
36. 3rd Generation of Neuron Models
(Spiking Neuron Models)
*Figure adopted from http://kzyjc.cnjournals.com/html/2018/5/20180512.htm
37. Spiking Neuron Models
Miscellaneous models (integrators / resonators):
Hodgkin-Huxley model
Izhikevish model
Leaky Integrate-and-Fire (LIF) model
Resonate-and-Fire model
Spike Response model (SRM)
……
*Further reading: https://en.wikipedia.org/wiki/Biological_neuron_model
& http://www.scholarpedia.org/article/Spike-response_model
38. Hodgkin-Huxley Model
*Figure adopted from Wulfram Gerstner & Werner M. Kistler. Spiking Neuron Models: Single Neurons,
Populations, Plasticity. Cambridge University Press. 2002. Page 34.
42. Leaky Integrate-and-Fire Model
*Figure adopted from Wulfram Gerstner, Werner M. Kistler, Richard Naud and Liam Paninski “Neuronal Dynamics:
From Single Neurons to Networks and Models of Cognition” Cambridge University Press. 2014. Page 11.
43. The Firing of a Leaky Integrate-and-
Fire Model Neuron
*Figure adopted from https://doi.org/10.1371/journal.pone.0001377
50. Sparse Coding
*Figure adopted from http://brainworkshow.sparsey.com/measuring-similarity-in-localist-vs-distributed-representations/
51. Sparse Coding with Inhibitory Neurons
Population sparseness: Few neurons are
active at any given time
Lifetime sparseness: Individual neurons
are responsive to few specific stimuli
*Figure adopted from https://doi.org/10.1523/JNEUROSCI.4188-12.2013
59. ANN-to-SNN Conversion
Train ANNs using standard supervised training
techniques like backpropagation to leverage
the superior performance of trained ANNs and
subsequently convert to event-driven SNNs for
inference operation on neuromorphic platform.
Rate-encoded spikes are approximately
proportional to the magnitude of the original
ANN inputs.
62. Further Reading
Wulfram Gerstner & Werner M. Kistler, “Spiking Neuron Models:
Single Neurons, Populations, Plasticity”. Cambridge University
Press (2002)
Wulfram Gerstner, Werner M. Kistler, Richard Naud and Liam
Paninski, “Neuronal Dynamics: From Single Neurons to Networks
and Models of Cognition”. Cambridge University Press (2014)
Eugene M. Izhikevich, “The Dynamical Systems in Neuroscience:
Geometry of Excitability and Bursting”. The MIT Press (2007)
Nikola K. Kasabov, “Time-Space, Spiking Neural Networks and
Brain-Inspired Artificial Intelligence”. Springer International
Publishing (2018)
蔺想红、王向文, “脉冲神经网络原理及应用”. 科学出版社 (2018)