1. Introduction
Sources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF Neuron
Conclusion
Capacity Analysis of Neurons with Descending
Action Potential Thresholds
Prapun Suksompong
Electrical and Computer Engineering
Cornell University, Ithaca, NY 14853
ps92@cornell.edu
Final Examination for the Doctoral Degree (“B” Exam)
July 24, 2008
Prapun Suksompong Capacity Analysis of Neurons
2. Introduction
Sources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF Neuron
Conclusion
Outline
Introduction
Sources of Variability for the ISIs and Derivation of the Threshold
Information-Theoretic Analysis of IF Neuron
Conclusion
Prapun Suksompong Capacity Analysis of Neurons
3. Introduction
Neuron Morphology
Sources of Variability and Threshold Derivation
Integrate-and-Fire Neurons
Information-Theoretic Analysis of IF Neuron
Goal
Conclusion
Introduction
Neuron Morphology
Integrate-and-Fire Neurons
Goal
Sources of Variability for the ISIs and Derivation of the Threshold
Information-Theoretic Analysis of IF Neuron
Conclusion
Prapun Suksompong Capacity Analysis of Neurons
4. Introduction
Neuron Morphology
Sources of Variability and Threshold Derivation
Integrate-and-Fire Neurons
Information-Theoretic Analysis of IF Neuron
Goal
Conclusion
Neuron Morphology
A neuron is the basic working unit of the nervous system.
Dendrite
A typical neuron has three Nucleus
Axon Hillock
functionally distinct parts, called Axon
dendrites, Axon
Axon from Terminals
soma, and another neuron Cell Body
(Soma)
Synapse
axon. Myelin
Sheath
Node of
Ranvier
The junction between two Synaptic Vesicle
Presynaptic
neurons is called a synapse. Axom
Terminal
Synaptic Cleft
Postsynaptic
Dendrite
Ion Channel
Prapun Suksompong Capacity Analysis of Neurons
5. Introduction
Neuron Morphology
Sources of Variability and Threshold Derivation
Integrate-and-Fire Neurons
Information-Theoretic Analysis of IF Neuron
Goal
Conclusion
Action Potentials (Spikes)
Looking at a synapse, we refer to the sending neuron as the
presynaptic neuron and to the receiving neuron as the
postsynaptic neuron.
presynaptic postsynaptic
synapse
axon
The neuronal signals consist of short electrical pulses called action
potentials (APs) or spikes. A chain of APs emitted by a single
neuron is called a spike train.
Prapun Suksompong Capacity Analysis of Neurons
6. Introduction
Neuron Morphology
Sources of Variability and Threshold Derivation
Integrate-and-Fire Neurons
Information-Theoretic Analysis of IF Neuron
Goal
Conclusion
Quantal Synaptic Failure (QSF)
presynaptic postsynaptic
synapse
axon
Synaptic failure: It is possible that an AP fails to get
“across” the synapse.
We may model a synapse as a Z -channel.
Spikes which successfully cross the synapse then propagate
down to soma.
Prapun Suksompong Capacity Analysis of Neurons
7. Introduction
Neuron Morphology
Sources of Variability and Threshold Derivation
Integrate-and-Fire Neurons
Information-Theoretic Analysis of IF Neuron
Goal
Conclusion
Integrate-and-Fire Neurons
Assumption: ∼ 104
pre-synaptic neurons.
True in cortex (higher brain
functions).
Prapun Suksompong Capacity Analysis of Neurons
8. Introduction which are drawn here are in fact decreasing. We will return to t
Neuron Morphology
Sources of Variability and Threshold Derivation
Integrate-and-Fire Neurons
Information-Theoretic Analysis of IF Neuron
Goal
Conclusion
Integrate-and-Fire Neurons
Descending Threshold
Ascending
Membrane
Potential
time
Spike Train
Spikes generated when the
membrane potentials hit the
time
thresholds.
• First jitter: Spike generation
Descending thresholds. • Poisson approximation
Prapun Suksompong Capacity Analysis of Neurons
9. Introduction
Neuron Morphology
Sources of Variability and Threshold Derivation
Integrate-and-Fire Neurons
Information-Theoretic Analysis of IF Neuron
Goal
Conclusion
(Leaky) Integrate-and-Fire Model: LIF or IF
Let τ1 , τ2 , τ3 , . . . be the sequence of time that the spikes arrive at
the spike generating region. The membrane potential at time t is
then
X (t) = h (t, τm , Ym ) = Ym h(t − τm ).
m m
This is the “integrate” part of the integrate-and-fire neuron.
X (t )
Ym is the weight for the mth
spike due to propagation
loss, synaptic strength,
synaptic failure, etc. Yi + 2 h ( t − τ i + 2 )
Yi h ( t − τ i ) Yi +1h ( t − τ i +1 )
h is the shape function.
τi τ i +1 τ i+2 time
Prapun Suksompong Capacity Analysis of Neurons
10. Introduction
Neuron Morphology
Sources of Variability and Threshold Derivation
Integrate-and-Fire Neurons
Information-Theoretic Analysis of IF Neuron
Goal
Conclusion
Integrate-and-Fire Neurons (Con’t)
Constant bombardment of spikes
leads to increase in membrane
potential.
As soon as the membrane potential
reaches a critical value or
threshold, the neuron “fires” an time (t)
action potential. Then, everything
resets.
Refractory period: The time after
a AP is produced, during which it
is impossible to generate another
Threshold
AP. time (t)
Set T (t) to be ∞ during this
period.
Prapun Suksompong Capacity Analysis of Neurons
11. Introduction
Neuron Morphology
Sources of Variability and Threshold Derivation
Integrate-and-Fire Neurons
Information-Theoretic Analysis of IF Neuron
Goal
Conclusion
Integrate-and-Fire Neurons (Summary)
∼ 104 pre-synaptic neurons. Descending Threshold
Ascending
Membrane
Potential
time
Spike Train
time
• First jitter: Spike generation
Descending thresholds.
• Poisson approximation
Prapun Suksompong Capacity Analysis of Neurons
12. Introduction
Neuron Morphology
Sources of Variability and Threshold Derivation
Integrate-and-Fire Neurons
Information-Theoretic Analysis of IF Neuron
Goal
Conclusion
Theoretical Approaches to Neuroscience
We use IF model, but more biologically-realistic models exist
(e.g. Hodgkin and Huxley [’52] model).
1. Too many parameters.
Physical measurements “fundamentally disturb cell properties”
2. Provide less insight.
Biological structures have evolved via natural selection to
operate optimally.
See the book Optima for Animals by R. McNeill Alexander.
What is the best strength for a bone?
At what speed should humans change from walking to
running?
Prapun Suksompong Capacity Analysis of Neurons
13. Introduction
Neuron Morphology
Sources of Variability and Threshold Derivation
Integrate-and-Fire Neurons
Information-Theoretic Analysis of IF Neuron
Goal
Conclusion
Information-Theoretic Optimization
Application of information theory has already found success in
many areas of neuroscience.
Barlow’s “economy of impulses”[’59, ’69]
Minimize redundancy.
Linsker’s InfoMax principle [’88, ’89]
Maximize the mutual information.
Levy and Baxter’s energy-efficient coding [’96, ’02]
Maximize mutual information per unit energy expended.
Prapun Suksompong Capacity Analysis of Neurons
14. Introduction
Neuron Morphology
Sources of Variability and Threshold Derivation
Integrate-and-Fire Neurons
Information-Theoretic Analysis of IF Neuron
Goal
Conclusion
Motivation and Goal
Integrate-and-Fire (IF) model is very popular.
The threshold function is a crucial element of the IF model.
Little amount of work exists on deriving the form of the
threshold curve.
In fact, using constant thresholding is also popular.
This leads to large jitter in the spike timing and hence
discourages the use of time coding.
Goal: Find (1) an expression for threshold curve under biologically
realistic constraints and (2) the optimal operating point of neuron
under such threshold.
Prapun Suksompong Capacity Analysis of Neurons
15. Introduction
First Jitter: Spike generation
Sources of Variability and Threshold Derivation
Second and Third Jitters
Information-Theoretic Analysis of IF Neuron
The Threshold Curve
Conclusion
Introduction
Sources of Variability for the ISIs and Derivation of the Threshold
First Jitter: Spike generation
Second and Third Jitters
The Threshold Curve
Information-Theoretic Analysis of IF Neuron
Conclusion
Prapun Suksompong Capacity Analysis of Neurons
16. Introduction
First Jitter: Spike generation
Sources of Variability and Threshold Derivation
Second and Third Jitters
Information-Theoretic Analysis of IF Neuron
The Threshold Curve
Conclusion
Three sources of variability for inter-spike intervals
1. Spike generation
2. Spike propagation
3. Time-of-arrival estimation
Prapun Suksompong Capacity Analysis of Neurons
17. Introduction
First Jitter: Spike generation
Sources of Variability and Threshold Derivation
Second and Third Jitters
Information-Theoretic Analysis of IF Neuron
The Threshold Curve
Conclusion
First jitter: Spike generation
formula that governs how the membrane potential of this middle neuron rises given the
combined incoming rate λ .
λ1
λ2
⊕
λ3
λ = λ1 + λ2 + λ3 +
Now, of course, there is some jitter in the timing of the
paper today.
Large number of presynaptic neurons source of jitter comes from the fact that the in
The first allows Poisson
approximation for the superposed in fact some jitter in large. That assumption allows
has process.
presynaptic neurons are
them. Now, you may recall th
The membrane potential is governed by a filtered Poisson to a Poisson p
.. the superposed spike trains … is close
process. spike train coming out of a single neuron is not a Poiss
a tractable formula that governs how the membrane po
Prapun Suksompong given the combined incoming rate λ .
Capacity Analysis of Neurons
18. Introduction
First Jitter: Spike generation
Sources of Variability and Threshold Derivation
Second and Third Jitters
Information-Theoretic Analysis of IF Neuron
The Threshold Curve
Conclusion
Approximation for the first timing jitter
For fixed λ, different realizations of the membrane potential
correspond to different spiking times.
Membrane potential X ( t ) Filtered Poisson
σX approximation for
amount of variation
in vertical direction
[Parzen’62].
Linear approximation
Threshold T ( t ) for amount of
σ time variation in horizontal
Time
direction.
Prapun Suksompong Capacity Analysis of Neurons
19. Introduction
First Jitter: Spike generation
Sources of Variability and Threshold Derivation
Second and Third Jitters
Information-Theoretic Analysis of IF Neuron
The Threshold Curve
Conclusion
Membrane potential X ( t )
σX
Approximation for the first timing jitter (con’t)
Threshold T ( t )
σ time
Time
H (τ ) T (τ ) c2 H2 (τ )
σtime (τ ) ≈ The figure here shows different realizations of the membrane potentials for a fixed
combined incoming rate λ . Here, we see that the randomness from the Poisson arrivals
T (τ ) h (τ ) − T (τ ) H (τ ) c1 H (τ )
causes fluctuation in the time that the membrane potentials hit the threshold. Under som
linear approximation, we can relate the jitter in the vertical direction to the one in
horizontal direction. This then gives us the formula for the magnitude of the timing jitte
as a function of the spike time τ .
h : shape function.
Membrane potential X ( t ) Here, h is the shape function which describes how the membrane potential changes in
e.g. exponential
response to a single input spike. For the usual leaky integrate-and-fire model, this h star
with some amplitude and then decay exponentially.
σX
h (t )
For conciseness, we define these two integrations which get used in the formula here.
t
H (t) = h (µ)dµ.
0
Threshold T ( t ) t
σ time H2 (t) = h2 (µ)dµ.
Time 0
Prapun Suksompong Capacity Analysis of Neurons
20. Introduction
First Jitter: Spike generation
Sources of Variability and Threshold Derivation
Second and Third Jitters
Information-Theoretic Analysis of IF Neuron
The Threshold Curve
Conclusion
Approximation for the first timing jitter (con’t)
H (τ ) T (τ ) c2 H2 (τ )
σtime (τ ) ≈
T (τ ) h (τ ) − T (τ ) H (τ ) c1 H (τ )
c1 and c2 are constants
Membrane potential X ( t ) which depend on the
σX distribution of the weight
(Ym ) for each spike.
Recall:
X (t) = m Ym h(t − τm ).
Threshold T ( t )
σ time
Time
Prapun Suksompong Capacity Analysis of Neurons
21. Introduction
First Jitter: Spike generation
Sources of Variability and Threshold Derivation
Second and Third Jitters
Information-Theoretic Analysis of IF Neuron
The Threshold Curve
Conclusion
Jitter in rate estimation
The only information contained in a Poisson process is its rate
λ.
Different λ’s ⇒ different spiking times τ ’s.
Descending Threshold
Spike times vary inversely with λ.
PSP for Large λ
T (τ )
λ(τ ) ≈
c1 H (τ )
PSP for Small λ Error in rate estimation:
1 T (τ ) c2 H2 (τ )
σλ (τ ) ≈ .
Time c1 H (τ ) c1 H (τ )
However, this spike time has some jitter, so the $lambda$ estimation also have some
error. We then go on and approximate this error:
Prapun Suksompong Capacity Analysis of Neurons
22. The second jitter is the randomness in the length of time a spike takes to propa
Introduction
First Jitter: Spike generation
Sources of Variability and Threshold Derivationanother neuron. It is on the order of 10 microseconds. The third jitt
synapse on
Second and Third Jitters
time-of-arrival estimation error; that is, if this neuron tries to measure the inter
Information-Theoretic Analysis of IF Neuron
The Threshold Curve
Conclusion
interval, it needs to find out what time a spike arrives. We borrow some formu
Radar guys shown here because this is exactly the problem that they call the ra
Second and Third Jitters over N0 here. Iterror depends on the signal-to-noise the shape ofis show
ranging problem. The
this Es also depends on the bandwidth for
ratio which
the acti
potential. The amount of error here is about 10 microseconds as well.
Propagation Time.
Time-of-Arrival Estimation
Error (radar ranging
problem):
−1
2Es 2
4π 2 f .
N0
f 2 : Gabor-bandwidth.
ES : Signal energy.
N0
2 : Spectral height of the
Noise.
Small: < 10µs.
Prapun Suksompong Capacity Analysis of Neurons
23. Introduction
First Jitter: Spike generation
Sources of Variability and Threshold Derivation
Second and Third Jitters
Information-Theoretic Analysis of IF Neuron
The Threshold Curve
Conclusion
Deriving The Threshold Curves
Recall: Deviation in time:
H (τ ) T (τ ) c2 H2 (τ )
σtime (τ ) = .
T (τ ) h (τ ) − T (τ ) H (τ ) c1 H (τ )
We consider the thresholds which
1) preserve timing jitter σtime (τ ) ≡ σtime,0 , or
σtime (τ )
2) preserve relative timing jitter τ ≡ σ%time,0
across spiking times (or spiking frequencies) of interest.
Prapun Suksompong Capacity Analysis of Neurons
24. Introduction
First Jitter: Spike generation
Sources of Variability and Threshold Derivation
Second and Third Jitters
Information-Theoretic Analysis of IF Neuron
The Threshold Curve
Conclusion
Deriving The Threshold Curves
Recall:
H(τ ) T (τ )c2 H2 (τ )
(a) Deviation in time: σtime (τ ) = T (τ )h(τ )−T (τ )H(τ ) c1 H(τ ) .
1 T (τ )c2 H2 (τ )
(b) Deviation in λ estimation: σλ (τ ) = c1 H(τ ) c1 H(τ ) .
We consider the thresholds which
1) preserve timing jitter σtime (τ ) ≡ σtime,0 , or
σtime (τ )
2) preserve relative timing jitter τ ≡ σ%time,0 , or
3) preserve jitter in λ estimation σλ (τ ) ≡ σλ,0 , or
σλ (τ )
4) preserve relative jitter in λ estimation λ ≡ σ%λ,0 , or
5) preserve jitter in ln λ estimation
across spiking times (or spiking frequencies) of interest.
Prapun Suksompong Capacity Analysis of Neurons
25. Introduction
First Jitter: Spike generation
Sources of Variability and Threshold Derivation
Second and Third Jitters
Information-Theoretic Analysis of IF Neuron
The Threshold Curve
Conclusion
Deriving The Threshold Curves (con’t)
Constant timing jitter level:
h (t) 1 c2 H2 (t)
T (t) = T (t) − T (t) .
H (t) σtime,0 c1 H (t)
Constant relative-timing-jitter level:
h (t) 1 c2 H2 (t)
T (t) = T (t) − T (t) .
H (t) tσ%time,0 c1 H (t)
Preserve relative error in λ estimation:
H2 (t)
T (t) = c0 .
H (t)
Prapun Suksompong Capacity Analysis of Neurons
26. Introduction
First Jitter: Spike generation
Sources of Variability and Threshold Derivation
Second and Third Jitters
Information-Theoretic Analysis of IF Neuron
The Threshold Curve
Conclusion
The differential equations are Bernoulli equations of the form
T (t) = T (t) P (t) − T (t)Q (t) .
They can be reduced to linear equation by introducing
v (t) = T (t) which gives
1 1
v (t) = P (t) v (t) − Q (t) .
2 2
Linear! The solution is
t
1
v (t) = v (t0 ) φ (t, t0 ) − φ (t, τ ) Q (τ )dτ,
2
t0
t
1
2
P(τ )dτ
where φ (t, s) = e s .
Prapun Suksompong Capacity Analysis of Neurons
27. Introduction
First Jitter: Spike generation
Sources of Variability and Threshold Derivation
Second and Third Jitters
Information-Theoretic Analysis of IF Neuron
The Threshold Curve
Conclusion
Comparison between derived thresholds
Constant
timing jitter
Constant relative
timing jitter
Exponential
Heavy-tail
Linear
6 7 8
Time [ms]
Prapun Suksompong Capacity Analysis of Neurons
28. Introduction
First Jitter: Spike generation
Sources of Variability and Threshold Derivation
Second and Third Jitters
Information-Theoretic Analysis of IF Neuron
The Threshold Curve
Conclusion
Summary
Analyze and quantify three sources of timing jitter
Predict shape of threshold curves
Constant
timing jitter
Constant relative
timing jitter
Exponential
Heavy-tail
Linear
6 7 8
Time [ms]
Prapun Suksompong Capacity Analysis of Neurons
29. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Introduction
Sources of Variability for the ISIs and Derivation of the Threshold
Information-Theoretic Analysis of IF Neuron
OPT1: Maximization of Mutual Information
OPT2: Mutual Information per Unit Energy Cost
Rate Matching
Conclusion
Prapun Suksompong Capacity Analysis of Neurons
30. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Conditional density Q(t|λ) = fτ |Λ (t|λ)
We have formula(s) for the threshold curve T (t).
Assumption: λ stays constant during each ISI.
Given Poisson input intensity λ, can find the conditional
density Q(t|λ) = fτ |Λ (t|λ).
τ = g (Λ)+jitter.
Λ λ τ
τ |Λ |λ
Prapun Suksompong Capacity Analysis of Neurons
31. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Optimization 1: Mutual Information
OPT1:
sup I (Λ; τ )
where
fΛ,τ (Λ, τ )
I (Λ; τ ) = E log
fΛ (Λ)fτ (τ )
and the supremum is taken over all possible fΛ (λ).
Blahut-Arimoto Algorithm (BAA)
Prapun Suksompong Capacity Analysis of Neurons
32. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Exponential “Constant-Jitter” “Constant-Relative-Jitter”
Threshold Threshold Threshold
-3 -3 -3
x 10 x 10 x 10
6 6 6
4 4 4
f ()
f ()
f ()
2 2 2
0 0 0
0 500 1000 1500 2000 0 500 1000 1500 2000 0 500 1000 1500 2000
0.8 0.8 0.8
0.6 0.6 0.6
f(t)
f(t)
f(t)
0.4 0.4 0.4
0.2 0.2 0.2
0 0 0
4 6 8 10 12 14 4 5 6 7 8 4 5 6 7 8 9
t t t
C = 5.457 C = 4.931 C = 5.109
(a) (b) (c)
Prapun Suksompong Capacity Analysis of Neurons
33. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Capacity-achieving input densities look similar.
−3
x 10
6
exponential
constant jitter
5 constant relative jitter
4
fΛ(λ)
3
2
1
0
0 200 400 600 800 1000 1200 1400 1600 1800 2000
λ
Prapun Suksompong Capacity Analysis of Neurons
34. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Input-Intensity Density Approximation
BAA does not provide any
insight.
Our simpler formula:
σ0 (c1 H (t))3
fΛ (λ) ≈
d T (t) c2 H2 (t)
t=g (λ)
where g (λ) = E [τ |λ].
Prapun Suksompong Capacity Analysis of Neurons
35. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Input-Intensity Density Approximation
−3
BAA does not provide any 6
x 10
exponential
insight. 5
constant jitter
constant relative jitter
approximation
Our simpler formula: 4
fΛ(λ)
(c1 H (t))3
3
σ0
fΛ (λ) ≈
d T (t) c2 H2 (t) 2
t=g (λ) 1
where g (λ) = E [τ |λ]. 0
0 200 400 600 800 1000 1200 1400 1600 1800 2000
λ
Prapun Suksompong Capacity Analysis of Neurons
36. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Input-Intensity Density Approximation
BAA does not provide any exponential
constant jitter
insight. constant relative jitter
approximation
Our simpler formula:
fΛ(λ)
−3
10
σ0 (c1 H (t))3
fΛ (λ) ≈
d T (t) c2 H2 (t)
t=g (λ)
where g (λ) = E [τ |λ]. 1
10 10
2
λ
10
3
Prapun Suksompong Capacity Analysis of Neurons
37. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Approximation Strategy
Assume that g (Λ) is uniform and then find the corresponding fΛ .
Exponential “Constant-Jitter” “Constant-Relative-Jitter”
Threshold Threshold Threshold
-3 -3 -3
x 10 x 10 x 10
6 6 6
4 4 4
f ()
f ()
f ()
2 2 2
0 0 0
0 500 1000 1500 2000 0 500 1000 1500 2000 0 500 1000 1500 2000
0.8 0.8 0.8
0.6 0.6 0.6
f(t)
f(t)
f(t)
0.4 0.4 0.4
0.2 0.2 0.2
0 0 0
4 6 8 10 12 14 4 5 6 7 8 4 5 6 7 8 9
t t t
C = 5.457 C = 4.931 C = 5.109
(a) (b) (c)
Exponential “Constant-Jitter” “Constant-Relative-Jitter”
Threshold Threshold Threshold
Prapun Suksompong Capacity Analysis of Neurons
38. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Approximation Strategy
Assume that g (Λ) is uniform and then find the corresponding fΛ .
������������
������ ������
������������ ������
*
Convolution
For invertible function g , the pdf of Z = g (Λ) is given by
d −1 1
fZ (z) = g (z) fΛ g −1 (z) = fΛ (λ) ,
dz |g (λ)|
where z = g (λ).
Prapun Suksompong Capacity Analysis of Neurons
39. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
OPT2: Energy-Efficient Neuron
Brains consume 20% of energy consumption for adults and
60% for infant [Laughlin and Sejnowski’03].
Suppose neuron spends
1 unit of energy per ms when it is idle, and
e unit of energy per ms when AP is produced.
e >> 1.
If the time to the next spike is τ = t, the energy expended is
bo (t) = 1 × (t − ∆) + e × ∆ = t + (e − 1)∆ = t + r .
where ∆ is the time used to produce a spike.
The value of r depends on the type of neurons under
consideration.
Prapun Suksompong Capacity Analysis of Neurons
40. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Optimization 2: I/E
OPT2:
I (Λ; τ )
sup
E [bo (τ )]
where the supremum is taken over all possible fΛ (λ).
I (Λ;τ )
Jimbo-Kunisawa algorithm (JKA) maximizes E[b(Λ)] .
b is a function of input.
Our bo is a function of output.
We define b(λ) = E [bo (τ )|Λ = λ] and apply JKA.
Because bo (τ ) = τ + r , we have
b(λ) = E [τ |λ] + r = g (λ) + r .
Prapun Suksompong Capacity Analysis of Neurons
41. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Input-Intensity Density Approximation
Can use the same technique as in OPT1 to do approximation
of input-intensity density.
In stead of uniform density, consider bounded exponential
density of the form
γ
f (t; γ, α, β) = e −γt 1[α,β] (t) .
e −γα − e −γβ
Prapun Suksompong Capacity Analysis of Neurons
42. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Input-Intensity Density Approximation
Can use the same technique as in OPT1 to do approximation
of input-intensity density.
Result:
(c1 H (t))3
fΛ (λ) ≈ σ0 f (t; γ, g (b), g (a)) ,
T (t) c2 H2 (t)
t=g (λ)
where f (t; γ, g (b), g (a)) is the bounded exponential pdf with
support on the interval [g (b), g (a)] and parameter γ.
Prapun Suksompong Capacity Analysis of Neurons
43. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
−3
x 10
6
exponential exponential
constant jitter constant jitter
5 constant relative jitter 0.5 constant relative jitter
approximation
4 0.4
fΛ(λ)
fτ(t)
3 0.3
2 0.2
1 0.1
0 0
0 500 1000 1500 2000 4 6 8 10 12 14
λ t [ms]
exponential
constant jitter
constant relative jitter
approximation
fΛ(λ)
−3
fτ(t)
10
−1
10
exponential
constant jitter
constant relative jitter
1 2 3 4 5 6 7 8 9 10
10 10 10
λ t [ms]
Prapun Suksompong Capacity Analysis of Neurons
44. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Free Parameters - Revisited
Recall, for example, the differential equation that define our
“constant-relative-jitter” threshold:
h (t) 1 c2 H2 (t)
T (t) = T (t) − T (t) .
H (t) tσ%time,0 c1 H (t)
There are a couple of parameters which we want to revisit.
The constants c1 and c2 .
Embedded in them is the effect of QSF.
σtime,0
σ%time,0 = t0 . What value should we set σtime,0 to be?
> 10µs.
Prapun Suksompong Capacity Analysis of Neurons
45. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
By scaling the unit of the voltage, we can make c1 = 1.
The scaling makes
1 1
c2 ∝ =
psuccess 1 − pfailure
where pfailure is the QSF probability.
pfailure depends on the type of neurons under consideration.
Let σtime,0 = σ1 and play with it.
Prapun Suksompong Capacity Analysis of Neurons
46. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Average Rates
From our optimization (either OPT1 or OPT2), we get the optimal
input-intensity density fΛ (λ) and spiking-time density fτ (t) for each
value of σ1 .
¯
Each value of σ1 gives average arrival rate λin and the average
¯ out .
spiking rate λ
¯
λout = 1 . E[τ ]
¯
λin = E [Λ]?
Prapun Suksompong Capacity Analysis of Neurons
47. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Average Afferent Rate
High value of Λ corresponds to low value of τ .
Neuron experiences large Λ value for short amount of time.
¯
λin should be lower than E [Λ].
Let (Λi , τi ) be the pair of input-intensity and the length of the
ISI associated with the ith spike.
The number of input spikes during the ith ISI is a Poisson
random variable Ni with mean Λi τi .
The long-term average input rate is then
k 1 k
i=1 Ni k i=1 Ni E [Λ1 τ1 ]
k
= 1 k
→ .
i=1 τi i=1 τi
E [τ1 ]
k
¯ E[Λτ ]
λin = E[τ ] .
Prapun Suksompong Capacity Analysis of Neurons
48. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Rate Matching
On average, our neuron in consideration is bombarded with an
¯
average input-intensity λin .
Suppose our neuron is receiving input from n other neurons.
Then, on average, each of the sending neurons fire at a rate
1¯
of n λin .
Our neuron should also generate spikes at this rate.
Therefore, we must have
1¯ ¯
λin = λout .
n
Prapun Suksompong Capacity Analysis of Neurons
49. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Rate Matching (Con’t)
Average spiking rate [spikes/s]
200
input
150 output
100
50
0
10 20 30 40 50 60 70 80 90 100
σ1 [microsec]
5.4
Capacity [bits]
5.2
5
4.8
10 20 30 40 50 60 70 80 90 100
σ1 [microsec]
Prapun Suksompong Capacity Analysis of Neurons
50. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Rate Matching (Con’t)
Capacity value increases as
we increase the noise level
Average spiking rate [spikes/s]
200
σ1 . 150
input
output
100
Larger value of σ1 implies 50
threshold decays slower This 0
10 15 20 25 30 35 40 45 50 55 60
σ1 [microsec]
gives larger support for the
spiking time. 5.8
Capacity [bits]
5.6
Noise is small. Effect of 5.4
increasing the support of the 5.2
10 15 20 25 30 35 40 45 50 55 60
output is stronger than the σ1 [microsec]
effect of increased noise.
Prapun Suksompong Capacity Analysis of Neurons
51. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
60
55
(c) max I/E, r = 100
50
Optimal rate [spikes per second]
45
40
(b) max I/E, r = 400
35
30
25 exponential
constant relative jitter (a) max I
20
15
1 1.5 2 2.5 3 3.5 4 4.5 5
c2
Prapun Suksompong Capacity Analysis of Neurons
52. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Rate Matching (Con’t)
60
55
(c) max I/E, r = 100
Firing rate decreases as the 50
Optimal rate [spikes per second]
failure probability increases. 45
Smaller value of r 40
(b) max I/E, r = 400
35
corresponds to higher rate.
30
As r → ∞, the rate 25 exponential
constant relative jitter (a) max I
converges to the max-I case. 20
15
1 1.5 2 2.5 3 3.5 4 4.5 5
c2
Prapun Suksompong Capacity Analysis of Neurons
53. Introduction
OPT1: Maximization of Mutual Information
Sources of Variability and Threshold Derivation
OPT2: Mutual Information per Unit Energy Cost
Information-Theoretic Analysis of IF Neuron
Rate Matching
Conclusion
Optimal Threshold: c2 = 5 and r = 400
800
700 exponential
constant relative timing jitter
600
500
400
300
200
100
0
0 20 40 60 80 100 120 140 160 180 200
Time [ms]
Prapun Suksompong Capacity Analysis of Neurons
54. Introduction
Sources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF Neuron
Conclusion
Introduction
Sources of Variability for the ISIs and Derivation of the Threshold
Information-Theoretic Analysis of IF Neuron
Conclusion
Prapun Suksompong Capacity Analysis of Neurons
55. Introduction
Sources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF Neuron
Conclusion
Contributions
1. Quantify the amount of timing jitter in neuron.
2. Construct threshold functions.
3. Provide optimal operating points for neurons which are close
to experimentally observed values.
Formulas to approximate the optimal input-intensity densities.
Prapun Suksompong Capacity Analysis of Neurons
56. Introduction
Sources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF Neuron
Conclusion
References I
P. Suksompong and T. Berger.
Jitter Analysis of Timing Codes for Neurons with Descending
Action Potential Thresholds.
ISIT, 2006.
P. Suksompong and T. Berger.
Capacity Analysis of Neurons with Descending Action
Potential Thresholds.
In preparation for special issue of IEEE Tran. on Info. Theory,
2009.
Prapun Suksompong Capacity Analysis of Neurons
57. Introduction
Sources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF Neuron
Conclusion
References II
P. Suksompong and T. Berger.
Energy-Efficient Neurons with Descending Action Potential
Thresholds.
In preparation for Journal of Comp. Neuroscience, 2009.
T. Berger and W.B. Levy.
Encoding of excitation via dynamic thresholding.
Society for Neuroscience, 2004.
W.B. Levy and R.A. Baxter.
Energy-Efficient Neuronal Computation via Quantal Synaptic
Failures.
Journal of Neuroscience, 2002.
Prapun Suksompong Capacity Analysis of Neurons
58. Introduction
Sources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF Neuron
Conclusion
References III
W.B. Levy and R.A. Baxter.
Energy Efficient Neural Codes
Neural Computation, 1996.
Patrick Crotty and William Levy.
Biophysical limits on axonal transmission rates in axons.
CNS, 2005.
E. Parzen.
Stochastic Processes.
Holden Day, 1962.
H. Vincent Poor.
An introduction to signal detection and estimation (2nd ed.).
Springer-Verlag New York, Inc., New York, NY, USA, 1994.
Prapun Suksompong Capacity Analysis of Neurons
59. Introduction
Sources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF Neuron
Conclusion
References IV
Dale Purve et al.
Neuroscience (3rd ed.).
Sinauer Associates Inc., Sunderland, MA USA, 1997.
Prapun Suksompong Capacity Analysis of Neurons
60. Introduction
Sources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF Neuron
Conclusion
THE
END
Prapun Suksompong Capacity Analysis of Neurons