Linking brain systems and mental functions requires accurate descriptions of behavioral tasks and fine demarcations of brain regions. Functional Magnetic Resonance Imaging (fMRI) has opened the possibility to investigate how brain activity is modulated by behavior. However, to date, no data collection has systematically addressed the functional mapping of cognitive mechanisms at a fine spatial scale. Most studies so far are bound to one single task, in which functional responses to a handful of contrasts are analyzed and reported as a group average brain map. The Individual Brain Charting (IBC) project stands for a high-resolution (1.5mm), multi-task fMRI dataset, intended to provide an objective basis for the establishment of a neurocognitive atlas based on the individual mapping of the human brain. This data collection refers to a permanent cohort during performance of a wide variety of tasks across many sessions. Data up to the third release---comprising 28 tasks---are publicly available in the OpenNeuro repository (ds002685). Derived statistical maps from the first and second releases can be found in NeuroVault (id6618) and they amount for 205 canonical contrasts described on the basis of 113 cognitive concepts taken from the Cognitive Atlas. These derivatives reveal all together a comprehensive brain coverage of regions engaged in cognitive processes as well as a successful encoding of the functional networks reported by the original studies. As the dataset becomes larger and the ensuing collection of concepts gets richer, finer subject-specific, cognitive topographies can be extracted from the data. We thus explore this individual-functional-atlasing approach in order to link functional segregation of specialized brain regions to elementary mental functions. Results show that individual topographies---common to all tasks---are consistently mapped within and, to a lesser extent, across participants. Besides, prediction scores associated with the reconstruction of contrasts of one task from the remaining ones reveal the quantitative contribution of each task to these common representations. Yet, scores decreased when subjects were permuted between train and test, confirming that topographies are driven by subject-specific variability. Lastly, we demonstrate how cognitive mapping can benefit from contrasts accumulation, by analyzing the functional fingerprints of a set of individualized regions-of-interest from the language network.
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learning
Individual functional atlasing of the human brain with multitask fMRI data: leveraging the IBC dataset
1. Online seminar
@ALuisaPinho
Individual functional atlasing of the human brain with
multitask fMRI data: leveraging the IBC dataset
Ana Luı́sa Pinho, Ph.D.
Parietal Team
Inria Saclay – Île-de-France
NeuroSpin, CEA-Saclay
France
25th of March, 2021
3. Background and motivations (1/2)
In cognitive neuroscience:
Brain systems
⇐⇒
Mental functions
Experiments typically shall:
I tackle one psychological domain
2/25
4. Background and motivations (1/2)
In cognitive neuroscience:
Brain systems
⇐⇒
Mental functions
Experiments typically shall:
I tackle one psychological domain
I be specific enough to accurately isolate brain processes
2/25
5. Background and motivations (1/2)
In cognitive neuroscience:
Brain systems
⇐⇒
Mental functions
Experiments typically shall:
I tackle one psychological domain
I be specific enough to accurately isolate brain processes
⇓
Very hard to achieve!
Lack of generality.
6. Background and motivations (1/2)
In cognitive neuroscience:
Brain systems
⇐⇒
Mental functions
Task-fMRI experiments allow to:
I link brain systems to behavior
I map neural activity at mm-scale
2/25
7. Background and motivations (2/2)
Data-pooling analysis
I Meta-analysis:
pooling data derivatives
I Mega-analysis:
pooling raw data
Requisites for cognitive mapping
Minimize variability of Successful interpretation of
spatial location combined results
same processing no loss of info from sparse
routines peak-coord. representation
same experimental consistency of
settings cognitive annotations
low inter-subject variability sufficient multi-task data
3/25
8. Background and motivations (2/2)
Data-pooling analysis
I Meta-analysis:
pooling data derivatives
I Mega-analysis:
pooling raw data
Requisites for cognitive mapping
Minimize variability of Successful interpretation of
spatial location combined results
same processing no loss of info from sparse
routines peak-coord. representation
same experimental consistency of
settings cognitive annotations
low inter-subject variability sufficient multi-task data
3/25
9. Background and motivations (2/2)
Data-pooling analysis
I Meta-analysis:
pooling data derivatives
I Mega-analysis:
pooling raw data
Requisites for cognitive mapping
Minimize variability of Successful interpretation of
spatial location combined results
same processing no loss of info from sparse
routines peak-coord. representation
same experimental consistency of
settings ( ) cognitive annotations
low inter-subject variability sufficient multi-task data
3/25
10. Background and motivations (2/2)
Data-pooling analysis
I Meta-analysis:
pooling data derivatives
I Mega-analysis:
pooling raw data
Requisites for cognitive mapping
Minimize variability of Successful interpretation of
spatial location combined results
same processing no loss of info from sparse
routines peak-coord. representation
same experimental consistency of
settings ( ) cognitive annotations
low inter-subject variability sufficient multi-task data
Large-scale repositories:
I OpenNeuro
I NeuroVault
I EBRAINS
3/25
11. Background and motivations (2/2)
Data-pooling analysis
I Meta-analysis:
pooling data derivatives
I Mega-analysis:
pooling raw data
Requisites for cognitive mapping
Minimize variability of Successful interpretation of
spatial location combined results
same processing no loss of info from sparse
routines peak-coord. representation
same experimental consistency of
settings ( ) cognitive annotations
low inter-subject variability sufficient multi-task data
Large-scale repositories:
I OpenNeuro
I NeuroVault
I EBRAINS
Individual analysis:
I Fedorenko, E. et al. (2011)
I Haxby, J. et al. (2011)
I Hanke, M. et al. (2014)
3/25
12. Background and motivations (2/2)
Data-pooling analysis
I Meta-analysis:
pooling data derivatives
I Mega-analysis:
pooling raw data
Requisites for cognitive mapping
Minimize variability of Successful interpretation of
spatial location combined results
same processing no loss of info from sparse
routines peak-coord. representation
same experimental consistency of
settings ( )( ) cognitive annotations
low inter-subject variability sufficient multi-task data
Large-scale repositories:
I OpenNeuro
I NeuroVault
I EBRAINS
Individual analysis:
I Fedorenko, E. et al. (2011)
I Haxby, J. et al. (2011)
I Hanke, M. et al. (2014)
Large-scale datasets:
I HCP
I studyforrest
I CONNECT/Archi
3/25
13. Background and motivations (2/2)
Data-pooling analysis
I Meta-analysis:
pooling data derivatives
I Mega-analysis:
pooling raw data
Requisites for cognitive mapping
Minimize variability of Successful interpretation of
spatial location combined results
same processing no loss of info from sparse
routines peak-coord. representation
same experimental consistency of
settings ( )( ) cognitive annotations
low inter-subject variability sufficient multi-task data
Large-scale repositories:
I OpenNeuro
I NeuroVault
I EBRAINS
Individual analysis:
I Fedorenko, E. et al. (2011)
I Haxby, J. et al. (2011)
I Hanke, M. et al. (2014)
Large-scale datasets:
I HCP
I studyforrest
I CONNECT/Archi
IBC dataset: a facility that
meets the requisites all
together
3/25
15. The IBC dataset
I High spatial-resolution fMRI data (1.5mm)
I TR = 2s
4/25
16. The IBC dataset
I High spatial-resolution fMRI data (1.5mm)
I TR = 2s
I Task-wise dataset:
I Many tasks
4/25
17. The IBC dataset
I High spatial-resolution fMRI data (1.5mm)
I TR = 2s
I Task-wise dataset:
I Many tasks
I Fixed cohort - 12 healthy adults
4/25
18. The IBC dataset
I High spatial-resolution fMRI data (1.5mm)
I TR = 2s
I Task-wise dataset:
I Many tasks
I Fixed cohort - 12 healthy adults
I Fixed environment
NeuroSpin platform, CEA-Saclay, France
Siemens 3T Magnetom Prismafit
64-channel coil
4/25
19. The IBC dataset
I High spatial-resolution fMRI data (1.5mm)
I TR = 2s
I Task-wise dataset:
I Many tasks
I Fixed cohort - 12 healthy adults
I Fixed environment
I Inclusion of other MRI modalities
NeuroSpin platform, CEA-Saclay, France
Siemens 3T Magnetom Prismafit
64-channel coil
4/25
20. The IBC dataset
I High spatial-resolution fMRI data (1.5mm)
I TR = 2s
I Task-wise dataset:
I Many tasks
I Fixed cohort - 12 healthy adults
I Fixed environment
I Inclusion of other MRI modalities
I Not a longitudinal study!
NeuroSpin platform, CEA-Saclay, France
Siemens 3T Magnetom Prismafit
64-channel coil
4/25
21. Tasks
First release:
I ARCHI battery
Pinel, P. et al. (2007)
I Standard
I Spatial
I Social
I Emotional
I HCP battery
Barch, D. M. et al. (2013)
I Emotion
I Gambling
I Motor
I Language
I Relational
I Social
I WM
I RSVP Language task
Humphries, C. et al. (2006)
Second release:
I Mental Time Travel battery
Gauthier, B., & van Wassenhove, V. (2016a,b)
I Preference battery
Lebreton, M. et al. (2015)
I ToM + Pain Matrices battery
Dodell-Feder, D. et al. (2010)
Jacoby, N. et al. (2015)
Richardson, H. et al. (2018)
I Visual Short-Term Memory +
Enumeration tasks
Knops, A. et al. (2014)
I Self-Reference Effect task
Genon, S. et al. (2014)
I “Bang!” task
Campbell, K. L. et al. (2015)
Third release:
I Clips task
Nishimoto, S. et al. (2011)
I Retinotopy task
Sereno, M. et al. (1995)
I “Raiders” task
Haxby, J. V. et al. (2011)
Fourth release
I Lyon battery
Hamamé, C. M. et al. (2012) / Ossandón, T. et al. (2012)
Saignavongs, M. et al. (2017) / Vidal, J. R. et al. (2010)
Perrone-Bertolotti, M. et al. (2012)
I Realistic Sounds task
Santoro, R. et al. (2017)
I Stanford battery
Ward, G. and Allport, A. (1997)
Shallice, T. (1992) / Stroop, J. R. (1935)
Bissett, P. G. and Logan, G. D. (2011)
Eriksen, B. A. and Eriksen, C. W. (1974) 5/25
22. Tasks
First release:
I ARCHI battery
Pinel, P. et al. (2007)
I Standard
I Spatial
I Social
I Emotional
I HCP battery
Barch, D. M. et al. (2013)
I Emotion
I Gambling
I Motor
I Language
I Relational
I Social
I WM
I RSVP Language task
Humphries, C. et al. (2006)
Second release:
I Mental Time Travel battery
Gauthier, B., & van Wassenhove, V. (2016a,b)
I Preference battery
Lebreton, M. et al. (2015)
I ToM + Pain Matrices battery
Dodell-Feder, D. et al. (2010)
Jacoby, N. et al. (2015)
Richardson, H. et al. (2018)
I Visual Short-Term Memory +
Enumeration tasks
Knops, A. et al. (2014)
I Self-Reference Effect task
Genon, S. et al. (2014)
I “Bang!” task
Campbell, K. L. et al. (2015)
Third release:
I Clips task
Nishimoto, S. et al. (2011)
I Retinotopy task
Sereno, M. et al. (1995)
I “Raiders” task
Haxby, J. V. et al. (2011)
Fourth release
I Lyon battery
Hamamé, C. M. et al. (2012) / Ossandón, T. et al. (2012)
Saignavongs, M. et al. (2017) / Vidal, J. R. et al. (2010)
Perrone-Bertolotti, M. et al. (2012)
I Realistic Sounds task
Santoro, R. et al. (2017)
I Stanford battery
Ward, G. and Allport, A. (1997)
Shallice, T. (1992) / Stroop, J. R. (1935)
Bissett, P. G. and Logan, G. D. (2011)
Eriksen, B. A. and Eriksen, C. W. (1974)
1st rel. + 2nd rel. + Retinotopy
All contrasts: 216
Elementary contrasts: 120
Cognitive concepts: 113
5/25
25. Behavioral Protocols
Software Tools:
I More public repositories of behavioral protocols to
reproduce experiments!
I Normatives to describe the experimental paradigms!
6/25
30. IBC reproduces ARCHI and HCP
t
a
l
e
v
s
.
m
e
n
t
a
l
a
d
d
i
t
i
o
n
m
e
n
t
a
l
m
o
t
i
o
n
v
s
.
r
a
n
d
o
m
m
o
t
i
o
n
p
u
n
i
s
h
m
e
n
t
v
s
.
r
e
w
a
r
d
l
e
f
t
f
o
o
t
v
s
.
a
n
y
m
o
t
i
o
n
l
e
f
t
h
a
n
d
v
s
.
a
n
y
m
o
t
i
o
n
r
i
g
h
t
f
o
o
t
v
s
.
a
n
y
m
o
t
i
o
n
r
i
g
h
t
h
a
n
d
v
s
.
a
n
y
m
o
t
i
o
n
t
o
n
g
u
e
v
s
.
a
n
y
m
o
t
i
o
n
f
a
c
e
i
m
a
g
e
v
s
.
s
h
a
p
e
o
u
t
l
i
n
e
r
e
l
a
t
i
o
n
a
l
p
r
o
c
e
s
s
i
n
g
v
s
.
v
i
s
u
a
l
m
a
t
c
h
i
n
g
2
-
b
a
c
k
v
s
.
0
-
b
a
c
k
b
o
d
y
i
m
a
g
e
v
s
.
a
n
y
i
m
a
g
e
f
a
c
e
i
m
a
g
e
v
s
.
a
n
y
i
m
a
g
e
p
l
a
c
e
i
m
a
g
e
v
s
.
a
n
y
i
m
a
g
e
t
o
o
l
i
m
a
g
e
v
s
.
a
n
y
i
m
a
g
e
h
o
r
i
z
o
n
t
a
l
c
h
e
c
k
e
r
b
o
a
r
d
v
s
.
v
e
r
t
i
c
a
l
c
h
e
c
k
e
r
b
o
a
r
d
m
e
n
t
a
l
s
u
b
t
r
a
c
t
i
o
n
v
s
.
s
e
n
t
e
n
c
e
r
e
a
d
s
e
n
t
e
n
c
e
v
s
.
l
i
s
t
e
n
t
o
s
e
n
t
e
n
c
e
r
e
a
d
s
e
n
t
e
n
c
e
v
s
.
c
h
e
c
k
e
r
b
o
a
r
d
l
e
f
t
h
a
n
d
v
s
.
r
i
g
h
t
h
a
n
d
s
a
c
c
a
d
e
v
s
.
f
i
x
a
t
i
o
n
g
u
e
s
s
w
h
i
c
h
h
a
n
d
v
s
.
h
a
n
d
p
a
l
m
o
r
b
a
c
k
o
b
j
e
c
t
g
r
a
s
p
i
n
g
v
s
.
m
i
m
i
c
o
r
i
e
n
t
a
t
i
o
n
m
e
n
t
a
l
m
o
t
i
o
n
v
s
.
r
a
n
d
o
m
m
o
t
i
o
n
f
a
l
s
e
-
b
e
l
i
e
f
s
t
o
r
y
v
s
.
m
e
c
h
a
n
i
s
t
i
c
s
t
o
r
y
f
a
l
s
e
-
b
e
l
i
e
f
t
a
l
e
v
s
.
m
e
c
h
a
n
i
s
t
i
c
t
a
l
e
f
a
c
e
t
r
u
s
t
y
v
s
.
f
a
c
e
g
e
n
d
e
r
e
x
p
r
e
s
s
i
o
n
i
n
t
e
n
t
i
o
n
v
s
.
e
x
p
r
e
s
s
i
o
n
g
e
n
d
e
r
tale vs. mental addition
mental motion vs. random motion
punishment vs. reward
left foot vs. any motion
left hand vs. any motion
right foot vs. any motion
right hand vs. any motion
tongue vs. any motion
face image vs. shape outline
relational processing vs. visual matching
2-back vs. 0-back
body image vs. any image
face image vs. any image
place image vs. any image
tool image vs. any image
horizontal checkerboard vs. vertical checkerboard
mental subtraction vs. sentence
read sentence vs. listen to sentence
read sentence vs. checkerboard
left hand vs. right hand
saccade vs. fixation
guess which hand vs. hand palm or back
object grasping vs. mimic orientation
mental motion vs. random motion
false-belief story vs. mechanistic story
false-belief tale vs. mechanistic tale
face trusty vs. face gender
expression intention vs. expression gender
HCP contrasts ARCHI contrasts
IBC
contrasts
1.00
0.75
0.50
0.25
0.00
0.25
0.50
0.75
1.00 ARCHI batteries:
Pinel, P. et al. (2007)
HCP batteries:
Barch, D. M. et al. (2013)
n = 13
Pinho, A.L. et al. Hum Brain Mapp(2021)
10/25
31. Effect of subject and task on brain activity
Per-voxel one-way ANOVA qFDR < 0.05
x=10
L R
z=10 -28
-14
0
14
28
L R
y=-50
Subject effect
x=10
L R
z=10 -37
-19
0
19
37
L R
y=-50
Condition effect
x=-6
L R
z=3 -12
-5.9
0
5.9
12
L R
y=45
Phase encoding effect
Pinho, A.L. et al. SciData(2018) Pinho, A.L. et al. SciData(2020)
11/25
32. Effect of subject and task on brain activity
Per-voxel one-way ANOVA qFDR < 0.05
x=10
L R
z=10 -28
-14
0
14
28
L R
y=-50
Subject effect
x=10
L R
z=10 -37
-19
0
19
37
L R
y=-50
Condition effect
x=-6
L R
z=3 -12
-5.9
0
5.9
12
L R
y=45
Phase encoding effect
Pinho, A.L. et al. SciData(2018) Pinho, A.L. et al. SciData(2020)
I IBC data is suitable for cognitive mapping
and individual-brain modeling!
11/25
33. Activation similarity fits task similarity
n = 11
Similarity between
activation maps
of elementary contrasts
Similarity between
cognitive description
of elementary contrasts
Pinho, A.L. et al. SciData(2020) 12/25
34. Activation similarity fits task similarity
n = 11
Similarity between
activation maps
of elementary contrasts
Similarity between
cognitive description
of elementary contrasts
Pinho, A.L. et al. SciData(2020)
Spearman correlation
First Release: 0.21 (p ≤ 10−17)
Second Release: 0.21 (p ≤ 10−13)
First+Second Releases: 0.23 (p ≤ 10−72)
12/25
37. Variability of Functional Signatures
Pinho, A.L. et al. Hum Brain Mapp(2021) n = 13
Individual z-maps
15/25
38. Variability of Functional Signatures
Pinho, A.L. et al. Hum Brain Mapp(2021) n = 13
0.00 0.25 0.50
read sentence vs. listen to sentence
read sentence vs. checkerboard
left hand vs. right hand
horizontal checkerboard vs. vertical checkerboard
mental subtraction vs. sentence
saccade vs. fixation
guess which hand vs. hand palm or back
object grasping vs. mimic orientation
mental motion vs. random motion
false-belief story vs. mechanistic story
false-belief tale vs. mechanistic tale
expression intention vs. expression gender
face trusty vs. face gender
face image vs. shape outline
punishment vs. reward
0.00 0.25 0.50
tongue vs. any motion
right foot vs. any motion
left foot vs. any motion
right hand vs. any motion
left hand vs. any motion
tale vs. mental addition
relational processing vs. visual matching
mental motion vs. random motion
tool image vs. any image
place image vs. any image
face image vs. any image
body image vs. any image
2-back vs. 0-back
read pseudowords vs. consonant strings
read words vs. consonant strings
read words vs. read pseudowords
read sentence vs. read jabberwocky
read sentence vs. read words
inter-subject correlation
intra-subject correlation
Intra- and inter- subject correlation of brain maps
15/25
40. Dictionary of cognitive components
Decomposition of 51 contrasts
with dictionary learning
Individual topographies of
20 components (n = 13)
Each component gets the name
of the active condition from the
contrast with the highest value in
the dictionary.
Multi-subject, sparse dictionary learning:
min(Us )s=1...n,V∈C
n
X
s=1
kXs
− Us
Vk2
+ λkUs
k1
,
with Xs
p×c , Us
p×k and Vk×c
I Functional correspondence: dictionary
of functional profiles (V) common to
all subjects
I Sparsity: `1−norm penalty and
Us ≥ 0 , ∀s ∈ [n]
17/25
41. Dictionary of cognitive components
Pinho, A.L. et al. Hum Brain Mapp(2021) n = 13
Components are consistently mapped across subjects.
17/25
42. Dictionary of cognitive components
Pinho, A.L. et al. Hum Brain Mapp(2021) n = 13
Components are consistently mapped across subjects.
17/25
43. Dictionary of cognitive components
Pinho, A.L. et al. Hum Brain Mapp(2021) n = 13
0.25 0.30 0.35 0.40 0.45 0.50 0.55
Intra-subject
correlation
Inter-subject
correlation
Correlations of the dictionary components on split-half data
Variability of topographies linked to individual differences.
17/25
45. Reconstruction of functional contrasts
Leave-p-out CV (p=3 subjects)
experiment to learn the shared
representations from contrasts of
eleven tasks. (n = 13)
Predict all contrasts from the
remaining task
19/25
46. Reconstruction of functional contrasts
Leave-p-out CV (p=3 subjects)
experiment to learn the shared
representations from contrasts of
eleven tasks. (n = 13)
Predict all contrasts from the
remaining task
Train a Ridge-regression model with individual
contrast maps i of tasks −j to predict task j on
individual contrast-maps s 6= i:
b
ws,λ,j
= argminw∈Rc−1
X
i6=s
kXi
j − Xi
−j wk2
+ λkwk2
Prediction output for one contrast of task j in
subject s:
b
Xs
j = Xs
−j b
ws,λ,j
.
Cross-validated R-squared for task j at location i:
R2
i (j) = 1 − means∈[n]
kb
Xs
i,j − Xs
i,j k2
kXs
i,j k2
19/25
47. Reconstruction of functional contrasts
Pinho, A.L. et al. Hum Brain Mapp(2021)
n = 13
max R2
Most of the brain regions
are covered by the
predicted functional
signatures.
19/25
48. Reconstruction of functional contrasts
n = 13
Pinho, A.L. et al. Hum Brain Mapp(2021)
Ridge-Regression model
for the scrambled case:
b
ws,λ,j
= argminw∈Rc−1
X
i,k 6= s
kXi
j −Xk
−j wk2
+λkwk2
Cross-validated R-squared:
R2
i (j) = 1 − means∈[n]
kb
Xs
i,j − Xs0
i,j k2
kXs0
i,j k2
Permutations of subjects
decrease the proportion of
well-predicted voxels in all
tasks, showing that
topographies are driven by
subject-specific variability.
19/25
50. Ex: Functional mapping of the language network
Goal: Cognitive profile of ROIs based on IBC language-related contrasts
Select ROIs / Select IBC contrasts
Individualize ROIs using dual-regression
and the left-out contrasts
R(s) = R pinv X(s)
X(s)
Voxelwise z-scores average for each
ROI at every selected contrast
Pinho, A.L. et al. Hum Brain Mapp(2021)
21/25
51. Ex: Functional mapping of the language network
Linear SVC (upper triangle)
Dummy Classifier (lower triangle)
LOGOCV scheme
Prediction within pairs of ROIs
13 groups = 13 participants
Pinho, A.L. et al. Hum Brain Mapp(2021)
21/25
52. Concluding remarks
Functional atlasing using a large dataset in the task dimension
I Investigation of common functional profiles between tasks
I Common
functional profiles
Shared
behavioral responses
Mental
functions
22/25
53. Concluding remarks
Functional atlasing using a large dataset in the task dimension
I Investigation of common functional profiles between tasks
I Common
functional profiles
Shared
behavioral responses
Mental
functions
Individual brain modeling using data with higher spatial resolution
I generalize across subjects
I elicit variability between subjects
22/25
54. Future outcomes
I Article on the IBC-dataset third-release
I Fifth and sixth releases out this year
23/25