Talk 4: Ordering events in time and space: similar algorithms, different implementations? Virginie van Wassenhove1, Baptiste Gauthier2, Pooja Prabhu3; 1CEA DRF/Joliot NeuroSpin, INSERM Cognitive Neuroimaging Unit, France, 2Laboratory of Cognitive Neuroscience, Brain Mind Institute Ecole, Polytechnique Fédérale de Lausanne, Campus Biotech, Genève, Switzerland, 3Department of Computer Applications, Manipal Insititute of Technology, Manipal Academy of Higher Education, India When moving, the spatiotemporal unfolding of events is bound to our physical trajectory, and time and space become entangled in episodic memory. When imagining past or future events, or being in different geographical locations, the temporal and spatial dimensions of mental events can be independently accessed and manipulated. How the human brain represents time and space is essential to understand the conscious mind. Using psychophysics (Gauthier & van Wassenhove, Cognition, 2016), fMRI (Gauthier & van Wassenhove, J Neurosci, 2016) and magnetoencephalograhy (MEG; Gauthier, Petske & van Wassenhove, bioRxiv, 2017), we characterized chronometry, performance, and brain activity while participants ordered historical events from different mental perspectives in time (i.e. from a past or future imagined viewpoint) or in space (i.e. from a western or eastern imagined viewpoint). We report similar behavioral patterns for ordering events in time and space, but substantial differences in the neuroanatomical and dynamic implementations of the cognitive operations implicated in this task. In addition to the convergence of behavioral, fMRI and MEG results indicating distance effects between an imagined self position (in time and space) and the ordering of retrieved events, we also report signed distance effects in MEG results enabling to dissociate the reconstruction of events in the past from those in the future, including their distances to self. Preliminary reconstructions of deep brain sources using MEG suggests the implication of hippocampal structures during the conscious representation of ordinality notably in time, i.e. in the elaboration of a mental time arrow. The directionality of the psychological time arrow thus appears to rely on distinct neural implementations dissociable from spatial directionalities.

1. Ordering events in time & space
similar algorithms, different implementations?
Baptiste Gauthier, PhD
Postdoctoral Research Fellow
Laboratory of Cognitive Neuroscience
Foundation Bertarelli, Campus Biotech
Geneva, Switzerland
Pooja Prabhu, MSc
PhD Student
Dpt of Computer Applications
Manipal Institute of Technology MAHE
Manipal, India
Virginie van Wassenhove
Cognition & Brain Dynamics lab
CEA, DRF/Joliot, NeuroSpin
INSERM Cognitive Neuroimaging Unit

2. Psychological Time & Consciousness
It is indisputable that our consciousness of the passage of time is determined by our
consciousness of the sequence of events […] time-consciousness being nothing more than
the memory of a series of successive changes in consciousness […]
- Romanes (1878)
XIXe

3. Exogenous vs. Endogenous Serial Order
It is indisputable that our consciousness of the passage of time is determined by our
consciousness of the sequence of events […] time-consciousness being nothing more than
the memory of a series of successive changes in consciousness […]
- Romanes (1878)
 The endogenous temporal hierarchy
of mental events does not follow the
serial ordering of sensory events
Ashley(1951) The Problem of Serial Order in Behavior
Hayek(1952) The Sensory Order
Gallistel (1990) The Organization of Learning
Dennett & Kinsbourne(1992) Time & the Observer
XIXe
XXe

4. Time vs. Timing
It is indisputable that our consciousness of the passage of time is determined by our
consciousness of the sequence of events […] time-consciousness being nothing more than
the memory of a series of successive changes in consciousness […]
- Romanes (1878)
« the representation of a sequence is not a sequence of representations »
- Buzsáki & Friston (2016) TiCS
 the consciousness of time is not the timing of consciousness
Dennett & Kinsbourne (1992, Behav Brain Sci)
XIXe
XXe
XXIe

5. egocentric & allocentric cognitive maps
space (and time?)
d = vt
distance = speed x duration
Buzsáki & Moser (2013) Nat Neurosci

6. Serial ordering
path integration
Schacter & Addis
A memory that works by piecing together bits
of the past may be better suited to simulating
future events than one that is a store of
perfect records
Tulving
Mental time travel is an episodic extension
towards the future
Suddendorf & Corballis
Episodic memory as generative process
Nielson et al 2015 PNAS
Endogenous ordering
Deuker et al 2016 eLife
Jenkins & Ranganath (2010) J Neurosci

7. In a purely mental task, would there be a
distinctive signature of past-to-future in the mind?

8.  The consciousness of time is not the timing of consciousness
Does thinking about events in time and space interfere?
Are workspaces for thinking about time and space shared?
Is ordering time the same as ordering space?

9. absolute distance effects
self-projection slows down
and increases errrors
relative distance effects
the more distant in time a memory is,
the faster its ordering w.r.t self-position
Behavioral markers of mental time travel
[Arzy et al, 2008, 2009a,b]
see also D’Argembeau & Van der Linden 2004; Spreng & Levine, 2006

10. XP1
Event Set
a shared cognitive map for mental time & space?
are there reliable interferences between temporal and
spatial judgements of non-episodic world events?

11. General Experimental Design
(1) SELF-PROJECTION
in time or in space
(2) DIMENSION of
ORDINALITY judgement
(3) EVENT RETRIEVAL,
ORDINALITY decision, response
36 events
NOW = 2012
HERE = PARIS

12. [absolute distance effects]
RT and Error Rates increase
when self-projection away
from the ‘here and now’
Distance effects in mental time & space travels
Fig 3
Gauthier & van Wassenhove (2016) Cognition

13. [absolute distance effects]
RT and Error Rates increase
when self-projection away
from the ‘here and now’
[relative distance effects]
RT and Error Rates decrease
with distance from the mental
reference
Distance effects in mental time & space travels
Fig 3
Gauthier & van Wassenhove (2016) Cognition

14. Self-projection in time slows down RT and increases errors in space
Self-projection in space slows down RT and increases errors in time
**the behavioral cost is significant larger within dimension [Experiments 2 & 3]
Fig 4C Fig 6A
Gauthier & van Wassenhove (2016) Cognition

15. Self-projection in time slows down RT and increases errors in space
Self-projection in space slows down RT and increases errors in time
**although the behavioral cost is significantly larger within dimension [Experiments 2 & 3]
Fig 4C Fig 6A
Switching to the non self-projected dimension is chronometrically costly
Fig 4A Fig 4B
Gauthier & van Wassenhove (2016) Cognition

16.  The consciousness of time is not the timing of consciousness
 Time & space interfere during self-projection
Time & space do not interfere during ordering
No signature of past-to-future in behavior
Are workspaces to think about time and space the same?
Is ordering time the same as ordering space?

17. • Historical event distribution
balanced for response side
• Historical event were centered on
the imagined self-position
fMRI
same event set, same design
Control task = valence judgment
controling for
2-AFC
Memory retrieval
General executive processes
Motor processes
Gauthier & van Wassenhove (2016) J Neurosci

18. Brain regions implicated in judging the ordinality vs. valence
of world events
core network
• medial parietal
• IPL/AG
• Precuneus/RSC
• right superior frontal
T-value
Gauthier & van Wassenhove (2016) J Neurosci

19. Ordering time differs from ordering space
Gauthier & van Wassenhove J Neurosci 2016

20. Self-projection
as
allo- to ego-centric mapping
Ordering involves
egocentric distance
Gauthier & van Wassenhove J Neurosci 2016

21. IPL/AG as distance in relation to self
Peer et al. PNAS 2015Parkinson et al J Neurosci 2014
Gauthier & van Wassenhove J Neurosci 2016

22. Committeri et al
J Cogn Neurosci 2004
Landmark-referenced
spatial judgement
participant’s
actual location
landmarks
(allocentric)
landmarks
egocentric
Translation
alloego
Hippocampus
Entorhinal
Parahippocampus
medial parietal
precuneus
post. parietal
retrosplenial
Nyberg & Tulving
PNAS 2010
Mental imagery
short walks
Mental Time Travel
≠ Spatial translation
≠ Arithmetics
Mental Spatial Navigation
Gauthier & van Wassenhove
J Neurosci 2016
Gauthier & van Wassenhove
J Neurosci 2016
Consistent with prior literature
Burgess, N., Becker, S., King, J. A., & Keefe, J. O.
(2001). Memory for events and their spatial
context : models and experiments.

23.  The consciousness of time is not the timing consciousness
 Time and space interfere during self-projection
Time and space do not interfere during ordering
No signature of past-to-future in behavior
 Self-projection as allo- to ego-centric mapping
Automaticity of egocentric distance computation
No signature of past-to-future in fMRI
Is ordering time the same as ordering space?

24. Gauthier, Pestke, van Wassenhove (2018) Cereb Cortex
Early self-
projection in time
after the REF
Late self-projection
in space after the
QU

25. All REF All QU
ongoing work
Hippocampal structures (Hipp, MT, EC) active during self-projection
but not selective to self-position or the task dimension
Prabuh et al (in prep)
corrected p<0.05
corrected p<0.05

26. ACTUAL
FUTURE
ACTUAL
PAST
self-projection in PAST
mental now = -9 years (2021)
NO self-projection
mental now = actual now (2012)
self-projection in FUTURE
mental now = + 9 years (2003)
MENTAL NOW
MENTAL
FUTURE
MENTAL
PAST
aligned on: physical now self

27. self-projection in PAST
mental now = -9 years
NO self-projection
mental now = actual now
self-projection in FUTURE
mental now = + 9 year
MENTAL NOW
MENTAL
FUTURE
MENTAL
PAST
Graded past-to-future activity w.r.t mental now

28. self-projection in PAST
mental now = -9 years
NO self-projection
mental now = actual now
self-projection in FUTURE
mental now = + 9 year
MENTAL NOW
MENTAL
FUTURE
MENTAL
PAST
Gauthier, Petske, van Wassenhove (2018) Cereb Cortex
Graded past-to-future activity w.r.t mental now

29. self-projection in PAST
mental now = -9 years
NO self-projection
mental now = actual now
self-projection in FUTURE
mental now = + 9 year
MENTAL NOW
MENTAL
FUTURE
MENTAL
PAST
Gauthier, Pestke, van Wassenhove (2018) Cereb Cortex
Graded activity distance to mental now

30. ACTUAL
EAST
ACTUAL
WEST
self-projection in EAST
mental east = Dubai
NO self-projection
mental here = Paris
self-projection in WEST
mental west = Cayenne
MENTAL HERE
MENTAL
EAST
MENTAL
WEST
Gauthier, Pestke, van Wassenhove (2018) Cereb Cortex
aligned on: physical here self

31. self-projection in EAST
mental east = Dubai
NO self-projection
mental here = Paris
self-projection in WEST
mental west = Cayenne
MENTAL HERE
MENTAL
EAST
MENTAL
WEST
Gauthier, Pestke, van Wassenhove (2018) Cereb Cortex
Graded west-to-east activity w.r.t mental here

32. self-projection in EAST
mental east = Dubai
NO self-projection
mental here = Paris
self-projection in WEST
mental west = Cayenne
MENTAL HERE
MENTAL
EAST
MENTAL
WEST
Gauthier, Pestke, van Wassenhove (2018) Cereb Cortex
Graded activity distance to mental here

33.  The consciousness of time is not the timing consciousness
 Self-projection in time & space interfere
Ordinality in time & space does not
No signatures of past-to-future in behavior
 Self-projection as allo- to ego-centric mapping
Automatic egocentric distance computation
No signatures of past-to-future in fMRI
 Hippocampal structures contribute to the endogenous construction of a
mental time arrow centered on the self
 Egocentric mapping, symbolic distance, & ordinality when
thinking in space and time (same algorithmic steps) are computed
at different latencies in different brain regions (different
implementations)
Take home messages

34. Conclusion Time & The Observer
the consciousness of time is not the timing of consciousness
when time-align brain activity on an exogenous now
Dennett &
Kinsbourne(1992)

35. Conclusion Time & The Observer
the consciousness of time is not the timing of consciousness
when we time-align brain activity on an exogenous now
the consciousness of time may be the timing of consciousness
when we time-align brain activity on an endogenous now
Dennett &
Kinsbourne(1992)