Human Sensory Adaptation To The Ecological Structure of Environmental Statistics

Journal of Vision (2024) 24(3):3, 1–23 1
Human sensory adaptation to the ecological structure of

environmental statistics
Peter Neri Laboratoire des Systèmes Perceptifs (UMR8248),
École normale supérieure, PSL Research University,
Paris, France
Humans acquire sensory information via fast, highly Subsequent research in both physiology and
specialized detectors: For example, edge detectors psychophysics has delineated a much more complex
monitor restricted regions of visual space over picture of these processes. Visual detectors may
timescales of 100–200 ms. Surprisingly, this study be primarily driven by localized spatiotemporal
demonstrates that their operation is nevertheless information, but the manner in which they process said
shaped by the ecological consistency of slow global information is influenced by information from other
statistical structure in the environment. In the spatiotemporal regions (Bolz & Gilbert, 1986). For
experiments, humans acquired feature information from example, visual neurons may respond differently to an
brief localized elements embedded within a virtual identical stimulus, depending on what came before the
environment. Cast shadows are important for stimulus (Kohn, 2007) and on what is presented in the
determining the appearance and layout of the surrounding region (Allman, Miezin, & McGuinness,
environment. When the statistical reliability of shadows
1985). These effects can be complex (Roelfsema,
was manipulated, human feature detectors implicitly
adapted to these changes over minutes, adjusting their Lamme, & Spekreijse, 1998; Zhou, Friedman, &
response properties to emphasize either “image-based” von der Heydt, 2000; Roelfsema, Tolboom, & Khayat,
or “object-based” anchoring of local visual elements. 2007; Gilbert & Li, 2013), to the extent that they may
More specifically, local visual operators were more incorporate specific aspects of natural scenes (Felsen,
firmly anchored around object representations when Touryan, Han, & Dan, 2005; Pecka, Han, Sader, &
shadows were reliable. As shadow reliability was Mrsic-Flogel, 2014; Coen-Cagli, Kohn, & Schwartz,
reduced, visual operators disengaged from objects and 2015; Hesse & Tsao, 2016). Similarly, visual percepts
became anchored around image features. These results are influenced by preceding stimuli to produce strong
indicate that the notion of sensory adaptation must be phenomenological experiences (Blakemore & Campbell,
reframed around complex statistical constructs with 1969), such as aftereffects (Anstis, Verstraten, &
ecological validity. These constructs far exceed the Mather, 1998). Perceptual judgments of elementary
spatiotemporal selectivity bandwidth of sensory image elements are impacted by global characteristics
detectors, thus demonstrating the highly integrated of the surrounding image (Schwartz, Hsu, & Dayan,
nature of sensory processing during natural behavior. 2007; Choung, Bornet, Doerig, & Herzog, 2021), often
incorporating higher-level inferences about object
structure (Bar, 2004; Neri, 2014; Neri, 2017).
It is now evident that visual sensors adapt to the
Introduction wider context within which they are embedded, such as
the highly structured nature of environmental signals
Early understanding of visual processing was (Simoncelli & Olshausen, 2001). However, existing
dominated by the notion of specificity (Hubel, 1982): measurements are largely restricted to constrained
cortical neurons respond to stimuli within a restricted viewing conditions in which human observers cannot
region of visual space (their “receptive field”), and display the full repertoire of natural vision. For
record stimulus characteristics within the timescale of example, our current understanding of large-scale
stable vision (<300 ms). The corresponding perceptual contextual interactions for visual processing of image
mechanisms display similar characteristics. For features is limited to settings in which humans view
example, humans perceive motion via mechanisms that flat images on a computer monitor, with their heads
integrate signals over ∼100 ms (Burr & Ross, 1986), immobilized and their eyes fixed (Choung et al., 2021;
and effective manipulation of the associated percepts is Neri, 2017). The conditions of these experiments
often strictly local (Curran, Clifford, & Benton, 2006). present the advantage of fine experimental control
Citation: Neri, P. (2024). Human sensory adaptation to the ecological structure of environmental statistics. Journal of Vision,
24(3):3, 1–23, https://doi.org/10.1167/jov.24.3.3.
https://doi.org/10.1167/jov.24.3.3 Received March 27, 2023; published March 5, 2024 ISSN 1534-7362 Copyright 2024 The Authors
This work is licensed under a Creative Commons Attribution 4.0 International License.
Downloaded from jov.arvojournals.org on 03/23/2024
Journal of Vision (2024) 24(3):3, 1–23 Neri 2
(Rust & Movshon, 2005), but they do not adequately Base Stations) were placed on opposite sides of the
capture natural vision in terms of stimulus richness and physical test area (measuring approximately 3 × 3 m)
ecological validity of human behavior (Felsen & Dan, at a height of ∼2 m. The HMD was cabled to an
2005). Alienware Area 51 desktop and directly controlled from
I overcome the above limitations by performing Unity. All virtual elements, shaders, active routines, and
quantitative measurements of visual discrimination in image-processing modules were supported by in-house
virtual reality (VR). To my knowledge, this study is software written in C#. Participants were monitored
the first to characterize low-level visual mechanisms at all times during experimentation to ensure safe
during complex exploration of ecologically valid navigation.
environments. VR affords the opportunity to define
structural consistency of the stimulus on a much more
complex level than available with traditional laboratory
Participants and data mass (number of trials)
methods (Warren, Rothman, Schnapp, & Ericson,
All protocols received ethics approval (agreement
2017). For example, we can define a statistical construct
RIPH3 #2022-A00963-40), and all participants gave
that specifies the ecological validity of shadow casting:
informed consent. I tested 10 naive participants
whether it is consistent with a natural light source that
and 1 non naive participant (author PN). Each
behaves like the sun (Mamassian, Knill, & Kersten,
participant collected between 600 and 1,200 trials,
1998) and/or whether it aligns with active goal-oriented
equally split across the three shadow configurations,
exploration of the environment (Gibson, 1979). VR
where “trial” refers to one sensory discrimination event.
allows fine control over complex constructs of this kind
I collected a total of 9,300 trials. Different shadow
(de Gelder, Tsyri, & de Borst, 2018).
configurations were run in different blocks of 100 trials
Surprisingly, I find that the long-term statistical
in pseudo-random order. Participants were paid 15
reliability of such global environmental constructs
euros per hour.
is able to steer local processes that analyze visual
signals on the subsecond scale. More specifically,
local edge information is anchored more effectively Virtual room design
to objects when shadows are statistically reliable.
When shadow information is unreliable, observers The rendered dimensions of the virtual room were
shift their perceptual weight toward image-based 6 × 6 × 3 m (width × length × height). The interior
cues to the detriment of object-based cues. Dynamic of all sides (including floor and ceiling) was lined with
reallocation of cue information happens slowly, zigzag patterns of alternating black and white stripes.
exposing the operation of a temporally extended Each stripe was ∼70 cm wide and changed orientation
adaptive mechanism that gathers environmental data by 90 degrees every ∼1.5 m (see pattern in Figures 1B,
over timescales of minutes. Overall, the results indicate K). I selected zigzag patterns from a wider range
that the concept of sensory adaptation should be of high-contrast patterns, which I tested during the
expanded and reframed to incorporate not only the piloting stages of this project. Some other patterns, such
proximal spatiotemporal context of sensory stimuli, but as thresholded low-pass two-dimensional (2D) noise,
also the full perceptual experience of agents engaged in carried the advantage of more natural appearance, but
active exploration of complex environments (Gibson, they posed serious challenges for the probe insertion
1979). algorithm, because they often lacked clear edges and did
not support the corner exclusion strategy adopted by
the algorithm (see below). The piloting results indicated
that zigzag patterns were the only feasible option
Methods (among those tested) that would ensure successful probe
insertion on a sufficient number of trials for viable
VR hardware and software experimentation. The room was filled with 10 boxes,
created and specified sequentially according to the
I collected data with two head-mounted devices following rules. For each box, the computer algorithm
(HMDs): HTC Vive and HTC Vive Pro Eye. The latter selected a random location within the region defined
supports eye-tracking capabilities. The nominal field by the floor, such that the box was not located within
of view provided by the manufacturer is 110 degrees 80 cm from the center of the room, and its distance
(horizontal), but the actual field of view is known to from all other boxes was at least 80 cm. The algorithm
be much smaller (∼65 degrees) under typical viewing then selected four values for width, length, height, and
conditions (Lange, Stratmann, Gruenefeld, & Boll, rotation around the vertical axis by randomly sampling
2020). In the conditions of the experiments, I estimated from uniform distributions over the following ranges:
a usable (relatively undistorted) horizontal field of view 40–120 cm (width), 40–120 cm (length), 40–240 cm
of ∼60 degrees. Two tracking cameras (HTC Vive (height), and 0–180 degrees, respectively. These rules

Figure 1. Sensory/memory tasks in an edge-rich VR world. Observers operate within a virtual room containing numerous edges (B).
Every few seconds, the scene viewed by observers (A) is analyzed to extract an image-driven map (D), reflecting edge-contrast energy
in the 2D scene, and an object-driven map (E), reflecting object edges defined by 3D layout (see Methods). Guided by these maps,
oriented probes (circled in F–G) are locally inserted into either image-defined edges (F, red) or object-defined edges (G, blue) and
corrupted by orientation noise (H). In the sensory task, observers must determine whether probe orientation is congruent (I) or
incongruent (J) with the orientation locally defined by the scene (indicated by green line in I–J). After 100 such discriminations
(indicated by green segments in N), observers are teleported above the room (blue head icon in B) and asked to perform a memory
task (indicated by blue segment in N): They must identify which of two boxes moved during teleportation (C, K) and are provided with
immediate feedback (correct/incorrect, L/M). In different experiments, cast shadows are structured according to three
configurations: “reliable” (P), in which shadows are consistent with one fixed source of illumination throughout the 5-min block;
“absent” (Q), in which shadows are completely removed from the environment; and “unreliable” (O), in which shadows are recast to
conform with a different source of illumination half a second before the presentation of each probe (events indicated by connecting
lines between N and O).

ensured sufficient separation between boxes and seconds. The probe lasted 200 ms, and different
avoided instances in which participants (who always probe presentations were separated by random time
started at the center of the room) would find themselves intervals uniformly distributed between 2.0 and 3.6 s.
inside a box at the start of the block. The algorithm Immediately after probe disappearance, observers
created specifications for 11 boxes but only instantiated were given 1 s to produce a valid binary response
10 boxes during the experimental block. At the end of by pressing one of two buttons on the handheld
each block, one of the 10 instantiated boxes was moved HTC Vive Controller. Their response choice was
to the location specified by the 11th box, without between “congruent” and “incongruent” (defined
modifying its size and rotation (see section below on further below). At the end of the 1-s response interval,
memory task). All sides of all boxes were lined with a they were provided with visual feedback indicating
zigzag pattern similar to the one applied to the room, whether their response was correct, incorrect, or
except scaled down by a factor of 2. invalid (no response within the 1-s window), and how
many consecutive incorrect/invalid responses they
had produced. See below for further details on probe
Lighting construction, feedback delivery, and rules associated
with correct/incorrect responses.
When present, the point light source was located at
an elevation of 45 degrees and a randomly selected
azimuth value spanning the entire directional range Probe object construction
(0–360 degrees). In the reliable shadow configuration Although the probe was inserted into the virtual
(Figure 1P), the light source was created once at the environment, which was defined in three dimensions,
beginning of each block and remained fixed throughout it was essentially a 2D object: It consisted of a disc
the block (it was only removed after teleportation). In with zero thickness and no identifiable depth. In this
the unreliable shadow configuration (Figure 1O), 500 ms sense, the probe lived in retinal space. This dissociation
before each probe appearance, the light source was between the three-dimensional (3D) environment
destroyed and immediately re-created with a different and the 2D nature of the probe was achieved via
azimuth value. In the absent shadow configuration the following specifications. First, the disc element
(Figure 1Q), the light source was never created, and supporting the probe was always placed at a fixed
ambient intensity was doubled to compensate for the distance from the observer (50 cm), and always oriented
associated loss in general lighting. More specifically, in 3D space to face the observer: When viewed from
ambient luminance in the reliable/unreliable shadow the HMD, the probe always appeared as a disc of
configurations ranged between ∼20 cd/m2 and ∼90 constant diameter defined in visual space, spanning
cd/m2 with an average of ∼60 cd/m2 , depending on ∼3 degrees of visual angle. Second, the disc was
which part of the room was viewed at the time of not subjected to shadow rendering and was only
measurement. I obtained these estimates indirectly illuminated by a designated light source placed at
by placing a photometer against one the two lenses the center of the HMD, ensuring that the luminance
inside the HMD and by sampling the environment pattern within the probe was under direct control
for different HMD directions/positions similar to of the texture pattern applied to the disc. More
those adopted by observers during the experiments. specifically, the disc was not illuminated by the light
These figures are therefore only indicative of the broad source that illuminated the room, and the room was
luminance range spanned by the stimuli and should not illuminated by the light source that illuminated the
not be regarded as accurate measures. Based on similar disc. Third, the disc was presented monocularly to a
measurements in the absent shadow configuration, I randomly selected eye, accompanied by a uniform gray
determined that it was necessary to increase ambient disc of matched average luminance presented to the
intensity for this configuration to match the other other eye. I adopted a slightly larger diameter for the
two shadow configurations. This match could only uniform disc (∼3.3 degrees) to accommodate potential
be achieved in terms of average luminance intensity, misalignments associated with vergence. Under this
not in terms of range: Reliable/unreliable shadow dichoptic configuration, informal assessment by all
configurations inevitably spanned a larger range participants indicated that the probe was clearly
compared with the absent shadow configuration. visible and did not engage in binocular rivalry with the
uniform disc. Furthermore, it did not produce a percept
of stereoscopic depth.
Sensory task
General procedure Probe texture specification
Observers were asked to judge the orientation The texture applied to the probe consisted of 16
of a visual probe that was briefly flashed every few superimposed gratings, windowed by a disc with a

Gaussian edge profile of SD ∼15 arcmin (Figure 1H). the distance of the corresponding environmental
Probe design conformed to extensively validated location from the observer. Based on calibration tests
specifications that yielded meaningful results in prior with fiducial markers placed at the corners of the
applications outside VR (Neri, 2014; Neri, 2017). Each captured area, I estimate this area to cover 45 × 50
probe was defined by two vectors, each containing 16 degrees (width × height) of the central visual field.
elements: an amplitude vector a and a phase vector The image-driven map was converted to grayscale, so
p. Different elements ai of a specified the different that the two maps were both defined by 2D matrices
amplitude values of an array of 2D gratings spanning of matched dimensions and aligned coordinates in
the entire orientation axis in 16 equally spaced steps. A visual space. They were downsampled to 90 × 100
similar convention applied to p. For example, a1 and p1 pixels (width × height) to achieve rapid image
specified the amplitude and phase of a grating oriented processing, which was implemented primarily using
at 0 degrees, a2 and p2 specified the amplitude and the OpenCV for Unity package (C# port). Each map
phase of a grating oriented at 11.25 degrees, a3 and p3 was subjected to edge detection via Sobel filtering. The
specified the amplitude and phase of a grating oriented resulting image-driven and object-driven edge maps
at 22.5 degrees, and so on in steps of 11.25 degrees (Figures 1D–E) were then used to identify two potential
until a16 and p16 specified the amplitude and phase of insertion points: an image-driven insertion point (red
a grating oriented at 168.75 degrees. Frequency was dashed circles) and an object-driven insertion point
fixed at ∼1.2 cycles/deg. The probe simply consisted (blue dashed circles). When describing the identification
of the sum of all gratings specified by a and p. Phase of an image-driven insertion point below, I refer to
was randomly drawn from a uniform distribution the image-driven edge map as “driver” map M↑ and
between 0 and 360 degrees for each element of p. to the object-driven edge map as “modulator” map
The amplitude vector a was obtained by summing M↓ . The role of the “driver” map is to direct the
two amplitude vectors n and s. Each element of the probe insertion point toward the information content
noise vector n was drawn from a Gaussian distribution represented by said map (image-based if the driver map
typically centered on 3% contrast with SD of 1% is defined by the image-driven edge map, object-based
contrast, although these exact values were adjusted if the driver map is defined by the object-driven edge
individually to target threshold performance (d ∼1). map). The role of the “modulator” map is to direct
Each element of the signal vector s was set to 0, except the probe insertion point away from the information
for the two elements associated with gratings oriented content represented by said map. Identification of an
at 0 degrees and 90 degrees: Either one was assigned a object-driven insertion point simply involves swapping
value of 40% contrast (the other element was set to 0). denominations for the two maps: The object-driven
When the probe was rendered, the 16 orientation values edge map becomes “driver” (M↑ ), and the image-driven
associated with the above specification were shifted edge map becomes modulator (M↓ ). Both maps were
by an amount equivalent to the orientation specified initially normalized to range between 0 (minimum
locally by the environment (see below for further details value within map) and 1 (maximum value within
on how this characteristic was determined): 0 degrees map). M↓ was blurred with a Gaussian kernel of SD
above corresponds to the orientation specified by equal to 5 pixels and sign-inverted as M↓ → 1 − M↓
the environment, which I term “congruent,” and 90 (0 corresponds to maximum value within original
degrees corresponds to the orientation orthogonal to map, and 1 corresponds to minimum value within
the orientation specified by the environment, which I original map). Each entry of M↑ was corrupted by
term “incongruent.” When rendered, average probe a noise source uniformly distributed between 0 and
luminance was ∼50 cd/m2 as estimated via indirect 0.2 to introduce a small degree of variability in the
photometric measurements (see above). insertion point for a given view. The algorithm then
set corner regions within M↑ to zero. These regions
were identified by subjecting M↑ to Harris corner
Probe insertion algorithm detection with block size 10, aperture parameter 3, and
The overall orientation of the probe was specified Harris detector-free parameter 0.04, to obtain a corner
by the local orientation defined by the environment map M . This map was thresholded to retain only the
at the location where the probe was inserted. I refer top 10% of its values, which were set to 0 while all
to said location with the term “insertion point.” remaining values were set to 1, and M↑ → M↑ × M .
Right before probe presentation, two maps of the The algorithm then combined M↑ and M↓ to obtain
environment were captured from the HMD: an the insertion map M = M↑ × M↓ . M was windowed
image-driven map, consisting of a 2D snapshot of the by a circular envelope to include only the central disc
scene viewed by the observer, taken by a cyclopean of diameter equal to 80% of image width, with a
camera positioned between the two eyes, and an tapering Gaussian edge of SD equal to 10% of image
object-driven map, consisting of a depth map of the width. The final probe insertion point corresponded
scene where the intensity of a given pixel scaled with to the pixel with maximum intensity within M. Local

orientation was defined as the orientation returned by consequences for their final performance score (see
the Sobel detector for that location within M↑ . On below).
each trial, the decision as to whether the probe would The potential role of feedback in determining the
be inserted at the image-driven insertion point or at strategy adopted by participants was not specifically
the object-driven insertion point was taken randomly explored in this study. Feedback was provided to
with equal probability, meaning that (on average) the ensure that the above monitoring procedure could
number of trials containing image-driven insertions be implemented, facilitate familiarization with task
matched the number of trials containing object-driven requirements, and prompt observers to maximize
insertions. their sensory performance. It is possible that it also
played a role in steering participants toward adopting
strategies that, had feedback been withheld, they may
Online correction of probe insertion point not have otherwise adopted. However, it is unlikely
that this phenomenon would be specifically connected
I estimated that probe insertion involved a delay
with the image-based/object-based distinction, because
of ∼60 ms with respect to the time at which image
participants were not aware of this distinction in
information was acquired for determining the insertion
relation to how the insertion algorithm operated. When
point (see above). During this time, observers could
feedback was incorrect on trials for which they were
move their head to face a slightly different scene,
confident that their response was correct, they never
causing potential mismatches between intended and
reported that they explicitly ascribed this inconsistency
actual insertions. To compensate for this spatial
to the probe insertion algorithm. Furthermore, any
misalignment, the correction algorithm calculated
potential impact of feedback is expected to remain
the difference in HMD direction between the time of
unchanged across shadow configurations, thus
image acquisition and probe insertion, and applied
mitigating the importance of this issue for the present
an equivalent shift of spatial coordinates to the
investigation. At the same time, the potential role of
nominal insertion point. For example, if observers
feedback remains an interesting and important topic
moved/rotated their head to the right, this would cause
for further inquiry in future work.
a leftward shift of the intended insertion point in
retinal coordinates. The correction algorithm therefore
modified the horizontal coordinate returned by the
insertion algorithm described above to reflect this
Memory task
shift in retinal image. Based on qualitative evaluation
The memory task was introduced to engage
of the insertion algorithm while wearing the HMD,
observers in active encoding of scene layout. After 100
this procedure improved the accuracy of the insertion
presentations of the sensory probe (approximately 5
algorithm. However, it did not completely eliminate all
min), position tracking was disabled and the HMD
misalignments, which occasionally occurred particularly
was teleported to a virtual location above the room
in the case of fast HMD movements.
that corresponded to the observer’s feet standing
on a transparent ceiling. Rotation tracking was left
active. From this position, observers could inspect
Feedback element the room immediately below their vantage point by
Feedback was delivered by a colored disc presented rotating their head (and moving their eyes), but they
at fixation for 140 ms, using the same 2D support could not navigate laterally. Cast shadows were absent
implementation techniques adopted for the probe in this configuration, regardless of which shadow
(HMD-facing orientation, independent illumination, configuration was adopted during the preceding block.
monocular presentation). The feedback disc had a fixed During teleportation, one of the 10 boxes in the room
diameter of ∼9 degrees and opacity of 50%. It appeared (randomly selected) was moved to the 11th box location
at the end of the response period (see above) and was (which had not been previously occupied by any box),
solid green when preceded by a correct response, solid without changing its size and rotation. This event
red when preceded by an incorrect response, and open was not seen by the observer. The displaced box and
red when preceded by an invalid response. On incorrect one other randomly selected box were highlighted by
trials, the red disc included a numeric digit placed at applying a blue tint to their texture, and were randomly
its center, which indicated the number of consecutive labeled with the letters “A” and “B” (see Figure 1K).
incorrect trials at that point throughout the block. This Observers were asked to identify the displaced box by
number was reset to 0 every time observers produced pressing one of two buttons on the controller (one
a correct response. Observers were provided with button for A, the other button for B). Immediately
real-time information about the number of consecutive after responding, feedback was provided in the form
incorrect trials because this parameter determined the of duplicating the displaced box at its original position
occurrence of a highly disruptive event with important and via application of specific color changes to the

boxes. The duplicate box was semi-transparent to points at the end of the block. If they choose to rely
distinguish it from the displaced box and to emphasize on the residual correct responses generated by chance
its belonging to a past event. In the event of a correct in the sensory task, they nevertheless run into the risk
response, a green tint was applied to the displaced box of producing several consecutive incorrect responses,
and to its duplicate. The two boxes were connected by a with the resulting room scrambling event nullifying
green element indicating the direction of displacement their efforts to memorize room layout up to that point.
in the form of an expanding strip (Figure 1L). In the Conversely, if observers only focus on the sensory task
event of an incorrect response, a red tint was applied to while ignoring the memory task, they risk halving their
the selected box, and a yellow tint was applied to the score (and missing out on potentially doubling it) at the
displaced box, to its duplicate, and to their connecting end of the block. Observers clearly understood these
element (Figure 1M). Observers were introduced to implications during data collection. As a consequence,
both sensory and memory tasks during brief training they never ignored either task.
sessions preceding data collection. They were therefore
aware that they would be required to perform the
memory task at the end of each block. Memory task without sensory task
In additional experiments, I removed all probe
Rules for preventing single-task strategies presentations and instructed observers to perform only
the memory task. Under these conditions, I found that
I introduced a set of rules specifically designed to the 5-min block duration adopted in the dual-task
ensure that observers would perform both sensory experiments resulted in ceiling performance for the
and memory tasks while inhabiting the virtual room. memory task (100% correct responses). To target
This design feature is important, because failure to threshold performance comparable with the levels
perform either task would significantly impact the achieved in the dual task (∼75% correct responses), I
behavior intended for the sensory task, which is reduced block duration to 30 s. Except for removing the
the task of primary interest for the measurements sensory probes, all other experimental manipulations
adopted here. More specifically, if participants were to were left unmodified.
focus on the memory task while ignoring the sensory
task, performance in the latter task would drop to
chance. Conversely, if participants were to focus on Estimation of sensitivity and response bias
the sensory task while ignoring the memory task, the
most productive strategy for performing the latter task I estimated sensitivity and response bias from valid
involves staring at a fixed region of the room, possibly a trials (I excluded trials with invalid responses) using
wall, to reduce uncertainty about probe appearance and the two metrics d and criterion c, computed using
facilitate sensory performance. This scenario would not standard formula from signal detection theory (SDT;
be representative of natural vision and would produce Green & Swets, 1966): d =−1 (phit ) − −1 (pfa ) and
redundant insertions with poor probing power, making c = −[−1 (phit ) + −1 (pfa )]/2, where phit is the hit
any associated results difficult to interpret in meaningful probability, pfa is the false alarm probability, and −1
ways. I therefore wished to avoid both scenarios and is the inverse of the normal cumulative distribution
prompted observers to perform both tasks at the same function. Our goal is to determine whether the sensory
time. process probed by the measurements is strictly visual
The experiment was presented to observers in the or decisional. I formulate these two positions in
form of a game with a point-based score. To ensure that the following (deliberately accentuated) terms: By
observers would not be able to ignore the sensory task, “strictly visual,” I mean that the sensory mechanisms
I introduced the following rules for point assignment. underlying edge detection/representation undergo
Each correct sensory discrimination counted 1 point. measurable alterations of their response characteristics,
At the end of the block, their total score was doubled such as changes in their orientation tuning and/or
if they responded correctly in the memory task selectivity for spatial frequency; by “decisional,” I
or halved if they responded incorrectly. After four refer to changes in the criterion adopted by observers
consecutive incorrect/invalid responses, all boxes within to produce a binary response. The latter possibility
the room disappeared and immediately reappeared in a must be explicitly entertained because the adopted
completely new configuration (using the algorithm for task conforms to the yes/no protocol (Green & Swets,
box placement/sizing/rotation detailed above). I refer to 1966), and the two possible responses (congruent
this event as “room scrambling.” vs. incongruent) are not symmetric: Under these
The above rules undermine single-task strategies conditions, it is entirely reasonable to expect response
for the following reasons. If observers only focus on bias.
the memory task while ignoring the sensory task, To arbitrate between the two scenarios outlined
their drop in sensory performance will result in fewer above, behavioral scientists typically estimate two

Figure 2. Sensitivity shifts in favor of image-driven information when shadow information is absent/unreliable. When shadow casting
is reliable (A), sensitivity (y-axis) and response bias (x-axis) for performing the sensory task (congruent/incongruent orientation
discrimination; see Figures 1A–J) are similar between image-driven (red symbols) and object-driven (blue symbols) probe insertions:
Data points scatter around a sensitivity (d ) value of 1 (indicated by horizontal green line) and no response bias (0 on the x-axis,
indicated by vertical dashed line). Diagonal arrows (top-right region of the plot) connect object-driven (arrow base) and image-driven
(arrow end) projections of sensitivity/bias values onto the negative diagonal for each observer. Arrow color is red if image-driven
projected value falls to the left of object-driven projected value, blue otherwise. When shadow casting is absent/unreliable (B–C),
image-driven and object-driven data clusters shift away from each other along the negative diagonal. This shift may be informally
characterized as a perceptual “flattening” effect (see main text). Contours reflect data spread using an automated
smoothing/thresholding procedure and are intended for visualization only (see main text for statistical analysis).
independent quantities (Green & Swets, 1966): changes Modeling

in d , which are meant to capture genuine changes in
the ability of the underlying sensory mechanism to The proposed model should not be intended as a
extract signal-relevant information from the stimulus, mechanistic explanatory account of the underlying
and changes in criterion c, which are meant to reflect cognitive processes. It is best intended as an illustrative
changes in the disposition of the participant toward tool for clarifying my interpretation of the results,
more lax (inclined toward reporting congruent) or which is based on cue reweighting (Hillis, Watt,
more conservative behavior (inclined toward reporting Landy, & Banks, 2004). In this sense, the proposed
incongruent). Estimation of these two quantities from implementation of the early visual processing stages
binary responses typically involves committing to the (e.g., edge detection, object identification) is neither
equal-variance SDT model. Because the equal-variance veridical nor exclusive, and the proposed model does
assumption is likely violated under many real-world not carry much explanatory power beyond the specific
scenarios (Green & Swets, 1966), the resulting d and conditions of the experiments reported here.
c estimates reflect mixtures of sensory and decisional
effects. In other words, these metrics cannot be
expected to achieve full orthogonalization of the
cognitive mechanisms they are meant to capture Stimulus/response information fed to the model
independently. The model was provided with the following data
To assess/address the crossover between sensitivity for every experimental trial (I excluded trials with
and response bias at the level of d and c metric invalid responses): a monochrome (average intensity
estimates, and determine whether the main effects are across color guns) image-based snapshot Mimage of
attributable to one or the other cognitive mechanism, I the scene right before probe insertion (Figure 4A), a
evaluated the size and consistency of the two metrics. depth-based snapshot Mdepth of the scene right before
As detailed in the Supplementary Text and shown in probe insertion (not shown), an image-based snapshot
Supplementary Figure S1, the data decisively support Mprobe of the scene during probe insertion (Figure 4F),
the conclusion that the cluster separation effects in and the binary congruent/incongruent response
Figure 2 are driven by sensitivity changes and not generated by the human observer (Figure 4G). The
changes in response bias. More specifically, I find that three input images were used by the model to generate
c estimates produce small and inconsistent effects, a congruent/incongruent response (Figure 4D), which
thus likely reflecting estimation errors deriving from was then compared against the response generated by
imperfect separation of sensitivity and response bias. the observer (see below).

Map construction
The image-processing strategy used for the model
largely mirrored the strategy adopted for probe
insertion (see above), albeit with some modifications
meant to reflect known properties of primary visual
cortex. The algorithm submitted Mimage to energy
extraction by a quadrature pair of Gabor wavelets with
carrier frequency equal to 1 cycle per 2% of image
width (equivalent to ∼1 cycle/degree when referred
back to visual space), SD of Gaussian envelope equal
to 2% of image width (equivalent to ∼1 degree), and
oriented at one of eight values uniformly spanning the
0–180 degree range. For each pixel across the image,
the algorithm selected the orientation corresponding to
the largest energy output to construct an image-driven
orientation map (red in Figure 4B). The algorithm
repeated this process for Mdepth and computed the
corresponding object-driven orientation map (blue in
Figure 4B). The algorithm also repeated this process for
Mprobe after halving the SD of the Gaussian envelope
and computed the corresponding probe orientation
map (green in Figure 4E). I reduced the spatial extent
of the Gabor filters (smaller SD) for the latter map to
reflect more localized processing of the probe, which
we expect to simulate the perceptual processes engaged
by observers. More specifically, we expect that, before
gaining knowledge of probe location, observers would
carry out relatively coarse orientation estimation
across the visual field (indicated by the magenta circle
in Figure 4A). When the probe appears, we expect
observers to engage in more localized processing of
the region around the probe (indicated by the green
circle in Figure 4F), hence the smaller SD value
adopted to simulate the latter process. This strategy is
effectively prompted by the nature of the sensory task:
Observers are essentially asked to compare the local
orientation of the probe with the overall orientation of
Figure 3. Absent/unreliable shadow configurations slowly drift the surrounding scene, because this comparison forms
away from reliable configuration. Difference in sensitivity the basis for the congruent/incongruent judgment.
(Sensitivity) between image-driven and object-driven
insertions (y-axis) is plotted for different time points (x-axis)
Decisional stage
during the 5-min experimental block over a 2-min sliding
window centered on each point. This quantity, which The algorithm computed the circular difference
corresponds to the distance between red and blue clusters between the image-driven orientation map and the
along the y-axis in Figure 2, is similar for different shadow probe orientation map. The resulting image-driven
configurations at the beginning of the block (leftmost data differential map contains orientation values between
points). As time progresses (moving rightward along the x-axis), 0 and 90 degrees. The algorithm also computed the
data for the absent/unreliable shadow configurations corresponding object-driven differential map (circular
(orange/magenta) depart from corresponding data for the difference between the object-driven orientation map
reliable-shadow configuration (black), until they diverge to
significantly positive values of Sensitivity toward the end of
the block (rightmost data points) that reflect the shifts reported ←
in Figures 2B–C. Histograms within insets plot Sensitivity regions within main panel indicate ±SEM (open symbols refer
values for different observers (one bar per observer) in the to data points for which SEM could not be computed because of
three shadow configurations (color-coded as detailed above), insufficient data); those within insets indicate 95% confidence
computed separately from two equal subsets of data intervals across observers. Solid curves plot univariate spline
corresponding to “early” and “late” halves of the block. Shaded fits of degree 2 and are intended for visualization only.
→

Figure 4. Trial-by-trial modeling of human responses via flexible cue combination. For every experimental trial, the model is provided
with image information captured directly from the HMD right before (A) and during probe insertion (F). It then constructs an
image-driven energy map (red regions within surface plot in B) with associated orientation map (red segments in B), and an
object-driven map (blue regions) with associated orientation map (blue segments). The model then combines these two orientation
maps via weight w (C): When w = 1, only the image-driven map is used; when w = 0, only the object-driven map is used.
Intermediate values of w involve mixtures of the two maps (see Methods). Based on this weighted combination, the model estimates
the overall orientation of a local region around the probe insertion point (indicated by magenta circle in A). Different values of w may
correspond to different orientation estimates (blue/red segments in D; hue value between blue and red reflects w value). The model
also constructs a descriptor from F: an image-driven energy map (green regions within surface plot in E) with associated orientation
map (green segments in E). The model then uses this orientation map to estimate probe orientation from the local region occupied by
the probe (indicated by green circle in F). This estimate is directly compared with the estimate obtained from A to determine whether
probe orientation is congruent or incongruent, which in general will depend on the value of w: For this example trial, w = 0
corresponds to an “incongruent” response, and w = 1 to a “congruent” response (D). The same procedure is applied to the next trial,
as shown in I–N. On this trial, the value of w does not impact the final response (L) because image-driven and object-driven
orientation maps agree within the probe region (J). Model responses are matched against human responses on corresponding trials
(G, O) to compute human–model agreement (fraction of matching responses) across all trials. Agreement (y-axis in H) reaches its
maximum for different values of w (x-axis in H) when computed from different shadow configurations (compare traces of different
colors). Histograms (top of H) plot distributions of w values that maximize agreement across multiple bootstrap iterations: They are
mostly distributed to the left of w = 0.5 (this value corresponds to equal weight between image-driven and object-driven
information, indicated by vertical green line) for the reliable shadow configuration (black histogram), and mostly to the right of w =
0.5 for absent/unreliable configurations (orange/magenta histograms). Shaded regions in H show ±1 SD across iterations.
and the probe orientation map) and combined the which represented a sizable proportion of the data
two differential maps via weighted averaging: Factor set (∼30%), were excluded from analysis. Attempts at
w was applied to the image-driven differential map, reducing this exclusion rate were unsuccessful because
and factor 1 – w was applied to the object-driven they required increasing the envelope size of the local
differential map (Figure 4C). I refer to the resulting filters and/or increasing the size of the pooling box
combined map as the incongruency map: It reflects described immediately above, which in turn produced
the degree of incongruency (orientation distance) extremely noisy estimates of local orientation content
between scene and probe. The algorithm averaged that did not support sensible outputs at the decisional
values from this map within a 3 × 3 box centered on stage.
probe location and produced an incongruent response
if the resulting value was >45 degrees, or a congruent
response otherwise (<45 degrees). On some trials, the Human–model agreement
generated orientation maps were not viable because the Human–model agreement is the fraction of trials on
corresponding energy maps (see above) contained no which the model generates the same response produced
energy within the selected pooling region. These trials, by human observers, over all experimental trials across

makers, 2011). I first carried out an exploratory two-way

repeated-measures analysis of variance (ANOVA) with
shadow pattern (reliable/absent/unreliable) and inser-
tion type (image-driven/object-driven) as factors and the
diagonal shift (projected values onto negative diagonal
returned by d −c) as the dependent variable. This anal-
ysis confirmed the presence of an interaction between
factors (p = 0.03), as suggested by visual inspection of
Figure 2 (I also found an effect of insertion type [p =
0.0034] but no effect of shadow pattern [p = 0.3], again
consistent with the pattern exposed by the visualization).
To identify which conditions produced measurable
shifts, I then carried out Wilcoxon tests on diagonal
shifts between image-driven and object-driven measure-
ments, separately for each shadow configuration. I ap-
plied a 3× Bonferroni correction to compensate for the
comparison across shadow configurations. The experi-
ments were designed so that the null hypothesis adopted
for the Wilcoxon tests would be transparently and un-
ambiguously defined as involving no difference between
two measurements of the same variable under two dif-
ferent conditions. When reporting data across individual
observers for subset analysis (insets to Figure 3 and
Figures 5B–C), relevant figures display 95% confidence
Figure 5. Memory performance is impacted by the sensory task, intervals across observers in the form of shaded regions.
but sensory performance is not impacted by the memory task. I adopt a combination of confidence intervals and
Recall performance in the memory task (y-axis in A) is at p-values to avoid the limitations associated with p-values
threshold levels for absent/unreliable configurations alone (Cumming, 2014; Wasserstein & Lazar, 2016).
(orange/magenta data points in A) but drops to chance for the
reliable configuration (solid black data point). When the
sensory task is removed (open symbols in A), performance is
similar across all shadow configurations. Differential sensitivity Results
in the sensory task follows similar patterns for blocks on which
participants responded correctly in the memory task (B) and for Brief overview
those on which they responded incorrectly (C). More
specifically, differential sensitivity is not different from 0 for the Before unpacking the details of my measurements
reliable shadow configuration (black histograms in B–C) and is and their implications, I summarize their most
positive for the absent/unreliable configurations (cyan important features here. Sensitivity for discriminating
histograms). Histograms are plotted to the conventions the local orientation of a brief probe embedded
adopted in Figure 3. in a virtual environment (Figures 1F–G) is plotted
on the y-axes of Figure 2. When object shadows
in the virtual environment are stable, sensitivity to
all observers. An agreement value of 0.5 indicates
image-based information is comparable with sensitivity
complete decoupling between model and human
to object-based information: Red data points largely
responses, while an agreement value of 1.0 indicates
overlap with blue data points in Figure 2A. When
perfect match. In general, values fell within a tight range
shadows are removed (Figure 1Q) or rendered unreliable
between 0.55 and 0.65. I computed agreement separately
by an erratic light source (Figure 1O), sensitivity to
for different shadow configurations and for different
image-based information is higher than sensitivity
values of w (Figure 4H). Agreement generally peaked
to object-based information: Red data points scatter
for intermediate values of w, between 0.3 and 0.7.
above blue data points in Figures 2B–C. This difference
in sensitivity, plotted on the y-axis of Figure 3, emerges
Statistical analysis slowly (timescale of minutes plotted on the x-axis) and
only for the absent/unreliable shadow configurations
To evaluate the statistical significance of the data (orange/magenta elements), not for the reliable shadow
pattern suggested by the visualizations in Figure 2, I configuration (black elements). These results are
followed accepted guidelines for statistical assessment successfully captured by a model that allocates more
of interactions (Nieuwenhuis, Forstmann, & Wagen- weight to image-based cues versus object-based cues

when shadow information is degraded (Figure 4). for details). During teleportation, one box in the room
Overall, these results demonstrate that slow and was displaced to a different location (Figure 1C).
global characteristics of the environment (shadow Observers were informed that this event would take
distribution/reliability) can impact the operation of fast place, but they did not know which box would be
and local sensory processes (orientation discrimination displaced. As they looked down onto the room, 2 of the
of transient localized target). 10 boxes were highlighted and labeled with the letters
A and B (blue-tinted boxes in Figure 1K). One of the
labeled boxes had been moved (box labeled A in the
example shown in Figure 1K), while the other one was a
Image-based versus object-based cues for visual randomly selected box (box labeled B). Observers were
discrimination asked to indicate which, of those two boxes, had been
displaced. The memory task was introduced to engage
Participants operated within a virtual room observers in explicit reconstruction of environmental
containing different boxes at random locations. This 3D layout (this process is not necessarily engaged by the
environment was designed to present numerous sensory task). Observers were therefore simultaneously
and varied edges, alongside a method to quantify performing two separate tasks on completely different
whether each edge is generated by “image-based” or spatiotemporal scales: the sensory task, reliant on local
“object-based” elements. For example, a high-contrast (retinotopic) fast information (subsecond timescale),
edge lining the wallpaper (region indicated by red and the memory task, reliant on global (room-wide)
dashed circle in Figure 1A) is image-based but not information from slow exploration (several seconds to
object-based; conversely, the transition between a box minutes).
and its background (region indicated by blue dashed
circle in Figure 1A) may carry no image-based contrast
but marks an object-based edge. I locally perturbed
edge regions via the insertion of Gabor-like probes In the presence of reliable shadows, both cues
(Figures 1F–H) and asked observers to discriminate are used for discrimination
probe orientation (“sensory task”). More specifically,
observers were asked to determine whether probe Figure 2A plots sensitivity (d ) for discriminating
orientation was congruent (aligned) or incongruent probe orientation (congruent vs. incongruent) on
(orthogonal) with the orientation locally specified by the y-axis, against response bias on the x-axis (see
the environment (Figures 1I–J). Probes were selectively Methods for details on how these quantities were
inserted at either image-based or object-based edges. computed). Data points fall within optimal ranges
When they were inserted along an image-based edge for performing accurate behavioral measurements:
(region indicated by red dashed circle in Figure 1F), Sensitivity falls around 1, and bias values scatter
the congruent orientation was specified by image-based around 0. This is the sweet spot targeted by traditional
information in the environment (Figure 1D). Similarly, psychophysical experiments, but it is not to be taken
when probes were inserted along object-based edges for granted that such conditions would be achieved
(region indicated by blue dashed circle in Figure 1G), in VR. Given the large number of extraneous factors
the congruent orientation was specified by object-based that inevitably intervene in this medium, there are
environmental information (Figure 1E). Probes numerous reasons for expecting strong response bias.
appeared briefly every few seconds (probe events are Furthermore, given the difficulty of performing a
indicated by green vertical segments along the timeline low-level discrimination task in a complex environment
in Figure 1N). Over a period of approximately 5 min, while concomitantly being challenged with a demanding
observers therefore performed 100 instances of the high-level cognitive task (memory task), it is not at
sensory task. I refer to each instance as one “trial” and all trivial that observers would be able to achieve
to the 5-min period as a “block.” sensitivity levels substantially above chance. Finally, the
While performing the sensory task, observers were technical procedure for inserting probes must operate
also simultaneously engaged in a separate “memory on the fly (see Methods), which may lead to alignment
task.” In the memory task, observers were asked to failures and real-time glitches that may preclude
memorize the layout of the room for the purpose of effective performance of the sensory task. Overall, the
answering a recall question at the end of each block. data in Figure 2A demonstrate that these potential
When the block finished, observers were teleported difficulties were successfully overcome by the adopted
above the room (blue elements in Figure 1B) so that protocol.
they could view the space they previously occupied from When we compare the above performance metrics
a different vantage point (as seen in Figure 1K). Their between insertions that were image-based (red data
end location was such that they were standing on the points) and those that were object-based (blue), we find
ceiling, which was rendered transparent (see Methods that they are largely comparable (red and blue data

clusters overlap in Figure 2A). This result demonstrates observers viewed the shadowless environment as being
that observers were able to retrieve edge information “flatter” (Tarr et al., 1998). Again, I emphasize that
from the scene with reference to both texture elements this characterization is rather imprecise, because it
and object boundaries. Using imprecise terminology, fails to capture a number of important points about
we could rephrase this result by stating that observers the measurements reported in Figures 2A–B. More
viewed the environment as both “flat” and “in-depth.” specifically, observers were never asked to explicitly
These terms do not entirely capture the nature of the rate the appearance of the environment in terms
phenomena under study and may to some extent even of its flatness/depthness. Rather, they were asked to
misrepresent them. However, they are useful for an perform a low-level visual discrimination task with
intuitive summary of the data, which motivates us to reference to a specified local signal (Figures 1H–J). The
occasionally adopt them here. resulting measurements are therefore properly visual
and of the quantitative kind that is normally accessible
only via well-controlled psychophysics. Because these
measurements can be referred back to the complexity
In the absence of shadows, the world becomes of the scene, however, they carry implications for
flat interpreting the role played by the virtual environment.
I repeated the above measurements in the absence of

cast shadows (Tarr, Kersten, & Bülthoff, 1998). Under
these conditions, the environment is uniformly lit from Sensitivity shift is connected with shadow
within, without any point light source (see example reliability and not an artifact of low-level
in Figure 1Q). It remains in clear relief: Stereoscopic differences
information is still available, in addition to pictorial
cues about depth. However, there are no shadows. It may be argued that any comparison between
Phenomenologically, observers do not experience much Figure 2A and Figure 2B is saddled with serious
difference with respect to the “reliable” configuration interpretational challenges associated with the many
tested above (Figure 1P), in which shadows are present differences between shadow-rich and shadowless
and consistent with a fixed sun-like source. environments: When shadows are removed, the
In the absence of shadows, sensitivity/bias for associated visual changes span a wide range of
image-based versus object-based probe insertions dimensions and cues, such as spatial frequency, edge
falls within different regions of the plot in Figure 2B: density, orientation content, and potentially many
Sensitivity is higher for image-based insertions (red others. From a low-level perspective, there are too
data cluster is shifted upward compared with blue data many cues changing at the same time to allow informed
cluster), and bias is slightly more lax (red data cluster statements about the origin of the shift discussed
is shifted leftward compared with blue data cluster). above. On the one hand, it is possible that this shift
This upward-to-the-left shift can be quantified by reflects an interesting phenomenon pertaining to
projecting data coordinates onto the negative diagonal. higher-level reconstruction and interpretation of
The resulting values are indicated by red/blue arrows, environmental layout: Our visual system must sort
which clearly demonstrate the shift associated with the out edges into “structural” ones belonging to objects
object-based→image-based transition (p = 0.0058; and “nonstructural” ones deriving from shadows or
the same analysis applied to Figure 2A returns p = textures (Marr, 1982; Neri, 2017). The operation of this
0.12; see Methods for details on statistical tests). This process would be impacted by shadow manipulations.
shift is in fact dominated by a change in sensitivity, On the other hand, it is equally possible that the shift
while the concomitant shift in bias likely reflects is caused by any of several low-level image differences
imperfect dissociation between these two metrics (see between the two environments, such as edge density
Methods and Supplementary Figure S1 for further (Bex, Solomon, & Dakin, 2009). Based on the data in
quantification). Figures 2A–B, one cannot say for sure.
When we compare Figure 2A with Figure 2B, we To address the above criticism, I designed an
find that data clusters are pushed away from each other additional condition, which I term the “unreliable”
along the direction of higher sensitivity for image-based shadow configuration. In this variant of the experiment,
insertions and slightly lower sensitivity for object-based shadows are present and consistent with a single light
insertions. In other words, removing shadows prompts source, but only for a period of a few seconds at a time:
observers to increase their reliance on image-based cues, Every 3–4 seconds, and just before the probe is flashed,
to the detriment of object-based cues (I formalize this the light source is teleported to a different location
concept further below with computational modeling). around the room (Figure 1O). As a consequence,
Adopting the intuitive terminology introduced the shadow pattern projected onto the environment
earlier, we could summarize this result by stating that immediately changes to a completely new configuration:

It is as if the sun jumped to a different part of the sky Sensitivity shift is not an artifact of sudden
every few seconds. environmental changes
A critical feature of the design presented above is
that, over the timescale of the sensory task, there is Although the unreliable-shadow configuration
no difference between reliable and unreliable shadow allows for the exclusion of potential low-level artifacts
configurations: In both cases, observers experience associated with the absent-shadow configuration,
a shadow pattern that is consistent with the natural it raises concerns with relation to other potential
laws of everyday lighting (compare leftmost scene in low-level artifacts of its own: In the unreliable-shadow
Figure 1O with scene in Figure 1P). More importantly, configuration, observers experience a sudden
overall there are no low-level differences in visual environmental change preceding the appearance of
stimulation between these two configurations: If the sensory probe, while they do not experience any
one were to take a snapshot of the room under the such event in the reliable-shadow configuration. It
two configurations at any given time, it would be may be argued that this event introduces an aspecific
impossible to tell them apart. On the longer timescale difference between the two shadow configurations that
of several seconds, however, the reliable configuration is unrelated to shadow reliability and may drive the
is associated with a stable shadow pattern that is data shift in Figure 2C. There are at least two ways of
consistent with everyday experience, while the unreliable formulating this hypothesis.
shadow configuration does not allow for a dependable First, the environmental change may act as a
representation of light projection from the sun, as this disruptive event that destabilizes perceptual processing,
light source hops around in a fashion that cannot be leading to an imbalance in the way sensory cues are
experienced in the natural world. combined by observers. Under this interpretation, any
Based on the above design considerations, we can sudden disruptive change would generate the data
make specific predictions about the manner in which pattern in Figure 2C, without requiring any specificity
data should scatter for image-based versus object-based for shadow statistics. For example, one may apply a
probe discrimination. If the shift reported in Figure 2B brief “earthquake” to the room and make all boxes
is entirely caused by low-level image differences between wobble. This manipulation would not impact shadow
reliable and shadowless environments, we should reliability but would cause some degree of disruption.
not observe such shift when shadows are unreliable: Second, the environmental change may act as a
Under this configuration, there are no low-level image cueing event that warns observers about the upcoming
differences from the perspective of the fast local appearance of the sensory probe. The resulting
process that is engaged by the sensory task, and the reduction in temporal uncertainty may underlie the
experimental pattern should mirror Figure 2A. If, on the shift in Figure 2C. Under this interpretation, any
other hand, the shift in Figure 2B reflects the operation stimulus that is time-locked to the probe would produce
of a higher-level process that attempts reconstruction the same effect, for example, an auditory cue.
of 3D layout based on complex considerations about We can reasonably exclude a role for the
object placement and light casting, we may expect to factors discussed above via reference to the results
measure a similar shift for unreliable shadows. Under obtained from the shadowless configuration: In this
this scenario, rendering shadows unreliable is not too configuration, the appearance of the sensory probe
dissimilar from removing them altogether: In both is not preceded by any disruptive/cueing event, yet
cases, information about shadow casting is degraded. we observe the same data shift that is observed in
Under this scenario, the experimental pattern should the unreliable-shadow configuration (Figure 2B).
conform with the shift reported in Figure 2B. We conclude that the measured differences between
As demonstrated in Figure 2C, the empirical reliable- and absent/unreliable-shadow configurations
measurements support the latter prediction: The cannot be readily ascribed to the low-level differences
image-based data cluster is shifted away from the introduced by the unreliable-shadow manipulation. At
object-based data cluster (p = 0.0088), similarly the same time, we must recognize that the similarity
to Figure 2B. It appears that observers process the between the two sets of measurements obtained
environment as “flatter” when shadows are unreliable, from absent- and unreliable-shadow configurations
even though its visual appearance has not changed. (Figures 2B–C) does not in itself guarantee that they
What does change is the long-term statistical reliability reflect the same underlying cognitive mechanism: It
of shadows. This factor is somehow incorporated by remains possible that they originate from entirely
observers over time. It is then reverberated back to the different mechanisms, which nevertheless map to the
sensory process that supports local discrimination of same experimental signature. We cannot categorically
flashed targets, to the extent that the resulting effect can exclude this possibility, but the more parsimonious
be measured via behavioral metrics. From whichever interpretation appears to involve one and the
perspective they are viewed, whether conceptual or same mechanism, thus making it unlikely that the
experimental, these effects are not trivially expected. factors discussed above played a role in determining

the pattern observed for the unreliable-shadow Figure 3 shows that, at the beginning of the block
configuration. (zero on the x-axis), the three shadow configurations
produce similar results: The corresponding three traces
show little difference (y values slightly above zero)
between image-based and object-based insertions. As
Impact of environmental stability on low-level time progresses along the x-axis, the absent/unreliable-
visual discrimination takes minutes to develop shadow configurations (orange/magenta) depart from
the reliable-shadow configuration (black) to produce
In principle, we may expect the impact of the the data arrangement shown in Figure 2: separated
unreliable-shadow manipulation to appear within a red-blue clusters for absent/unreliable configurations
few seconds of each block: After observers experience (Figures 2B–C), corresponding to positive values for
the first sudden change in sun location, they may differential sensitivity on the y-axis in Figure 3 (trailing
become aware that shadows are not stable indicators end of orange/magenta traces); overlapping clusters
of environmental layout. At the same time, further for the reliable-shadow configuration (Figure 2A),
consideration raises the possibility that this impact corresponding to near-zero values for differential
may be delayed, for at least two important reasons. sensitivity in Figure 3 (black trace). Notice that this
First, it is natural for observers to incorporate shadows transition occurs over a timescale of minutes: It takes
into their representation of the environment as an approximately 2 min before the three traces separate to
automatic process, and one that they engage almost a substantial degree in this plot.
instantly as shadows are recast following sudden To assess the statistical reliability across observers
displacement of the light source. Therefore, even if of the effect demonstrated above, I split the data set
they experience repeated disruption from the changing into two halves: The early epoch, corresponding to
light source, observers will nevertheless engage in active the first half of the block (first 2.5 min), and the late
reconstruction of room layout that takes shadows into epoch, corresponding to the second half of the block
account, simply because this is what their visual system (remaining 2.5 min). Differential sensitivity values are
is built to do. Second, the experiments in this study do not substantially different from zero during the early
not measure the impact of environmental stability on epoch (see confidence intervals around individual
the perception of environmental stability: Observers observer data for inset histograms on the left side of
were not asked to explicitly rate shadow reliability, for Figure 3, indicated by shaded regions), while they stay
example. The adopted protocols measure the impact clear of the zero line for absent/unreliable-shadow
of this manipulation on sensory discrimination of configurations during the late epoch (orange/magenta
localized probes. The question of whether and how insets on the right side of Figure 3). The results of
these two processes interact, and on what timescale they this analysis support the observations made earlier
may do so, remains completely open. in relation to the slow dynamics of the effects under
To gain some insight into the above question, I scrutiny (notice that this analysis does not depend
repeated the sensitivity analysis separately for different on the choice of sliding window adopted for the
epochs throughout each block: I applied a sliding visualization in the main panel).
window over the duration of each block and computed
sensitivity values for each window separately (similar
trends are visible for window sizes between 20 s and
2 min, except for inevitably noisier traces with shorter Does image-based sensitivity increase, or
window durations). In this way, we can track how object-based sensitivity decrease, or both?
sensitivity changes throughout the 5-min duration of
each block. The y-axis in Figure 3 plots the difference A relevant question at this stage is whether the
in sensitivity between image-based and object-based increase in differential sensitivity for the orange/magenta
measurements: The upward shift between red and traces in Figure 3 reflects an increase in image-based
blue data clusters from Figure 2. As we have seen sensitivity with no increase in object-based sensitivity,
previously, this is the metric of interest for studying a decrease in object-based sensitivity with no decrease
the impact of shadow manipulations on sensory in image-based sensitivity, or a combination of these
measurements. We can therefore map Figure 2 and/or similar effects. Answering this question is not
directly onto Figure 3: Overlapping clusters (as in straightforward, because absolute sensitivity varies
Figure 2A) correspond to near-zero values on the greatly across observers and fluctuates substantially
y-axis of Figure 3, while separated clusters (like those over the duration of a given block, most likely as
in Figures 2B–C) correspond to positive values in a consequence of differences in task engagement,
Figure 3. The important difference between Figure 2 alertness, and attention. By computing differential
and Figure 3 is that, with the latter, we can study how sensitivity, these effects are largely factored out, making
this effect evolves throughout the block. the differential measurement more robust/reliable. For

example, different participants often produce very model that may at least capture some important aspects
different patterns of sensitivity over the duration of of the experimental setting we are considering here.
the block: In one participant, sensitivity may decrease Furthermore, this model may support a type of data
for both image-based and object-based cues over the analysis that accounts for factors overlooked by the
duration of the block, while in another participant, sensitivity/bias measurements. I illustrate one such
both measurements may increase. However, the factor below.
difference between image-based and object-based Consider the image in Figure 4A. This image shows
sensitivity may show similar patterns between the two a scene actually viewed by one of the observers during
participants. the experiments (as seen from inside the HMD). On
For the above reason, when the data are analyzed this occasion, the probe was presented at the location
at the level of individual observers (as in the insets indicated by the magenta circle. If one were to estimate
to Figure 3), no clear trend emerges. If we ignore the orientation defined by the environment at/near
interindividual differences and only consider aggregate that location, the answer would be substantially
traces of sensitivity over time (such as those in different depending on the type of information that
the main panel of Figure 3), and if we measure is used to carry out the estimate. If we construct an
time-dependence in the form of correlation between object-based map of the scene, the orientation defined
sensitivity and time, the following trends are suggested by the environment around the location of probe
by the data: Image-driven sensitivity increases for the insertion is near-vertical. This assessment is based
absent/unreliable-shadow configurations (correlation on the presence of a clear object-based boundary
coefficients of 0.46/0.49) and decreases for the defined by the box on the right (blue map/segments
reliable-shadow configuration (correlation coefficient in Figure 4B) but runs in opposition to the indication
of −0.56); object-driven sensitivity decreases for the provided by image-based information: On the basis
unreliable-shadow configuration (−0.91). The above of image information alone, and disregarding object
trends return minuscule p-values (all <10−6 ) for a test of structure, the orientation defined by the scene at/around
the null hypothesis of lack of correlation under normal probe location is predominantly horizontal (red
distribution (see Kowalski [1972] for a discussion of map/segments in Figure 4B). On this trial, observers
various issues associated with this type of test), but we are therefore expected to produce substantially
must be cautious in interpreting those values because different responses depending on whether they rely on
they inevitably depend on the choice of sliding window object-based or image-based information to make their
adopted for computing the time traces, and they do not judgment.
reflect significant structure at the level of individual Consider now Figure 4I. This image shows a
observer analysis. different scene. Here, the orientation defined by
From the above, we must conclude that the data the environment at probe location is unmistakably
set associated with this study does not carry enough horizontal, regardless of which information is used
resolving power to conclusively answer the question to assess it. When we generate object-based and
of whether/how absolute sensitivity changes over time image-based maps from the scene, they both assign
for image-based and object-based characteristics. horizontal orientation to that location (Figure 4J). On
Furthermore, even if one were to collect more data, it this trial, observers are therefore expected to produce
remains unlikely that this question may be answered the same response, regardless of which cue information
decisively as a consequence of large fluctuations in (whether image-based or object-based) they use. As
engagement/attention across participants and over time a consequence, trials of this kind are less informative
for the same participant, if not by concluding that about observer strategy than trials of the kind examined
no consistent trend can be identified that adequately in the previous paragraph. When computing sensitivity
captures behavior at the population level. values from the data, however, all trials are treated in
the same way, without differentiating between those
that are more similar to Figure 4A and those that are
more similar to Figure 4I. Furthermore, the sensitivity
Qualitative model based on flexible cue calculations do not take into account trial-by-trial
combination fluctuations associated with orientation noise within
the probe (see Methods): On different trials, the physical
We currently lack sufficient knowledge to construct orientation energy carried by the probe may differ
an explicit computational model of the observer substantially from its target orientation (aligned with,
performing sensory/memory tasks while freely or orthogonal to, the congruent orientation). This
navigating a complex environment. Even if we trial-by-trial variation must impact the sensory task
attempted to design such a model, it would require to varying extents, but it is entirely overlooked by the
specification of so many parameters that the available sensitivity calculation.
data set would not be able to constrain it effectively. To address the above issues, and possibly use them
We can, however, restrict our efforts to a simplified as starting point for gaining further insight into the

strategy adopted by observers, I implemented a simple model depicted in Figure 4, we may interpret shadow
model that essentially retraces the logic laid out in availability/reliability as a higher-level factor controlling
previous paragraphs. The model operates in three steps: w and therefore adjusting perceptual reliance on
First, it estimates local orientation from the noisy image-based versus object-based cues in the scene.
probe presented on a given trial; second, it estimates the The location of the peak in human–model agreement
congruent orientation defined by the scene on the same for the reliable-shadow configuration (black trace in
trial; and third, it compares the two estimates to make Figure 4H), which is shifted toward greater weight on
a determination as to whether the probe is congruent object-based cues (w < 0.5), may lead to the reasonable
or incongruent on that specific trial. These steps rely expectation that the model should display greater
on three actual snapshots of the scene, taken on every object-based sensitivity than image-based sensitivity for
trial during the experiments. One snapshot (Figure 4F) this configuration. This pattern would be inconsistent
is used to estimate the absolute orientation of the with the human measurements, which do not show
local region occupied by the probe (Step 1 above), significant differences between image-based and
without reference to whether it may be congruent or object-based sensitivity (Figure 2A). The connection
incongruent. The remaining two snapshots (one is between w and model sensitivity is, however, not
shown in Figure 4A) are used to construct image-based straightforward, in the same way that the connection
and object-based maps for the scene, as it appeared to between w and human–model agreement is not obvious:
observers immediately preceding probe presentation Many factors contribute to this connection, such as
(see Methods for details). The model must then produce trial-to-trial variations in the reliability of image-based
an estimate of “congruency” (Step 2 above) to serve versus object-based cues, response bias, and differences
as reference orientation against which to evaluate in the efficacy of the selected model parameters for
the orientation of the probe (Step 3). This estimate extracting the two cues. For example, even though equal
is constructed by combining separate estimates from weight may be allocated to the two cues (w = 0.5),
image-based and object-based maps (Figure 4C). object-based cues may be noisier than image-based
The weighted combination is controlled by factor w: cues, so that object-based sensitivity may be lower than
When w = 0, only object-based information is used to image-based sensitivity. Under this scenario, w = 0.5
define “congruent” (vertical blue segment on the left of would not transparently translate to equal sensitivity for
Figure 4D); when w = 1, only image-based information the two cues. In the simulations, the model sensitivity
is used (horizontal red segment on the right of values associated with peak human–model agreement
Figure 4D). For a given value of w, the model proceeds were virtually identical for the reliable-shadow
to define an estimate for the congruent orientation and configuration, differing by less than 1% between cues.
compares it against the estimate obtained from the For the other two shadow configurations, image-based
probe. Based on this comparison, it produces a binary sensitivity was 40%/58% (absent/unreliable) higher
congruent/incongruent response (Figure 4D) that is than object-based sensitivity. These results mirror the
directly comparable with those produced by human human measurements (Figure 2) and provide a useful
observers (Figure 4G). illustration of the difficulties involved in establishing
We can then fix all model parameters except w a transparent connection between specific cognitive
(see Methods) and identify w values that maximize constructs, such as cue weight allocation, and final
the match between human and model responses on behavioral metrics, such as sensitivity. In a sense, those
a trial-by-trial basis (agreement). Figure 4H shows difficulties justify modeling exercises such as the one
that, as w varies along the x-axis, the corresponding presented here, because they equip the formulation of
human–model agreement (y-axis) for the reliable- cognitive theories with more tangible tools that can be
shadow configuration (black trace) reaches a peak tested and evaluated in software.
for w < 0.5 (to the left of vertical green line), Notwithstanding the utility of implementing
indicating that slightly more weight is allocated to simulated observers of the kind exemplified by the
object-based cues compared with image-based cues. tools in Figure 4, I emphasize that the model proposed
In the absent/unreliable-shadow configurations, the above is not really a model, for several reasons. First
optimal w values are >0.5 (orange/magenta traces and foremost, it does not provide a mechanistic
peak to the right of vertical green line), indicating that account of how shadow information is incorporated
more weight is allocated to image-based cues. These into scene reconstruction or, for that matter, of scene
results are broadly consistent with those obtained reconstruction itself. It merely produces w values that
from the sensitivity/bias measurements (Figure 2; see project to trial-by-trial responses in close vicinity of
next paragraph for further consideration of specific the human data. In this sense, it represents little more
features that do not conform to this statement). At the than a fitting exercise, rather than a computational
same time, they offer a more tangible interpretation for model with genuine explanatory power. An additional
those effects in terms of weight allocation to different limitation comes from the many arbitrary choices
sources of visual information. If we subscribe to the that went into designing specific steps, such as the

spatial extent over which map information is pooled shadow configurations. In the Discussion, I propose
to generate an orientation estimate for the congruent an interpretation of this finding based on prior results
reference. For these reasons, the proposed model from the VR literature.
should not be regarded as a critical component of A relevant question at this stage is whether the
the conclusions we draw from the data or as an above-documented interaction between shadow
instrument for adding genuine depth of understanding configuration and task performance is determined
to this study. It should be intended in the spirit of an by these two factors alone, or whether there may be
interpretational framework for illustrative purposes. more complex effects at play. For example, we may
In essence, it amounts to rephrasing the intuitive query the role of the sensory task: What happens when
terminology adopted earlier (flatness/depthness, observers are not engaged in this task, and therefore
object-based→image-based shift) using more tangible able to devote their entire efforts to the memory
concepts. task? To answer this question directly, I repeated the
experiments in the absence of the sensory task: Probes
were removed altogether, and observers were only
asked to identify which box moved at the end of each
Memory task and sensory task interact in block. Because the memory task can be performed
complex ways more easily under these conditions, it was necessary
to reduce block duration substantially (below 1 min)
The focus of this study is centered primarily on to achieve threshold performance (see Methods). The
the sensory task: Our goal is to understand whether resulting measurements show no difference across
and how sensory processing may be impacted by shadow configurations (open symbols in Figure 5A):
complex behavioral phenomena that are not normally Performance in the reliable-shadow configuration
understood as relevant to the acquisition of proximal recovers to the level measured for the other two
sensory information. In this context, the memory task configurations. It therefore appears that the sensory
does not play a critical conceptual role, but it does fulfill task can impact the memory task in complex ways,
an important practical role: It prompts observers to by only affecting configurations with reliable shadow
engage in active reconstruction of scene layout. For information.
this reason, not only observers were asked to perform The above results indicate a measurable impact of
two tasks at the same time, but additional task rules the sensory task on the memory task. We may wonder
were specifically designed to ensure that ignoring one about the opposite direction: Does performance in the
task would impact performance in the other task (see memory task impact the sensory task? We can attempt
Methods). to answer this question by splitting blocks into those
Notwithstanding our focus on the sensory task, for which observers produced a correct recall response
the memory task equally supports an explicit measure and those for which the recall response was incorrect.
of performance (percentage of correct responses in When comparing different shadow configurations
a two-alternative forced choice). At the same time, following this split, however, we find mismatched
it comes with a much smaller data mass: For every data mass for the different configurations: Because
block, I recorded 100 measurements from the sensory the reliable-shadow configuration was associated with
task and only 1 measurement from the memory task. chance performance, the split is roughly even, while the
Because of this sizable reduction in data mass, we other two shadow configurations are associated with
cannot analyze results from the memory task with the uneven splits of 3/4 to 1/4 (correct to incorrect). To
same level of detail afforded by the sensory task. More ensure that data mass is more appropriately matched
specifically, we can only examine values aggregated when comparing shadow configurations, I combined
across observers, with no insight into interobserver absent-shadow and unreliable-shadow configurations
variability/consistency. Notwithstanding this limitation, into one configuration (absent/unreliable).
we observe internally consistent patterns that also Figures 5B–C plots differential sensitivity for both
present interesting peculiarities. absent/unreliable- and reliable-shadow configurations.
Figure 5A plots the percentage of correct box As expected, this metric is near zero for the reliable-
choices for the three different shadow configurations. shadow configuration (black elements in Figures 5B–C),
We first notice that performance never reaches and positive for the absent/unreliable configuration
ceiling. This result provides direct evidence that the (cyan elements). More importantly with regard to the
memory recall task was sufficiently difficult to require specific question we are presently asking, this difference
active engagement on the part of observers, as it persists regardless of performance in the memory task:
was specifically designed to do. Perhaps surprisingly, Estimates in Figure 5B, obtained from blocks on which
performance was at chance (50%) for this task in observers responded correctly to the memory task, are
the reliable-shadow configuration, while it reached virtually identical to those in Figure 5C, obtained from
threshold values around 75% for the absent/unreliable- blocks on which observers responded incorrectly. We

therefore find no connection between success/failure factors such as shadow casting). This is not to say
in the memory task and shadow effects in the sensory that shadows are useless for inferring environmental
task. A more direct approach to exploring the potential structure — to the contrary, they can provide important
impact of the memory task on the sensory task information about 3D shape and other properties
would involve complete removal of the memory task. of the scene (Kersten, Mamassian, & Knill, 1997;
Unfortunately, this manipulation would severely alter Mamassian et al., 1998; Dee & Santos, 2011). The
the general exploratory behavior engaged by observers process by which this information is inferred from
in the arena (see Methods for a detailed explanation), shadows, however, cannot be merely characterized as
which would in turn make it impossible to obtain detection and involves more complex knowledge about
interpretable measurements from the sensory task. the environment (Gibson, 1979; Cavanagh, 1991). From
the viewpoint of early extraction of elementary features
from the proximal image (Morgan, 2011), shadows (and
more generally image rendering under different lighting
Discussion conditions) pose serious challenges for recovering
scene structure (Rensink & Cavanagh, 2004). When it
The most significant result of this study is the comes to shadows, these challenges are encountered
demonstration that complex, long-term environmental all the time during natural vision. For the above
statistics is implicitly incorporated into early sensory reasons, shadows have played a dominant role in early
processing by freely navigating human agents. In thinking about biological vision from a computational
this context, early sensory processing refers to visual standpoint (Marr, 1982), and they represent a natural
extraction of elementary features, such as edges and tool for interrogating the higher-level properties of
bars in the scene (Marr, 1982; Morgan, 2011). As a scene reconstruction. These considerations have shaped
consequence, the results presented here pertain to the the design of the experiments.
point of contact between two extremes of sensory In approaching existing literature of potential
behavior: on the one hand, immersive exploration of relevance to this study, I notice that (to my knowledge)
3D environments for spatial retrieval of scene layout; no prior study has attempted to characterize early
on the other hand, local analysis of the retinal input vision via low-level psychophysical measurements while
for proximal image reconstruction. Below I elaborate observers actively engage in virtual scene exploration.
further on this result, its connections with existing However, several existing studies have examined either
literature, and its implications for understanding one of those two aspects of sensory processing. Below I
sensory processes during natural behavior. select and discuss studies that are most directly relevant
Before embarking on a detailed discussion of specific to the present one.
issues and relevant studies, I address the theoretical Previous work has provided useful knowledge
motivation behind my choice of experimental design. about detection/discrimination of localized wavelets
Perhaps the most obvious question in relation to in the presence of complex contexts, such as those
the adopted paradigm is: Why shadows? There are experienced during natural vision. If we adopt the
many other environmental characteristics that one terminology introduced earlier, existing evidence may
may choose to manipulate, such as spatial consistency be characterized in the following terms: detection is
(Warren, 2019; Muryy & Glennerster, 2021) or primarily determined by image-based cues (Sebastian,
texture composition (Isoyama, Sakuragi, Terada, & Abrams, & Geisler, 2017), while discrimination is
Tsukamoto, 2021). My motivation for manipulating primarily determined by object-based cues (Neri,
shadows stems from the pivotal role played by this scene 2017) (with some exceptions; Teufel, Dakin, &
component in formulating early theories of biological Fletcher, 2018). For example, efficiency for detecting
image reconstruction (Gibson, 1979; Marr, 1982). targets/distortions within local regions of natural
Shadows perfectly exemplify the difficulties involved in scenes can be satisfactorily accounted for on the basis
successfully segmenting natural images (Neri, 2017): On of low-level image properties (Sebastian et al., 2017)
the one hand, they cast sharp edges across the scene, such as edge contrast (Bex, Mareschal, & Dakin,
thus generating salient features that vigorously drive 2007), density (Bex et al., 2009), and phase (Rideaux,
local edge detectors (Morgan, 2011); on the other hand, West, Wallis, Bex, Mattingley, & Harrison, 2022).
they do not directly inform the visual system about When observers are required to engage in orientation
fundamental properties of the environment, such as discrimination for determining congruency, however,
object boundaries (Bansal, Kowdle, Parikh, Gallagher, other factors come into play, many of which are
& Zitnick, 2013). not directly connected with low-level image-based
In a sense, the most burdensome job for the early content (Neri, 2017). These factors range from those
visual system is to sort out edges into whether they are associated with object segmentation (Neri, 2017) to
structural (associated with genuine stable properties of those impacted by semantic interpretation (Neri, 2011;
the environment) or incidental (generated by transitory Neri, 2014). At the same time, not all results fall into

the dichotomy above, which inevitably provides an fall within their goal-oriented focus (Triesch, Ballard,
oversimplified picture of current literature. To provide Hayhoe, & Sullivan, 2003). In other VR experiments,
one example, detection of oriented wavelets inserted observers may fail to notice large changes in the
into degraded scenes may improve following acquisition surrounding environment, such as rooms undergoing
of higher-level knowledge about the scene, despite no implausible transformations (Glennerster, Tcheang,
change in physical content of the image (Teufel et al., Gilson, Fitzgibbon, & Parker, 2006), or they may
2018). disregard gross inconsistencies in the spatial layout of
At least superficially, the adopted sensory task is a virtual maze (Muryy & Glennerster, 2021; Warren
comparable with previous orientation discrimination et al., 2017). Results of this kind appear to share some
tasks involving a congruent/incongruent determination broad characteristics with the results presented here,
(Neri, 2014; Neri, 2017). In prior experiments with but it is not easy to establish a clear connection between
static natural scenes, performance was superior for the two sets of studies (see below).
object-based probe insertions compared with image- On the one hand, the VR findings reviewed above
based insertions (Neri, 2017). I do not measure this remind us of the extent to which scene perception is
effect here: Performance for object-based insertions is at about gist rather than detail (as also demonstrated
best equivalent to that for image-based insertions (in the by the phenomenon of change blindness; Simons &
reliable-shadow configuration, Figure 2A). Although Levin, 1997; Triesch et al., 2003). In this sense, they
I do not have a ready explanation for this apparent emphasize the persistent connection between low-level
discrepancy, I must emphasize the innumerable and higher-level processes in perception, a result that
differences between the conditions of prior experiments aligns well with those presented here. In particular, they
and those used here. In addition to differences relating may provide a tentative explanation for the decrease
to scene appearance, spatial uncertainty, timing, and in recall performance associated with the sensory task
other stimulus-specific parameters, the major difference in the reliable-shadow configuration (solid vs. open
clearly relates to the VR medium. Previous studies black symbols in Figure 5A): When observers are
involved flat static natural scenes flashed in front required to take action with relation to specific regions
of fixating observers (Neri, 2017) and are therefore of the scene to determine probe congruency, their
not comparable with the experiments described here. strategy for memorizing room layout may shift from a
Even if one were to hypothesize that the underlying detail-oriented representation toward a gist-oriented
perceptual processes should behave similarly, the VR representation of the scene, thus freeing up resources for
implementation involves technical difficulties that make detail-oriented processing of the local region occupied
it impossible to compare measurements across studies. by the probe. This shift in strategy would lead to reduced
More specifically, probe alignment can be substantially performance in the memory task (under the assumption
inaccurate on some trials, and the minimal requirements that a gist-based map of the room would not support
for satisfactory insertion (see Methods) are sometimes accurate identification of box displacements) and
met only partially in real time. These technical glitches would be broadly consistent with the perceptual theory
do not apply to experiments with preprocessed images that has emerged from previous work in VR (Ballard
outside VR (Neri, 2014; Neri, 2017), and they do et al., 1995; Triesch et al., 2003). However, this account
not necessarily impact image-based insertions to the does not explain the different results obtained for
same extent that they impact object-based insertions. different shadow configurations, unless we introduce
As a consequence, it is impossible to make confident further interpretational elements. For example, we may
predictions about the relationship between performance hypothesize that gist-oriented encoding of the scene is
within VR and performance outside VR (Draschkow, not engaged when environmental statistics is violated
Nobre, & van Ede, 2022). by implausible manipulations, thus preventing the shift
An iconic result in the VR literature with potential from detail-oriented to gist-oriented representation
connections to the paradigm adopted here comes when shadow statistics is absent/unreliable. This
from early studies in which agents were asked to interpretation, however, remains highly speculative,
complete specific tasks involving simple objects, with little direct evidence in the data to support or
such as blocks of different shape and color (Ballard, exclude it.
Hayhoe, & Pelz, 1995). Overall, this line of research On the other hand, those same VR findings indicate
has established that perception happens “online,” in that the long-term statistical consistency of the
the sense that observers do not constantly engage in environment is largely ignored by observers (Ballard
detailed reconstruction of the scene, but only do so et al., 1995), which appears at odds with the present
when they are asked to take specific action in relation finding of measurable differences between reliable-
to a given aspect/portion/element of a scene (Hayhoe, and unreliable-shadow configurations (although see
Bensinger, & Ballard, 1998). As a consequence, they Hayhoe et al. [1998] for evidence of overlapping
may overlook substantial changes in the physical processes operating at different timescales). Those
structure of the scene, when those changes do not differences must reflect the long-term statistical degree

of reliability associated with shadow information, reliability associated with subelements of this structure.
because the two shadow configurations did not differ In the present experiments, for example, it is not only
in any other respect. More specifically, on the short relevant whether shadows are consistent or inconsistent
timescale (∼200 ms) of the sensory task from which with natural everyday experience: Their reliability
the sensitivity measurements are derived, there is no (or lack thereof) also plays a role in structuring their
difference in environmental statistics between these two impact on perception. At this stage, we do not hold
configurations. It is important to recognize, however, a complete account of how the representation of
that observers were never asked to make explicit this environmental “Umwelt” affects perception. The
judgments about changes in the environment: The experiments in this study provide quantitative evidence
sensitivity measurements reflect long-term statistics that it exists and offer specifics on how it operates with
implicitly, via their effect on sensory performance in the regard to shadow statistics. More importantly, these
probe discrimination task. In other words, they reflect a results emphasize the highly sophisticated nature of
form of implicit statistical leaning (Perruchet & Pacton, the underlying processes and their pervasive reach
2006). This is an important difference with respect to into perception at all levels (whether early or late)
classic VR studies in which observers were explicitly and scales (whether local or global, whether fast
asked to report on potential changes in the appearance or slow). Future experiments will be necessary to
of objects within the environment (Ballard et al., 1995; explore and characterize the full extent of this complex
Triesch et al., 2003), and may explain the apparent phenomenon.
inconsistency between their results and those reported
Keywords: virtual reality, object segmentation,
here. For example, it is conceivable that in the present
statistical learning
experiments, observers may not register environmental
changes to the extent of being able to describe them
in explicit detail (Nightingale, Wade, Farid, & Watson,
2019), and that asking them to do so may reset their Acknowledgments
implicit adaptation to said changes.
The timescale associated with the perceptual accrual I thank Felix Wichmann for insightful comments
of shadow information and its impact on sensory that contributed material to the Discussion. I thank
processing, which extends over minutes (Figure 3), Larry Cormack and Reuben Rideaux for feedback on
may seem implausibly long. The nature of this finding various aspects of the manuscript that greatly improved
is particularly surprising in consideration of the fact its clarity and quality.
that the primary metric of differential sensitivity
relates to the sensory task, which completes on the Supported by the Agence Nationale de la Recherche
subsecond timescale. This result, however, is reminiscent (ANR-16-CE28-0016, ANR-19-CE28-0010-01,
of early findings on storage of aftereffects: Under ANR-17-EURE-0017, ANR-10-LABX-0087 IEC, and
some conditions, a few minutes of exposure to the ANR-10-IDEX-0001-02 PSL*).
adapting stimulus may produce measurable aftereffects
3 months later (Jones & Holding, 1975; Favreau, 1979). Commercial relationships: none.
These findings are often overlooked by contemporary Corresponding author: Peter Neri.
literature, because they are difficult to reconcile with Email: [email protected].
modern notions of sensory adaptation. The present Address: Laboratoire des Systèmes Perceptifs
experiments differ in many respects from those (UMR8248), École normale supérieure, PSL Research
carried out to test aftereffects and related phenomena, University, 29 rue d’Ulm, Paris, France.
but they share the same surprising characteristic
of demonstrating interactions across perceptual
processes that may seem incommensurable according to
mainstream views of perception.
I propose a novel conceptual framework for
References
understanding sensory adaptation. In this framework,
which shares numerous features with Gibsonian notions Allman, J., Miezin, F., & McGuinness, E. (1985).
of ecological perception (Gibson, 1979; Warren, Stimulus specific responses from beyond the
2021), sensory recalibration is driven by the ecological classical receptive field: Neurophysiological
structure of environmental statistics. This structure mechanisms for local-global comparisons in
must be understood in the broadest sense to incorporate visual neurons. Annual Review of Neuroscience, 8,
various aspects of how the physical environment 407–430.
behaves and how it interfaces with the perceptual Anstis, S., Verstraten, F. A., & Mather, G. (1998). The
systems of the observer (Gibson, 1979). Furthermore, it motion aftereffect. Trends in Cognitive Sciences,
is not just structure that matters, but also the statistical 2(3), 111–117.

Ballard, D. H., Hayhoe, M. M., & Pelz, J. B. (1995). Felsen, G., & Dan, Y. (2005). A natural approach to
Memory representations in natural tasks. Journal of studying vision. Nature Neuroscience, 8, 1643–1646.
Cognitive Neuroscience, 7(1), 66–80. Felsen, G., Touryan, J., Han, F., & Dan, Y. (2005).
Bansal, A., Kowdle, A., Parikh, D., Gallagher, A., & Cortical sensitivity to visual features in natural
Zitnick, L. (2013). Which edges matter? Proceedings scenes. PLoS Biology, 3, e342.
of ICCV ’13 Workshop on 3D Representation and de Gelder, B., Tsyri, J., & de Borst, A. W. (2018). Virtual
Recognition, 578–585. reality and the new psychophysics. British Journal
Bar, M. (2004). Visual objects in context. Nature of Psychology, 109(3), 421–426.
Reviews Neuroscience, 5(8), 617–629. Gibson, J. J. (1979). The ecological approach to visual
Bex, P. J., Mareschal, I., & Dakin, S. C. (2007). Contrast perception. Boston, MA: Houghton Mifflin.
gain control in natural scenes. Journal of Vision, 7, Gilbert, C. D., & Li, W. (2013). Top-down influences
1–12, https://doi.org/10.1167/7.11.12. on visual processing. Nature Reviews Neuroscience,
Bex, P. J., Solomon, S. G., & Dakin, S. C. (2009). 14(5), 350–363.
Contrast sensitivity in natural scenes depends on Glennerster, A., Tcheang, L., Gilson, S. J., Fitzgibbon,
edge as well as spatial frequency structure. Journal A. W., & Parker, A. J. (2006). Humans ignore
of Vision, 9, 1–19, https://doi.org/10.1167/9.10.1. motion and stereo cues in favor of a fictional stable
Blakemore, C., & Campbell, F. W. (1969). On the world. Current Biology, 16(4), 428–432.
existence of neurones in the human visual system Green, D. M., & Swets, J. A. (1966). Signal detection
selectively sensitive to the orientation and size theory and psychophysics. New York, NY: Wiley.
of retinal images. Journal of Physiology, 203(1),
Hayhoe, M. M., Bensinger, D. G., & Ballard, D. H.
237–260.
(1998). Task constraints in visual working memory.
Bolz, J., & Gilbert, C. D. (1986). Generation of Vision Research, 38(1),125–137.
end-inhibition in the visual cortex via interlaminar
Hesse, J. K., & Tsao, D. Y. (2016). Consistency of
connections. Nature, 320(6060), 362–365.
border-ownership cells across artificial stimuli,
Burr, D. C., & Ross, J. (1986). Visual processing of natural stimuli, and stimuli with ambiguous
motion. Trends in Neuroscience, 9, 304–306. contours. Journal of Neuroscience, 36(44),
Cavanagh, P. (1991). What’s up in top-down processing. 11338–11349.
In A. Gorea (Ed.), Representations of vision (pp. Hillis, J. M., Watt, S. J., Landy, M. S., & Banks, M.
295–304). Cambridge, UK: Cambridge University S. (2004). Slant from texture and disparity cues:
Press. Optimal cue combination. Journal of Vision, 4(12),
Choung, O. H., Bornet, A., Doerig, A., & Herzog, M. H. 967–992, https://doi.org/10.1167/4.12.1.
(2021). Dissecting (un)crowding. Journal of Vision, Hubel, D. H. (1982). Exploration of the primary
21(10), 10, https://doi.org/10.1167/jov.21.10.10. visual cortex, 1955-78. Nature, 299(5883),
Coen-Cagli, R., Kohn, A., & Schwartz, O. (2015). 515–524.
Flexible gating of contextual influences in natural Isoyama, N., Sakuragi, Y., Terada, T., & Tsukamoto,
vision. Nature Neuroscience, 18(11), 1648–1655. M. (2021). Effects of augmented reality
Cumming, G. (2014). The new statistics: Why and how. object and texture presentation on walking
Psychological Science, 25(1), 7–29. behavior. Electronics, 10(6), 702, https:
Curran, W., Clifford, C. W., & Benton, C. P. (2006). The //doi.org/10.3390/electronics10060702.
direction aftereffect is driven by adaptation of local Jones, P. D., & Holding, D. H. (1975). Extremely
motion detectors. Vision Research, 46(25), 4270– long-term persistence of the McCollough
4278, https://doi.org/10.1016/j.visres.2006.08.026. effect. Journal of Experimental Psychology:
Dee, H., & Santos, P. (2011). The perception and content Human Perception and Performance, 1(4), 323–
of cast shadows: An interdisciplinary review. 327.
Spatial Cognition & Computation, 11, 226–253, Kersten, D., Mamassian, P., & Knill, D. C. (1997).
https://doi.org/10.1080/13875868.2011.565396. Moving cast shadows induce apparent motion in
Draschkow, D., Nobre, A. C., & Ede, F. van. (2022). depth. Perception, 26(2), 171–192.
Multiple spatial frames for immersive working Kohn, A. (2007). Visual adaptation: Physiology,
memory. Nature Human Behavior, 6(4), 536–544. mechanisms, and functional benefits. Journal of
Favreau, O. E. (1979). Persistence of simple and Neurophysiology, 97, 3155–3164.
contingent motion aftereffects. Perception & Kowalski, C. J. (1972). On the effects of non-normality
Psychophysics, 26(3), 187–194. on the distribution of the sample product-moment

correlation coefficient. Journal of the Royal natural images. Journal of Vision, 22(1), 4,
Statistical Society. Series C (Applied Statistics), https://doi.org/10.1167/jov.22.1.4.
21(1), 1–12, http://www.jstor.org/stable/2346598. Roelfsema, P. R., Lamme, V. A., & Spekreijse, H.
Lange, D., Stratmann, T. C., Gruenefeld, U., & (1998). Object-based attention in the primary visual
Boll, S. (2020). Hivefive: Immersion preserving cortex of the macaque monkey. Nature, 395(6700),
attention guidance in virtual reality. In Proceedings 376–381.
of the 2020 CHI Conference on Human Factors Roelfsema, P. R., Tolboom, M., & Khayat, P. S. (2007).
in Computing Systems (pp. 1–13). New York, Different processing phases for features, figures,
NY: Association for Computing Machinery, and selective attention in the primary visual cortex.
https://doi.org/10.1145/3313831.3376803. Neuron, 56(5), 785–792.
Mamassian, P., Knill, D. C., & Kersten, D. Rust, N. C., & Movshon, J. A. (2005). In praise of
(1998). The perception of cast shadows. artifice. Nature Neuroscience, 8, 1647–1650.
Trends in Cognitive Sciences, 2(8), 288–295,
Schwartz, O., Hsu, A., & Dayan, P. (2007). Space and
https://doi.org/10.1016/S1364-6613(98)01204-2.
time in visual context. Nature Reviews Neuroscience,
Marr, D. C. (1982). Vision: A computational investigation 8(7), 522–535.
into the human representation and processing of
Sebastian, S., Abrams, J., & Geisler, W. S. (2017).
visual information. New York, NY: Freeman.
Constrained sampling experiments reveal principles
Morgan, M. J. (2011). Features and the ‘primal sketch’. of detection in natural scenes. Proceedings of the
Vision Research, 51, 738–753. National Academy of Sciences of the United States
Muryy, A., & Glennerster, A. (2021). Route selection in of America, 114(28), E5731–E5740.
non-Euclidean virtual environments. PLoS ONE, Simoncelli, E. P., & Olshausen, B. A. (2001). Natural
16(4), e0247818. image statistics and neural representation. Annual
Neri, P. (2011). Global properties of natural scenes Review of Neuroscience, 24, 1193–1216.
shape local properties of human edge detectors. Simons, D. J., & Levin, D. T. (1997). Change blindness.
Frontiers in Psychology, 2, 172. Trends in Cognitive Sciences, 1(7), 261–267.
Neri, P. (2014). Semantic control of feature extraction Tarr, M. J., Kersten, D., & Bülthoff, H. H. (1998). Why
from natural scenes. Journal of Neuroscience, 34(6), the visual recognition system might encode the
2374–2388. effects of illumination. Vision Research, 38(15–16),
Neri, P. (2017). Object segmentation controls image 2259–2275.
reconstruction from natural scenes. PLoS Biology, Teufel, C., Dakin, S. C., & Fletcher, P. C. (2018). Prior
15(8), e1002611. object-knowledge sharpens properties of early
Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, visual feature-detectors. Scientific Reports, 8(1),
E. J. (2011). Erroneous analyses of interactions in 10853.
neuroscience: A problem of significance. Nature Triesch, J., Ballard, D. H., Hayhoe, M. M., & Sullivan,
Neuroscience, 14(9), 1105–1107. B. T. (2003). What you see is what you need. Journal
Nightingale, S. J., Wade, K. A., Farid, H., & Watson, D. of Vision, 3(1), 86–94, https://doi.org/10.1167/3.1.9.
G. (2019). Can people detect errors in shadows and Warren, W. H. (2019). Non-Euclidean navigation.
reflections? Attention, Perception, & Psychophysics, Journal of Experimental Biology, 222(Pt., Suppl. 1),
81(8), 2917–2943. 1–10.
Pecka, M., Han, Y., Sader, E., & Mrsic-Flogel, T. D. Warren, W. H. (2021). Information is where you
(2014). Experience-dependent specialization of find it: Perception as an ecologically well-posed
receptive field surround for selective coding of problem. i-Perception, 12(2), 20416695211000366,
natural scenes. Neuron, 84(2), 457–469. https://doi.org/10.1177/20416695211000366.
Perruchet, P., & Pacton, S. (2006). Implicit learning Warren, W. H., Rothman, D. B., Schnapp, B. H., &
and statistical learning: One phenomenon, two Ericson, J. D. (2017). Wormholes in virtual space:
approaches. Trends in Cognitive Sciences, 10(5), From cognitive maps to cognitive graphs. Cognition,
233–238. 166, 152–163.
Rensink, R. A., & Cavanagh, P. (2004). The influence of Wasserstein, R. L., & Lazar, N. A. (2016). The ASA’s
cast shadows on visual search. Perception, 33(11), statement on p-values: Context, process, and
1339–1358. purpose. The American Statistician, 70(2), 129–133.
Rideaux, R., West, R. K., Wallis, T. S. A., Bex, P. Zhou, H., Friedman, H. S., & Heydt, R. von der. (2000).
J., Mattingley, J. B., & Harrison, W. J. (2022). Coding of border ownership in monkey visual
Spatial structure, phase, and the contrast of cortex. Journal of Neuroscience, 20, 6594–6611.

Human Sensory Adaptation To The Ecological Structure of Environmental Statistics

Uploaded by

Copyright:

Available Formats

Human Sensory Adaptation To The Ecological Structure of Environmental Statistics

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Human Sensory Adaptation To The Ecological Structure of Environmental Statistics

Uploaded by

Copyright:

Available Formats

Journal of Vision (2024) 24(3):3, 1–23 1

Human sensory adaptation to the ecological structure of

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

independent quantities (Green & Swets, 1966): changes Modeling

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

makers, 2011). I first carried out an exploratory two-way

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

I repeated the above measurements in the absence of

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

Downloaded from jov.arvojournals.org on 03/23/2024

You might also like