Most studies of memory and perceptual learning in humans have employed unisensory settings to sim... more Most studies of memory and perceptual learning in humans have employed unisensory settings to simplify the study paradigm. However, in daily life we are often surrounded by complex and cluttered scenes made up of many objects and sources of sensory stimulation. Our experiences are, therefore, highly multisensory both when passively observing the world and when acting and navigating. We argue that human learning and memory systems are evolved to operate under these multisensory and dynamic conditions. The nervous system exploits the rich array of sensory inputs in this process, is sensitive to the relationship between the sensory inputs, and continuously updates sensory representations, and encodes memory traces based on the relationship between the senses. We review some recent findings that demonstrate a range of human learning and memory phenomena in which the interactions between visual and auditory modalities play an important role. We outline open questions as well as direction...
Supplemental material, supplement_material for Disturbed Eating and Body Dysmorphic Symptoms in a... more Supplemental material, supplement_material for Disturbed Eating and Body Dysmorphic Symptoms in a Young Adult Sample Are Separable Constructs That Each Show a Mixture of Distributions by Majed Samad, Christina Ralph-Nearman, Gerhard Hellemann, Sahib S. Khalsa, Ladan Shams and Jamie D. Feusner in Assessment
Visuotactile ventriloquism is a recently reported effect showing that somatotopic tactile represe... more Visuotactile ventriloquism is a recently reported effect showing that somatotopic tactile representations (namely, representation of location along the surface of one’s arm) can be biased by simultaneous presentation of a visual stimulus in a spatial localization task along the surface of the skin. Here we investigated whether the exposure to discrepancy between tactile and visual stimuli on the skin can induce lasting changes in the somatotopic representations of space. We conducted an experiment investigating this question by asking participants to perform a localization task that included unisensory and bisensory trials, before and after exposure to spatially discrepant visuotactile stimuli. Participants localized brief flashes of light and brief vibrations that were presented along the surface of their forearms, and were presented either individually (unisensory conditions) or were presented simultaneously at the same location or different locations. We then compared the localiz...
This is the author manuscript accepted for publication and has undergone full peer review but has... more This is the author manuscript accepted for publication and has undergone full peer review but has not been through the copyediting, typesetting, pagination and proofreading process, which may lead to differences between this version and the Version of Record. Please cite this article as
The main question that Firestone & Scholl (F&S) pose is whether "what and how we see is func... more The main question that Firestone & Scholl (F&S) pose is whether "what and how we see is functionally independent from what and how we think, know, desire, act, and so forth" (sect. 2, para. 1). We synthesize a collection of concerns from an interdisciplinary set of coauthors regarding F&S's assumptions and appeals to intuition, resulting in their treatment of visual perception as context-free.
Associating names to faces can be challenging, but it is an important task that we engage in thro... more Associating names to faces can be challenging, but it is an important task that we engage in throughout our lives. An interesting feature of this task is the lack of an inherent, semantic relationship between a face and name. Previous scientific research, as well as common lay theories, offer strategies that can aid in this task (e.g., mnemonics, semantic associations). However, these strategies are either impractical (e.g., spaced repetition) or cumbersome (e.g., mnemonics). The current study seeks to understand whether bolstering names with cross-modal cues—specifically, name tags—may aid memory for face and name pairings. In a series of five experiments, we investigated whether the presentation of congruent auditory (vocal) and written names at encoding might benefit subsequent cued recall and recognition memory tasks. The first experiment consisted of short video clips of individuals verbally introducing themselves (auditory cue), presented with or without a name tag (visual cue...
Individuals vary in their tendency to bind signals from multiple senses. For the same set of sigh... more Individuals vary in their tendency to bind signals from multiple senses. For the same set of sights and sounds, one individual may frequently integrate multisensory signals and experience a unified percept, whereas another individual may rarely bind them and often experience two distinct sensations. Thus, while this binding/integration tendency is specific to each individual, it is not clear how plastic this tendency is in adulthood, and how sensory experiences may cause it to change. Here, we conducted an exploratory investigation which provides evidence that (1) the brain’s tendency to bind in spatial perception is plastic, (2) that it can change following brief exposure to simple audiovisual stimuli, and (3) that exposure to temporally synchronous, spatially discrepant stimuli provides the most effective method to modify it. These results can inform current theories about how the brain updates its internal model of the surrounding sensory world, as well as future investigations s...
A major puzzle about visual system is how to deal with image variations in terms of lighting, def... more A major puzzle about visual system is how to deal with image variations in terms of lighting, deformation and others. It is important to have a code that is robust with respect to such changes and yet can distinguish between different shapes. We present here arguments that show that the complex cells of the visual cortex are precisely that. There is increasing evidence for an important role of complex cells in object representation and recognition. On the other hand, complex cells are insensitive to spatial phase information, and thus can lead to ambiguities with respect to position, contrast polarity and mirror symmetry, aspects which appear to be important in characterizing natural images. We investigate the richness of information in such a neural coding scheme, by using Gabor magnitudes as a model of complex cells, through a mathematical treatment of the ambiguities of such filter responses, and by empirical examination of alternate reconstructions of images from their Gabor mag...
BACKGROUND An electrical accident victim's recollection is often distorted by Bayesian infere... more BACKGROUND An electrical accident victim's recollection is often distorted by Bayesian inference in multisensory integration. For example, hearing the sound and seeing the bright flash of an electrical arc can create the false impression that someone had experienced an electrical shock. These subjects will often present to an emergency department seeking either treatment or reassurance. CASE REPORTS We present seven cases in which the subjects were startled by an electrical shock (real or perceived) and injury was reported. Calculations of the current and path were used to allocate causality between the shock and a history of chronic disease or previous trauma. In all seven cases, our analysis suggests that no current was passed through the body. WHY SHOULD AN EMERGENCY PHYSICIAN BE AWARE OF THIS?: Symptomology seen as corroborating may actually be confounding. Witness and survivor descriptions of electrical shocks are fraught with subjectivity and misunderstanding. Available current is usually irrelevant and overemphasized, such as stress on a 100-ampere welding source, which is orders of magnitude beyond lethal limits. History can also be biased for a number of reasons. Bayesian inference in multisensory perception can lead to a subject sincerely believing they had experienced an electrical shock. Determination of the current pathway and calculations of the amplitude and duration of the shock can be critical for understanding the limits and potential causation of electrical injury.
Visual primitives have long been proposed, under the name of geons or generalized cylinders, as c... more Visual primitives have long been proposed, under the name of geons or generalized cylinders, as components for object models in the brain and in the computer. Geons constitute a structural level intermediate between features and whole objects, and can form a basis for powerful generalization between different view points. We here present a system for the extraction of geon models from images of composite objects. We use synthetic gray-level images, each composed of several shape primitives. Our system finds geons as partial matches between several images. Images and models are represented as arrays of complex cells of Gabor type.
It is well known that the nervous system combines information from different cues within and acro... more It is well known that the nervous system combines information from different cues within and across sensory modalities to improve performance on perceptual tasks. In this article, we present results showing that in a visual motion-detection task, concurrent auditory motion stimuli improve accuracy even when they do not provide any useful information for the task. When participants judged which of two stimulus intervals contained visual coherent motion, the addition of identical moving sounds to both intervals improved accuracy. However, this enhancement occurred only with sounds that moved in the same direction as the visual motion. Therefore, it appears that the observed benefit of auditory stimulation is due to auditory-visual interactions at a sensory level. Thus, auditory and visual motion-processing pathways interact at a sensory-representation level in addition to the level at which perceptual estimates are combined.
Vision is generally considered the dominant sensory modality; self-contained and independent of o... more Vision is generally considered the dominant sensory modality; self-contained and independent of other senses. In this article, we will present recent results that contradict this view, and show that visual perception can be strongly altered by sound and touch, and such alterations can occur even at early stages of processing, as early as primary visual cortex. We will first review the behavioral evidence demonstrating modulation of visual perception by other modalities. As extreme examples of such modulations, we will describe two visual illusions induced by sound, and a visual illusion induced by touch. Next, we will discuss studies demonstrating modulation of activity in visual areas by stimulation of other modalities, and discuss possible pathways that could underpin such interactions. This will be followed by a discussion of how crossmodal interactions can affect visual learning and adaptation. We will review several studies showing crossmodal effects on visual learning. We will...
Likelihoods P(A|Za)=P(Za|A)P(A)/Σ AP(Za|A)P(A) P(V|Zv)=P(Zv|V)P(V)/ΣV P(Zv|V)P(V) Abstract: To op... more Likelihoods P(A|Za)=P(Za|A)P(A)/Σ AP(Za|A)P(A) P(V|Zv)=P(Zv|V)P(V)/ΣV P(Zv|V)P(V) Abstract: To optimally react to its surroundings, the nervous system must decide what the relations are between stimuli of different modalities. Given the statistics of a variable environment, the best statistical strategy to decide whether and how to integrate or segregate sensory information is a Bayesian one that optimally combines the available evidence (likelihood) with
Most studies of memory and perceptual learning in humans have employed unisensory settings to sim... more Most studies of memory and perceptual learning in humans have employed unisensory settings to simplify the study paradigm. However, in daily life we are often surrounded by complex and cluttered scenes made up of many objects and sources of sensory stimulation. Our experiences are, therefore, highly multisensory both when passively observing the world and when acting and navigating. We argue that human learning and memory systems are evolved to operate under these multisensory and dynamic conditions. The nervous system exploits the rich array of sensory inputs in this process, is sensitive to the relationship between the sensory inputs, and continuously updates sensory representations, and encodes memory traces based on the relationship between the senses. We review some recent findings that demonstrate a range of human learning and memory phenomena in which the interactions between visual and auditory modalities play an important role. We outline open questions as well as direction...
Supplemental material, supplement_material for Disturbed Eating and Body Dysmorphic Symptoms in a... more Supplemental material, supplement_material for Disturbed Eating and Body Dysmorphic Symptoms in a Young Adult Sample Are Separable Constructs That Each Show a Mixture of Distributions by Majed Samad, Christina Ralph-Nearman, Gerhard Hellemann, Sahib S. Khalsa, Ladan Shams and Jamie D. Feusner in Assessment
Visuotactile ventriloquism is a recently reported effect showing that somatotopic tactile represe... more Visuotactile ventriloquism is a recently reported effect showing that somatotopic tactile representations (namely, representation of location along the surface of one’s arm) can be biased by simultaneous presentation of a visual stimulus in a spatial localization task along the surface of the skin. Here we investigated whether the exposure to discrepancy between tactile and visual stimuli on the skin can induce lasting changes in the somatotopic representations of space. We conducted an experiment investigating this question by asking participants to perform a localization task that included unisensory and bisensory trials, before and after exposure to spatially discrepant visuotactile stimuli. Participants localized brief flashes of light and brief vibrations that were presented along the surface of their forearms, and were presented either individually (unisensory conditions) or were presented simultaneously at the same location or different locations. We then compared the localiz...
This is the author manuscript accepted for publication and has undergone full peer review but has... more This is the author manuscript accepted for publication and has undergone full peer review but has not been through the copyediting, typesetting, pagination and proofreading process, which may lead to differences between this version and the Version of Record. Please cite this article as
The main question that Firestone & Scholl (F&S) pose is whether "what and how we see is func... more The main question that Firestone & Scholl (F&S) pose is whether "what and how we see is functionally independent from what and how we think, know, desire, act, and so forth" (sect. 2, para. 1). We synthesize a collection of concerns from an interdisciplinary set of coauthors regarding F&S's assumptions and appeals to intuition, resulting in their treatment of visual perception as context-free.
Associating names to faces can be challenging, but it is an important task that we engage in thro... more Associating names to faces can be challenging, but it is an important task that we engage in throughout our lives. An interesting feature of this task is the lack of an inherent, semantic relationship between a face and name. Previous scientific research, as well as common lay theories, offer strategies that can aid in this task (e.g., mnemonics, semantic associations). However, these strategies are either impractical (e.g., spaced repetition) or cumbersome (e.g., mnemonics). The current study seeks to understand whether bolstering names with cross-modal cues—specifically, name tags—may aid memory for face and name pairings. In a series of five experiments, we investigated whether the presentation of congruent auditory (vocal) and written names at encoding might benefit subsequent cued recall and recognition memory tasks. The first experiment consisted of short video clips of individuals verbally introducing themselves (auditory cue), presented with or without a name tag (visual cue...
Individuals vary in their tendency to bind signals from multiple senses. For the same set of sigh... more Individuals vary in their tendency to bind signals from multiple senses. For the same set of sights and sounds, one individual may frequently integrate multisensory signals and experience a unified percept, whereas another individual may rarely bind them and often experience two distinct sensations. Thus, while this binding/integration tendency is specific to each individual, it is not clear how plastic this tendency is in adulthood, and how sensory experiences may cause it to change. Here, we conducted an exploratory investigation which provides evidence that (1) the brain’s tendency to bind in spatial perception is plastic, (2) that it can change following brief exposure to simple audiovisual stimuli, and (3) that exposure to temporally synchronous, spatially discrepant stimuli provides the most effective method to modify it. These results can inform current theories about how the brain updates its internal model of the surrounding sensory world, as well as future investigations s...
A major puzzle about visual system is how to deal with image variations in terms of lighting, def... more A major puzzle about visual system is how to deal with image variations in terms of lighting, deformation and others. It is important to have a code that is robust with respect to such changes and yet can distinguish between different shapes. We present here arguments that show that the complex cells of the visual cortex are precisely that. There is increasing evidence for an important role of complex cells in object representation and recognition. On the other hand, complex cells are insensitive to spatial phase information, and thus can lead to ambiguities with respect to position, contrast polarity and mirror symmetry, aspects which appear to be important in characterizing natural images. We investigate the richness of information in such a neural coding scheme, by using Gabor magnitudes as a model of complex cells, through a mathematical treatment of the ambiguities of such filter responses, and by empirical examination of alternate reconstructions of images from their Gabor mag...
BACKGROUND An electrical accident victim's recollection is often distorted by Bayesian infere... more BACKGROUND An electrical accident victim's recollection is often distorted by Bayesian inference in multisensory integration. For example, hearing the sound and seeing the bright flash of an electrical arc can create the false impression that someone had experienced an electrical shock. These subjects will often present to an emergency department seeking either treatment or reassurance. CASE REPORTS We present seven cases in which the subjects were startled by an electrical shock (real or perceived) and injury was reported. Calculations of the current and path were used to allocate causality between the shock and a history of chronic disease or previous trauma. In all seven cases, our analysis suggests that no current was passed through the body. WHY SHOULD AN EMERGENCY PHYSICIAN BE AWARE OF THIS?: Symptomology seen as corroborating may actually be confounding. Witness and survivor descriptions of electrical shocks are fraught with subjectivity and misunderstanding. Available current is usually irrelevant and overemphasized, such as stress on a 100-ampere welding source, which is orders of magnitude beyond lethal limits. History can also be biased for a number of reasons. Bayesian inference in multisensory perception can lead to a subject sincerely believing they had experienced an electrical shock. Determination of the current pathway and calculations of the amplitude and duration of the shock can be critical for understanding the limits and potential causation of electrical injury.
Visual primitives have long been proposed, under the name of geons or generalized cylinders, as c... more Visual primitives have long been proposed, under the name of geons or generalized cylinders, as components for object models in the brain and in the computer. Geons constitute a structural level intermediate between features and whole objects, and can form a basis for powerful generalization between different view points. We here present a system for the extraction of geon models from images of composite objects. We use synthetic gray-level images, each composed of several shape primitives. Our system finds geons as partial matches between several images. Images and models are represented as arrays of complex cells of Gabor type.
It is well known that the nervous system combines information from different cues within and acro... more It is well known that the nervous system combines information from different cues within and across sensory modalities to improve performance on perceptual tasks. In this article, we present results showing that in a visual motion-detection task, concurrent auditory motion stimuli improve accuracy even when they do not provide any useful information for the task. When participants judged which of two stimulus intervals contained visual coherent motion, the addition of identical moving sounds to both intervals improved accuracy. However, this enhancement occurred only with sounds that moved in the same direction as the visual motion. Therefore, it appears that the observed benefit of auditory stimulation is due to auditory-visual interactions at a sensory level. Thus, auditory and visual motion-processing pathways interact at a sensory-representation level in addition to the level at which perceptual estimates are combined.
Vision is generally considered the dominant sensory modality; self-contained and independent of o... more Vision is generally considered the dominant sensory modality; self-contained and independent of other senses. In this article, we will present recent results that contradict this view, and show that visual perception can be strongly altered by sound and touch, and such alterations can occur even at early stages of processing, as early as primary visual cortex. We will first review the behavioral evidence demonstrating modulation of visual perception by other modalities. As extreme examples of such modulations, we will describe two visual illusions induced by sound, and a visual illusion induced by touch. Next, we will discuss studies demonstrating modulation of activity in visual areas by stimulation of other modalities, and discuss possible pathways that could underpin such interactions. This will be followed by a discussion of how crossmodal interactions can affect visual learning and adaptation. We will review several studies showing crossmodal effects on visual learning. We will...
Likelihoods P(A|Za)=P(Za|A)P(A)/Σ AP(Za|A)P(A) P(V|Zv)=P(Zv|V)P(V)/ΣV P(Zv|V)P(V) Abstract: To op... more Likelihoods P(A|Za)=P(Za|A)P(A)/Σ AP(Za|A)P(A) P(V|Zv)=P(Zv|V)P(V)/ΣV P(Zv|V)P(V) Abstract: To optimally react to its surroundings, the nervous system must decide what the relations are between stimuli of different modalities. Given the statistics of a variable environment, the best statistical strategy to decide whether and how to integrate or segregate sensory information is a Bayesian one that optimally combines the available evidence (likelihood) with
Uploads
Papers by Ladan Shams