Sensory Substitution Devices

This paper argues that it is possible to combine enactivism and ecological psychology in a single post-cognitivist research framework if we highlight the common pragmatist assumptions of both approaches. These pragmatist assumptions or starting points are shared by ecological psychology and the enactive approach independently of being historically related to pragmatism, and they are based on the idea of organic coordination, which states that the evolution and development of the cognitive abilities of an organism are explained by appealing to the history of interactions of that organism with its environment. It is argued that the idea of behavioral or organic coordination within the enactive approach gives rise to the sensorimotor abilities of the organism, while the ecological approach emphasizes the coordination at a higher-level between organism and environment through the agent’s exploratory behavior for perceiving affordances. As such, these two different processes of organic coordination can be integrated in a postcognitivist research framework, which will be based on two levels of analysis: the subpersonal one (the neural dynamics of the sensorimotor contingencies and the emergence of enactive agency) and the personal one (the dynamics that emerges from the organism-environment interaction in ecological terms). If this proposal is on the right track, this may be a promising first step for offering a systematized and consistent postcognitivist approach to cognition that retain the full potential of both enactivism and ecological psychology.

Man, even though he may sometimes dream of being a pure spirit free from all material constraints, actually exists as a body in a world which is itself essentially constructed as a spatialized system of bodies; and in this constructed world features such as resistance, impenetrability, inertia, weight and forces are not accidents or exceptions but the rule.
The reflexions presented in this work aim at elucidating on a phenomenological level the relation that man entertains with the tangible aspects of his environment, and the function that this relation plays in the construction of the different sectors of his existence. The first of the sectors is perception: the opening on an ambient world that is pregnant with organisation and meaning. On the one hand, we shall take into account and conceptualize the way in which the tangibility of the world (the testing experience that the individual can have of resistance of his environment in the frame of a direct bodily relation with that environment, but more generally the understanding that the individual possesses of the very possibility of such a relation) participates in setting up the rationality by which man renders his world intelligible. On the other hand, we shall identify the structures of “subjectivity” (to speak the language of psychology, the structures of cognition) which make possible the specific way in which man constructs his experience and understanding of tangible resistance – whether this resistance is perceived in the context of an actual bodily engagement, or whether it is envisaged “indirectly” in the guise of a “simple” possibility.
We shall show here, on the basis of phenomenological analyses as well as elements coming from empirical psychology (experimental psychology as well as neuropsychology), that the relation that man has with the resistance of his ambient world depends on an opening towards the possible; and that the organisation of the ambient world in the form of a space which can contain material structures, or structures capable of opposing a resistance to the body, depends on a rationality which consists of making phenomena intelligible by interpreting them with reference to those capacities for action and for passion which the body confers on us, to the power that the body provides us with and the constraints to which it submits us.
Thus, even though the realm of material things – tangible objects, bodies – functions as an archetype of “presence” and of “being” (indeed, what is more “real” than a body?), in the last resort it gains its phenomenal character from the fact that it crystallizes for the person who perceives it a stream of virtual possibilities; hence, the here-and-now presence of the “tangible world” is in an essential way indebted to the realm of that which is not actually realized.

Despite the numerous studies and research programs devoted to their development and integration, sensory substitution devices have failed to live up to their goal of allowing one to “see with the skin” (White et al., 1970) or to “see with the brain” (Bach-y-Rita et al., 2003). The problem, we argue, is in the assumed or desired equivalence between using a sensory substitution device and seeing, or even perceiving through a particular sensory modality. Sensory substitution are closer to cultural extensions of existing perceptual skills such as reading. The parallel with the most recent two-route and interactive models of reading (e.g., Dehaene et al., 2005) generates a radically new way of approaching sensory substitution, and sets the stage for future work.

Le présent travail se propose de questionner les mécanismes de la perception de l’obstruction et de la résistance motrice. Comment puis-je comprendre que mon corps et mon action sont bloqués par un obstacle ? Quels sont les mécanismes par lesquels je prends conscience de la résistance que m'oppose le monde extérieur ?
Pour répondre à ces questions, nous tenterons d’articuler une approche phénoménologique et une approche expérimentale, basée sur un dispositif de suppléance perceptive et axée sur la perception d’obstacles.
Nous commencerons par présenter les travaux que le philosophe Maine de Biran (1766-1824) a consacrés à la question du tangible et de la résistance. Nous présenterons ensuite l’expérience que nous avons réalisée et, après avoir proposé une description en première personne de l'expérience perceptive rendue possible par ce dispositif expérimental, nous tenterons d’articuler nos résultats avec les acquis de la philosophie biranienne, en présentant leurs points de divergence.

This paper presents a multimodal interactive system for non-visual (auditory-haptic) exploration of virtual maps. The system is able to display haptically the height profile of a map, through a tactile mouse. Moreover, spatial auditory information is provided in the form of virtual anchor sounds located in specific points of the map, and delivered through headphones using customized Head-Related Transfer Functions (HRTFs). The validity of the proposed approach is investigated through two experiments on non-visual exploration of virtual maps. The first experiment has a preliminary nature and is aimed at assessing the effectiveness and the complementarity of auditory and haptic information in a goal reaching task. The second experiment investigates the potential of the system in providing subjects with spatial knowledge: specifically in helping with the construction of a cognitive map depicting simple geometrical objects. Results from both experiments show that the proposed concept, design, and implementation allow to effectively exploit the complementary natures of the " proximal " haptic modality and the " distal " auditory modality. Implications for orientation & mobility (O&M) protocols for visually impaired subjects are discussed.

The phenomenon of sensory substitution is not new. Blind subjects decode tactile signs or mechanically augmented learning materials, such as printed Braille, refreshable Braille displays, or tactile maps, through their fingertips. Tactile-visual sensory substitution (TVSS) is a functional equivalence, if not an absolute analogue, between vision and touch for the purposes of textual comprehension and wayfinding. However, such sensory-substitution technologies have spurred some philosophers to make large claims about the underlying neurophysiological architecture of the brain. In particular, the kinds of historically recognized functional analogues between vision and touch, when facilitated by recent generations of sensory-substitution device (SSD) technology, are often explained in terms of brain plasticity, as adaptation to these technologies requires fast perceptual learning and, over the long term, substantial cortical remapping. "Sensory substitution is […] only possible because of brain plasticity," assert the neuroscientists Bach-y- Rita and Kercel (2003: 541). The significance of TVSS specifically to highlight possibilities of neurological plasticity has captured the public's imagination to some degree, featuring in the New Yorker (Twilley 2017) and in books of popular neuroscience by Doidge (The Brain That Changes Itself, 2007; The Brain's Way of Healing, 2015). Philosophical interest started relatively early and has persisted

In this review we explore the relationship between synaesthesia and sensory substitution and argue that sensory substitution does indeed show properties of synaesthesia. Both are associated with atypical perceptual experiences elicited by the processing of a qualitatively different stimulus to that which normally gives rise to that experience. In the most common forms of sensory substitution, perceptual processing of an auditory or tactile signal (which has been converted from a visual signal) is experienced as visual-like in addition to retaining auditory/tactile characteristics. We consider different lines of evidence that support, to varying degrees, the assumption that sensory substitution is associated with visual-like experiences. We then go on to analyse the key similarities and differences between sensory substitution and synaesthesia. Lastly, we propose two testable predictions: firstly that, in an expert user of a sensory substitution device, the substituting modality should not be lost. Secondly that stimulation within the substituting modality, but by means other than a sensory substitution device, should still produce sensation in the normally substituted modality.

Researchers in postwar industrial laboratories such as Bell Labs and the Smith-Kettlewell Institute pioneered solutions to compensate for sensory loss through so-called sensory substitution systems, premised on an assumption of cortical and sensory plasticity. The paper tracks early discussions of plasticity in psychology literature from William James, acknowledged by Wiener, but explicitly developed by Bach-y-Rita and his collaborators. After discussing the conceptual foundations of the principles of sensory substitution, two examples are discussed. First, 'Project Felix' was an experiment in vibrotactile communication by means of 'hearing gloves' for the deaf at Norbert Wiener's laboratory at MIT, demonstrated to Helen Keller in 1950. Second, the tactile-visual sensory substitution (TVSS) system for the blind pioneered by Paul Bach-y-Rita from 1968 onwards. Cumulatively, this paper underlines the crucial yet occluded history of research on sensory impairments in the discovery of underlying neurophysiological processes of plasticity and the emergent discourse of neuroplastic subjectivity.

People often say things like the following: Cabernet Sauvignon tastes differently to an expert wine taster, or Beethoven's Ninth Symphony sounds differently to a seasoned conductor. The examples just described are cases of perceptual learning-cases of long-term changes in perception that result from practice or experience. Philosophers have been discussing such cases for centuries, from the 14 th-century Indian philosopher Vedānta Deśika to Thomas Reid as well as to contemporary philosophers like John McDowell, Ned Block, Susanna Siegel, and Christopher Peacocke. In the book, I use recent evidence from psychology and neuroscience to show that perceptual learning in fact occurs. I also offer a way for philosophers to distinguish between various different types of it. In some cases, perceptual learning involves changes in how one attends; in other cases, it involves a learned ability to differentiate two properties, or to perceive two properties as unified. I show how this taxonomy helps to classify cases in the philosophical literature. I then go on to rethink several domains in the philosophy of perception in terms of perceptual learning, including multisensory perception, color perception, speech perception, and sensory substitution. As a whole, the book offers a theory of the function of perceptual learning. Perceptual learning embeds into our quick perceptual systems what would be a slower task were it to be done in a controlled, cognitive manner. This frees up cognitive resources for other tasks. For instance, a novice wine taster drinking a Cabernet Sauvignon might have to think about its features first and then infer the type of wine, while an expert would be able to identify it immediately. This learned ability to immediately identify the wine enables the expert to think about other things like the vineyard or the vintage of the wine. My account gives us a new way to understand such cases in terms of cognitive resources and cognitive economy.

Traditionally, the pedagogical design for teaching and learning practices has been characterized as a process during which an active expert supports passive learner for the accomplishment of a specific goal or task. Nowadays, however, the accessibility of information technologies and the understanding of the learner’s active role have caused that interactive, embodied and contextual learning perspectives have begun to gain room. Here, we contribute with a technical report of a pilot study based on the Enactive Torch, a tool for the scientific study of perception, which aimed to investigate the crucial role of embodied process in the generation of perceptual experience for sensory substitution. In using this technological scaffolding, a group of students, from various academic disciplines, have coordinated and conducted three projects using different methods, each of them analyzing quantitative and qualitative data recorded from the participants’ firstand third-person perspective. By...

People sense the world by exploiting correlations between their physical actions and the changing sensory input that results from those actions. Interfaces that translate non-human sensor data to signals that are compatible with the human senses can therefore augment our abilities to make sense of the world. This insight has recently sparked an increase in projects that explore sensemaking and the creation of novel human experiences across scientific and artistic disciplines. However, there currently exists no constructive dialogue between artists and scientists that conduct research on this topic. In this position paper, we identify the theory and practice of sensory augmentation as a domain that could benefit from such a dialogue. We argue that artistic and scientific methods can complement each other within research on sensory augmentation and identify six thematic starting points for a dialogue between the arts and sciences. We conducted a case study to explore these conjectures, in which we instigated such a dialogue on a small scale. The case study revealed that the six themes we identified as relevant for a dialogue on sensory augmentation emerge rather spontaneously in such a dialogue and that such an exchange may facilitate progress on questions that are central to the theory and practice of sensory augmentation. Overall, this position paper contributes preliminary evidence for the potential of, and a starting point for, a dialogue between the arts and sciences that advances our understanding of sensory augmentation and the development of applications that involve it.

Sensorimotor theories of perception have been widely investigated in the context of the perceiver's normal environment, but not in the context of virtual environments. There are clearly identified differences between perception of pictures and that of a real-world environment, but these differences have not been studied in the light of sensorimotor theory. Nagel et al's studies of sensory augmentation included a trial of their feelSpace belt in a computer-game environment, but with inconclusive results. We propose that the sensorimotor contingencies that apply in the context of a virtual environment are significantly different from those in the 'real world', and might account for the differences found between 'normal' and picture perception. Building on Froese et al's work on Enactive Interfaces, and on Visell's structure for sensory substitution, we consider how interfacing a sensory augmentation device with a computer game environment might provide the basis for fruitful research in this area.

In this paper we wish to bring together two seemingly independent areas of research: synaesthesia and sensory substitution. Synaesthesia refers to a rare condition where a sensory stimulus elicits not only the sensation that stimulus evokes in its own modality, but an additional one; a synaesthete may thus hear the word "Monday", and, in addition to hearing it, have a concurrent visual experience of a red color. Sensory substitution, in contrast, attempts to substitute a sensory modality that a person has lost by transforming the information it provided so that it can be accessed through another, intact sensory modality. To make visual information accessible to a blind person, for example, data taken by a camera would be transformed into tactile or auditory information. What do synaesthesia and sensory substitution have in common? Research in both of these areas contributes to our understanding both of cross-modal cooperation and of sensory sensations or qualia, asking under what circumstances these can arise in a modality that is not stimulated. Synaesthesia reveals that this "sensory cross-activation" is possible, and sensory substitution research hopes to induce it. In this article, we will review briefly the literature on synaesthesia, and discuss the issue of qualia for this domain of research. We will then address the evidence for synaesthesia and visual qualia in the blind, and the research on sensory substitution, to finally ask whether sensory substitution may induce a 'synthetic' form of synaesthesia by taking advantage of the nervous system's capacity for generating visual images in the absence of retinal input.

In this article the problem of perception is highlighted from a philosophical, physiological and psychological point of view. The essay (second update) is followed by a list of publications, journals and websites about perception. The author would be delighted to receive comments. He can be contacted at j[underscore]kuijper[at]online[dot]nl.

Sensory substitution devices provide through an unusual sensory modality (the substituting modality, e.g., audition) access to features of the world that are normally accessed through another sensory modality (the substituted modality, e.g., vision). In this article, we address the question of which sensory modality the acquired perception belongs to. We have recourse to the four traditional criteria that have been used to define sensory modalities: sensory organ, stimuli, proper- ties, and qualitative experience (Grice, 1962), to which we have added the criteria of behavioral equivalence (Morgan, 1977), dedication (Keeley, 2002), and sensorimotor equivalence (O’Regan & Noe ̈, 2001). We discuss which of them are fulfilled by perception through sensory substitution devices and whether this favors the view that perception belongs to the substituting or to the substituted modality. Though the application of a number of criteria might be taken to point to the conclusion that perception with a sensory substitution device belongs to the substituted modality, we argue that the evidence leads to an alternative view on sensory substitution. Accord- ing to this view, the experience after sensory substitution is a transformation, extension, or aug- mentation of our perceptual capacities, rather than being something equivalent or reducible to an already existing sensory modality. We develop this view by comparing sensory substitution devices to other ‘‘mind-enhancing tools’’ such as pen and paper, sketchpads, or calculators. An analysis of sensory substitution in terms of mind-enhancing tools unveils it as a thoroughly trans- forming perceptual experience and as giving rise to a novel form of perceptual interaction with the environment.

Might the perception of real color include stimuli from any sensory system whose input was within the visible and near-visible spectrum? For instance, might color content be processed by an auditory system with color receptors? I argue that subjects wearing a color sonification device that processes color content using the auditory system hear real color much like a parrot or a human sees real color, a view called, “Radical Pluralistic Realism”. What makes it radical is that, unlike Mohan Matthen’s (2005) view, color content can be processed by a visual system that did not evolve out of processes relevant to an ecological niche.

Age-related Macular Degeneration (AMD) is a Low Vision (LV) disorder that causes vision impairment in older people. It can have an impact on their overall independence as changes in lighting or unfamiliar locations require more time for the eye to adapt and focus on the surrounding visual information.

According to proponents of the sensorimotor contingency theory of perception , active control of camera movement is necessary for the emergence of distal attribution in tactile-visual sensory substitution (TVSS) because it enables the subject to acquire knowledge of the way stimulation in the substituting modality varies as a function of self-initiated, bodily action. This chapter, by contrast, approaches distal attribution as a solution to a causal inference problem faced by the subject's perceptual systems. Given all of the available endogenous and exogenous evidence available to those systems, what is the most probable source of stimulation in the substituting modality? From this perspective, active control over the camera's movements matters for rather different reasons. Most importantly, it generates proprioceptive and efference-copy based information about the camera's body-relative position necessary to make use of the spatial cues present in the stimulation that the subject receives for purposes of egocentric object localization.

Visual information can be provided to blind users through sensory substitution devices that convert images into sound. Through extensive use to develop expertise, some blind users have reported visual experiences when using such a device. These blind expert users have also reported visual phenomenology to other sounds even when not using the device. The blind users acquired synthetic synaesthesia, with visual experience evoked by sounds only after gaining such expertise. Sensorimotor learning may facilitate and perhaps even be required to develop expertise in the use of multimodal information. Furthermore, other areas where expertise is acquired in dividing attention amongst cross-modal information or integrating such information might also give rise to synthetic synaesthesia.

One of the most exciting recent findings in neuroscience has been the capacity for neural plasticity in adult humans and animals. Studies of perceptual learning have provided key insights into the mechanisms of neural plasticity and the changes in functional neuroanatomy that it affords. Key questions in this field of research concern how practice of a task leads to specific or general improvement. Although much of this work has been carried out with a focus on a single sensory modality, primarily visual, there is increasing interest in multisensory perceptual learning. Here we will examine how advances in perceptual learning research both inform and can be informed by the development and advancement of sensory substitution devices for blind persons. To allow…

This study investigates how active exploration helps users of sensory substitution devices (SSDs) to detect action-relevant information. A vibrotactile SSD was developed that generates stimulation that is contingent on the users' movements. Target direction was specified by the location of the vibratory stimulation, and target distance by the size and intensity of the pattern of stimulation. A series of experiments was performed with blindfolded participants. In Experiments 1a to 1c, participants used the SSD to align their central body axis with prespecified targets. These experiments differed in the number of actuators that were used and whether online perception-action coupling was present. In Experiment 2, participants approached targets with forward locomotion along a straight line. Experiment 3 combined the previous experiments and studied the concomitant walking and steering toward targets. Oscillatory movements, which facilitated information pickup, were observed in all ...

Sensory substitutions systems offer to blind persons (and blindfolded subjects) a new kind of perception, a new sensorial modality which can compensate a lost sense and/ or be added to the others modalities in order to perceive the world. As such sensorial modality, perception with these devices can be evaluated by acuity tests. Thus, in this paper we describe an experimental study intended to specify limits of perception by using a sensory substitution device named Tactos. Blindfolded subjects had to recognize objects of different sizes by using a tactile feedback (mechanical stimulators) and a graphics tablet with its stylus. The results show that the Snellen acuity ratios are comprise between 20/120 and 20/560 and are closely dependant of the spatial acuity of the device which is also measured.

Synesthetic interfaces—interfaces that enact a sensory substitution by translating information normally accessed through one sense modality into the phenomenal forms associated with another—constitute a fascinating and little-theorized corner of videogame UI design. Frustrating the distinctions laid out in dichotomies of "diegetic" versus "non-diegetic" UI elements, or "ecological" versus "emphatic" elements, synesthetic interfaces have been poorly served by the terminological frameworks that have typically dominated discussions of the functional and fictional status of game UI. This paper examines two employments of synesthetic interfaces—to communicate human perceptual expertise, and to depict the experience of nonhuman organisms—as a way of illuminating aspects of synesthetic interfaces that evade more terminologically rigid approaches.

We consider here two modes of relation to an instrument: “in-hand” and “put-down”. For the user, these two modes are phenomenologically mutually exclusive; we discuss the conditions for passing from one mode to the other. More precisely, the constitution and the effectiveness of the “in-hand” mode for a new tool depends on a process of appropriation that needs to be characterized and conceptualized. For the design of interaction, taking into account this appropriation process translates into writing a scenario for a disappearance, that of the tool itself, in favour of the space of action that the tool makes possible.

Enactive cognitive science (ECS) and ecological psychology (EP) agree that active movement is important for perception, but they remain ambiguous regarding the precise role of agency. EP has focused on the notion of sensorimotor invariants, according to which bodily movements play an instrumental role in perception. ECS has focused on the notion of sensorimotor contingencies, which goes beyond an instrumental role because skillfully regulated movements are claimed to play a constitutive role. We refer to these two hypotheses as instrumental agency and constitutive agency, respectively. Evidence comes from a variety of fields, including neural, behavioral, and phenomenological research, but so far with confounds that prevent an experimental distinction between these hypotheses. Here we advance the debate by proposing a novel double-participant setup that aims to isolate agency as the key variable that distinguishes bodily movement in active and passive conditions of perception. We pilot this setup with a psychological study of width discrimination using the Enactive Torch, a haptic sensory substitution device. There was no evidence favoring the stronger hypothesis of constitutive agency over instrumental agency. However, we caution that during debriefing several participants reported using cognitive strategies that did not rely on spatial perception. We conclude that this approach is a viable direction for future research, but that greater care is required to establish and confirm the desired modality of first-person experience.

A large number of technical devices attempt to help blind persons improve their spatial perception and facilitate their mobility. We wish to present here the principles on which these prosthetic perceptual devices function, the conditions of their appropriation, and the general perspectives they open concerning the role of technical objects and systems in the constitution of human experience. A technical device for assisting perception has to compensate for a sensory deficit by mobilizing other subsisting sensory modalities which remain available. We therefore have to understand the relationship between perceptual activity and sensory input. In addition, we shall see that the technical devices which assist perceptual activity can in return serves as tools for experimental scientific research on the mechanisms of perception in general. Our aim in this chapter is to propose several hypotheses to explain the mechanisms by which perception can be transformed according to the tools and prosthetic devices employed, and to show that in each case this transformation amounts, finally, to a reconfiguration of a field of possibilities. To do this, we shall start from the extreme situation of the so-called ‘sensory substitution’ systems; we will then decline several variations so as to better understand in each case the way in which the technical mediation transforms the relations between sensation and perception, and thereby modifies the field of accessible possibilities.

Effective training techniques for sensory substitution devices (SSDs), devices that translate information for one sensory modality into information that can be used by another, are rarely discussed.  In this paper, we compare three training strategies on a particular SSD, known as the vOICe, in a minimal search and localization task and show that an interactive paradigm, which combines the efforts of two trainees at the same time in a tag-like game,  is more effective than passive training at a computer console or active training involving search and localization of luminescent discs.  This finding suggests support for philosophical views of perception that take interaction with the environment seriously.  In particular, we argue that these findings are in support of the embodied mind view as proposed by Andy Clark (2008).

Interfaces that subvert, substitute, or augment the relationship between acting and sensing make interesting tools for experimentation in artistic research. Amongst them, tactile vision interfaces, which enable people to experience to "see with their skin", have particular appeal. However, surprisingly few such interfaces are used by artists. In the present work we take a first look at how the artistic potential of tactile vision interfaces could be unlocked. On the basis of related artistic work, we argue that an accessible, generalizable and learnable interface is needed to enable artists to explore the possibilities of tactile vision. Subsequently, we present ongoing research that revolves around the development of such a tactile vision interface. A case study is presented that informs the development of a new prototype of this interface that overcomes two key limitations in its purpose to unlock the artistic potential of tactile vision.

The objective of this work concerns the design and the implementation of a zoomable interface implying the haptic modality. The initial postulate is that the zoom experience is not a natural, a direct experience, but supposes instrumentation and learning. In other words, the zoom experience is built by the appropriation of a technical substitution which makes it possible to modify the properties of the space-time flow; these properties which bind the subject to his (real or virtual) world are relational. To conceive this new interface, directly inspired from technologies known as of sensory substitution, we carried out a set of experiments allowing to define and to qualify the technical conditions and of use which favour the emergence of a perceptive experience of the zoom type. More generally, it concerns the proposition of more intuitive or immediate modes of instrumental interaction engaging explicitly the body in action.

Bio-mimetic approaches to restoring sensory function show great promise in that they rapidly produce perceptual experience, but have the disadvantage of being invasive. In contrast, sensory substitution approaches are non-invasive, but may lead to cognitive rather than perceptual experience. Here we introduce a new non-invasive approach that leads to fast and truly perceptual experience like bio-mimetic techniques. Instead of building on existing circuits at the neural level as done in bio-mimetics, we piggy-back on sensorimotor contingencies at the stimulus level. We convey head orientation to geomagnetic North, a reliable spatial relation not normally sensed by humans, by mimicking sensorimotor contingencies of distal sounds via head-related transfer functions. We demonstrate rapid and long-lasting integration into the perception of self-rotation. Short training with amplified or reduced rotation gain in the magnetic signal can expand or compress the perceived extent of vestibular self-rotation, even with the magnetic signal absent in the test. We argue that it is the reliability of the magnetic signal that allows vestibular spatial recalibration, and the coding scheme mimicking sensorimotor contingencies of distal sounds that permits fast integration. Hence we propose that contingency-mimetic feedback has great potential for creating sensory augmentation devices that achieve fast and genuinely perceptual experiences. Starting from the seminal work of Bach-y-Rita 1 , sensory substitution and augmentation research has aimed to restore sensory functionality from non-invasive afferent signals of artificial sensors. However, as noted in an extensive recent review by Deroy & Auvray 2 , there has been little concrete evidence that truly perceptual experiences have ever been obtained via this approach. Evidence robustly shows abilities to locate and identify objects and shapes using sensory substitution devices 3–6 , yet these abilities are typically constrained to a small number of (known) stimuli and do not match the speed or the accuracy of natural perceptual object identification 2. Thus, it is debatable whether they really involve perceptual experiences similar to the source modality, or are based on higher-level cognitive decision strategies for stimulus discrimination. Deroy & Auvray conclude that the skills achieved with sensory substitution devices should not be interpreted as being 'perceptual' but rather should be described as " acquired cognitive extensions to existing perceptual skills " 2. This seems similarly to be the case for approaches to sensory augmentation which reduce the complexity of the artificial afferents by using simpler artificial sensors 6,7. For instance in the case of magnetic North, behavioural integration has been obtained in rats using neuroprosthetics 8. However, in humans using sensory augmentation König et al. 9 could not demonstrate low-level sensory effects using a vibro-tactile approach, and subjective reports indicate high cognitive involvement as described for sensory substitution 2,9,10. Why has it, beyond stimulus discrimination abilities, not been possible by using sensory substitution and augmentation to create truly modal perceptual experience that (begins to) resemble that of a natural modality? A key to creating perceptual sensations from an artificial afferent signal seems to be that the signal should find an 'entry point' to interface with sensory processes that are already in place 11. Yet many classical approaches to sensory substitution have contacted the sensory apparatus with more or less arbitrary sensory coding schemes that do not necessarily have a direct correspondence to the low-level processes of the source modality. This is the case even when the codes are based on genetically, statistically or culturally established analogies or associations 12,13. As a consequence, an entry point into existing sensory processing might have to be established via perceptual learning and cross-modal neural plasticity 2,12,14–16 , for instance by a mechanism proposed in a recent cross-modal extension to reversed hierarchy theory 11,17. However, perceptual learning typically requires hundreds and thousands

Since 1692, Molyneux’s question to John Locke has been a focus for the discussion of perception in philosophy and psychology. In this chapter we introduce  a methodological question inspired by the Molyneux problem. Can a conceptual framework developed to theorise one sense modality generalise to other modalities? Previous philosophical accounts of the senses have assumed that the presence in perception of an external spatial field, or of bodily awareness, is a stable characteristic of sensory modalities, the former being associated with vision, the latter with touch (Martin 1992; O'Shaugnessy 1989). From this, Martin (1992) argues against the possibility of a general theory of perception. However, findings of the plasticity of neural systems underlying sensory performance are a challenge to this assumption. Haptic awareness can become more "spatial" and less "corporeal" following the experience of optically driven touch stimulation in Tactile-Visual Sensory Substitution (TVSS). This removes one obstacle to a positive response to the MMQ regarding spatial perception. Yet neuroplasticity is not the whole story behind sensory substitution. We argue that the inherent similarity of certain visual and haptic spatial functions makes these more “substitutable” than others, and that it is valuable to apply a unified approach to these functions.