Individual Differences in Sensory Substitution

in Multisensory Research
Restricted Access
Get Access to Full Text
Rent on DeepDyve

Have an Access Token?

Enter your access token to activate and access content online.

Please login and go to your personal user account to enter your access token.


Have Institutional Access?

Access content through your institution. Any other coaching guidance?


Sensory substitution devices were developed in the context of perceptual rehabilitation and they aim at compensating one or several functions of a deficient sensory modality by converting stimuli that are normally accessed through this deficient sensory modality into stimuli accessible by another sensory modality. For instance, they can convert visual information into sounds or tactile stimuli. In this article, we review those studies that investigated the individual differences at the behavioural, neural, and phenomenological levels when using a sensory substitution device. We highlight how taking into account individual differences has consequences for the optimization and learning of sensory substitution devices. We also discuss the extent to which these studies allow a better understanding of the experience with sensory substitution devices, and in particular how the resulting experience is not akin to a single sensory modality. Rather, it should be conceived as a multisensory experience, involving both perceptual and cognitive processes, and emerging on each user’s pre-existing sensory and cognitive capacities.

Individual Differences in Sensory Substitution

in Multisensory Research



AmediA.SternW. M.CamprodonJ. A.BermpohlF.MerabetL.RotmanS.HemondC.MeijerP.Pascual-LeoneA. (2007). Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complexNat. Neurosci. 10687689.

ArnoldG.SpenceC.AuvrayM. (2016). Taking someone else’s spatial perspective: natural stance or effortful decentring? Cognition 1482733.

AuvrayM.FarinaM. (2017). Patrolling the boundaries of synaethesia: a critical appraisal of transient and artificially-acquired forms of synaesthetic experiences in: Sensory Blendings: New Essays on SynaesthesiaDeroyO. (Ed.). Oxford University PressOxford, UK. (Publication scheduled for April 2017).

AuvrayM.MyinE. (2009). Perception with compensatory devices. From sensory substitution to sensorimotor extensionCogn. Sci. 3310361058.

AuvrayM.HannetonS.O’ReganJ. K. (2007). Learning to perceive with a visuo-auditory substitution system: localization and object recognition with the voicePerception 36416430.

Bach-y-RitaP.CollinsC. C.SaundersF. A.WhiteB.ScaddenL. (1969). Vision substitution by tactile image projectionNature 221(5184) 963964.

Bach-y-RitaP.TylerM. E.KaczmarekK. A. (2003). Seeing with the brainInt. J. Hum-Comput. Interact. 2285295.

BavelierD.NevilleH. J. (2002). Cross-modal plasticity: where and how? Nat. Rev. Neurosci. 3443452.

BlockN. (2003). Tactile sensation via spatial perceptionTrends Cogn. Sci. 7285286.

BolognaG.DevilleB.PunT. (2009). On the use of the auditory pathway to represent image scenes in real-timeNeurocomputing 72839849.

BoussaoudD.BremmerF. (1999). Gaze effects in the cerebral cortex: reference frames for space coding and actionExp. Brain Res. 128170180.

BrownD. J.MacphersonT.WardJ. (2011). Seeing with sound? Exploring different characteristics of a visual-to-auditory sensory substitution devicePerception 4011201135.

CohenY. E.AndersenR. A. (2002). A common reference frame for movement plans in the posterior parietal cortexNat. Rev. Neurosci. 3553562.

DanilovY.TylerM.SkinnerK.HogleR.Bach-y-RitaP. (2007). Efficacy of electrotactile vestibular substitution in patients with peripheral and central vestibular lossJ. Vestib. Res. 17119130.

DeroyO.AuvrayM. (2012). Reading the world through the skin and ears: a new perspective on sensory substitutionFron. Psychol. 3457. DOI:10.3389/fpsyg.2012.00457.

DeroyO.AuvrayM. (2014). A crossmodal perspective on sensory substitution in: Perception and Its ModalitiesBiggsS.MatthenM.StokesD. (Eds) pp.  327349. Oxford University PressOxford, UK.

DeroyO.FasielloI.HaywardV.AuvrayM. (2016). Differentiated audio-tactile correspondences in sighted and blind individualsJ. Exp. Psychol. Hum. Percept. Perform. 4212041214.

DiotB.HalavackovaP.DemongeotJ.VuillermeN. (2014). Sensory substitution for balance control using a vestibular-to-tactile deviceMultisens. Res. 27313336.

ElliG. V.BenettiS.CollignonO. (2014). Is there a future for sensory substitution outside academic laboratories? Multisens. Res. 27271291.

FarcyR.LerouxR.JuchaA.DamaschiniR.GrégoireC.ZogaghiA. (2006). Electronic travel aids and electronic orientation aids for blind people: technical, rehabilitation and everyday life points of view in: Proceedings of the Conference and Workshop on Assistive Technology for Vision and Hearing ImpairmentHershM. A. (Ed.).

FarinaM. (2013). Neither touch nor vision: sensory substitution as artificial synaesthesia? Biol. Philos. 28639655.

GallayM.DenisM.AuvrayM. (2013). Navigation assistance for blind pedestrians: guidelines for the design of devices and implications for spatial cognition in: Representing Space in Cognition: Interrelations of Behaviour Language and Formal ModelsTenbrinkT.WienerJ.ClaramuntC. (Eds). Oxford University PressOxford, UK.

GoekeC.KornpetpaneeS.KösterM.Fernández-RevellesA.GramannK.KönigP. (2015). Cultural background shapes spatial reference frame proclivitySci. Rep. 511426. DOI:10.1038/srep11426.

GuarnieroG. (1974). Experience of tactile visionPerception 3101104.

HaighA.BrownD. J.MeijerP.ProulxM. J. (2013). How well do you see what you hear? The acuity of visual-to-auditory sensory substitutionFront. Psychol. 4330.

HannetonS.AuvrayM.DuretteB. (2010). The vibe: a versatile vision-to-audition sensory substitution deviceAppl. Bionics Biomech. 7269276.

HannetonS.HerquelP.AuvrayM. (2015). Intermodal recoding of a video game: learning to process signals for motion perception in a pure auditory environmentInt. J. Adapt. Contr. Signal Proc. 2914751483.

HarrarV.HarrisL. R. (2010). Touch used to guide action is partially coded in a visual reference frameExp. Brain Res. 203615620.

Hartcher-O’BrienJ.AuvrayM. (2014). The process of distal attribution illuminated through studies of sensory substitutionMultisens. Res. 27421441.

HeimlerB.Striem-AmitE.AmediA. (2015). Origins of task-specific sensory-independent organization in the visual and auditory brain: neuroscience evidence, open questions and clinical implicationsCurr. Opin. Neurobiol. 35169177.

HoC.SpenceC. (2008). The Multisensory Driver: Implications for Ergonomic Car Interface Design. Ashgate PublishingAldershot, UK.

HöttingK.RöderB. (2004). Hearing cheats touch, but less in congenitally blind than in sighted individualsPsychol. Sci. 156064.

HöttingK.RöslerF.RöderB. (2004). Altered auditory-tactile interactions in congenitally blind humans: an event-related potential studyExp. Brain Res. 159370381.

HumphreyN. (1992). A History of the Mind. Simon and SchusterNew York, NY, USA.

HurleyS.NoëA. (2003). Neural plasticity and consciousnessBiol. Philos. 18131168.

JonesL. A.KunkelJ.PiateskiE. (2009). Vibrotactile pattern recognition on the arm and backPerception 385268.

KlatzkyR. L. (1998). Allocentric and egocentric spatial representations: definitions, distinctions, and interconnections in: Spatial Cognition IFreksaC.HabelC.WenderK. F. (Eds) pp.  118. SpringerNew York, NY, USA.

KristjánssonÁ.MoldoveanuA.JóhannessonÓ.BalanO.SpagnolS.ValgeirsdóttirV. V.UnnpórssonR. (2016). Designing sensory-substitution devices: principles, pitfalls and potentialRestor. Neurol. Neurosci. 34769787.

KupersR.FumalA.Maertens de NoordhoutA.GjeddeA.SchoenenJ.PtitoM. (2006). Transcranial magnetic stimulation of the visual cortex induces somatotopically organized qualia in blind subjectsProc. Natl Acad. Sci. 351325613260.

Levy-TzedekS.NovickI.ArbelR.AbboudS.MaidenbaumS.VaadiaE.AmediA. (2012). Cross-sensory transfer of sensory-motor information: visuomotor learning affects performance on an audiomotor task, using sensory-substitutionSci. Rep. 2949. DOI:10.1038/srep00949.

Levy-TzedekS.RiemerD.AmediA. (2014). Color improves “visual” acuity via soundFront. Neurosci. 8358. DOI:10.3389/fnins.2014.00358.

Levy-TzedekS.MaidenbaumS.AmediA.LacknerJ. (2016). Aging and sensory substitution in a virtual navigation taskPloS One 11e0151593. DOI:10.1371/journal.pone.0151593.

LoomisJ. M.KlatzkyR. L.GolledgeR. G.CicinelliJ. G.PellegrinoJ. W.FryP. A. (1993). Nonvisual navigation by blind and sighted: assessment of path integration abilityJ. Exp. Psychol. Gen. 1227391.

LoomisJ. M.KlatzkyR. L.GiudiceN. A. (2012). Sensory substitution of vision: importance of perceptual and cognitive processing in: Assistive Technology for Blindness and Low VisionManduchiR.KurniawanS. (Eds) pp.  162191. CRC PressBoca Raton, FL, USA.

MaidenbaumS.HanassyS.AbboudS.BuchsG.ChebatD. R.Levy-TzedekS.AmediA. (2014). The “EyeCane”, a new electronic travel aid for the blind: technology, behavior and Swift learningRestor. Neurol. Neurosci. 32813824.

MarksD. F. (1973). Visual imagery in the recall of picturesBr. J. Psychol. 641724.

MarksL. E. (1989). On cross-modal similarity: the perceptual structure of pitch, loudness, and brightnessJ. Exp. Psychol. Hum. Percept. Perform. 15586602.

MeijerP. B. L. (1992). An experimental system for auditory image representationsIEEE Trans. Biomed. Eng. 39112121.

MillarS. (1994). Understanding and Representing Space: Theory and Evidence From Studies With Blind and Sighted Children. Clarendon PressOxford, UK.

MurphyM. C.NauA. C.FisherC.KimS. G.SchumanJ. S.ChanK. C. (2016). Top-down influence on the visual cortex of the blind during sensory substitutionNeuroImage 125932940.

NoëA. (2004). Action in Perception. MIT PressCambridge, MA, USA.

NoordzijM. L.ZuidhoekS.PostmaA. (2006). The influence of visual experience on the ability to form spatial mental models based on route and survey descriptionsCognition 100321342.

O’KeefeJ.NadelL. (1978). The Hippocampus as a Cognitive Map. Oxford University PressLondon, UK.

O’ReganJ. K. (2011). Why Red Doesn’t Sound Like a Bell. Understanding the Feel of Consciousness. Oxford University PressOxford, UK.

OrtizT.PochJ.SantosJ. M.RequenaC.MartínezA. M.Ortiz-TeránL.TurreroA.BarciaJ.NogalesR.CalvoA.MartínezJ. M.CórdobaJ. L.Pascual-LeoneA. (2011). Recruitment of occipital cortex during sensory substitution training linked to subjective experience of seeing in people with blindnessPLoS One 6e23264. DOI:10.1371/journal.pone.0023264.

Pascual-LeoneA.HamiltonR. (2001). The metamodal organization of the brainProg. Brain Res. 134427445.

PasqualottoA.EsenkayaT. (2016). Sensory substitution: the spatial updating of auditory scenes “mimics” the spatial updating of visual scenesFront. Behav. Neurosci. 1079. DOI:10.3389/fnbeh.2016.00079.

PasqualottoA.ProulxM. J. (2012). The role of visual experience for the neural basis of spatial cognitionNeurosci. Biobehav. Rev. 3611791187.

PasqualottoA.SpillerM. J.JansariA. S.ProulxM. J. (2013). Visual experience facilitates allocentric spatial representationBehav. Brain Res. 236175179.

Pesnot-LerousseauJ.ArnoldG.AuvrayM. (subm.). Training induced plasticity with visual-to-auditory sensory substitution devices.

PoirierC.De VolderA. G.ScheiberC. (2007). What neuroimaging tells us about sensory substitutionNeurosci. Biobehav. Rev. 3110641070.

PollokB.SchnitzlerI.StoerigP.MierdorfT.SchnitzlerA. (2005). Image-to-sound conversion: experience-induced plasticity in auditory cortex of blindfolded adultsExp. Brain Res. 167287291.

PrinzJ. (2006). Putting the brakes on enactive perceptionPsyche 12119.

ProulxM. J.StoerigP.LudowigE.KnollI. (2008). Seeing ‘where’ through the ears: effects of learning-by-doing and long-term sensory deprivation on localization based on image-to-sound substitutionPloS One 3e1840. DOI:10.1371/journal.pone.0001840.

ProulxM. J.BrownD. J.PasqualottoA.MeijerP. (2014). Multisensory perceptual learning and sensory substitutionNeurosci. Biobehav. Rev. 411625.

ProulxM. J.GwinnuttJ.Dell’ SousaA. A.BrownD. J. (2016). Other ways of seeing: from behavior to neural mechanisms in the online “visual” control of action with sensory substitutionRestor. Neurol. Neurosci. 342944.

PtitoM.MoesgaardS. M.GjeddeA.KupersR. (2005). Cross-modal plasticity revealed by electrotactile stimulation of the tongue in the congenitally blindBrain 128606614.

RenierL.LaloyauxC.CollignonO.TranduyD.VanlierdeA.BruyerR.De VolderA. G. (2005). The Ponzo illusion using auditory substitution of vision in sighted and early blind subjectsPerception 34857867.

RieserJ. J.GumD. A.HillE. W. (1986). Sensitivity to perspective structure while walking without visionPerception 15173188.

SathianK.StillaR. (2010). Cross-modal plasticity of tactile perception in blindnessRestor. Neurol. Neurosci. 28271281.

ShoreD. I.SpryE.SpenceC. (2002). Confusing the mind by crossing the handsCogn. Brain Res. 14153163.

SpenceC.DeroyO. (2013). Crossmodal mental imagery in: Multisensory ImageryLaceyS.LawsonR. (Eds) pp.  157183. SpringerNew York, NY, USA.

StilesN. R.ShimojoS. (2015). Sensory substitution in: The Oxford Handbook of Perceptual OrganizationWagemansJ. (Ed.) pp.  775798. Oxford University PressOxford, UK.

StilesN. R.ZhengY.ShimojoS. (2015). Length and orientation constancy learning in 2-dimensions with auditory sensory substitution: the importance of self-initiated movementFront. Psychol. 6842. DOI:10.3389/fpsyg.2015.00842.

Striem-AmitE.AmediA. (2014). Visual cortex extrastriate body-selective area activation in congenitally blind people “seeing” by using soundsCurr. Biol. 6687692.

Striem-AmitE.GuendelmanM.AmediA. (2012). ‘Visual’ acuity of the congenitally blind using visual-to-auditory sensory substitutionPloS One 7e33136. DOI:10.1371/journal.pone.0033136.

TverskyB. (2003). Structures on mental spaces: how people think about spaceEnviron. Behav. 356680.

VandenbergS. G. (1971). Mental Rotation Test. University of ColoradoBoulder, CO, USA.

WanC. Y.WoodA. G.ReutensD. C.WilsonS. J. (2010). Early but not late-blindness leads to enhanced auditory perceptionNeuropsychologia 48344348.

WardJ.MeijerP. (2010). Visual experiences in the blind induced by an auditory sensory substitution deviceConsc. Cogn. 19492500.

WestR. L. (1996). An application of prefrontal cortex function theory to cognitive agingPsychol. Bull. 120272292.

WhiteB. W.SaundersF. A.ScaddenL.Bach-y-RitaP.CollinsC. C. (1970). Seeing with the skinPercept. Psychophys. 72327.

ZwiersM. P.Van OpstalA. J.PaigeG. D. (2003). Plasticity in human sound localization induced by compressed spatial visionNat. Neurosci. 6175181.


  • View in gallery

    Three alternative views (deference, dominance, and vertical integration) on the processes involved in sensory substitution, with their specificities at the behavioural, neural, and phenomenological levels. Illustrative examples are given in parentheses.

  • View in gallery

    Illustration of the two neural streams described by Poirier et al. (2007). Early-blind participants use the cross-modal pathway (solid line). Sighted participants use the mental-imagery pathway (dashed line). We propose three hypotheses to explain this difference. (A) Blind participants already have enhanced cross-modal connections before training with the device. (B) Sensory deprivation allows the recruitment of visual areas. (C) Auditory or tactile abilities determine the recruitment of visual processes. As blind participants have better low-level auditory and tactile capacities, the recruitment of visual areas is facilitated.


Content Metrics

Content Metrics

All Time Past Year Past 30 Days
Abstract Views 24 24 13
Full Text Views 76 76 69
PDF Downloads 4 4 1
EPUB Downloads 0 0 0