Crossmodal Correspondences: Standing Issues and Experimental Guidelines
In: Multisensory ResearchSearch for other papers by Cesare V. Parise in
Current site
Google Scholar
Crossmodal correspondences refer to the systematic associations often found across seemingly unrelated sensory features from different sensory modalities. Such phenomena constitute a universal trait of multisensory perception even in non-human species, and seem to result, at least in part, from the adaptation of sensory systems to natural scene statistics. Despite recent developments in the study of crossmodal correspondences, there are still a number of standing questions about their definition, their origins, their plasticity, and their underlying computational mechanisms. In this paper, I will review such questions in the light of current research on sensory cue integration, where crossmodal correspondences can be conceptualized in terms of natural mappings across different sensory cues that are present in the environment and learnt by the sensory systems. Finally, I will provide some practical guidelines for the design of experiments that might shed new light on crossmodal correspondences.
Purchase
Buy instant access (PDF download and unlimited online access):
Institutional Login
Log in with Open Athens, Shibboleth, or your institutional credentials
Personal login
Log in with your brill.com account
-
Backus B. T. (2011). Recruitment of new visual cues for perceptual appearance, in: Sensory Cue Integration, Trommershäuser J., Körding K., Landy M. (Eds), pp. 101–119. Oxford University Press, New York, NY, USA.
-
Baier B., Kleinschmidt A., Müller N. G. (2006). Cross-modal processing in early visual and auditory cortices depends on expected statistical relationship of multisensory information, J. Neurosci. 26, 12260–12265.
-
Bernstein I. H., Edelstein B. A. (1971). Effects of some variations in auditory input upon visual choice reaction time, J. Exp. Psychol. 87, 241–247.
-
Bien N., Ten Oever S., Goebel R., Sack A. T. (2012). The sound of size: crossmodal binding in pitch–size synesthesia: a combined TMS, EEG and psychophysics study, Neuroimage 59, 663–672.
-
Bien N., Ten Oever S., Goebel R., Sack A. T. (2013). Corrigendum to ‘The sound of size: crossmodal binding in pitch–size synesthesia: a combined TMS, EEG and psychophysics study’, Neuroimage 72, 325.
-
Bremner A. J., Caparos S., Davidoff J., De Fockert J., Linnell K. J., Spence C. (2013). “Bouba” and “Kiki” in Namibia? A remote culture make similar shape–sound matches, but different shape–taste matches to Westerners, Cognition 126, 165–172.
-
Crisinel A. S., Spence C. (2012). A fruity note: crossmodal associations between odors and musical notes, Chem. Senses 37, 151–158.
-
Deroy O., Spence C. (2013). Why we are not all synesthetes (not even weakly so), Psychonom. Bull. Rev. 20, 1–22.
-
Di Luca M., Ernst M. O., Backus B. (2010). Learning to use an invisible visual signal for perception, Curr. Biol. 20, 1860–1863.
-
Dolscheid S., Hunnius S., Casasanto D., Majid A. (2014). Prelinguistic infants are sensitive to space–pitch associations found across cultures, Psychol. Sci. 25, 1256–1261.
-
Ernst M. O. (2007). Learning to integrate arbitrary signals from vision and touch, J. Vis. 7, 7. DOI:10.1167/7.5.7.
-
Ernst M. O., Banks M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion, Nature 415, 429–433.
-
Evans K. K., Treisman A. (2010). Natural cross-modal mappings between visual and auditory features, J. Vis. 10, 6. DOI:10.1167/10.1.6.
-
Fechner G. T. (1860). Elemente der Psychophysik (Elements of Psychophysics). Breitkopf and Hartel, Leipzig, Germany.
-
Flanagan J. R., Beltzner M. A. (2000). Independence of perceptual and sensorimotor predictions in the size–weight illusion, Nat. Neurosci. 3, 737–741.
-
Flanagan J. R., Bittner J. P., Johansson R. S. (2008). Experience can change distinct size–weight priors engaged in lifting objects and judging their weights, Curr. Biol. 18, 1742–1747.
-
Gallace A., Spence C. (2006). Multisensory synesthetic interactions in the speeded classification of visual size, Percept. Psychophys. 68, 1191–1203.
-
Gates G. A., Mills J. H. (2005). Presbycusis, Lancet 366, 1111–1120.
-
Gescheider G. A. (2013). Psychophysics: the Fundamentals. Lawrence Erlbaum, Mahwah, NJ, USA.
-
Green D., Swets J. (1966). Signal Detection Theory and Psychophysics. Wiley, New York, NY, USA.
-
Guzman-Martinez E., Ortega L., Grabowecky M., Mossbridge J., Suzuki S. (2012). Interactive coding of visual spatial frequency and auditory amplitude-modulation rate, Curr. Biol. 22, 383–388.
-
Haijiang Q., Saunders J. A., Stone R. W., Backus B. T. (2006). Demonstration of cue recruitment: change in visual appearance by means of Pavlovian conditioning, Proc. Natl Acad. Sci. USA 103, 483–488.
-
Helmholtz H. V. (1909). Handbuch der Physiologischen Optik (Handbook of Physiological Optics). Voss, Hamburg, Germany.
-
Hillis J., Ernst M. O., Banks M., Landy M. (2002). Combining sensory information: mandatory fusion within, but not between, senses, Science 298, 1627–1630.
-
Holway A. H., Boring E. G. (1941). Determinants of apparent visual size with distance variant, Am. J. Psychol. 54, 21–37.
-
Kaliuzhna M., Prsa M., Gale S., Lee S. J., Blanke O. (2015). Learning to integrate contradictory multisensory self-motion cue pairings, J. Vis. 15, 10. DOI:10.1167/15.1.10.
-
Keetels M., Vroomen J. (2011). No effect of synesthetic congruency on temporal ventriloquism, Atten. Percept. Psychophys. 73, 1–10.
-
Knill D. C., Richards W. (1996). Perception as Bayesian Inference. Cambridge University Press, Cambridge, UK.
-
Knöferle K. M., Woods A., Käppler F., Spence C. (2015). That sounds sweet: using crossmodal correspondences to communicate gustatory attributes, Psychol. Marketing 32, 107–120.
-
Köhler W. (1929). Gestalt Psychology. Liveright, New York, NY, USA.
-
Köhler W. (1947). Gestalt Psychology: an Introduction to New Concepts in Modern Psychology. Liveright, New York, NY, USA.
-
Landy M. S., Maloney L. T., Johnston E. B., Young M. (1995). Measurement and modeling of depth cue combination: in defense of weak fusion, Vis. Res. 35, 389–412.
-
Landy M. S., Ho Y.-X., Serwe S., Trommershäuser J., Maloney L. T. (2011). Cues and pseudocues in texture and shape perception, in: Sensory Cue Integration, Trommershäuser J., Körding K., Landy M. (Eds), pp. 263–278. Oxford University Press, New York, NY, USA.
-
Lewkowicz D. J., Minar N. J. (2014). Infants are not sensitive to synesthetic cross-modality correspondences: a comment on Walker et al. (2010), Psychol. Sci. 25, 832–834.
-
Marks L. E. (1989). On cross-modal similarity: the perceptual structure of pitch, loudness, and brightness, J. Exp. Psychol. Hum. Percept. Perform. 15, 586–602.
-
Marks L. E. (2004). Cross-modal interactions in speeded classification, in: The Handbook of Multisensory Processes, Calvert G. A., Spence C., Stein B. E. (Eds), pp. 85–106. MIT Press, Cambridge, MA, USA.
-
Nielsen A., Rendall D. (2011). The sound of round: evaluating the sound-symbolic role of consonants in the classic Takete–Maluma phenomenon, Can. J. Exp. Psychol. 65, 115–124.
-
Nielsen A. K., Rendall D. (2013). Parsing the role of consonants versus vowels in the classic Takete–Maluma phenomenon, Can. J. Exp. Psychol. 67, 153–163.
-
Occelli V., Esposito G., Venuti P., Arduino G. M., Zampini M. (2013). The takete–maluma phenomenon in autism spectrum disorders, Perception 42, 233–241.
-
Orchard-Mills E., Van der Burg E., Alais D. (2013). Amplitude-modulated auditory stimuli influence selection of visual spatial frequencies, J. Vis. 13, 6. DOI:10.1167/13.3.6.
-
Parise C., Pavani F. (2011). Evidence of sound symbolism in simple vocalizations, Exp. Brain Res. 214, 373–380.
-
Parise C., Spence C. (2008). Synesthetic congruency modulates the temporal ventriloquism effect, Neurosci. Lett. 442, 257–261.
-
Parise C., Spence C. (2009). When birds of a feather flock together: synesthetic correspondences modulate audiovisual integration in non-synesthetes, PLoS One 4, e5664. DOI:10.1371/journal.pone.0005664.
-
Parise C. V., Spence C. (2012). Audiovisual crossmodal correspondences and sound symbolism: a study using the implicit association test, Exp. Brain Res. 220, 319–333.
-
Parise C., Spence C. (2013). Audiovisual cross-modal correspondences in the general population, in: Oxford Handbook of Synaesthesia, Simner J., Hubbard E. M. (Eds), pp. 790–815. Oxford University Press, Oxford, UK.
-
Parise C. V., Knorre K., Ernst M. O. (2014). Natural auditory scene statistics shapes human spatial hearing, Proc. Natl Acad. Sci. USA 111, 6104–6108.
-
Peters M. A. K., Balzer J., Shams L. (2015). Smaller = denser, and the brain knows it: natural statistics of object density shape weight expectations, PLoS One 10, e0119794. DOI:10.1371/journal.pone.0119794.
-
Plaisier M. A., Smeets J. B. J. (2012). Mass is all that matters in the size–weight illusion, PLoS One 7, e42518. DOI:10.1371/journal.pone.0042518.
-
Pratt C. C. (1930). The spatial character of high and low tones, J. Exp. Psychol. 13, 278–285.
-
Rader C. M., Tellegen A. (1987). An investigation of synesthesia, J. Pers. Soc. Psychol. 52, 981–987.
-
Ramachandran V., Hubbard E. (2001). Synaesthesia: a window into perception, thought and language, J. Conscious. Stud. 8, 3–34.
-
Rammsayer T. H., Verner M. (2015). Larger visual stimuli are perceived to last longer from time to time: the internal clock is not affected by nontemporal visual stimulus size, J. Vis. 15, 5. DOI:10.1167/15.3.5.
-
Roffler S. K., Butler R. A. (1968a). Factors that influence the localization of sound in the vertical plane, J. Acoust. Soc. Am. 43, 1255–1259.
-
Roffler S. K., Butler R. A. (1968b). Localization of tonal stimuli in the vertical plane, J. Acoust. Soc. Am. 43, 1260–1266.
-
Rogers M. E., Butler R. A. (1992). The linkage between stimulus frequency and covert peak areas as it relates to monaural localization, Percept. Psychophys. 52, 536–546.
-
Ross H. E. (1969). When is a weight not illusory? Q. J. Exp. Psychol. 21, 346–355.
-
Rusconi E., Kwan B., Giordano B. L., Umilta C., Butterworth B. (2006). Spatial representation of pitch height: the SMARC effect, Cognition 99, 113–129.
-
Sadaghiani S., Maier J. X., Noppeney U. (2009). Natural, metaphoric, and linguistic auditory direction signals have distinct influences on visual motion processing, J. Neurosci. 29, 6490–6499.
-
Sapir E. (1929). A study in phonetic symbolism, J. Exp. Psychol. 12, 225–239.
-
Spence C. (2011). Crossmodal correspondences: a tutorial review, Atten. Percept. Psychophys. 73, 1–25.
-
Spence C., Deroy O. (2012). Crossmodal correspondences: innate or learned? i-Perception 3, 316–318.
-
Spence C., Parise C. V. (2012). The cognitive neuroscience of crossmodal correspondences, i-Perception 3, 410–412.
-
Stevens J. C., Marks L. E. (1965). Cross-modality matching of brightness and loudness, Proc. Natl Acad. Sci. USA 54, 407–411.
-
Stumpf K. (1883). Tonpsychologie (Tone Psychology). Hirzel, Leipzig, Germany.
-
Suzuki Y. I., Takeshima H. (2004). Equal-loudness-level contours for pure tones, J. Acoust. Soc. Am. 116, 918–933.
-
Trommershäuser J., Körding K., Landy M. (Eds) (2011). Sensory Cue Integration. Oxford University Press, New York, NY, USA.
-
Van Dam L. C. J., Ernst M. O. (2015). Mapping shape to visuomotor mapping: learning and generalisation of sensorimotor behaviour based on contextual information, PLoS Comput. Biol. 11, e1004172. DOI:10.1371/journal.pcbi.1004172.
-
Van Dam L. C. J., Parise C. V., Ernst M. O. (2014). Modeling multisensory integration, in: Sensory Integration and the Unity of Consciousness, Bennett D., Hill C. (Eds), pp. 209–229. MIT Press, Cambridge, MA, USA.
-
Walker R. (1987). The effects of culture, environment, age, and musical training on choices of visual metaphors for sound, Percept. Psychophys. 42, 491–502.
-
Walker P., Smith S. (1984). Stroop interference based on the synaesthetic qualities of auditory pitch, Perception 13, 75–81.
-
Walker P., Smith S. (1985). Stroop interference based on the multimodal correlates of haptic size and auditory pitch, Perception 14, 729–736.
-
Walker P., Bremner J., Mason U., Spring J., Mattock K., Slater A., Johnson S. (2010). Preverbal infants’ sensitivity to synaesthetic cross-modality correspondences, Psychol. Sci. 21, 21–25.
-
Yates M. J., Loetscher T., Nicholls M. E. R. (2012). A generalized magnitude system for space, time, and quantity? A cautionary note, J. Vis. 12, 9. DOI:10.1167/12.7.9.
| All Time | Past 365 days | Past 30 Days | |
|---|---|---|---|
| Abstract Views | 3314 | 699 | 80 |
| Full Text Views | 719 | 117 | 27 |
| PDF Views & Downloads | 724 | 172 | 29 |
Crossmodal Correspondences: Standing Issues and Experimental Guidelines
In: Multisensory ResearchSearch for other papers by Cesare V. Parise in
Current site
Google Scholar
PubMed
Crossmodal correspondences refer to the systematic associations often found across seemingly unrelated sensory features from different sensory modalities. Such phenomena constitute a universal trait of multisensory perception even in non-human species, and seem to result, at least in part, from the adaptation of sensory systems to natural scene statistics. Despite recent developments in the study of crossmodal correspondences, there are still a number of standing questions about their definition, their origins, their plasticity, and their underlying computational mechanisms. In this paper, I will review such questions in the light of current research on sensory cue integration, where crossmodal correspondences can be conceptualized in terms of natural mappings across different sensory cues that are present in the environment and learnt by the sensory systems. Finally, I will provide some practical guidelines for the design of experiments that might shed new light on crossmodal correspondences.
| All Time | Past 365 days | Past 30 Days | |
|---|---|---|---|
| Abstract Views | 3314 | 699 | 80 |
| Full Text Views | 719 | 117 | 27 |
| PDF Views & Downloads | 724 | 172 | 29 |
