The renewed interest that has emerged around the topic of crossmodal correspondences in recent years has demonstrated that crossmodal matchings and mappings exist between the majority of sensory dimensions, and across all combinations of sensory modalities. This renewed interest also offers a rapidly-growing list of ways in which correspondences affect — or interact with — metaphorical understanding, feelings of ‘knowing’, behavioral tasks, learning, mental imagery, and perceptual experiences. Here we highlight why, more generally, crossmodal correspondences matter to theories of multisensory interactions.
Purchase
Buy instant access (PDF download and unlimited online access):
Institutional Login
Log in with Open Athens, Shibboleth, or your institutional credentials
Personal login
Log in with your brill.com account
Adeli M., Rouat J., Molotchnikoff S. (2014). Audiovisual correspondence between musical timbre and visual shapes, Front. Hum. Neurosci. 8, 352.
Alais D., Burr D. (2004). The ventriloquist effect results from near-optimal bimodal integration, Curr. Biol. 14, 257–262.
Albertazzi L., Malfatti M., Canal L., Micciolo R. (2015). The hue of angles — was Kandinsky right? Art Percept. 3, 81–92.
Alexander K. R., Shansky M. S. (1976). Influence of hue, value, and chroma on the perceived heaviness of colors, Percept. Psychophys. 19, 72–74.
Altieri N., Stevenson R. A., Wallace M. T., Wenger M. J. (2013). Learning to associate auditory and visual stimuli: behavioral and neural mechanisms, Brain Topogr. DOI:10.1007/s10548-013-0333-7.
Asano M., Imai M., Kita S., Kitajo K., Okada H., Thierry G. (2015). Sound symbolism scaffolds language development in preverbal infants, Cortex 63, 196–205.
Athanasopoulos G., Moran N. (2013). Cross-cultural representations of musical shape, Empir. Musicol. Rev. 8, 185–199.
Baudelaire C. (1857). Les Fleurs du mal [Flowers of Evil]. Poulet-Malassis, Paris, France.
Berkeley G. (1709/1975). An essay toward a new theory of vision, in: Philosophical Works, Ayers M. R. (Ed.), pp. 1–59. Dent, London, UK.
Botvinick M., Cohen J. (1998). Rubber hands ‘feel’ touch that eyes see, Nature 391(6669), 756.
Bremner A., Caparos S., Davidoff J., de Fockert J., Linnell K., Spence C. (2013). Bouba and Kiki in Namibia? A remote culture makes similar shape–sound matches, but different shape–taste matches to Westerners, Cognition 126, 165–172.
Brunel L., Carvalho P., Goldstone R. L. (in press). It does belong together: cross-modal correspondences influence cross-modal integration during perceptual learning, Front. Psychol. 6, 358. DOI:10.3389/fpsyg.2015.00358.
Calvert G., Spence C., Stein B. E. (Eds) (2004). The Handbook of Multisensory Processing. MIT Press, Cambridge, MA, USA.
Chen L. (2013). Synaesthetic correspondence between auditory clips and colors: an empirical study, in: Intelligent Science and Intelligent Data Engineering, Second Sino-Foreign-Interchange Workshop, IScIDE 2011, Xi’an, China, October 23–25, 2011, Revised Selected Papers, pp. 507–513. Springer, Berlin, Germany.
Chen Y.-C., Spence C. (2011). Crossmodal semantic priming by naturalistic sounds and spoken words enhances visual sensitivity, J. Exp. Psychol. Hum. Percept. Perform. 37, 1554–1568.
Chen N., Tanaka K., Watanabe K. (2015). Color–shape associations revealed with implicit association tests, PLoS ONE 10, e0116954. DOI:10.1371/journal.pone.0116954.
Chiou R., Rich A. N. (2012). Cross-modality correspondence between pitch and spatial location modulates attentional orienting, Perception 41, 339–353.
Clark A. (1993). Sensory Qualities. Oxford University Press, Oxford, UK.
Cohen N. E. (1934). Equivalence of brightness across modalities, Am. J. Psychol. 46, 117–119.
Connell L., Cai Z. G., Holler J. (2013). Do you see what I’m singing? Visuospatial movement biases pitch perception, Brain Cogn. 81, 124–130.
Connolly K. (2014). Multisensory perception as an associative learning process, Front. Psychol. 5, 1095. DOI:10.3389/fpsyg.2014.01095.
Crisinel A.-S., Spence C. (2012). A fruity note: crossmodal associations between odors and musical notes, Chem. Sens. 37, 151–158.
Crisinel A.-S., Cosser S., King S., Jones R., Petrie J., Spence C. (2012). A bittersweet symphony: systematically modulating the taste of food by changing the sonic properties of the soundtrack playing in the background, Food Qual. Pref. 24, 201–204.
Deroy O. (2014). The unity assumption and the many unities of consciousness, in: Sensory Integration and the Unity of Consciousness, Bennett D., Hill C. (Eds), pp. 105–124. MIT Press, Cambridge, MA, USA.
Deroy O., Spence C. (2013a). Are we all born synaesthetic? Examining the neonatal synaesthesia hypothesis, Neurosci. Biobehav. Rev. 37, 1240–1253.
Deroy O., Spence C. (2013b). Why we are not all synesthetes (not even weakly so), Psychon. Bull. Rev. 20, 643–664.
Deroy O., Spence C. (submitted). The role of vision in crossmodal matchings.
Deroy O., Valentin D. (2011). Tasting shapes: investigating the sensory basis of cross-modal correspondences, Chemosens. Percept. 4, 80–90.
Deroy O., Crisinel A.-S., Spence C. (2013). Crossmodal correspondences between odors and contingent features: odors, musical notes, and geometrical shapes, Psychon. Bull. Rev. 20, 878–896.
Doehrmann O., Naumer M. J. (2008). Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration, Brain Res. 1242, 136–150.
Dolscheid S., Hunnius S., Casasanto D., Majid A. (2014). Prelinguistic infants are sensitive to space-pitch associations found across cultures, Psychol. Sci. 25, 1256–1261.
Duck M. J. (1988). Newton and Goethe on colour: physical and physiological considerations, Ann. Sci. 45, 507–519.
Eitan Z. (2013a). How pitch and loudness shape musical space and motion: new findings and persisting questions, in: The Psychology of Music in Multimedia, Tan S. L., Cohen A., Lipscomb S., Kendall R. (Eds), pp. 161–187. Oxford University Press, Oxford, UK.
Eitan Z. (2013b). Which is more? Pitch height, parametric scales, and the intricacies of cross-domain magnitude relationships, in: Musical I: Essays in Honor of Eugene Narmour, Brenstein L., Rozin A. (Eds), pp. 131–148. Pendragon Press, Hillsdale, NY, USA.
Eitan Z. (2013c). Musical objects, cross-domain correspondences, and cultural choice: commentary on “Cross-cultural representations of musical shape” by George Athanasopoulos and Nikki Moran, Empir. Musicol. Rev. 8, 204–207.
Eitan Z., Schupak A., Gotler A., Marks L. E. (2014). Lower pitch is larger, yet falling pitches shrink: interaction of pitch change and size change in speeded discrimination, Exp. Psychol. 61, 273–284.
Eitan Z., Timmers R., Adler M. (in press). The mapping of musical shapes to multiple dimensions, in: Music and Shape, Leech-Wilkinson D., Prior H. (Eds). Oxford University Press, Oxford, UK.
Ernst M. O., Bülthoff H. H. (2004). Merging the senses into a robust percept, Trends Cogn. Sci. 8, 162–169.
Evans K. K., Treisman A. (2010). Natural cross-modal mappings between visual and auditory features, J. Vis. 10(6), 1–12.
Faragó T., Pongrácz P., Miklósi Á., Huber L., Virányi Z., Range F. (2010). Dogs’ expectation about signalers’ body size by virtue of their growls, PLoS ONE 5, e15175. DOI:10.1371/journal.pone.0015175.
Fernández-Prieto I., Vera-Constán F., García-Morera J., Navarra J. (2012). Spatial recoding of sound: pitch-varying auditory cues modulate up/down visual spatial attention, See. Perceiv. 25, 150–151.
Forceville C. (2006). Non-verbal and multimodal metaphor in a cognitivist framework: agendas for research, in: Cognitive Linguistics: Current Applications and Future Perspectives, Kristiansen G., Achard M., Dirven R., Ruiz de Mendoza Ibàñez F. (Eds), pp. 379–402. Mouton de Gruyter, New York, NY, USA.
Gallace A., Spence C. (2006). Multisensory synesthetic interactions in the speeded classification of visual size, Percept. Psychophys. 68, 1191–1203.
Garrett C. (1984). Swedenborg and the mystical enlightenment in late eighteenth-century England, J. Hist. Ideas 45, 67–81.
Ghazanfar A. A., Schroeder C. E. (2006). Is neocortex essentially multisensory? Trends Cogn. Sci. 10, 278–285.
Gibson J. J. (1966). The Senses Considered as Perceptual Systems. Houghton Mifflin, Boston, MA, USA.
Guerdoux E., Trouillet R., Brouillet D. (2014). Olfactory–visual congruence effects stable across ages: yellow is warmer when it is pleasantly lemony, Atten. Percept. Psychophys. 76, 1280–1286.
Hanson-Vaux G., Crisinel A.-S., Spence C. (2013). Smelling shapes: crossmodal correspondences between odors and shapes, Chem. Sens. 38, 161–166.
Haverkamp M. (2014). Synesthetic Design: Handbook for a Multi-Sensory Approach. Birkhäuser Verlag, Basel, Switzerland.
Hedger S. C., Nusbaum H. C., Lescop O., Wallisch P., Hoeckner B. (2013). Music can elicit a visual motion aftereffect, Atten. Percept. Psychophys. 75, 1039–1047.
Held R., Ostrovsky Y., de Gelder B., Gandhi T., Ganesh S., Mathur U., Sinha P. (2011). The newly sighted fail to match seen with felt, Nat. Neurosci. 14, 551–553.
Helmholtz H. (1878/1971). The facts of perception, in: Selected Writings of Hermann Helmholtz. Wesleyan University Press, Middletown, CT, USA.
Henrich J., Heine S. J., Norenzayan A. (2010). The weirdest people in the world? Behav. Brain Sci. 33, 61–135.
Henschke J. U., Noesselt T., Scheich H., Budinger E. (2014). Possible anatomical pathways for short-latency multisensory integration processes in primary sensory cortices, Brain Struct. Funct. 219, 1–23.
Ho H. N., Van Doorn G. H., Kawabe T., Watanabe J., Spence C. (2014). Colour–temperature correspondences: when reactions to thermal stimuli are influenced by colour, PLoS ONE 9, e91854. DOI:10.1371/journal.pone.0091854.
Imai M., Kita S. (2014). The sound symbolism bootstrapping hypothesis for language acquisition and language evolution, Phil. Trans. R. Soc. B Biol. Sci. 369(1651), 20130298.
Jeschonek S., Pauen S., Babocsai L. (2013). Cross-modal mapping of visual and acoustic displays in infants: the effect of dynamic and static components, Eur. J. Dev. Psychol. 10, 337–358.
Johnson M., Lakoff G. (1980). Metaphors We Live by. University of Chicago Press, Chicago, IL, USA.
Klapetek A., Ngo M. K., Spence C. (2012). Do crossmodal correspondences enhance the facilitatory effect of auditory cues on visual search? Atten. Percept. Psychophys. 74, 1154–1167.
Köhler W. (1929). Gestalt Psychology. Liveright, New York, NY, USA.
Köhler W. (1947). Gestalt Psychology: An Introduction to New Concepts in Modern Psychology. Liveright, New York, NY, USA.
Kohn D., Eitan Z. (2009). Musical parameters and children’s movement responses, in: Proceedings of the 7th Triennial Conference of European Society for the Cognitive Sciences of Music, J. Louhivuori, T. Eerola, S. Saarikallio, T. Himberg and P. S. Eerola (Eds), pp. 233–241. ESCOM 2009, Jyväskylä, Finland.
Koriat A. (1975). Phonetic symbolism and feeling of knowing, Mem. Cogn. 3, 545–548.
Koriat A. (2008). Subjective confidence in one’s answers: the consensuality principle, J. Exp. Psychol. Learn. Mem. Cogn. 34, 945–959.
Kovic V., Plunkett K., Westermann G. (2010). The shape of words in the brain, Cognition 114, 19–28.
Langlois T., Schloss K., Palmer S. (2013). Music–color associations to simple melodies in synesthetes and non-synesthetes, J. Vis. 13, 1325.
Laurienti P. J., Kraft R. A., Maldjian J. A., Burdette J. H., Wallace M. T. (2004). Semantic congruence is a critical factor in multisensory behavioral performance, Exp. Brain Res. 158, 405–414.
Liang M., Mouraux A., Hu L., Iannetti G. D. (2013). Primary sensory cortices contain distinguishable spatial patterns of activity for each sense, Nat. Commun. 4, 1979.
Ludwig V. U., Adachi I., Matzuzawa T. (2011). Visuoauditory mappings between high luminance and high pitch are shared by chimpanzees (Pan troglodytes) and humans, Proc. Natl Acad. Sci. USA 108, 20661–20665.
Maes P. J., Leman M. (2013). The influence of body movements on children’s perception of music with an ambiguous expressive character, PLoS ONE 8, e54682. DOI:10.1371/journal.pone.0054682.
Marks L. E. (1974). On associations of light and sound: the mediation of brightness, pitch, and loudness, Am. J. Psychol. 87, 173–188.
Marks L. E. (1978). The Unity of the Senses: Interrelations Among the Modalities. Academic Press, New York, NY, USA.
Marks L. E. (1982). Bright sneezes and dark coughs, loud sunlight and soft moonlight, J. Exp. Psychol. Hum. Percept. Perform. 8, 177–193.
Marks L. E. (1996). On perceptual metaphors, Metaphor Symb. 11, 39–66.
Marks L. (2004). Crossmodal interactions in speeded classifications, in: The Handbook of Multisensory Processes, Calvert G., Spence C., Stein B. E. (Eds), pp. 85–105. MIT Press, Cambridge, MA, USA.
Martino G., Marks L. E. (1999). Perceptual and linguistic interactions in speeded classification: tests of the semantic coding hypothesis, Perception 28, 903–924.
McGurk H., MacDonald J. (1976). Hearing lips and seeing voices, Nature 264(5588), 746–748.
Meier B. P., Robinson M. D. (2004). Why the sunny side is up: associations between affect and vertical position, Psychol. Sci. 15, 243–247.
Melara R. D. (1989). Dimensional interaction between color and pitch, J. Exp. Psychol. Hum. Percept. Perform. 15, 69–79.
Melara R. D., Marks L. E. (1990). Processes underlying dimensional interactions: correspondences between linguistic and nonlinguistic dimensions, Mem. Cogn. 18, 477–495.
Miller G. (2012). The smartphone psychology manifesto, Perspect. Psychol. Sci. 7, 221–237.
Monroe M. (1926). The apparent weight of color and correlated phenomena, Am. J. Psychol. 36, 192–206.
Morton E. (1994). Sound symbolism and its role in non-human vertebrate communication, in: Sound Symbolism, Hinton L., Nichols J., Ohala J. (Eds), pp. 348–365. Cambridge University Press, Cambridge, UK.
Newton I. (1675/1757). Hypothesis explaining the properties of light, in: History of the Royal Society, Vol. III, Birch T. (Ed.), pp. 247–305. A. Millar, London, UK.
Orchard-Mills E., Alais D., Van der Burg E. (2013a). Amplitude-modulated auditory stimuli influence selection of visual spatial frequencies, J. Vis. 13, 6. DOI:10.1167/13.3.6.
Orchard-Mills E., Alais D., Van der Burg E. (2013b). Cross-modal associations between vision, touch and audition influence visual search through top-down attention, not bottom-up capture, Atten. Percept. Psychophys. 75, 1892–1905.
Palmer S. E., Schloss K. B., Xu Z., Prado-León L. R. (2013). Music–color associations are mediated by emotion, Proc. Natl Acad. Sci. USA 110, 8836–8841.
Parise C. V., Spence C. (2009). When birds of a feather flock together: synesthetic correspondences modulate audiovisual integration in non-synesthetes, PLoS ONE 4, e5664. DOI:10.1371/journal.pone.0005664.
Parise C. V., Knorre K., Ernst M. O. (2014). Natural auditory scene statistics shapes human spatial hearing, Proc. Natl Acad. Sci. USA 111, 6104–6108.
Pesic P. (2006). Isaac Newton and the mystery of the major sixth: a transcription of his manuscript ’Of Musick’ with commentary, Interdiscipl. Sci. Rev. 31, 291–306.
Ramachandran V. S., Hubbard E. M. (2001). Synaesthesia — a window into perception, thought and language, J. Conscious. Stud. 8, 3–34.
Revill K. P., Namy L. L., DeFife L. C., Nygaard L. C. (2014). Cross-linguistic sound symbolism and crossmodal correspondence: evidence from fMRI and DTI, Brain Lang. 128, 18–24.
Roach N. W., Heron J., McGraw P. V. (2006). Resolving multisensory conflict: a strategy for balancing the costs and benefits of audio-visual integration, Proc. R. Soc. B 273, 2159–2168.
Rouw R., Case L., Gosavi R., Ramachandran V. (2014). Color associations for days and letters across different languages, Front. Psychol. 5, 369. DOI:10.3389/fpsyg.2014.00369.
Sadaghiani S., Maier J. X., Noppeney U. (2009). Natural, metaphoric, and linguistic auditory direction signals have distinct influences on visual motion processing, J. Neurosci. 29, 6490–6499.
Schutz M., Lipscomb S. (2007). Hearing gestures, seeing music: vision influences perceived tone duration, Perception 36, 888–892.
Sekuler R., Sekuler A. B., Lau R. (1997). Sound alters visual motion perception, Nature 385(6614), 308.
Seo H.-S., Arshamian A., Schemmer K., Scheer I., Sander T., Ritter G., Hummel T. (2010). Cross-modal integration between odors and abstract symbols, Neurosci. Lett. 478, 175–178.
Shams L., Kamitani Y., Shimojo S. (2000). Illusions. What you see is what you hear, Nature 408(6814), 788.
Shayan S., Ozturk O., Sicoli M. A. (2011). The thickness of pitch: crossmodal metaphors in Farsi, Turkish, and Zapotec, Sens. Soc. 6, 96–105.
Simner J., Ward J., Lanz M., Jansari A., Noonan K., Glover L., Oakley D. A. (2005). Non-random associations of graphemes to colours in synaesthetic and non-synaesthetic populations, Cogn. Neuropsychol. 22, 1069–1085.
Spence C. (2007). Audiovisual multisensory integration, Acoust. Sci. Technol. 28, 61–70.
Spence C. (2010). Multisensory integration: solving the crossmodal binding problem. Comment on “Crossmodal influences on visual perception” by Shams & Kim, Physics of Life Rev. 7, 285–286.
Spence C. (2011). Crossmodal correspondences: a tutorial review, Atten. Percept. Psychophys. 73, 971–995.
Spence C. (2013). Just how important is spatial coincidence to multisensory integration? Evaluating the spatial rule, Ann. NY Acad. Sci. 1296, 31–49.
Spence C., Deroy O. (2013a). How automatic are crossmodal correspondences? Conscious. Cogn. 22, 245–260.
Spence C., Deroy O. (2013b). Crossmodal mental imagery, in: Multisensory Imagery, Lacey S., Lawson R. (Eds), pp. 157–183. Springer, New York, NY, USA.
Spence C., Driver J. (Eds) (2004). Crossmodal Space and Crossmodal Attention. Oxford University Press, Oxford, UK.
Spence C., Ngo M. (2012). Assessing the shape symbolism of the taste, flavour, and texture of foods and beverages, Flavour 1, 12.
Stein B. E. (Ed.) (2012). The New Handbook of Multisensory Processing. MIT Press, Cambridge, MA, USA.
Stein B. E., Meredith M. A. (1993). The Merging of the Senses. MIT Press, Cambridge, MA, USA.
Stevens J. C., Marks L. E. (1965). Cross-modality matching of brightness and loudness, Proc. Natl Acad. Sci. USA 54, 407–411.
Sweeny T. D., Guzman-Martinez E., Ortega L., Grabowecky M., Suzuki S. (2012). Sounds exaggerate visual shape, Cognition 124, 194–200.
Takeshima Y., Gyoba J. (2013). Changing pitch of sounds alters perceived visual motion trajectory, Multisens. Res. 26, 317–332.
Tomasik-Krótki J., Strojny J. (2008). Scaling of sensory impressions, J. Sens. Stud. 23, 251–266.
Treisman A. (1996). The binding problem, Curr. Opin. Neurobiol. 6, 171–178.
Vatakis A., Ghazanfar A., Spence C. (2008). Facilitation of multisensory integration by the ‘unity assumption’: is speech special? J. Vis. 8, 1–11.
Velasco C., Salgado-Montejo A., Marmolejo-Ramos F., Spence C. (2014). Predictive packaging design: tasting shapes, typographies, names, and sounds, Food Qual. Pref. 34, 88–95.
Velasco C., Woods A. T., Deroy O., Spence C. (2015). Hedonic mediation of the crossmodal correspondence between taste and shape, Food Qual. Pref. 41, 151–158.
Von Hornbostel E. M. (1931). Über Geruchshelligkeit [On smell brightness], Pflügers Arch. gesamte Physiol. Menschen Tiere 227, 517–538.
Von Hornbostel E. M. (1950). The unity of the senses, in: A Source Book of Gestalt Psychology, Ellis W. D. (Ed.), pp. 210–216. Routledge and Kegan Paul, London, UK.
Walker P. (2012). Cross-sensory correspondences and cross talk between dimensions of connotative meaning: visual angularity is hard, high-pitched, and bright, Atten. Percept. Psychophys. 74, 1792–1809.
Wan X., Woods A. T., Van den Bosch J., Mckenzie K. J., Velasco C., Spence C. (2014). Cross-cultural differences in crossmodal correspondences between tastes and visual features, Front. Psychol. Cogn. 5, 1365.
Welch R. B., Warren D. H. (1980). Immediate perceptual response to intersensory discrepancy, Psychol. Bull. 88, 638–667.
Welch R. B., Warren D. H. (1986). Intersensory interactions, in: Handbook of Perception and Performance. Vol. 1. Sensory Processes and Perception, Boff K. R., Kaufman L., Thomas J. P. (Eds), pp. 25-1–25-36. Wiley, New York, NY, USA.
Werning M., Fleischhauer J., Beseoglu H. (2006). The cognitive accessibility of synaesthetic metaphor, in: Proceedings of the Twenty-Eighth Annual Conference of the Cognitive Science Society, pp. 2365–2370. Québec City, QC, Canada.
Westbury C. (2005). Implicit sound symbolism in lexical access: evidence from an interference task, Brain Lang. 93, 10–19.
Wheatley J. (1973). Putting colour into marketing, Marketing 67, 24–29.
Woods A. T., Spence C., Butcher N., Deroy O. (2013). Fast lemons and sour boulders: testing the semantic hypothesis of crossmodal correspondences using an internet-based testing methodology, i-Perception 4, 365–369.
Yu N. (2003). Synesthetic metaphor: a cognitive perspective, J. Lit. Semantics 32, 19–34.
All Time | Past 365 days | Past 30 Days | |
---|---|---|---|
Abstract Views | 2441 | 426 | 50 |
Full Text Views | 503 | 30 | 8 |
PDF Views & Downloads | 355 | 57 | 13 |
The renewed interest that has emerged around the topic of crossmodal correspondences in recent years has demonstrated that crossmodal matchings and mappings exist between the majority of sensory dimensions, and across all combinations of sensory modalities. This renewed interest also offers a rapidly-growing list of ways in which correspondences affect — or interact with — metaphorical understanding, feelings of ‘knowing’, behavioral tasks, learning, mental imagery, and perceptual experiences. Here we highlight why, more generally, crossmodal correspondences matter to theories of multisensory interactions.
All Time | Past 365 days | Past 30 Days | |
---|---|---|---|
Abstract Views | 2441 | 426 | 50 |
Full Text Views | 503 | 30 | 8 |
PDF Views & Downloads | 355 | 57 | 13 |