The Time-Course of the Cross-Modal Semantic Modulation of Visual Picture Processing by Naturalistic Sounds and Spoken Words
In: Multisensory ResearchSearch for other papers by Yi-Chuan Chen in
Current site
Google Scholar
Search for other papers by Charles Spence in
Current site
Google Scholar
The time-course of cross-modal semantic interactions between pictures and either naturalistic sounds or spoken words was compared. Participants performed a speeded picture categorization task while hearing a task-irrelevant auditory stimulus presented at various stimulus onset asynchronies (SOAs) with respect to the visual picture. Both naturalistic sounds and spoken words gave rise to cross-modal semantic congruency effects (i.e., facilitation by semantically congruent sounds and inhibition by semantically incongruent sounds, as compared to a baseline noise condition) when the onset of the sound led that of the picture by 240 ms or more. Both naturalistic sounds and spoken words also gave rise to inhibition irrespective of their semantic congruency when presented within 106 ms of the onset of the picture. The peak of this cross-modal inhibitory effect occurred earlier for spoken words than for naturalistic sounds. These results therefore demonstrate that the semantic priming of visual picture categorization by auditory stimuli only occurs when the onset of the sound precedes that of the visual stimulus. The different time-courses observed for naturalistic sounds and spoken words likely reflect the different processing pathways to access the relevant semantic representations.
Purchase
Buy instant access (PDF download and unlimited online access):
Institutional Login
Log in with Open Athens, Shibboleth, or your institutional credentials
Personal login
Log in with your brill.com account
-
Alais D., Morrone C., Burr D. (2006). Separate attentional resources for vision and audition, Proc. R. Soc. B 273, 1339–1345.
-
Arnell K. M., Larson J. M. (2002). Cross-modality attentional blinks without preparatory task-set switching, Psychon. B. Rev. 9, 497–506.
-
Arrighi R., Lunardi R., Burr D. (2011). Vision and audition do not share attentional resources in sustained tasks, Front. Psychol. 2, 56.
-
Ballas J. A. (1993). Common factors in the identification of an assortment of brief everyday sounds, J. Exp. Psychol. Human 19, 240–267.
-
Bates E., D’Amico S., Jacobsen T., Székely A., Andonova E., Devescovi A., Herron D., Ching Lu C., Pechmann T., Pléh C., Wicha N., Federmeier K., Gerdjikova I., Gutierrez G., Hung D., Hsu J., Iyer G., Kohnert K., Mehotcheva T., Orozco-Figueroa A., Tzeng A., Tzeng O. (2003). Timed picture naming in seven languages, Psychon. Bull. Rev. 10, 344–380.
-
Chen Y.-C., Spence C. (2010). When hearing the bark helps to identify the dog: Semantically congruent sounds modulate the identification of masked pictures, Cognition 114, 389–404.
-
Chen Y.-C., Spence C. (2011). Cross-modal semantic priming by naturalistic sounds and spoken words enhances visual sensitivity, J. Exp. Psychol. Human 37, 1554–1568.
-
Coltheart M. (2000). Dual routes from print to speech and dual routes from print to meaning: some theoretical issues, in: Reading as a Perceptual Process, Kennedy A., Radach R., Pynte J., Heller D. (Eds), pp. 475–490. Elsevier, Oxford, UK.
-
Cummings A., Ceponiene R., Koyama A., Saygin A. P., Townsend J., Dick F. (2006). Auditory semantic networks for words and natural sounds, Brain Res. 1115, 92–107.
-
Dick F., Bussiere J., Saygin A. P. (2002). The effects of linguistic mediation on the identification of environmental sounds, Center for Research in Language Newsletter 14, 3–9.
-
Donohue S. E., Appelbaum L. G., Park C. J., Roberts K. C., Woldorff M. G. (submitted). Cross-modal stimulus conflict: the effects of stimulus input timing in a visual–auditory Stroop task.
-
Duncan J., Martens S., Ward R. (1997). Restricted attentional capacity within but not between sensory modalities, Nature 387, 808–810.
-
Elliott E. M., Cowan N., Valle-Inclan F. (1998). The nature of cross-modal color–word interference effects, Percept. Psychophys. 60, 761–767.
-
Glaser W. R., Glaser M. O. (1989). Context effects in Stroop-like word and picture processing, J. Exp. Psychol. Gen. 118, 13–42.
-
Grill-Spector K., Kanwisher N. (2005). Visual recognition: as soon as you know it is there, you know what it is, Psychol. Sci. 16, 152–160.
-
Hays W. (1973). Statistics for the Social Sciences. Holt, Rinehart and Winston, New York, NY, USA.
-
Iordanescu L., Guzman-Martinez E., Grabowecky M., Suzuki S. (2008). Characteristic sound facilitates visual search, Psychon. B. Rev. 15, 548–554.
-
Iordanescu L., Grabowecky M., Franconeri S., Theeuwes J., Suzuki S. (2010). Characteristic sounds make you look at target objects more quickly, Atten. Percept. Psychophys. 72, 1736–1741.
-
Klemen J., Buchel C., Rose M. (2009). Perceptual load interacts with stimulus processing across sensory modalities, Eur. J. Neurosci. 29, 2426–2434.
-
Laurienti P. J., Kraft R. A., Maldjian J. A., Burdette J. H., Wallace M. T. (2004). Semantic congruence is a critical factor in multisensory behavioral performance, Exp. Brain Res. 158, 405–414.
-
Mack M. L., Gauthier I., Sadr J., Palmeri T. J. (2008). Object detection and basic-level categorization: sometimes you know it is there before you know what it is, Psychon. B. Rev. 15, 28–35.
-
Marcell M. M., Borella D., Greene M., Kerr E., Rogers S. (2000). Confrontation naming of environmental sounds, J. Clin. Exp. Neuropsyc. 22, 830–864.
-
Marcell M., Malatanos M., Leahy C., Comeaux C. (2007). Identifying, rating and remembering environmental sound events, Behav. Res. Methods 39, 561–569.
-
Murray M. M., Spierer L. (2009). Auditory spatio-temporal brain dynamics and their consequences for multisensory interactions in humans, Hearing Res. 258, 121–133.
-
Murray M. M., Camen C., Andino S. L. G., Bovet P., Clarke S. (2006). Rapid brain discrimination of sounds of objects, J. Neurosci. 26, 1293–1302.
-
Robertson I. H., Mattingley J. B., Rorden C., Driver J. (1998). Phasic alerting of neglect patients overcomes their spatial deficit in visual awareness, Nature 395, 169–173.
-
Roelofs A. (2005). The visual–auditory color–word Stroop asymmetry and its time course, Mem. Cognit. 33, 1325–1336.
-
Santangelo V., Belardinelli M. O., Spence C. (2007). The suppression of reflexive visual and auditory orienting when attention is otherwise engaged, J. Exp. Psychol. Human 33, 137–148.
-
Saygin A. P., Dick F., Bates E. (2005). An on-line task for contrasting auditory processing in the verbal and nonverbal domains and norms for younger and older adults, Behav. Res. Methods 37, 99–110.
-
Schneider T. R., Engel A. K., Debener S. (2008). Multisensory identification of natural objects in a two-way cross-modal priming paradigm, Exp. Psychol. 55, 121–132.
-
Shimada H. (1990). Effect of auditory presentation of words on color naming: the intermodal Stroop effect, Percept. Motor Skill 70, 1155–1161.
-
Shore D. I., Barnes M. E., Spence C. (2006). Temporal aspects of the visuotactile congruency effect, Neurosci. Lett. 392, 96–100.
-
Smith M. C., Magee L. E. (1980). Tracing the time course of picture–word processing, J. Exp. Psychol. Gen. 109, 373–392.
-
Snodgrass J. G., Vanderwart M. (1980). A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity, J. Exp. Psychol.-Hum. Learn. 6, 174–215.
-
Spence C., Driver J. (1997). Audiovisual links in exogenous covert spatial orienting, Percept. Psychophys. 59, 1–22.
-
Stuart D. M., Carrasco M. (1993). Semantic component of a cross-modal Stroop-like task, Am. J. Psychol. 106, 383–405.
-
Van Petten C., Coulson S., Rubin S., Plante E., Parks M. (1999). Time course of word identification and semantic integration in spoken language, J. Exp. Psychol. Learn. 25, 394–417.
-
Yuval-Greenberg S., Deouell L. Y. (2009). The dog’s meow: asymmetrical interaction in cross-modal object recognition, Exp. Brain Res. 193, 603–614.
| All Time | Past Year | Past 30 Days | |
|---|---|---|---|
| Abstract Views | 567 | 94 | 4 |
| Full Text Views | 187 | 25 | 1 |
| PDF Views & Downloads | 124 | 47 | 3 |
The Time-Course of the Cross-Modal Semantic Modulation of Visual Picture Processing by Naturalistic Sounds and Spoken Words
In: Multisensory ResearchSearch for other papers by Yi-Chuan Chen in
Current site
Google Scholar
PubMed
Search for other papers by Charles Spence in
Current site
Google Scholar
PubMed
The time-course of cross-modal semantic interactions between pictures and either naturalistic sounds or spoken words was compared. Participants performed a speeded picture categorization task while hearing a task-irrelevant auditory stimulus presented at various stimulus onset asynchronies (SOAs) with respect to the visual picture. Both naturalistic sounds and spoken words gave rise to cross-modal semantic congruency effects (i.e., facilitation by semantically congruent sounds and inhibition by semantically incongruent sounds, as compared to a baseline noise condition) when the onset of the sound led that of the picture by 240 ms or more. Both naturalistic sounds and spoken words also gave rise to inhibition irrespective of their semantic congruency when presented within 106 ms of the onset of the picture. The peak of this cross-modal inhibitory effect occurred earlier for spoken words than for naturalistic sounds. These results therefore demonstrate that the semantic priming of visual picture categorization by auditory stimuli only occurs when the onset of the sound precedes that of the visual stimulus. The different time-courses observed for naturalistic sounds and spoken words likely reflect the different processing pathways to access the relevant semantic representations.
| All Time | Past Year | Past 30 Days | |
|---|---|---|---|
| Abstract Views | 567 | 94 | 4 |
| Full Text Views | 187 | 25 | 1 |
| PDF Views & Downloads | 124 | 47 | 3 |