Auditory–Visual Aversive Stimuli Modulate the Conscious Experience of Fear

in Multisensory Research
Restricted Access
Get Access to Full Text
Rent on DeepDyve

Have an Access Token?

Enter your access token to activate and access content online.

Please login and go to your personal user account to enter your access token.


Have Institutional Access?

Access content through your institution. Any other coaching guidance?


In a natural environment, affective information is perceived via multiple senses, mostly audition and vision. However, the impact of multisensory information on affect remains relatively undiscovered. In this study, we investigated whether the auditory–visual presentation of aversive stimuli influences the experience of fear. We used the advantages of virtual reality to manipulate multisensory presentation and to display potentially fearful dog stimuli embedded in a natural context. We manipulated the affective reactions evoked by the dog stimuli by recruiting two groups of participants: dog-fearful and non-fearful participants. The sensitivity to dog fear was assessed psychometrically by a questionnaire and also at behavioral and subjective levels using a Behavioral Avoidance Test (BAT). Participants navigated in virtual environments, in which they encountered virtual dog stimuli presented through the auditory channel, the visual channel or both. They were asked to report their fear using Subjective Units of Distress. We compared the fear for unimodal (visual or auditory) and bimodal (auditory–visual) dog stimuli. Dog-fearful participants as well as non-fearful participants reported more fear in response to bimodal audiovisual compared to unimodal presentation of dog stimuli. These results suggest that fear is more intense when the affective information is processed via multiple sensory pathways, which might be due to a cross-modal potentiation. Our findings have implications for the field of virtual reality-based therapy of phobias. Therapies could be refined and improved by implicating and manipulating the multisensory presentation of the feared situations.

Auditory–Visual Aversive Stimuli Modulate the Conscious Experience of Fear

in Multisensory Research



BaumgartnerT.LutzK.SchmidtC. F.JänckeL. (2006). The emotional power of music: How music enhances the feeling of affective picturesBrain Res. 1075151164. DOI:10.1016/j.brainres.2005.12.065

BenedekM.KaernbachC. (2010). A continuous measure of phasic electrodermal activityJ. Neurosci. Meth. 1908091. DOI:10.1016/j.jneumeth.2010.04.028

BohilC. J.AliceaB.BioccaF. A. (2011). Virtual reality in neuroscience research and therapyNat. Rev. Neurosci. 12752762. DOI:10.1038/nrn3122

BotellaC.BañosR. M.PerpiñáC.VillaH.AlcañizM.ReyA. (1998). Virtual reality treatment of claustrophobia: a case reportBehav. Res. Ther. 36239246.

BradleyM. M.CodispotiM.CuthbertB. N.LangP. J. (2001). Emotion and motivation I: Defensive and appetitive reactions in picture processingEmotion 1276298. DOI:10.1037/1528-3542.1.3.276

BuettiS.LlerasA. (2012). Perceiving control over aversive and fearful events can alter how we experience those events: An investigation of time perception in spider-fearful individualsFront. Psychol. 3337. DOI:10.3389/fpsyg.2012.00337

ChapadosC.LevitinD. J. (2008). Cross-modal interactions in the experience of musical performances: Physiological correlatesCognition 108639651. DOI:10.1016/j.cognition.2008.05.008

ChenY.EdgarJ. C.HolroydT.DammersJ.ThönnessenH.RobertsT. P. L.MathiakK. (2010). Neuromagnetic oscillations to emotional faces and prosodyEur. J. Neurosci. 3118181827. DOI:10.1111/j.1460-9568.2010.07203.x

ClerkinE. M.CodyM. W.StefanucciJ. K.ProffittD. R.TeachmanB. A. (2009). Imagery and fear influence height perceptionJ. Anxiety Disord. 23381386. DOI:10.1016/j.janxdis.2008.12.002

CollignonO.GirardS.GosselinF.RoyS.Saint-AmourD.LassondeM.LeporeF. (2008). Audio-visual integration of emotion expressionBrain Res. 1242126135. DOI:10.1016/j.brainres.2008.04.023

DamasioA. R. (1998). Emotion in the perspective of an integrated nervous systemBrain Res. Brain Res. Rev. 268386.

De GelderB.BöckerK. B.TuomainenJ.HensenM.VroomenJ. (1999). The combined perception of emotion from voice and face: Early interaction revealed by human electric brain responsesNeurosci. Lett. 260133136.

De GelderB.PourtoisG.WeiskrantzL. (2002). Fear recognition in the voice is modulated by unconsciously recognized facial expressions but not by unconsciously recognized affective picturesProc. Natl Acad. Sci. USA 9941214126. DOI:10.1073/pnas.062018499

De GelderB.VroomenJ. (2000). The perception of emotion by ear and by eyeCognition Emotion 14289311.

De GelderB.VroomenJ.De JongS. J.MasthoffE. D.TrompenaarsF. J.HodiamontP. (2005). Multisensory integration of emotional faces and voices in schizophrenicsSchizophr. Res. 72195203. DOI:10.1016/j.schres.2004.02.013

De JongJ. J.HodiamontP. P. G.De GelderB. (2010). Modality-specific attention and multisensory integration of emotions in schizophrenia: Reduced regulatory effectsSchizophr. Res. 122136143. DOI:10.1016/j.schres.2010.04.010

De JongJ. J.HodiamontP. P. G.Van den StockJ.De GelderB. (2009). Audiovisual emotion recognition in schizophrenia: Reduced integration of facial and vocal affectSchizophr. Res. 107286293. DOI:10.1016/j.schres.2008.10.001

DolanR. J.MorrisJ. S.De GelderB. (2001). Crossmodal binding of fear in voice and faceProc. Natl Acad. Sci. USA 981000610010. DOI:10.1073/pnas.171288598

EmmelkampP. M.BruynzeelM.DrostL.Van der MastC. A. (2001). Virtual reality treatment in acrophobia: A comparison with exposure in vivoCyberpsychol. Behav. 4335339.

EthoferT.PourtoisG.WildgruberD. (2006). Investigating audiovisual integration of emotional signals in the human brainProgr. Brain Res. 156345361. DOI:10.1016/S0079-6123(06)56019-4

FöckerJ.GondanM.RöderB. (2011). Preattentive processing of audio-visual emotional signalsActa Psychol. 1373647. DOI:10.1016/j.actpsy.2011.02.004

Garcia-PalaciosA.HoffmanH.CarlinA.FurnessT. A.BotellaC. (2002). Virtual reality in the treatment of spider phobia: a controlled studyBehav. Res. Ther. 40983993.

GarrettA. S.MaddockR. J. (2001). Time course of the subjective emotional response to aversive pictures: Relevance to fMRI studiesPsychiat. Res. 1083948.

HaganC. C.WoodsW.JohnsonS.CalderA.GreenG. G. R.YoungA. W. (2009). MEG demonstrates a supra-additive response to facial and vocal emotion in the right superior temporal sulcusProc. Natl Acad. Sci. USA 1062001020015. DOI:10.1073/pnas.0905792106

JessenS.KotzS. A. (2011). The temporal dynamics of processing emotions from vocal, facial, and bodily expressionsNeuroImage 58665674. DOI:10.1016/j.neuroimage.2011.06.035

KoizumiA.TanakaA.ImaiH.HiramatsuS.HiramotoE.SatoT.De GelderB. (2011). The effects of anxiety on the interpretation of emotion in the face–voice pairsExp. Brain Res. 213275282. DOI:10.1007/s00221-011-2668-1

KreibigS. D. (2010). Autonomic nervous system activity in emotion: a reviewBiol. Psychol. 84394421. DOI:10.1016/j.biopsycho.2010.03.010

KreifeltsB.EthoferT.GroddW.ErbM.WildgruberD. (2007). Audiovisual integration of emotional signals in voice and face: An event-related fMRI studyNeuroImage 37144511456. DOI:10.1016/j.neuroimage.2007.06.020

LeottiL. A.IyengarS. S.OchsnerK. N. (2010). Born to choose: The origins and value of the need for controlTrends Cogn. Sci. 14457463. DOI:10.1016/j.tics.2010.08.001

LourencoS. F.LongoM. R.PathmanT. (2011). Near space and its relation to claustrophobic fearCognition 119448453. DOI:10.1016/j.cognition.2011.02.009

MagnéeM.De GelderB.Van EngelandH.KemnerC. (2007). Facial electromyographic responses to emotional information from faces and voices in individuals with pervasive developmental disorderJ. Child Psychol. Psychiat. 4811221130. DOI:10.1111/j.1469-7610.2007.01779.x

MagnéeM.De GelderB.Van EngelandH.KemnerC. (2008). Atypical processing of fearful face–voice pairs in Pervasive Developmental Disorder: an ERP studyClin. Neurophysiol. 11920042010. DOI:10.1016/j.clinph.2008.05.005

MagnéeM.De GelderB.Van EngelandH.KemnerC. (2011). Multisensory integration and attention in autism spectrum disorder: Evidence from event-related potentialsPloS One 6e24196. DOI:10.1371/journal.pone.0024196

MassaroD.EganP. (1996). Perceiving affect from the voice and the facePsychon. B Rev. 3215221.

MaurageP.PhilippotP.JoassinF.PauwelsL.PhamT.PrietoE. A.Palmero-SolerE.ZanowF.CampanellaS. (2008). The auditory–visual integration of anger is impaired in alcoholism: An event-related potentials studyJ. Psychiatr. Neurosci. 33111122.

MoeckT.BonneelN.TsingosN.DrettakisG.Viaud-DelmonI.AllozaD. (2007). Progressive Perceptual audio rendering of complex scenes in: Proceedings of the 2007 Symposium on Interactive 3D Graphics and Games pp. 189–196. ACM New York NY USA.

MühlbergerA.SperberM.WieserM. J.PauliP. (2008). A Virtual Reality Behavior Avoidance Test (VR-BAT) for the assessment of spider phobiaJ. CyberTher. Rehabil. 1147158.

MüllerV. I.CieslikE. C.TuretskyB. I.EickhoffS. B. (2012). Crossmodal interactions in audiovisual emotion processingNeuroImage 60553561. DOI:10.1016/j.neuroimage.2011.12.007

NesseR. M.CurtisG. C.BrownG. M.RubinR. T. (1980). Anxiety induced by flooding therapy for phobias does not elicit prolactin secretory responsePsychosom. Med. 422531.

PhillipsM. L.DrevetsW. C.RauchS. L.LaneR. (2003). Neurobiology of emotion perception I: The neural basis of normal emotion perceptionBiol. Psychiat. 54504514. DOI:10.1016/S0006-3223(03)00168-9

PitmanR. K.OrrS. P.AltmanB.LongpreR. E.PoiréR. E.MacklinM. L.MichaelsM. J.SteketeeG. S. (1996). Emotional processing and outcome of imaginal flooding therapy in Vietnam veterans with chronic posttraumatic stress disorderCompr. Psychiat. 37(6) 409418.

PourtoisG.De GelderB.BolA.CrommelinckM. (2005). Perception of facial expressions and voices and of their combination in the human brainCortex 414959.

PourtoisG.De GelderB.VroomenJ.RossionB.CrommelinckM. (2000). The time-course of intermodal binding between seeing and hearing affective informationNeuroreport 1113291333.

RivaG. (2005). Virtual reality in psychotherapy: ReviewCyberpsychol. Behav. 8220230; Discussion 231–240. DOI:10.1089/cpb.2005.8.220

RobinsD. L.HunyadiE.SchultzR. T. (2009). Superior temporal activation in response to dynamic audio-visual emotional cuesBrain Cognition 69269278. DOI:10.1016/j.bandc.2008.08.007

RothbaumB. O.HodgesL. F.KooperR.OpdykeD.WillifordJ. S.NorthM. (1995). Virtual reality graded exposure in the treatment of acrophobia: A case reportBehav. Ther. 26547554.

RudraufD.LachauxJ.-P.DamasioA.BailletS.HuguevilleL.MartinerieJ.DamasioH.RenaultB. (2009). Enter feelings: Somatosensory responses following early stages of visual induction of emotionInt. J. Psychophysiol. 721323. DOI:10.1016/j.ijpsycho.2008.03.015

SarlatL.WarusfelO.Viaud-DelmonI. (2006). Ventriloquism aftereffects occur in the rear hemisphereNeurosci. Lett. 404324329. DOI:10.1016/j.neulet.2006.06.007

SchubertT.FriedmannF.RegenbrechtH. (2001). The experience of presence: Factor analytic insightsPresence-Teleop. 10266281.

SpielbergerC. D.GorsuchR. L.LusheneP. R.VaggP. R.JacobsA. G. (1983). Manual for the State-Trait Anxiety Inventory (Form Y). Consulting Psychologists PressPalo Alto, CA.

SteinB. E.MeredithM. A. (1993). The Merging of the Senses. MIT PressCambridge, MA, USA.

SteinB. E.StanfordT. R. (2008). Multisensory integration: Current issues from the perspective of the single neuronNat. Rev. Neurosci. 9255266. DOI:10.1038/nrn2331

StreinerD. L.NormanG. R. (2011). Correction for multiple testing: Is there a resolution? Chest 1401618. DOI:10.1378/chest.11-0523

SuiedC.DrettakisG.WarusfeldO.Viaud-DelmonI. (2013). Auditory–visual virtual reality as a diagnostic and therapeutic tool for cynophobiaCyberpsychol. Behav. Soc. Netw. 16145152. DOI:10.1089/cyber.2012.1568

TanakaA.KoizumiA.ImaiH.HiramatsuS.HiramotoE.De GelderB. (2010). I feel your voice. Cultural differences in the multisensory perception of emotionPsychol. Sci. 2112591262. DOI:10.1177/0956797610380698

ThyerB. A.PapsdorfJ. D.DavisR.VallecorsaS. (1984). Autonomic correlates of the subjective anxiety scaleJ. Behav. Ther. Exp. Psy. 1537.

VagnoniE.LourencoS. F.LongoM. R. (2012). Threat modulates perception of looming visual stimuliCurr. Biol. 22R826R827. DOI:10.1016/j.cub.2012.07.053

Viaud-DelmonI.IvanenkoY. P.BerthozA.JouventR. (2000). Adaptation as a sensorial profile in trait anxiety: A study with virtual realityJ. Anxiety Disord. 14583601.

Viaud-DelmonI.VenaultP.ChapouthierG. (2011). Behavioral models for anxiety and multisensory integration in animals and humansProgr. Neuro-Psychopharmacol. Biol. Psychiat. 3513911399. DOI:10.1016/j.pnpbp.2010.09.016

Viaud-DelmonI.ZnaïdiF.BonneelN.DoukhanD.SuiedC.WarusfelO.NguyenK. V.DrettakisG. (2008). Auditory–visual virtual environments to treat dog phobia in: Proceedings of the 7th ICDVRAT — International Conference on Disability Virtual Reality and Associated Technologies Maia Portugal pp. 119–124. University of Reading Reading UK.

VinesB. W.KrumhanslC. L.WanderleyM. M.DalcaI. M.LevitinD. J. (2011). Music to my eyes: cross-modal interactions in the perception of emotions in musical performanceCognition 118157170. DOI:10.1016/j.cognition.2010.11.010

VinesB. W.KrumhanslC. L.WanderleyM. M.LevitinD. J. (2006). Cross-modal interactions in the perception of musical performanceCognition 10180113. DOI:10.1016/j.cognition.2005.09.003

VroomenJ.DriverJ.De GelderB. (2001). Is cross-modal integration of emotional expressions independent of attentional resources? Cogn. Affect. Behav. Neurosci. 1382387.

WaldJ.TaylorS. (2001). Efficacy of virtual reality exposure therapy to treat driving phobia: A case reportJ. Behav. Ther. Exp. Psychiat. 31249257.

WolpeJ. (1973). The Practice of Behavior Therapy2nd edn. PergamonNew York.


  • View in gallery

    (A) Picture of the iSpace setup used in the study. (B) A participant, equipped with polarized glasses, headphones and a wireless joystick, standing within the iSpace during immersion in an auditory–visual VE. This figure is published in colour in the online version.

  • View in gallery

    Pictures of the auditory–visual VEs used to measure the participants’ fear when encountering virtual dogs. On the left, the outdoor garden scene and on the right, the indoor hangar scene. This figure is published in colour in the online version.

  • View in gallery

    Pictures of the virtual dog stimuli used in this study: the Doberman dog model with (from left to right) Malamute, Miniature Pinscher and Doberman texture. This figure is published in colour in the online version.

  • View in gallery

    Mean reported fear (mean SUDs ± SEM) of the NoFear group (grey squares) and the DogFear group (black squares) at each of the 14 steps during BATs. The responses collected during BAT1 are presented on the left and the responses collected during BAT2 are presented on the right. In the NoFear group, Wilcoxon tests revealed a significant increase of fear between steps 12 and 13 of BAT1. In the DogFear group, fear increased globally in both BATs and Wilcoxon tests revealed significant increases of fear between steps 11 and 12 and between steps 12 and 13 in BAT1. Neither groups showed any increase of fear between steps in BAT2.

  • View in gallery

    (A) Mean reported fear (mean SUDs ± SEM) of the NoFear (grey diamonds) and DogFear group (black squares) in the auditory–visual VEs according to the sensory modality in which the dogs were presented. The SUDs reported in response to the auditory static, the visual static, the auditory moving and the visual moving dog stimuli were averaged for the unimodal condition. The SUDs in response to the auditory–visual static and the auditory–visual moving dog stimuli were averaged for the bimodal condition. In both groups, the experience of fear was higher in response to bimodal compared to unimodal stimuli. (B) Mean increase of reported fear in the bimodal condition compared to the unimodal one (mean difference between SUDs in response to bimodal and unimodal stimuli ± SEM) in each group. The increase of fear is greater in the DogFear group (black bar) than in the NoFear group (grey bar).

Index Card

Content Metrics

Content Metrics

All Time Past Year Past 30 Days
Abstract Views 11 11 5
Full Text Views 3 3 3
PDF Downloads 2 2 2
EPUB Downloads 0 0 0