Steady-State EEG and Psychophysical Measures of Multisensory Integration to Cross-Modally Synchronous and Asynchronous Acoustic and Vibrotactile Amplitude Modulation Rate

in Multisensory Research
Restricted Access
Get Access to Full Text
Rent on DeepDyve

Have an Access Token?

Enter your access token to activate and access content online.

Please login and go to your personal user account to enter your access token.


Have Institutional Access?

Access content through your institution. Any other coaching guidance?


According to the temporal principle of multisensory integration, cross-modal synchronisation of stimulus onset facilitates multisensory integration. This is typically observed as a greater response to multisensory stimulation relative to the sum of the constituent unisensory responses (i.e., superadditivity). The aim of the present study was to examine whether the temporal principle extends to the cross-modal synchrony of amplitude-modulation (AM) rate. It is well established that psychophysical sensitivity to AM stimulation is strongly influenced by AM rate where the optimum rate differs according to sensory modality. This rate-dependent sensitivity is also apparent from EEG steady-state response (SSR) activity, which becomes entrained to the stimulation rate and is thought to reflect neural processing of the temporal characteristics of AM stimulation. In this study we investigated whether cross-modal congruence of AM rate reveals both psychophysical and EEG evidence of enhanced multisensory integration. To achieve this, EEG SSR and psychophysical sensitivity to simultaneous acoustic and/or vibrotactile AM stimuli were measured at cross-modally congruent and incongruent AM rates. While the results provided no evidence of superadditive multisensory SSR activity or psychophysical sensitivity, the complex pattern of results did reveal a consistent correspondence between SSR activity and psychophysical sensitivity to AM stimulation. This indicates that entrained EEG activity may provide a direct measure of cortical activity underlying multisensory integration. Consistent with the temporal principle of multisensory integration, increased vibrotactile SSR responses and psychophysical sensitivity were found for cross-modally congruent relative to incongruent AM rate. However, no corresponding increase in auditory SSR or psychophysical sensitivity was observed for cross-modally congruent AM rates. This complex pattern of results can be understood in terms of the likely influence of the principle of inverse effectiveness where the temporal principle of multisensory integration was only evident in the context of reduced perceptual sensitivity for the vibrotactile but not the auditory modality.

Steady-State EEG and Psychophysical Measures of Multisensory Integration to Cross-Modally Synchronous and Asynchronous Acoustic and Vibrotactile Amplitude Modulation Rate

in Multisensory Research



BendorD.WangX. (2007). Differential neural coding of acoustic flutter within primate auditory cortexNat. Neurosci. 10763771.

Brett-GreenB. A.MillerL. J.GavinW. J.DaviesP. L. (2008). Multisensory integration in children: a preliminary ERP studyBrain Res. 1242283290.

CalvertG. A. (2001). Crossmodal processing in the human brain: insights from functional neuroimaging studiesCereb. Cortex 1111101123.

CalvertG. A.CampbellR.BrammerM. J. (2000). Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortexCurr. Biol. 10649657.

ColonE.LegrainV.MourauxA. (2012). Steady-state evoked potentials to study the processing of tactile and nociceptive somatosensory input in the human brainClin. Neurophysiol. 42315323.

De JongR.ToffaninP.HarbersM. (2010). Dynamic crossmodal links revealed by steady-state responses in auditory-visual divided attentionInt. J. Psychophysiol. 75315.

DenisonR. N.DriverJ.RuffC. C. (2013). Temporal structure and complexity affect audio-visual correspondence detectionFront. Psychol. 3619. DOI:10.3389/fpsyg.2012.00619.

DiederichA.ColoniusH. (2004). Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction timePercept. Psychophys. 6613881404.

FindlayJ. M. (1978). Esimates on probablilty functions: a more virulent PESTPercept. Psychophys. 23181185.

FoxeJ. J.MoroczI. A.MurrayM. M.HigginsB. A.JavittD. C.SchroederC. E. (2000). Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mappingBrain Res. Cogn. Brain Res. 107783.

FoxeJ. J.WylieG. R.MartinezA.SchroederC. E.JavittD. C.GuilfoyleD.RitterW.MurrayM. M. (2002). Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI studyJ. Neurophysiol. 88540543.

GalambosR.MakeigS.TalmachoffP. J. (1981). A 40-Hz auditory potential recorded from the human scalpProc. Natl Acad. Sci. 7826432647.

GescheiderG. A.NibletteR. K. (1967). Cross-modality masking for touch and hearingJ. Exp. Psychol. 74313320.

GiabbiconiC. M.DancerC.ZopfR.GruberT.MullerM. M. (2004). Selective spatial attention to left or right hand flutter sensation modulates the steady-state somatosensory evoked potentialBrain Res. Cogn. Brain Res. 205866.

GianiA. S.OrtizE.BelardinelliP.KleinerM.PreisslH.NoppeneyU. (2012). Steady-state responses in MEG demonstrate information integration within but not across the auditory and visual sensesNeuroimage 6014781489.

GillmeisterH.EimerM. (2007). Tactile enhancement of auditory detection and perceived loudnessBrain Res. 11605868.

HolmesN. P.SpenceC. (2005). Multisensory integration: space, time and superadditivityCurr. Biol. 15R762R764.

JorisP. X.SchreinerC. E.ReesA. (2004). Neural processing of amplitude-modulated soundsPhysiol. Rev. 84541577.

KellyE. F.FolgerS. E. (1999). EEG evidence of stimulus-directed response dynamics in human somatosensory cortexBrain Res. 815326336.

KingdonF. A. A.PrinsN. (2016). Psychophysics: a Practical Introduction2nd edn. Academic PressLondon, UK.

LakatosP.ChenC.-M.O’ConnellM. N.MillsA.SchroederC. E. (2007). Neuronal oscillations and multisensory interaction in primary auditory cortexNeuron 53279292.

LangdonA. J.BoonstraT. W.BreakspearM. (2011). Multi-frequency phase locking in human somatosensory cortexProgr. Biophys. Mol. Biol. 1055866.

LewisR.NoppeneyU. (2010). Audiovisual synchrony improves motion discrimination via enhanced connectivity between early visual and auditory areasJ. Neurosci. 301232912339.

Liegeois-ChauvelC.LorenziC.TrebuchonA.RegisJ.ChauvelP. (2004). Temporal envelope processing in the human left and right auditory corticesCereb. Cortex 14731740.

MeredithM. A.NemitzJ. W.SteinB. E. (1987). Determinants of multisensory integration in superior colliculus neurons. I. Temporal factorsJ. Neurosci. 732153529.

MullerG. R.NeuperC.PfurtschellerG. (2001). “Resonance-like” frequencies of sensorimotor areas evoked by repetitive tactile stimulationBiomed. Tech. (Berl.) 46186190.

NäätänenR.PictonT. (1987). The N1 wave of the human electric and magnetic response to sound: a review and an analysis of the component structurePsychophysiology 24375425.

NanginiC.RossB.TamF.GrahamS. J. (2006). Magnetoencephalographic study of vibrotactile evoked transient and steady-state responses in human somatosensory cortexNeuroimage 33252262.

NoesseltT.RiegerJ. W.SchoenfeldM. A.KanowskiM.HinrichsH.HeinzeH.-J.DriverJ. (2007). Audiovisual temporal correspondence modulates human multisensory superior temporal sulcus plus primary sensory corticesJ. Neurosci. 271143111441.

NourskiK. V.BruggeJ. F. (2011). Representation of temporal sound features in the human auditory cortexRev. Neurosci. 22187203.

NozaradanS.PeretzI.MissalM.MourauxA. (2011). Tagging the neuronal entrainment to beat and meterJ. Neurosci. 311023410240.

NozaradanS.PeretzI.MourauxA. (2012). Steady-state evoked potentials as an index of multisensory temporal bindingNeuroimage 602128.

NozaradanS.ZeroualiY.PeretzI.MourauxA. (2013). Capturing with EEG the neural entrainment and coupling underlying sensorimotor synchronization to the beatCereb. Cortex 25736747.

OostenveldR.FriesP.MarisE.SchoffelenJ. M. (2011). FieldTrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological dataComput. Intell. Neurosci. 2011156869. DOI:10.1155/2011/156869.

PictonT. W.SkinnerC. R.ChampagneS. C.KellettA. J.MaisteA. C. (1987). Potentials evoked by the sinusoidal modulation of the amplitude or frequency of a toneJ. Acoust. Soc. Am. 82165178.

PictonT. W.JohnM. S.DimitrijevicA.PurcellD. (2003). Human auditory steady-state responsesInt. J. Audiol. 42177219.

PorcuE.KeitelC.MüllerM. M. (2014). Visual, auditory and tactile stimuli compete for early sensory processing capacities within but not between sensesNeuroImage 97224235.

PrinsN.KingdonF. (2009). Palamedes: Matlab routines for analyzing psychophysical data. Available at

ReesA.GreenG. G.KayR. H. (1986). Steady-state evoked responses to sinusoidally amplitude-modulated sounds recorded in manHear. Res. 23123133.

RoT.HsuJ.YasarN. E.ElmoreL. C.BeauchampM. S. (2009). Sound enhances touch perceptionExp. Brain Res. 195135143.

RoßB.BorgmannC.DraganovaR.RobertsL. E.PantevC. (2000). A high-precision magnetoencephalographic study of human auditory steady-state responses to amplitude-modulated tonesJ. Acoust. Soc. Am. 108679691.

SaalH. P.WangX.BensmaiaS. J. (2016). Importance of spike timing in touch: an analogy with hearing? Curr. Opin. Neurobiol. 40142149.

SenkowskiD.Saint-AmourD.HofleM.FoxeJ. J. (2011). Multisensory interactions in early evoked brain activity follow the principle of inverse effectivenessNeuroimage 5622002208.

SenkowskiD.Gomez-RamirezM.LakatosP.WylieG. R.MolholmS.SchroederC. E.FoxeJ. J. (2007). Multisensory processing and oscillatory activity: analyzing non-linear electrophysiological measures in humans and simiansExp. Brain Res. 177184195.

SnyderA. Z. (1992). Steady-state vibration evoked potentials: description of technique and characterization of responsesElectroencephalogr. Clin. Neurophysiol. 84257268.

SteinB. E.MeredithM. A. (1993). The Merging of the Senses. MIT PressCambridge, MA, USA.

TobimatsuS.ZhangY. M.KatoM. (1999). Steady-state vibration somatosensory evoked potentials: physiological characteristics and tuning functionClin. Neurophysiol. 11019531958.

Van AtteveldtN. M.FormisanoE.BlomertL.GoebelR. (2007). The effect of temporal asynchrony on the multisensory integration of letters and speech soundsCereb. Cortex 17962974.

ViemeisterN. F. (1979). Temporal modulation transfer functions based upon modulation thresholdsJ. Acoust. Soc. Am. 6613641380.

WaldA. (1943). Tests of hypotheses concernding several parameters when the number of obervations is largeTrans. Am. Math. Soc. 54426482.

WeisenbergerJ. M. (1986). Sensitivity to amplitude-modulated vibrotactile signalsJ. Acoust. Soc. Am. 8017071715.

WilsonE. C.ReedC. M.BraidaL. D. (2009). Integration of auditory and vibrotactile stimuli: effects of phase and stimulus-onset asynchronyJ. Acoust. Soc. Am. 12619601974.

WilsonE. C.ReedC. M.BraidaL. D. (2010). Integration of auditory and vibrotactile stimuli: effects of frequencyJ. Acoust. Soc. Am. 12730443059.

YauJ. M.OlenczakJ. B.DammannJ. F.BensmaiaS. J. (2009). Temporal frequency channels are linked across audition and touchCurr. Biol. 19561566.


  • View in gallery

    Mean auditory and vibrotactile psychophysical thresholds (n=32, error = s.e.m.) for AM depth detection thresholds for 21 and 40 Hz AM stimuli across AM temporal congruence conditions (i.e., unimodal, congruent AM rates and incongruent AM rates). Asterisks highlight significantly different post-hoc pariwise comparisons between vibrotactile thresholds for each AM rate (p<0.05). Note: The more negative the threshold the greater the perceptual sensitivity.

  • View in gallery

    Grand average (n=32) event-related potential waveforms (top row) and the corresponding frequency spectrums (bottom row) for the AM conditions (Carrier Only, 21 Hz AM and 40 Hz AM) for both auditory and vibrotactile stimulation (FCz). The grand average frequency plots display both the uncorrected (blue) and noise-reduced (red) frequency spectrums for each waveform for the time period between 300 to 1500 ms post stimulus onset.

  • View in gallery

    Grand average (n=32) steady-state response (SSR) scalp topographies for both 21 Hz and 40 Hz noise reduced SSRs for the AM stimulation conditions (Carrier Only, 21 Hz AM and 40 Hz AM) for both auditory (top panel) and vibrotactile (bottom panel) stimulation.

  • View in gallery

    EEG power (n=32, error = s.e.m.) for 21 and 40 Hz auditory steady-state responses (SSRs) (electrode sites: FC1, FC2, FCz, Cz, P9, P10, M1 and M2) and vibrotactile SSRs (electrode sites: FC3, FC1, FCz, FC2 and FC4) for unimodal AM stimulation conditions (i.e., Carrier Only, 21 Hz and 40 Hz AM). Consistent with EEG entrainment, increases in SSR power were found to be dependent on AM rate with greater activity at SSR frequencies matching the AM rate of the stimulus. Asterisks highlight significantly different post-hoc pairwise comparisons for SSR power for the across AM conditions for both 21 and 40 Hz auditory and vibrotactile SSR measures (p<0.05). Note: Negative Fast Fourier Transformation values are the result of the noise correction procedure.

  • View in gallery

    Auditory (left panel) (electrode sites: FC1, FC2, FCz, Cz, P9, P10, M1 and M2) and vibrotactile (electrode sites: FC3, FC1, FCz, FC2 and FC4) (right panel) EEG power for 21 and 40 Hz steady-state response frequencies for the multisensory congruent conditions and the sum of constituent unimodal conditions (e.g., unimodal auditory 21 Hz AM + 21 Hz vibrotactile AM).

  • View in gallery

    EEG power for 21 and 40 Hz steady-state response (SSR) frequencies for the multisensory cross-modal AM congruence conditions (n=32, error = s.e.m.). AM congruence had no effect on the auditory SSR responses (left panel) (electrode sites: FC1, FC2, FCz, Cz, P9, P10, M1 and M2); however, congruent AM rates appeared to enhance the vibrotactile SSR activity (right panel) (electrode sites: FC3, FC1, FCz, FC2 and FC4). Asterisks highlight significantly difference post-hoc pairwise comparisons for 21 and 40 Hz vibrotactile SSR measures across the AM congruence conditions (p<0.05).

Index Card

Content Metrics

Content Metrics

All Time Past Year Past 30 Days
Abstract Views 18 18 15
Full Text Views 4 4 4
PDF Downloads 0 0 0
EPUB Downloads 0 0 0