Cue Integration for Continuous and Categorical Dimensions by Synesthetes

in Multisensory Research
Restricted Access
Get Access to Full Text
Rent on DeepDyve

Have an Access Token?

Enter your access token to activate and access content online.

Please login and go to your personal user account to enter your access token.


Have Institutional Access?

Access content through your institution. Any other coaching guidance?


For synesthetes, sensory or cognitive stimuli induce the perception of an additional sensory or cognitive stimulus. Grapheme–color synesthetes, for instance, consciously and consistently experience particular colors (e.g., fluorescent pink) when perceiving letters (e.g., u). As a phenomenon involving multiple stimuli within or across modalities, researchers have posited that synesthetes may integrate sensory cues differently than non-synesthetes. However, findings to date present mixed results concerning this hypothesis, with researchers reporting enhanced, depressed, or normal sensory integration for synesthetes. In this study we quantitatively evaluated the multisensory integration process of synesthetes and non-synesthetes using Bayesian principles, rather than employing multisensory illusions, to make inferences about the sensory integration process. In two studies we investigated synesthetes’ sensory integration by comparing human behavior to that of an ideal observer. We found that synesthetes integrated cues for both continuous and categorical dimensions in a statistically optimal manner, matching the sensory integration behavior of controls. These findings suggest that synesthetes and controls utilize similar cue integration mechanisms, despite differences in how they perceive unimodal stimuli.

Cue Integration for Continuous and Categorical Dimensions by Synesthetes

in Multisensory Research



AhissarM.HochsteinS. (2004). The reverse hierarchy theory of visual perceptual learningTrends Cogn. Sci. 8457464.

BankierisK.BejjankiV. R.AslinR. (subm.). Sensory cue-combination in the context of newly learned categories.

BargaryG.MitchellK. J. (2008). Synaesthesia and cortical connectivityTrends Neurosci. 31335342.

BejjankiV. R.ClayardsM.KnillD. C.AslinR. N. (2011). Cue integration in categorical tasks: insights from audio-visual speech perceptionPLoS One 6e19812. DOI:10.1371/journal.pone.0019812.

BrangD.WilliamsL. E.RamachandranV. S. (2012). Grapheme–color synesthetes show enhanced crossmodal processing between auditory and visual modalitiesCortex 48(5) 630637.

CalkinsM. W. (1893). A statistical study of pseudochromesthesia and of mental formsAm. J. Psychol. 5439466.

ClaparedeE. (1903). Persistance de l’audition coloréeC. R. Seances Soc. Biol. Fil. 5512571259.

ClayardsM.TanenhausM. K.AslinR. N.JacobsR. A. (2008). Perception of speech reflects optimal use of probabilistic speech cuesCognition 108804809.

DayS. (2005). Some demographic and socio-cultural aspects of synesthesia in: Synesthesia: Perspectives From Cognitive NeuroscienceRobertsonL. C.SagivN. (Eds) pp.  1133. Oxford University PressNew York, NY, USA.

DayS. (2009). Types of synesthesia in: Synesthesia. Retrieved from

EaglemanD. M.KaganA. D.NelsonS. S.SagaramD.SarmaA. K. (2007). A standardized test battery for the study of synesthesiaJ. Neurosci. Meth. 159139145.

ErnstM. O.BanksM. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashionNature 415(6870) 429433.

FeldmanN. H.GriffithsT. L.MorganJ. L. (2009). The influence of categories on perception: explaining the perceptual magnet effect as optimal statistical inferencePsychol. Rev. 116752782.

GrossenbacherP. G.LovelaceC. T. (2001). Mechanisms of synesthesia: cognitive and physiological constraintsTrends Cogn. Sci. 53641.

HänggiJ.WotrubaD.JänckeL. (2011). Globally altered structural brain network topology in grapheme-color synesthesiaJ. Neurosci. 3158165828.

HershensonM. (1962). Reaction time as a measure of intersensory facilitationJ. Exp. Psychol. 63289293.

HillisJ. M.ErnstM. O.BanksM. S.LandyM. S. (2002). Combining sensory information: mandatory fusion within, but not between, sensesScience 298(5598) 16271630.

HubbardE. M.BrangD.RamachandranV. S. (2011). The cross-activation theory at 10J. Neuropsychol. 5152177.

HupéJ. M.DojatM. (2015). A critical review of the neuroimaging literature on synesthesiaFront. Hum. Neurosci. 9103. DOI:10.3389/fnhum.2015.00103.

JacobsR. A.FineI. (1999). Experience-dependent integration of texture and motion cues to depthVis. Res. 3940624075.

JänckeL.BeeliG.EuligC.HänggiJ. (2009). The neuroanatomy of grapheme–color synesthesiaEur. J. Neurosci. 2912871293.

JewanskiJ.DayS.WardJ. (2009). A colorful albino: the first documented case of synaesthesia, by Georg Tobias Ludwig Sachs in 1812J. Hist. Neurosci. 18293303.

KördingK. P.WolpertD. M. (2004). Bayesian integration in sensorimotor learningNature 427(6971) 244247.

KördingK. P.BeierholmU.MaW. J.QuartzS.TenenbaumJ. B.ShamsL. (2007). Causal inference in multisensory perceptionPLoS One 2e943. DOI:10.1371/journal.pone.0000943.

LaurientiP. J.BurdetteJ. H.MaldjianJ. A.WallaceM. T. (2006). Enhanced multisensory integration in older adultsNeurobiol. Aging 2711551163.

MichelM. M.JacobsR. A. (2008). Learning optimal integration of arbitrary features in a perceptual discrimination taskJ. Vis. 8(3) 116. DOI:10.1167/8.2.3.

MillerJ. (1982). Divided attention: evidence for coactivation with redundant signalsCogn. Psychol. 14247279.

MuirD. W.CliftonR. K.ClarksonM. G. (1989). The development of a human auditory localization response: a U-shaped functionCan. J. Psychol. 43199216.

NeufeldJ.SinkeC.ZedlerM.EmrichH. M.SzycikG. R. (2012). Reduced audio–visual integration in synaesthetes indicated by the double-flash illusionBrain Res. 14737886.

O’HanlonE.NewellF. N.MitchellK. J. (2013). Combined structural and functional imaging reveals cortical deactivations in grapheme–color synaesthesiaFront. Psychol. 4755. DOI:10.3389/fpsyg.2013.00755.

OruçI.MaloneyL. T.LandyM. S. (2003). Weighted linear cue combination with possibly correlated errorVis. Res. 4324512468.

RamachandranV.HubbardE. M. (2001). Synaesthesia: a window into perception, thought and languageJ. Consc. Stud. 8334.

RouwR.ScholteH. S. (2007). Increased structural connectivity in grapheme-color synesthesiaNat. Neurosci. 10792797.

RouwR.ScholteH. S. (2010). Neural basis of individual differences in synesthetic experiencesJ. Neurosci. 3062056213.

RouwR.ScholteH. S.ColizoliO. (2011). Brain areas involved in synaesthesia: a reviewJ. Neuropsychol. 5214242.

ShamsL.KamitaniY.ShimojoS. (2000). Illusions: what you see is what you hearNature 408(6814) 788.

SinkeC.NeufeldJ.ZedlerM.EmrichH. M.BleichS.MünteT. F.SzycikG. R. (2012). Reduced audiovisual integration in synesthesia–evidence from bimodal speech perceptionJ. Neuropsychol. 894106.

TomsonS. N.NarayanM.AllenG. I.EaglemanD. M. (2013). Neural networks of colored sequence synesthesiaJ. Neurosci. 331409814106.

Van BeersR. J.SittigA. C.DenierJ. A. N. J. (1999). Integration of proprioceptive and visual position-information: an experimentally supported modelJ. Neurophysiol. 8113551364.

WeissP. H.FinkG. R. (2009). Grapheme-colour synaesthetes show increased grey matter volumes of parietal and fusiform cortexBrain 1326570.

WhittinghamK. M.McDonaldJ. S.CliffordC. W. (2014). Synesthetes show normal sound-induced flash fission and fusion illusionsVis. Res. 10519.

WichmannF. A.HillN. J. (2001). The psychometric function: I. Fitting, sampling, and goodness of fitPercept Psychophys. 6312931313.


  • View in gallery

    Experiment setup. (a) Schematic depiction of apparatus. Viewing the setup from above, the black curved line represents the screen onto which visual stimuli were projected. The gray semi-circle indicates the custom-built table upon which seven speakers sat (see text for details). The smiley face indicates a participant sitting at the center of the setup and facing ±45°. (b) Visual stimuli, showing the three noise levels used in the experiment. (c) Example of a visual only trial. Note that the visual stimuli flickered, which is not depicted in this figure.

  • View in gallery

    Audiovisual trials. Axes indicate the location of the probe with respect to the standard audiovisual stimulus (which was always aligned and presented at 0°). Dark grey = aligned, light grey = misaligned. Twenty-five repetitions of each stimulus were presented.

  • View in gallery

    Cumulative Gaussian fits of unimodal trials for a representative synesthete. The top left panel plots all four unimodal cumulative Gaussian fits with the PSE equalized for descriptive purposes, to allow for easier slope comparison. The remaining panels plot cumulative Gaussian fits along with data for each unimodal condition separately. The standard is always presented at 0°.

  • View in gallery

    Observed and predicted visual weights for audiovisual trials. Note that neither synesthetes’ nor controls’ observed visual weights differ from the predicted visual weights. Error bars are standard error.

  • View in gallery

    Cue combination involving categorization. A depiction of the categorization problem where each category is defined by two cues. The x and y axes represent the strength of each sensory cue. The circles labeled A and B represent the mean and covariance of each cue for categories A and B for a given participant. The grey diagonal line represents the linear discriminant vector D that an optimal categorizer projects the received bi-cue signal onto (see text).

  • View in gallery

    Training and test stimuli. Black circles represent the occurrence of exemplars of the two-cue stimuli during training. The elliptical clusters of black circles represent the Gaussian distributions of the two task-relevant categories. The size of each circle represents the number of exemplars of each stimulus that were presented during one learning block. Grey squares represent testing stimuli (bimodal in center, unimodal along the x- and y-axes). Twenty-five repetitions of each testing stimulus were presented. Category labels (taygoo and dohkah) and locations (as above or rotated 90°) were counterbalanced across participants.

  • View in gallery

    Trial structure. Training: example of audiovisual training trials with feedback. Testing: example of visual only, audio only, and audiovisual testing trials without feedback.

  • View in gallery

    Cumulative Gaussian fits of unimodal trials for a representative synesthete. The top left panel plots all five unimodal cumulative Gaussian fits with the PSE equalized for descriptive purposes, to allow for easier slope comparison. The remaining panels plot cumulative Gaussian fits along with data for each unimodal condition separately.

  • View in gallery

    Observed auditory weights for audiovisual trials alongside predictions from the categorical model and the continuous model. Note that synesthetes’ and controls’ actual auditory weights differ from the continuous model’s predictions but are indiscriminable from the categorical model’s predictions. Error bars are standard error. Lines are linear fits generated for visualization purposes only.

Index Card

Content Metrics

Content Metrics

All Time Past Year Past 30 Days
Abstract Views 14 14 6
Full Text Views 7 7 7
PDF Downloads 2 2 2
EPUB Downloads 0 0 0