The Effect of Visual and Auditory Information on the Perception of Pleasantness and Roughness of Virtual Surfaces

in Multisensory Research
Restricted Access
Get Access to Full Text
Rent on DeepDyve

Have an Access Token?

Enter your access token to activate and access content online.

Please login and go to your personal user account to enter your access token.


Have Institutional Access?

Access content through your institution. Any other coaching guidance?


Despite the large number of studies on the multisensory aspects of tactile perception, very little is known regarding the effects of visual and auditory sensory modalities on the tactile hedonic evaluation of textures, especially when the presentation of the stimuli is mediated by a haptic device. In this study, different haptic virtual surfaces were rendered by varying the static and dynamic frictional coefficients of a Geomagic® Touch device. In Experiment 1, the haptic surfaces were paired with pictures representing everyday materials (glass, plastic, rubber and steel); in Experiment 2, the haptic surfaces were paired with sounds resulting from the haptic exploration of paper or sandpaper. In both the experiments, participants were required to rate the pleasantness and the roughness of the virtual surfaces explored. Exploration times were also recorded. Both pleasantness and roughness judgments, as well as the durations of exploration, varied as a function of the combinations of the visuo-tactile and the audio-tactile stimuli presented. Taken together, these results suggest that vision and audition modulate haptic perception and hedonic preferences when tactile sensations are provided through a haptic device. Importantly, these results offer interesting suggestions for designing more pleasant, and even more realistic, multisensory virtual surfaces.

The Effect of Visual and Auditory Information on the Perception of Pleasantness and Roughness of Virtual Surfaces

in Multisensory Research



AltinsoyM. E. (2008). The effect of auditory cues on the audiotactile roughness perception: modulation frequency and sound pressure level in: International Workshop on Haptic and Audio Interaction Design HAID 2008PirhonenA.BrewsterS. (Eds) Lecture Notes in Computer ScienceVol. 5270 pp.  120129. SpringerBerlin and Heidelberg, Germany.

AvanziniF.CrosatoP. (2006). Haptic-auditory rendering and perception of contact stiffness in: International Workshop on Haptic and Audio Interaction Design HAID 2006McGookinD.BrewsterS. (Eds) Lecture Notes in Computer ScienceVol. 4129 pp.  2435. SpringerBerlin and Heidelberg, Germany.

Bergmann TiestW. M.KappersA. M. (2006). Analysis of haptic perception of materials by multidimensional scaling and physical measurements of roughness and compressibilityActa Psychol. 121120.

BrieberD.NadalM.LederH.RosenbergR. (2014). Art in time and space: context modulates the relation between art experience and viewing timePLoS One 9e99019. DOI:10.1371/journal.pone.0099019.

CampionG.GoslineA. H.HaywardV. (2008). Does judgment of haptic virtual texture roughness scale monotonically with lateral force modulation? in: International Conference on Human Haptic Sensing and Touch Enabled Computer ApplicationsFerreM. (Ed.) Lecture Notes in Computer ScienceVol. 5024 pp.  718723. SpringerBerlin and Heidelberg, Germany.

ColavitaF. B. (1974). Human sensory dominancePercept. Psychophys. 16409412.

ColavitaF. B.WeisbergD. (1979). A further investigation of visual dominancePercept. Psychophys. 25345347.

CroyI.D’AngeloS.OlaussonH. (2014). Reduced pleasant touch appraisal in the presence of a disgusting odorPLoS One 9e92975. DOI:10.1371/journal.pone.0092975.

DemattèM. L.SanabriaD.SugarmanR.SpenceC. (2006). Cross-modal interactions between olfaction and touchChem. Senses 31291300.

EllingsenD. M.WessbergJ.ChelnokovaO.OlaussonH.LaengB.LeknesS. (2014). In touch with your emotions: oxytocin and touch change social impressions while others’ facial expressions can alter touchPsychoneuroendocrinology 391120.

ErnstM. O.BanksM. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashionNature 415(6870) 429433.

ErnstM. O.BülthoffH. H. (2004). Merging the senses into a robust perceptTrends Cogn. Sci. 8162169.

EssickG. K.McGloneF.DancerC.FabricantD.RaginY.PhillipsN.JonesT.GuestS. (2010). Quantitative assessment of pleasant touchNeurosci. Biobehav. Rev. 34192203.

EtziR.GallaceA. (2016). The arousing power of everyday materials: an analysis of the physiological and behavioral responses to visually and tactually presented texturesExp. Brain Res. 23416591666.

EtziR.SpenceC.GallaceA. (2014). Textures that we like to touch: an experimental study of aesthetic preferences for tactile stimuliConsc. Cogn. 29178188.

GallaceA. (2011). Verso una definizione dell’estetica tattile [Towards a definition of tactile aesthetics] in: Vedere con ManoArgentonA. (Ed.) pp.  93108. Edizioni EricksonTrento, Italy.

GallaceA. (2015). Touch to sell: when packaging design meets cognitive neurosciencePackaging Eur. 104749.

GallaceA.SpenceC. (2011). Tactile aesthetics: towards a definition of its characteristics and neural correlatesSoc. Semiotics 21569589.

GallaceA.SpenceC. (2014). In Touch With the Future: the Sense of Touch From Cognitive Neuroscience to Virtual Reality. Oxford University PressOxford, UK.

GallaceA.NgoM. K.SulaitisJ.SpenceC. (2011). Multisensory presence in virtual reality: possibilities and limitations in: Multiple Sensorial Media Advances and Applications: New Developments in MulSeMediaGhineaG.AndresF.GulliverS. (Eds) pp.  138. IGI GlobalVancouver, BC, Canada.

GuestS.SpenceC. (2003a). Tactile dominance in speeded discrimination of texturesExp. Brain Res. 150201207.

GuestS.SpenceC. (2003b). What role does multisensory integration play in the visuotactile perception of texture? Int. J. Psychophysiol. 506380.

GuestS.CatmurC.LloydD.SpenceC. (2002). Audiotactile interactions in roughness perceptionExp. Brain Res. 146161171.

GuestS.DessirierJ. M.MehrabyanA.McGloneF.EssickG.GescheiderG.FontanaA.XiongR.AckerleyR.BlotK. (2011). The development and validation of sensory and emotional scales of touch perceptionAtten. Percept. Psychophys. 73531550.

Hartcher-O’BrienJ.GallaceA.KringsB.KoppenC.SpenceC. (2008). When vision ‘extinguishes’ touch in neurologically-normal people: extending the Colavita visual dominance effectExp. Brain Res. 186643658.

HellerM. A. (1982). Visual and tactual texture perception: intersensory cooperationPercept. Psychophys. 31339344.

HellerM. A. (1985). Effect of magnification on texture perceptionPercept. Mot. Skills 611242.

HoH. N.IwaiD.YoshikawaY.WatanabeJ.NishidaS. Y. (2014). Combining colour and temperature: a blue object is more likely to be judged as warm than a red objectSci. Rep. 45527.

HollinsM.FaldowskiR.RaoS.YoungF. (1993). Perceptual dimensions of tactile surface texture: a multidimensional scaling analysisPercept. Psychophys. 54697705.

HollinsM.BensmaïaS.KarlofK.YoungF. (2000). Individual differences in perceptual space for tactile textures: evidence from multidimensional scalingPercept. Psychophys. 6215341544.

JonesB.O’NeilS. (1985). Combining vision and touch in texture perceptionPercept. Psychophys. 376672.

JousmäkiV.HariR. (1998). Parchment-skin illusion: sound-biased touchCurr. Biol. 8R190R191.

KlatzkyR. L.LedermanS. J. (2006). The perceived roughness of resistive virtual textures: I. Rendering by a force-feedback mouseACM Trans. Appl. Percept. 3114.

KlatzkyR. L.LedermanS. J. (2010). Multisensory texture perception in: Multisensory Object Perception in the Primate BrainNaumerM. J.KaiserJ. (Eds) pp.  211230. SpringerNew York, NY, USA.

KoppenC.SpenceC. (2007). Seeing the light: exploring the Colavita visual dominance effectExp. Brain Res. 180737754.

KornbrotD.PennP.PetrieH.FurnerS.HardwickA. (2007). Roughness perception in haptic virtual reality for sighted and blind peoplePercept. Psychophys. 69502512.

LedermanS. J.AbbottS. G. (1981). Texture perception: studies of intersensory organization using a discrepancy paradigm, and visual versus tactual psychophysicsJ. Exp. Psychol. Hum. Percept. Perform. 7902915.

LedermanS. J.KlatzkyR. L. (2004). Multisensory texture perception in: The Handbook of Multisensory ProcessesCalvertG. A.SpenceC.SteinB. E. (Eds) pp.  107123. MIT PressCambridge, MA, USA.

LedermanS. J.ThorneG.JonesB. (1986). Perception of texture by vision and touch: multidimensionality and intersensory integrationJ. Exp. Psychol. Hum. Percept. Perform. 12169180.

LedermanS. J.KlatzkyR. L.TongC.HamiltonC. (2006). The perceived roughness of resistive virtual textures: II. Effects of varying viscosity with a force-feedback deviceACM Trans. Appl. Percept. 31530.

McCabeD. B.NowlisS. M. (2003). The effect of examining actual products or product descriptions on consumer preferenceJ. Consum. Psychol. 13431439.

MeredithM. A.SteinB. E. (1986). Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integrationJ. Neurophysiol. 56640662.

OkamotoS.NaganoH.YamadaY. (2013). Psychophysical dimensions of tactile perception of texturesIEEE Trans. Haptics 68193.

PicardD.DacremontC.ValentinD.GiboreauA. (2003). Perceptual dimensions of tactile texturesActa Psycholog. 114165184.

PosnerM. I.NissenM. J.KleinR. M. (1976). Visual dominance: an information-processing account of its origins and significancePsychol. Rev. 83157171.

PunpongsanonP.IwaiD.SatoK. (2015). Softar: visually manipulating haptic softness perception in spatial augmented realityIEEE Trans. Vis. Comput. Graph. 2112791288.

RockI.VictorJ. (1964). Vision and touch: an experimentally created conflict between the two sensesScience 143594596.

SinnettS.SpenceC.Soto-FaracoS. (2007). Visual dominance and attention: the Colavita effect revisitedPercept. Psychophys. 69673686.

SpenceC. (2009). Explaining the Colavita visual dominance effectProgr. Brain Res. 176245258.

SpenceC.GallaceA. (2008). Making sense of touch in: Touch in Museums: Policy and Practice in Object HandlingChatterjeeH. E. (Ed.) pp.  2140. BergLondon, UK.

SpenceC.GallaceA. (2011). Multisensory design: reaching out to touch the consumerPsychol. Marketing 28267308.

SteinB. E. (Ed.) (2012). The New Handbook of Multisensory Processing. MIT PressCambridge, MA, USA.

SteinB. E.StanfordT. R. (2008). Multisensory integration: current issues from the perspective of the single neuronNat. Rev. Neurosci. 9255266.

TanakaY.TanakaM.ChonanS. (2006). Development of a sensor system for measuring tactile sensation in: Proceedings of IEEE Conference on Sensors pp.  554557. Daegu, South Korea.

UjitokoY.BanY.NarumiT.TanikawaT.HirotaK.HiroseM. (2015). Yubi-Toko: finger walking in snowy scene using pseudo-haptic technique on touchpad in: SA ’15 SIGGRAPH Asia 2015 Emerging TechnologiesKobe, Japanart. 29.

WeisenbergerJ. M.PolingG. L. (2004). Multisensory roughness perception of virtual surfaces: effects of correlated cues in: Proceedings of the 12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systs pp.  161168. Chicago, IL, USA.

WelchR. B.WarrenD. H. (1980). Immediate perceptual response to intersensory discrepancyPsychol. Bull. 88638667.

WhitakerT. A.Simões-FranklinC.NewellF. N. (2008). Vision and touch: independent or integrated systems for the perception of texture? Brain Res. 12425972.

World Medical Association (2013). World Medical Association declaration of Helsinki: ethical principles for medical research involving human subjectsJAMA 31021912194.

ZampiniM.SpenceC. (2004). The role of auditory cues in modulating the perceived crispness and staleness of potato chipsJ. Sens. Stud. 19347363.


  • View in gallery

    Visual stimuli (from left to right and from up to down: plastic, rubber, steel, glass) used in Experiment 1.

  • View in gallery

    Picture of the experimental setup adopted in Experiment 1.

  • View in gallery

    Participants’ evaluations of Experiment 1. (A) Pleasantness and roughness mean ratings of the four virtual surfaces; (B) pleasantness mean ratings for the visuo-tactile combinations; (C) roughness mean ratings for the visuo-tactile combinations. Error bars represent the standard errors of the mean and asterisks indicate significant differences (p<0.05).

  • View in gallery

    Participants’ mean exploration times of Experiment 1. Error bars represent the standard errors of the mean and asterisks indicate significant differences (p<0.05).

  • View in gallery

    Participants’ evaluations of Experiment 2. (A) Pleasantness and roughness mean ratings of the three virtual surfaces; (B) pleasantness mean ratings for the visuo-tactile combinations; (C) roughness mean ratings for the visuo-tactile combinations. Error bars represent the standard errors of the mean and asterisks indicate significant differences (p<0.05).

  • View in gallery

    Participants’ mean exploration times of Experiment 2. Error bars represent the standard errors of the mean and asterisks indicate significant differences (p<0.05).

Index Card

Content Metrics

Content Metrics

All Time Past Year Past 30 Days
Abstract Views 20 20 17
Full Text Views 29 29 29
PDF Downloads 2 2 2
EPUB Downloads 0 0 0