Auditory Coding of Human Movement Kinematics

In: Multisensory Research
View More View Less
  • 1 Leibniz University Hanover, Institute of Sports Science, Hanover, Germany

Purchase instant access (PDF download and unlimited online access):

€25.00$30.00

Although visual perception is dominant on motor perception, control and learning, auditory information can enhance and modulate perceptual as well as motor processes in a multifaceted manner. During last decades new methods of auditory augmentation had been developed with movement sonification as one of the most recent approaches expanding auditory movement information also to usually mute phases of movement. Despite general evidence on the effectiveness of movement sonification in different fields of applied research there is nearly no empirical proof on how sonification of gross motor human movement should be configured to achieve information rich sound sequences. Such lack of empirical proof is given for (a) the selection of suitable movement features as well as for (b) effective kinetic–acoustical mapping patterns and for (c) the number of regarded dimensions of sonification. In this study we explore the informational content of artificial acoustical kinematics in terms of a kinematic movement sonification using an intermodal discrimination paradigm. In a repeated measure design we analysed discrimination rates of six everyday upper limb actions to evaluate the effectiveness of seven different kinds of kinematic–acoustical mappings as well as short term learning effects. The kinematics of the upper limb actions were calculated based on inertial motion sensor data and transformed into seven different sonifications. Sound sequences were randomly presented to participants and discrimination rates as well as confidence of choice were analysed. Data indicate an instantaneous comprehensibility of the artificial movement acoustics as well as short term learning effects. No differences between different dimensional encodings became evident thus indicating a high efficiency for intermodal pattern discrimination for the acoustically coded velocity distribution of the actions. Taken together movement information related to continuous kinematic parameters can be transformed into the auditory domain. Additionally, pattern based action discrimination is obviously not restricted to the visual modality. Artificial acoustical kinematics might be used to supplement and/or substitute visual motion perception in sports and motor rehabilitation.

  • Abernethy B., Hanrahan S. J., Kippers V., Mackinnon L. T., Pandy M. G. (2005). The Biophysical Foundations of Human Movement, 2nd edn. Human Kinetics, Champaign, IL, USA.

    • Search Google Scholar
    • Export Citation
  • Beauchamp M. S. (2005). See me, hear me, touch me: multisensory integration in lateral occipital–temporal cortex, Curr. Opin. Neurobiol. 15, 145153.

    • Search Google Scholar
    • Export Citation
  • Bidet-Caulet A., Voisin J., Bertrand O., Fonlupt P. (2005). Listening to a walking human activates the temporal biological motion area, Neuroimage 28, 132139.

    • Search Google Scholar
    • Export Citation
  • Brock H., Schmitz G., Baumann J., Effenberg A. O. (2012). If motion sounds: movement sonification based on inertial motion data, in: Proceedia Engineering 34, Drahne P., Sherwood J. (Eds), pp.  556561. Elsevier, Amsterdam, Netherlands.

    • Search Google Scholar
    • Export Citation
  • Calvert G., Spence C., Stein B. E. (2004). The Handbook of Multisensory Processes. MIT Press, Cambridge, MA, USA.

  • Carey L. M., Abbott D. F., Puce A., Jackson G. D., Syngeniotis A., Donnan G. A. (2002). Reemergence of activation with poststroke somatosensory recovery: a serial fMRI case study, Neurology 59, 749752.

    • Search Google Scholar
    • Export Citation
  • Carlile S. (2011). Psychoacoustics, in: The Sonification Handbook, Hermann T., Hunt A., Neuhoff J. (Eds), pp.  549552. Logos, Berlin.

  • Cohen J. (1988). Statistical Power Analysis for the Behavioral Sciences, 2nd edn. Lawrence Erlbaum, New York, NY, USA.

  • Croix G., Lejeune L., Anderson D. I., Thouvarecq R. (2010). Light fingertip contact on thigh facilitates handstand balance in gymnasts, Psychol. Sport Exerc. 11, 330333.

    • Search Google Scholar
    • Export Citation
  • Effenberg A. O. (2005). Movement sonification — effects on perception and action, IEEE Multimed. 12, 5359.

  • Fong D. T.-P., Chan Y.-Y. (2010). The use of wearable inertial motion sensors in human lower limb biomechanics studies: a systematic review, Sensors 10, 1155611565.

    • Search Google Scholar
    • Export Citation
  • Gonzalez D. A., Dubrowski A., Carnahan H. (2010). The use of auditory cues in anticipatory control of grasping forces, Motor Control 14, 255264.

    • Search Google Scholar
    • Export Citation
  • Helten T., Brock H., Müller M., Seidel H.-P. (2011). Classification of trampoline jumps using inertial sensors, Sports Eng. 14, 155164.

  • Johannson G. (1973). Visual perception of biological motion and a model for its analysis, Percept. Psychophys. 14, 201211.

  • Kapralos B., Jenkin M. R. M., Milios E. (2003). Auditory Perception and Spatial (3D) Auditory Systems. Technical report CS-2003-07, York University, UK.

  • Kibele A. (2006). Non-consciously controlled decision making for fast motor reactions in sports — a priming approach for motor responses to non-consciously perceived movement features, Psychol. Sport Exerc. 7, 591610.

    • Search Google Scholar
    • Export Citation
  • Làdavas E. (2008). Multisensory-based approach to the recovery of unisensory deficit, Ann. N.Y. Acad. Sci. 1124, 98110.

  • Lahav A., Saltzman E., Schlaug G. (2007). Action representation of sound: audiomotor recognition network while listening to newly acquired actions, J. Neurosci. 27, 308314.

    • Search Google Scholar
    • Export Citation
  • Oldfield R. C. (1971). The assessment and analysis of handedness: the Edinburgh inventory, Neuropsychologia 9, 97113.

  • Parise C. V., Spence C., Ernst M. O. (2012). When correlation implies causation in multisensory integration, Curr. Biol. 22, 4649.

  • Rath M., Rocchesso D. (2005). Continuous sonic feedback from a rolling ball, IEEE Multimed. 12, 6069.

  • Ringenbach S. D. R., van Gemmert A. W. A., Shill H. A., Stelmach G. E. (2011). Auditory instructional cues benefit unimodal and bimanual drawing in Parkinson’s disease patients, Hum. Mov. Sci. 30, 770782.

    • Search Google Scholar
    • Export Citation
  • Rizzolatti G. (2005). The mirror neuron system and its function in humans, Anat. Embryol. 210, 419421.

  • Scheef L., Boecker H., Daamen M., Fehse U., Landsberg M. W., Granath D.-O., Mechling H., Effenberg A. O. (2009). Multimodal motion processing in area V5/MT — evidence from an artificial class of audio-visual events, Brain Res. 2152, 94104.

    • Search Google Scholar
    • Export Citation
  • Schmidt R. A., Lee T. (2011). Motor Control and Learning — A Behavioral Emphasis. Human Kinetics, Champaign, IL, USA.

  • Schmitz G., Effenberg A. O. (2012). Perceptual effects of auditory information about own and other movements, in: Proceedings of the 18th International Conference on Auditory Display, Atlanta, GA, USA, pp. 89–94.

  • Schmitz G., Mohammadi B., Hammer A., Heldmann M., Samii A., Münte T. F., Effenberg A. O. (2013). Observation of sonified movements engages a basal ganglia frontocortical network, Neuroscience 14, 111.

    • Search Google Scholar
    • Export Citation
  • Seitz A. R., Kim R., Shams L. (2006). Sound facilitates visual learning, Curr. Biol. 16, 14221427.

  • Shams L., Seitz A. R. (2008). Benefits of multisensory learning, Trends Cogn. Sci. 12, 411417.

  • Stein B. E., Meredith M. A. (1993). The Merging of the Senses. MIT Press, Cambridge, MA, USA.

  • Thornton I. M., Rensink R. A., Shiffrar M. (2002). Active versus passive processing of biological motion, Perception 31, 837853.

  • Vogt K., Pirro D., Kobenz I., Höldrich R., Eckel G. (2010). PhysioSonic — evaluated movement sonification as auditory feedback in physiotherapy, in: Auditory Display, 6th International Symposium, CMMR/ICAD 2009, Lecture Notes in Computer Science, S. Ystad, M. Aramaki, R. Kronland-Martinet and K. Jensen (Eds), pp. 103–120. Springer, Berlin, Germany.

  • Williams A. M., Ward P., Smeeton N. J. (2004). Perceptual and cognitive expertise in sport — implications for skill acquisition and performance enhancement, in: Skill Acquisition in Sport — Research, Theory and Practice, Williams A. M., Hodges N. J. (Eds), pp.  328347. Routledge, New York, NY, USA.

    • Search Google Scholar
    • Export Citation
  • Wilson-Bokowiec J., Bokowiec M. A. (2006). Kinaesonics: the intertwining relationship of body and sound, Contemp. Music Rev. 25, 4757.

    • Search Google Scholar
    • Export Citation
  • Young W., Rodger M., Craig C. M. (2012). Perceiving and reenacting spatiotemporal characteristics of walking sounds, J. Exp. Psychol. Hum. Percept. Perform. 39, 464476.

    • Search Google Scholar
    • Export Citation

Content Metrics

All Time Past Year Past 30 Days
Abstract Views 304 114 14
Full Text Views 136 7 0
PDF Views & Downloads 16 5 0