Exploration of Trans-Sensory Mappings

in Art & Perception
Restricted Access
Get Access to Full Text
Rent on DeepDyve

Have an Access Token?

Enter your access token to activate and access content online.

Please login and go to your personal user account to enter your access token.


Have Institutional Access?

Access content through your institution. Any other coaching guidance?


This paper explores the way music can be translated into a sequence of images. A general approach is described that associates various structures captured by different sense modalities. This approach is illustrated by examples from vision and audition. Music and sets of images are digitalized and represented as forms in multidimensional spaces. A mapping, which preserves the dissimilarities in each space, is then defined and an algorithm that can approximate the optimal fit is proposed. An example that maps a set of photos of different versions of the wire chair on various Mozart variations is given.

Exploration of Trans-Sensory Mappings

in Art & Perception



AymeA. (1984). N + 1 Variations sur une Empreinte de Viallat. Edition TraversièreParis, France.

BerenzweigA.EllisD. and LawrenceS. (s.a.). Anchor Space for Classification and Similarity Measurement of Music. Available from http://www.ee.columbia.edu/~dpwe/pubs/icme03-anchor.pdf.

BosseurJ.-Y. (2006). Musique et Arts Plastiques Interactions au XXe Siècle. Minerve, Paris, France.

DefaysD. (2015). Exploration on audio-visual mapping. Presented at the Workshop on Symbolic Data Analysis Orléans France November 2015. http://www.univ-orleans.fr/mapmo/colloques/sda2015/.

DefaysD. (in press). Appariement de matrices de dissimilarités. Modulad.

DidayE. and Noirhomme-FraitureM. (Eds) (2008). Symbolic Data Analysis and the SODAS Software. WileyChichester, UK.

HofstadterD. (1979). Gödel Escher Bach: An Eternal Golden Braid. Basic BooksNew York, NY, USA.

Hofstadter D. and the Fluid Analogies Research Group (1995). Fluid Concepts and Creative Analogies. Computer Models of the Fundamental Mechanisms of Thought . Basic BooksNew York, NY, USA.

HofstadterD. and SanderE. (2013). Surfaces and Essences. Analogy as the Fuel and Fire of Thinking. Basic BooksNew York, NY, USA.

KimR.PetersM. and ShamsL. (2012). 0 + 1 > 1: How adding noninformative sound improves performance on a visual task. Psychol. Sci.23612.

LoganB. (s.a.) Mel Frequency Cepstral Coefficients for Music Modelling. Available from http://ismir2000.ismir.net/papers/logan_paper.pdf.

MareschalD. and FrenchR. (2016). TRACX2: A connectionist autoencoder using graded chunks to model infant visual statistical learning. Philos. Trans. R. Soc. B Biol. Sci . 372 (1711).

MichalskiR.CarbonellJ. and MitchellM. (1986). Machine Learning. An Artificial Intelligence Approach. Volume II. Morgan Kaufmann Publishers, Los AltosCA, USA.

NewtonI. (1671–1672). Letter to Henry Oldenburg in: Philos. Trans.8030753087.

OxJ. and BrittonD. (2000). The 21st century virtual reality color organIEEE Multimedia769.

PhamN.-K.MorinA. and GrosP. (2008). CAViz, an interactive graphical tool for image miningJ. Comput. Inform. Technol.16295 302.

PhamN.-K.MorinA.GrosP. and LeQ.-T. (2009). Utilisation de l’analyse factorielle des correspondances pour la recherche d’images à grande échelle in: Actes d'EGC RNTI-E-15 Revue des Nouvelles Technologies de l'Information - Série Extraction et Gestion des Connaissances pp. 283294Cépaduès EditionsToulouse, France.

SacksO. (2007). Musicophilia Tales of Music and the Brain. KnopfNew York, NY, USA.

ThomsonW. (2015). Music Thought and Feeling. Understanding the Psychology of MusicOxford University PressNew York, NY, USA.

WidmerG. S.DixonW.GoeblE.PampalkA. and TobudicA. (2003). In search of the Horowitz factor. AI Mag.24111130.


  • View in gallery

    Which of these images corresponds best to the song Happy Birthday to You? And why? This figure is published in color in the online version.

  • View in gallery

    Unraveling Bolero by Anne Adams. UCSF Memory and Aging Center. This figure is published in color in the online version.

  • View in gallery

    Set of random points mapped on an inverted U-shape.

  • View in gallery

    Thirty-three variants of the wire chair (Paris, December 2015). This figure is published in color in the online version.

  • View in gallery

    Representation of the 12 variations of Mozart from Mel Frequency Cepstral Coefficients.

  • View in gallery

    Representation of 33 variants of the wire chair with an image analysis package. This figure is published in color in the online version.

  • View in gallery

    Association of an extract of Mozart variations with wire chairs. This figure is published in color in the online version.

  • View in gallery

    Quality of the representation of the six Mozart variations. High values indicate bad fits.


Content Metrics

Content Metrics

All Time Past Year Past 30 Days
Abstract Views 7 7 5
Full Text Views 62 62 52
PDF Downloads 3 3 1
EPUB Downloads 0 0 0