Touch-Screen Technology for the Dynamic Display of 2D Spatial Information Without Vision: Promise and Progress

in Multisensory Research
Restricted Access
Get Access to Full Text
Rent on DeepDyve

Have an Access Token?

Enter your access token to activate and access content online.

Please login and go to your personal user account to enter your access token.


Have Institutional Access?

Access content through your institution. Any other coaching guidance?


Many developers wish to capitalize on touch-screen technology for developing aids for the blind, particularly by incorporating vibrotactile stimulation to convey patterns on their surfaces, which otherwise are featureless. Our belief is that they will need to take into account basic research on haptic perception in designing these graphics interfaces. We point out constraints and limitations in haptic processing that affect the use of these devices. We also suggest ways to use sound to augment basic information from touch, and we include evaluation data from users of a touch-screen device with vibrotactile and auditory feedback that we have been developing, called a vibro-audio interface.

Touch-Screen Technology for the Dynamic Display of 2D Spatial Information Without Vision: Promise and Progress

in Multisensory Research



Apkarian-StielauP. M.LoomisJ. M. (1975). A comparison of tactile and blurred visual form perceptionPercept. Psychophys. 18362368.

AuvrayM.HannetonS.LenayC.O’ReganK. (2005). There is something out there: distal attribution in sensory substitution, twenty years laterJ. Integr. Neurosci. 4505521.

AuvrayM.HannetonS.O’ReganJ. K. (2007). Learning to perceive with a visuo-auditory substitution system: localisation and object recognition with ‘the vOICe’Perception 36416430.

Bach-y-RitaP. (1967). Sensory plasticity: applications to a vision substitution systemActa Neurol. Scand. 43417426.

Bach-y-RitaP. (1972). Brain Mechanisms in Sensory Substitution. Academic PressNew York, NY, USA.

Bach-y-RitaP.KaczmarekK. A.TylerM. E.Garcia-LaraJ. (1998). Form perception with a 49-point electrotactile stimulus array on the tongue: a technical noteJ. Rehabil. Res. Dev. 35427430.

Bach-y-RitaP.KercelS. (2003). Sensory substitution and the human machine interfaceTrends Cogn. Sci. 7541546.

BauO.PoupyrevI.IsrarA.HarrisonC. (2010). TeslaTouch: electrovibration for touch surfaces in: Proc. 23rd Ann. ACM Symp. User interface Software and Technology pp. 283–292. ACM New York NY USA.

BerlaE. (1973). Strategies in scanning a tactual pseudo-mapEduc. Vis. Handicap. 5819.

BlattnerM. M.SumikawaD. A.GreenbergR. M. (1989). Earcons and icons: their structure and common design principlesHum. Comput. Interact. 41144.

BlauertJ. (1997). Spatial Hearing: The Psychophysics of Human Sound Localization. MIT PressCambridge, MA, USA.

BrownD.MacphersonT.WardJ. (2011). Seeing with sound? Exploring different characteristics of a visual-to-auditory sensory substitution devicePerception 4011201135.

CarpenterP. A.ShahP. (1998). A model of the perceptual and conceptual processes in graph comprehensionJ. Exp. Psychol. Appl. 475100.

ChebatD. R.RainvilleC.KupersR.PtitoM. (2007). Tactile–‘visual’ acuity of the tongue in early blind individualsNeuroreport 1819011904.

ChebatD. R.SchneiderF. C.KupersR.PtitoM. (2011). Navigation with a sensory substitution device in congenitally blind individualsNeuroreport 22342347.

ChengM. F. (1968). Tactile–kinesthetic perception of lengthAm. J. Psychol. 817482.

CollinsC. C. (1970). Tactile television: mechanical and electrical image projection IEEE Trans. Man–Mach. Syst. MMS-11 65–71.

CollinsC. C. (1985). On mobility aids for the blind in: Electronic Spatial Sensing for the BlindWarrenD. H.StrelowE. R. (Eds) pp.  3564. Martinus NijhoffBoston, MA, USA.

CollinsC. C.MadeyJ. M. (1974). Tactile sensory replacementProc. San Diego Biomed. Symp. 131526.

CraigJ. C. (1999). Grating orientation as a measure of tactile spatial acuitySomatosens. Mot. Res. 16197206.

Cronly-DillonJ.PersaudK. C.BloreR. (2000). Blind subjects construct conscious mental images of visual scenes encoded in musical formProc. R. Soc. Lond. B Biol. Sci. 26722312238.

EitanZ.TimmersR. (2010). Beethoven’s last piano sonata and those who follow crocodiles: cross-domain mappings of auditory pitch in a musical contextCognition 114405422.

FerresL.LindgaardG.SumegiL. (2010). Evaluating a tool for improving accessibility to charts and graphs in: Proc. 12th Int. ACM SIGACCESS Conf. Computers and Accessibility (ASSETS ’10) pp. 83–90. ACM New York NY USA.

FranksF.BairdR. (1971). Geographical concepts and the visually handicappedExcept. Child. 38321324.

GalesicM.Garcia-RetameroR. (2011). Graph literacy: a cross-cultural comparisonMed. Decis. Making 31444457.

GardonyA. L.BrunyeT. T.TaylorH. A. (2013). Gardony map drawing analyzer: a novel approach to the analysis of sketch maps poster at the Annual Convention of the Association for Psychology Science Washington D.C. May 25.

GaverW. W. (1986). Auditory icons: using sound in computer interfacesHum. Comp. Interact. 2167177.

GiudiceN. A.LeggeG. E. (2008). Blind navigation and the role of technology in: Engineering Handbook of Smart Technology for Aging Disability and IndependenceHelalA.MokhtariM.AbdulrazakB. (Eds) pp.  479500. John Wiley and SonsNew York, NY, USA.

GiudiceN. A.PalaniH. P.BrennerE.KramerK. M. (2012). Learning non-visual graphical information using a touch-based vibro-audio interface in: Proc. 14th Int. ACM SIGACCESS Conf. Computers and Accessibility (ASSETS ’12) pp. 103–110. ACM New York NY USA.

GolledgeR. G. (1991). Tactual strip maps as navigational aidsJ. Vis. Impair. Blind. 85296301.

GoncuC.MarriottK. (2011). GraVVITAS: generic multi-touch presentation of accessible graphicsLect. Notes Comput. Sci. 69463048.

HogganE.BrewsterS. A.JohnstonJ. (2008). Investigating the effectiveness of tactile feedback for mobile touchscreens in: Proc. SIGCHI Conf Human Factors in Computing Systems (CHI ’08) pp. 1573–1582. ACM New York NY USA.

JanssonG. (1983). Tactile guidance of movementInt. J. Neurosci. 193746.

JohanssonR. S.VallboA. B. (1983). Tactile sensory coding in the glabrous skin of the human handTrends Neurosci. 62732.

KappersA. M. L. (2007). Haptic spatial processing: allocentric and egocentric reference framesCan. J. Exp. Psychol. 61208218.

KayeH. S.KangT.LaPlanteM. P. (2000). Mobility device use in the United States. Disability Statistics Report (14). Dept. of Education National Institute on Disability and Rehabilitation Research Washington D.C. USA.

KlatzkyR. L.LedermanS. J. (2003). Representing spatial location and layout from sparse kinesthetic contactsJ. Exp. Psychol.: Hum. Percept. Perform. 29310325.

KramerG. (1994). An introduction to auditory display in: Auditory DisplayKramerG. (Ed.) pp.  178. Addison-WesleyReading, MA, USA.

LedermanS. J.CampbellJ. I. (1982). Tangible graphs for the blindHum. Fact. 2485100.

LedermanS. J.JonesL. A. (2011). Tactile and haptic illusionsIEEE Trans. Haptics 4273294.

LedermanS. J.KlatzkyR. L. (2009). Haptic perception: a tutorialAtten. Percept. Psychophys. 7114391459.

LedermanS. J.KlatzkyR. L.ChatawayC.SummersC. (1990). Visual mediation and the haptic recognition of two-dimensional pictures of common objectsPercept. Psychophys. 475464.

LeggeG. E.MadisonC.VaughnB. N.CheongA. M. Y.MillerJ. C. (2008). Retention of high tactile acuity throughout the life span in blindnessPercept. Psychophys. 7014711488.

Levy-TzedekS.HanassyS.AbboudS.MaidenbaumS.AmediA. (2012). Fast, accurate reaching movements with a visual-to-auditory sensory substitution deviceRestor. Neurol. Neurosci. 30313323.

LoomisJ. L.KlatzkyR. L.GiudiceN. A. (2012). Sensory substitution of vision: importance of perceptual and cognitive processing in: Assistive Technology for Blindness and Low VisionManduchiR.KurniawanS. (Eds) pp.  162191. CRC PressBoca Raton, FL, USA.

LoomisJ. M. (1981). Tactile pattern perceptionPerception 10527.

LoomisJ. M. (1990). A model of character recognition and legibilityJ. Exp. Psychol. Hum. Percept. Perform. 16106120.

LoomisJ. M.KlatzkyR. L.LedermanS. J. (1991). Similarity of tactual and visual picture recognition with limited field of viewPerception 20167177.

MacLeanK. E. (2008). Haptic interaction design for everyday interfaces in: Reviews of Human Factors and ErgonomicsCarswellM. (Ed.) Vol. 4 pp.  149194. Human Factors and Ergonomics SocietySanta Monica, CA, USA.

MeijerP. B. L. (1992). An experimental system for auditory image representationsIEEE Tran. Biomed. Eng. 39112121.

MerriamE. P.GenoveseC. R.ColbyC. L. (2007). Remapping in human visual cortexJ. Neurophysiol. 9717381755.

MillarS. (1981). Self-referent and movement cues in coding spatial location by blind and sighted childrenPerception 10255264.

MillarS. (2008). Space and Sense. Psychology PressNew York, NY, USA.

NeesM. A.WalkerB. N. (2009). Auditory interfaces and sonification in: The Universal Access HandbookStephanidisC. (Ed.) pp.  507521. CRC PressNew York, NY, USA.

NolanC. Y.MorrisJ. E. (1971). Improvement of Tactual Symbols for Blind Children: Final Report American Printing House for the Blind Louisville KY USA.

Perkins Maps (2013).

Perkins Museum (2013).

PetitG.DufresneA.LevesqueV.HaywardV.TrudeauN. (2008). Refreshable tactile graphics applied to schoolbook illustrations for students with visual impairment in: Proc. 10th Int. ACM SIGACCESS Conf. Computers and Accessibility (ASSETS ’08) pp. 89–96. ACM New York NY USA.

PollokB.SchnitzlerI.MierdorfT.StoerigP.SchnitzlerA. (2005). Image-to-sound conversion: experience-induced plasticity in auditory cortex of blindfolded adultsExp. Brain Res. 167287291.

PoppingaB.MagnussonC.PielotM.Rassmus-GröhnK. (2011). TouchOver map: audio-tactile exploration of interactive maps in: Proc. 13th Int. Conf. Human Computer Interaction with Mobile Devices and Services (MobileHCI ’11) pp. 545–550. ACM New York NY USA.

PrattC. C. (1930). The spatial character of high and low tonesJ. Exp. Psychol. 13278285.

PtitoM.KupersR. (2005). Cross-modal plasticity in early blindnessJ. Integr. Neurosci. 4479488.

PtitoM.MoesgaardS. M.GjeddeA.KupersR. (2005). Cross-modal plasticity revealed by electrotactile stimulation of the tongue in the congenitally blindBrain 128606614.

RajaM. K. (2011). The development and validation of a new smartphone based non-visual spatial interface for learning indoor layouts Unpublished Masters thesis University of Maine Orono MN USA.

RastogiR.PawlukT. V. D.KetchumJ. (2013). Intuitive tactile zooming for graphics accessed by individuals who are blind and visually impairedIEEE Trans. Neural Syst. Rehabil. Eng. 21655663.

RenierL.CollignonO.PoirierC.TranduyD.VanlierdeA.BolA.VeraartC.De VolderA. G. (2005). Cross-modal activation of visual cortex during depth perception using auditory substitution of visionNeuroImage 26573580.

RosenbaumD. A.DawsonA. M.ChallisJ. H. (2006). Haptic tracking permits bimanual independenceJ. Exp. Psychol. Hum. Percept. Perform. 3212661275.

SampaioE.MarisS.Bach-y-RitaP. (2001). Brain plasticity: ‘visual’ acuity of blind persons via the tongueBrain Res. 908204207.

SandersA. F. J.KappersA. M. L. (2007). Haptically straight linesPerception 3616821697.

SegondH.WeissD.SampiaoE. (2005). Human spatial navigation via a visuo-tactile sensory substitution systemPerception 3412311249.

Striem-AmitE.DehaeneS.CohenL.AmediA. (2012). Reading with sounds: sensory substitution selectively activates the visual word form area in the blindNeuron 76640652.

TverskyB. (2003). Some ways graphics communicate in: Mobile Communication: Essays on Cognition and CommunityNyiriK. (Ed.) pp.  143156. Passagen VerlagWien, Austria.

WalkerB. N. (2002). Magnitude estimation of conceptual data dimensions for use in sonificationJ. Exp. Psychol. Appl. 8211221.

WalkerB. N.MauneyL. (2010). Universal design of auditory graphs: a comparison of sonification mappings for visually impaired and sighted listenersACM Trans. Access. Comput. 2Article 1216 pages.

WijntjesM. W. A.van LienenT.VerstijnenI. M.KappersA. M. L. (2008). Look what I have felt: unidentified haptic line drawings are identified after sketchingActa Psychol. 128255263.

WinfieldL.GlassmireJ.ColgateJ. E.PeshkinM. (2007). T-PaD: tactile pattern display through variable friction reduction in: Proc. 2nd Joint EuroHaptics Conf. Symp. Haptic Interfaces for Virtual Environment and Teleoperator Systems pp. 421–426. IEEE New York NY USA.

World Health Organization (2011). Visual Impairment and Blindness Fact Sheet.

XuC.IsrarA.PoupyrevI.BauO.HarrisonC. (2011). Tactile display for the visually impaired using TeslaTouch in: CHI ’11 Extended Abstracts on Human Factors in Computing Systems pp. 317–322. ACM New York NY USA.

YuW.BrewsterS. A. (2002a). Comparing two haptic interfaces for multimodal graph rendering in: Proc. 10th Symp. Haptic Interfaces for Virtual Environment and Teleoperator Systems pp. 3–9. IEEE New York NY USA.

YuW.BrewsterS. A. (2002b). Evaluation of multimodal graphs for blind peopleUnivers. Access Inform. Soc. 2105124.

ZacksJ.TverskyB. (1999). Bars and lines: a study of graphic communicationMem. Cogn. 2710731079.

ZahorikP.BrungartD. S.BronkhorstA. W. (2005). Auditory distance perception in humans: a summary of past and present researchActa Acustica United w. Acustica 91409420.


  • View in gallery

    A five-bend zigzag and one participant’s sketch, superimposed so that endpoints match.

Index Card

Content Metrics

Content Metrics

All Time Past Year Past 30 Days
Abstract Views 29 29 11
Full Text Views 13 13 11
PDF Downloads 5 5 3
EPUB Downloads 1 1 0