During conversations, we engage in turn-taking behaviour that proceeds back and forth effortlessly as we communicate. In any given day, we participate in numerous face-to-face interactions that contain social cues from our partner and we interpret these cues to rapidly identify whether it is appropriate to speak. Although the benefit provided by visual cues has been well established in several areas of communication, the use of visual information to make turn-taking decisions during conversation is unclear. Here we conducted two experiments to investigate the role of visual information in identifying conversational turn exchanges. We presented clips containing single utterances spoken by single individuals engaged in a natural conversation with another. These utterances were from either right before a turn exchange (i.e., when the current talker would finish and the other would begin) or were utterances where the same talker would continue speaking. In Experiment 1, participants were presented audiovisual, auditory-only and visual-only versions of our stimuli and identified whether a turn exchange would occur or not. We demonstrated that although participants could identify turn exchanges with unimodal information alone, they performed best in the audiovisual modality. In Experiment 2, we presented participants audiovisual turn exchanges where the talker, the listener or both were visible. We showed that participants suffered a cost at identifying turns exchanges when visual cues from the listener were not available. Overall, we demonstrate that although auditory information is sufficient for successful conversation, visual information plays an important role in the overall efficiency of communication.
MixdorffH.HonemannH.KimJ.DavisC. (2015). Anticipation of turn-switching in auditory-visual dialogs in: Proceedings of 1st Joint Conference on Facial Analysis Animation and Auditory-Visual Speech Processing pp. 52–56. Vienna Austria.
MunhallK. G.JonesJ. A.CallanD. E.KuratateT.Vatikiotis-BatesonE. (2004).
Visual prosody and speech intelligibility: head movement improves auditory speech perceptionPsychol. Sci.15133–137.
NenkovaA.GravanoA.HirschbergJ. (2008). High frequency word entrainment in spoken dialogue in: Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies pp. 169–172. Columbus OH USA.
Investigation of the relationship between turn-taking and prosodic features of spontaneous dialogue in:
Interspeech 2005 pp.
33–36. Lisbon, Portugal.
StiversT.EnfieldN. J.BrownP.EnglertC.HayashiM.HeinemannT.HoymannG.RossanoF.De RuiterJ. P.YoonK. E.LevinsonS. C. (2009).
Universals and cultural variation in turn-taking in conversationProc. Natl. Acad. Sci. USA10610587–10592.
VerbruggeR. R. (1985). Language and event perception: steps toward a synthesis in: Persistence and Change: Proceedings of the First International Conference on Event Perception W. H. Warren and R. E. Shaw (Eds) pp. 157–194. Lawrence Erlbaum Associates Publishers Hillsdale NJ USA.