CLASP
The Centre for Linguistic Theory and Studies in Probability

A Process Algebra Account of Speech-Gesture Interaction

The talk is based on extensive corpus work dealing with the interaction of gesture and speech in natural route-description dialogues. The issue discussed is how non-regimented gesture and speech processes can be modelled in a formal system. The main argument in the talk is that this cannot be achieved in structural paradigms currently in use. The proposal is to turn instead to process algebras in the tradition of Milner’s pi-calculus. The special algebra discussed in the talk is a newly developed hybrid lambda-psi calculus which can transport typed lambda-expressions over communicating input-output channels. Central for the account is the notion of agent: Agents encode speech information, gesture information or both. They can put information on channels and send it to other channels or take information from others and combine it with the information they have. Speech-gesture interaction is conceptualised via channel interactions of this sort. Interactions are allowed, postponed or blocked using a typing system. Successful communication among agents leads to a multi-modal meaning representation serving as logical form.

Bibliography: Rieser, H. (2014). Gesture and Speech as Autonomous Communicating Processes. Talk at the Stuttgart Workshop on “Embodied meaning goes public”. Stuttgart University, December 2014 Rieser, H. (2015). When Hands Talk to Mouth. Gesture and Speech as Autonomous Communicating Processes. Proceedings of Semdial 2015, Gothenburg Rieser, H. (2017). A Process Algebra Account of Speech-Gesture Interaction. Preliminary version. Ms, Bielefeld University