Postdoc Researcher in Computational Modeling of Multimodal Communication in Face-to-Face Dialogue

Are you interested in combining insights from cognitive science with computational linguistics? Do you have experience working with data-driven multimodal methods and / or computational approaches for analyzing dialogue? If you are excited about doing this kind of research in an interdisciplinary environment, with a team of friendly and enthusiastic colleagues, then you may want to join us.

We seek a Postdoctoral researcher for the project Understanding the Emergence and Maintenance of Cross-modal Convergence in Dialogue. Mutual understanding in dialogue requires convergence on mental representations of meaning between interlocutors. Such convergence emerges over time through the close coordination of interlocutors in real-time, and through building common ground by means of gesturing and speaking. Earlier conversational research has identified mechanisms of common ground building. However, it remains unclear how and when speakers converge when referents are signaled through naturalistic multimodal behaviors (i.e., speech and gesture). The project aims to advance the status quo by investigating the dynamics of multimodal referential processes and, ultimately, by developing a computational model powered by machine learning that captures multimodal referential convergence in dialogue.

What are you going to do

The project builds on a rich dataset of multimodal face-to-face dialogue that has already been collected, and seeks to combine natural language processing, body-motion analysis, and machine learning to identify and model, in a data-driven manner, processes of multimodal alignment in referential tasks. The project is part of the Language in Interaction consortium, in particular Big Question 3 . The postdoc will be led and advised by researchers at the Institute for Logic, Language and Computation (ILLC), University of Amsterdam, and the Donders Institute for Brain, Cognition and Behavior, Radboud University. The researchers coordinating the project include Raquel Fernández (ILLC, Amsterdam), Asli Ozyurek, Judith Holler, Ivan Toni, and Wim Pouw (Donders Institute, Nijmegen). The postdoc will be based at the ILLC, joining the Dialogue Modelling Group led by Raquel Fernández, and will visit the Donders Institute regularly.

Your tasks and responsibilities:

• carry out research as described in the project proposal, including developing data analysis algorithms for mapping out multimodal convergences and training multimodal deep neural network models for reference resolution;
• present and publish at key international conferences and in leading journals;
• contribute your expertise to joint research projects within the research groups;
• participate in and contribute to the organization of research activities and events at the ILLC, such as workshops and colloquia;
• make a small contribution to the ILLC's educational mission, by possibly taking part in teaching and student supervision activities in areas related to your expertise.