CLASP
The Centre for Linguistic Theory and Studies in Probability

What's in a translation model? Analyzing neural seq2seq models and the representations they learn

Neural sequence-to-sequence architectures are powerful models for various NLP tasks, machine translation being one of them. We are interested in exploring the representations that are learned by such models when trained on large and diverse multilingual data sets. It is still an open question what kind of linguistic properties are covered and how they are encoded in complex architectures such as a multi-layered transformer architecture, the current state-of-the-art in machine translation and many other tasks. Our main questions include the influence of multilinguality on linguistic abstraction, the traces of specific syntactic and semantic patterns in language representations and the differences of embeddings spaces trained with different objectives. In the talk I will discuss a few of our recent studies as part of the ERC project “Found in Translation” and the additional questions that they raise.