Unitary Matrices are Composable and Learnable Word Embeddings

• Event: Seminar
• Lecturer: Jean-Philippe Bernardy and Shalom Lappin from University of Gothenburg
• Date: 26 January 2022
• Duration: 2 hours
• Venue: Online

Abstract: “Unitary-evolution recurrent neural networks (URN) were previously introduced to address the problem of exploding and vanishing gradients, but have several other advantages. In this talk I will focus on the word embeddings that they learn. These are unitary matrices (unitary embeddings for short). Because of the absence of activation functions, the behaviour of the network is amenable to analysis using the methods of linear algebra. In particular unitary embeddings can be composed by multiplication. We develop and train a variant of the URN for two NLP-relevant tasks, and we achieve state of the art results on both. Our experiments show that they are able to track long distance dependencies, without additional storage or processing devices.”