CLASP
The Centre for Linguistic Theory and Studies in Probability

Scaling up a joint model of word meaning and sentence meaning: Situation Description Systems and the Visual Genome

Abstract How can a fine-grained representation of word meaning be integrated with a formal representation of sentence meaning? More specifically, how can such a representation address uncertainty and gradience? There are a number of approaches that address this question, including McNally (2015). Emerson (2016,2018,2020), Bernardy et al (2018, 2019), Sadrzadeh and Muskens (2018). Erk and Herbelot (2023). In this talk, we build on our own recent approach, Situation Description Systems (Erk and Herbelot 2023). Situation Description Systems describe meaning as both intensional and conceptual; the conceptual representation is a probabilistic graphical model representing dependencies between the underlying concepts of words in the sentence. So far, the approach was only toy-size, and could not be applied at scale. In this talk, we present a scaled-up variant of Situation Description Systems that uses a sizable lexicon derived from the Visual Genome database of annotated images. Concepts are represented by embeddings computed from labels in the image annotations.