Plenary talk – Matthew Crocker

A neurocomputational model of surprisal in comprehension

Matthew W. Crocker

Saarland University,
Zeit: Freitag, 10. März, 2017, 10:00 – 11:00
Ort: Gebäude B 4.1, Raum 0.01

People continuously assign meaning to the linguistic signal on a more or less word-by-word basis, building rich semantic representations of what has been encountered so far, which in turn conditions expectations about what is likely—or not—to follow. Surprisal Theory (Hale, 2001; Levy, 2008) has been particularly successful in offering a broad account of word-by-word processing difficulty. The theory asserts that the processing difficulty incurred by a word is inversely proportional to the expectancy (or surprisal) of a word, as estimated by probabilistic language models. Such models are limited, however, in that they assume expectancy is determined by linguistic experience alone, making it difficult to accommodate the influence of world and situational knowledge.

To address this issue, we1 present a neurocomputational model of language processing that seamlessly integrates linguistic experience and probabilistic world knowledge in online comprehension. The model is a simple recurrent network (SRN) that is trained to map sentences onto rich probabilistic meaning representations that are derived from a Distributed Situation-state Space (DSS: Frank et al, 2003). Crucially, these meaning representations go beyond literal propositional content and capture inferences driven by knowledge about the world (cf. a mental model (Johnson-Laird, 1983), or situation model (Zwaan, 1998)). The model is trained to construct the DSS representation for a sentence on an incremental, word-by-word basis, which results in the model being inherently sensitive to the frequency with which sentences are mapped onto specific DSSs (cf. Mayberry et al, 2009). Furthermore, the incremental construction of DSS representations allows for the computation of online surprisal based on the likelihood of the sentence meaning for the just processed word, given the sentence meaning up until before the word was encountered.

We demonstrate that our online surprisal metric integrates both the likelihood of a situation model (DSS)—thereby reflecting world knowledge—as well as linguistic experience. This ‘comprehension-centric’ characterisation of surprisal thus provides a more general index of the effort involved in mapping from the linguistic signal to rich and knowledge-driven situation models: Not only can this sentence-to-meaning mapping capture established surprisal phenomena reflecting linguistic experience, it also offers the potential for surprisal-based explanations of a range of findings that have demonstrated the importance of knowledge-, discourse-, and script-driven influences on processing difficulty.

Moreover, we will outline how our model can be related to neurophysiological indices of processing difficulty. Previous work has linked word-induced surprisal to the N400 component of the ERP signal (e.g., Frank et al. 2015). Our meaning-based notion of surprisal, however, reflects how unexpected an update is for the unfolding interpretation. This suggests a close link to the P600, which has been argued to index of the effort involved in updating the interpretation (Brouwer et al, in press). By supporting the straight-forward computation of meaning-based surprisal, as well as providing a direct linking hypothesis to ERP correlates, this work sheds much needed light on the neural mechanisms that underlie the otherwise still abstract notion of surprisal.

References: • Brouwer, H., Crocker, M.W., Venhuizen, N., Hoeks, J. (in press): A Neurocomputational Model of the N400 and the P600 in Language Processing. Cognitive Science. • Frank, S., Koppen, M., Noordman, L., Vonk, W. (2003): Modeling knowledge-based inferences in story compre- hension. Cognitive Science 27(6):875–910. • Frank, S., Otten, L., Galli, G., Vigliocco, G. (2015): The ERP response to the amount of information conveyed by words in sentences. Brain and language 140:1–11. • Hale, J. (2001): A probabilistic Earley parser as a psycholinguistic model. In: Proc. of the ACL 1–8. • Johnson-Laird, P.N. (1983): Mental Models. • Levy, R. (2008): Expectation-based syntactic comprehension. Cognition 106(3):1126–1177. • Mayberry, M., Crocker, M.W., Knoeferle, P. (2009). Learning to Attend: A Connectionist Model of Situated Language Comprehension. Cognitive Science 33(1):449–496. • Zwaan, R., Radvansky, G. (1998): Situation models in language comprehension and memory. Psych. Bul. 123:162–185.

1 This abstract presents joint work with Harm Brouwer and Noortje Venhuizen.