Context-sensitive word frequency: A context constructivist account of contextual diversity and word frequency effects

Abstract

The diversity of contexts in which a word occurred, operationalized as CD, better accounts for some sources of variance in measures of lexical processing previously attributed to token word frequency (WF), a result that has been found in research using a variety of paradigms, most of which show CD effects but no effects of WF. We revisit the relationship between CD and WF. We propose that lexical representations store context-sensitive frequencies and assume that language comprehension includes predictive processes that generate probability distributions for upcoming words. When variables that contribute to lexical processing are factored out, remaining effects reflect the need probability of a word in a specific context. This conceptualization, formalized, as an active context constructivist framework predicts that as contextual constraint increases CD effects will weaken. Moreover, WF effects should emerge in the three-condition design used in most experimental studies comparing CD and WF. We confirm these predictions using frequency judgments, eye-tracking during reading with three-condition designs, and a corpus analysis of eye-tracking data from natural texts. We discuss our framework within expectation-based models of sentence processing and the shift in sentence processing models towards enriched, context-sensitive lexical representations.

Publication
Psychology of Learning and Motivation