Research Article: The Frequency of Rapid Pupil Dilations as a Measure of Linguistic Processing Difficulty

Date Published: January 22, 2016

Publisher: Public Library of Science

Author(s): Vera Demberg, Asad Sayeed, Emmanuel Andreas Stamatakis.


While it has long been known that the pupil reacts to cognitive load, pupil size has received little attention in cognitive research because of its long latency and the difficulty of separating effects of cognitive load from the light reflex or effects due to eye movements. A novel measure, the Index of Cognitive Activity (ICA), relates cognitive effort to the frequency of small rapid dilations of the pupil. We report here on a total of seven experiments which test whether the ICA reliably indexes linguistically induced cognitive load: three experiments in reading (a manipulation of grammatical gender match / mismatch, an experiment of semantic fit, and an experiment comparing locally ambiguous subject versus object relative clauses, all in German), three dual-task experiments with simultaneous driving and spoken language comprehension (using the same manipulations as in the single-task reading experiments), and a visual world experiment comparing the processing of causal versus concessive discourse markers. These experiments are the first to investigate the effect and time course of the ICA in language processing. All of our experiments support the idea that the ICA indexes linguistic processing difficulty. The effects of our linguistic manipulations on the ICA are consistent for reading and auditory presentation. Furthermore, our experiments show that the ICA allows for usage within a multi-task paradigm. Its robustness with respect to eye movements means that it is a valid measure of processing difficulty for usage within the visual world paradigm, which will allow researchers to assess both visual attention and processing difficulty at the same time, using an eye-tracker. We argue that the ICA is indicative of activity in the locus caeruleus area of the brain stem, which has recently also been linked to P600 effects observed in psycholinguistic EEG experiments.

Partial Text

Pupil size has long been known to reflect arousal [1] and cognitive load in a variety of different tasks such as arithmetic problems [2], digit recall [3], attention [4] as well as language complexity [5–9], grammatical violations, context integration effects [10] and recently even pragmatic effects [11]. All of these studies have looked at the overall effect of pupil dilation; however, raw pupil dilation as a measure of cognitive load is always at risk of confounding the load reflex with the light reflex, especially in settings where the visual surroundings change or where the screen cannot in all conditions be fully controlled for luminosity of all objects on the screen (note also that the light reflex can even pose a problem in constant lighting conditions, because the pupil exhibits irregular oscillation under the influence of constant light). The “Index of Cognitive Activity” or ICA [12–14] proposes to solve this problem by identifying only those rapid dilations which are related to cognitive load but not the light reflex. The ICA separates the effect of the light reflex on pupil size (which causes larger and slower changes in pupil size) from the effect of the load reflex (observable in the frequency of rapid small dilations) by decomposing the raw pupil size signal with different wavelets to obtain high vs. low frequency components of the signal. The ICA is therefore more robust with respect to changes in ambient light than macro-level pupil dilation [12].

The Index of Cognitive Activity is a measure of cognitive load which has previously only been evaluated on a small range of tasks [12–14, 25, 26], including digit span tasks, visual tasks, and a simulated driving task. Using the ICA as a measure of processing load is motivated by the finding that pupil size can be affected by two different processes: lighting conditions and cognitive activity. In overall pupil dilation, these two effects are confounded, even when light conditions is stable due to the so-called “light reflex”, meaning that the pupil oscillates irregularly and continually. Pupil dilation is controlled by two groups of muscles: circular muscles, which make the pupil contract, and radial muscles, which make the pupil dilate. Furthermore, we know that the activation and inhibition patterns are different for reactions to light and reactions to cognitive activity [12, 27]: dilations due to cognitive activity are very rapid and small, while changes in pupil size due to lighting are slower and larger. The ICA disentangles these patterns by performing a wavelet analysis on the pupil dilation record to remove all large oscillations and retain only the small and rapid dilations. Among all small dilations and constrictions of the pupil, there are then the ones related to cognitive load in which we are interested as well as some random noise. [12] applies a denoising technique which tests for significance of changes in the signal and sets all non-significant changes to zero. The resulting signal then contains rapid dilations and constrictions larger than the threshold for denoising. For the calculation of the ICA, only the rapid dilations are considered. For an example of a 2-second recording from our own experiment along with marks for where a rapid dilation was detected, see Fig 1.

The Head of the Saarland University ethics committee confirmed that if no confidential information is collected, if the experiments do not induce a stressful situation and do not involve negative, or emotionally adverse stimuli, then such a study does not need approval to be conducted. All data were anonymized before the authors had access for analysis. Written informed consent was obtained from all participants prior to the start of the experiment. The person to whom the eye image (Striking image) belongs provided consent to have the image of their eye published.

We found higher rates of rapid pupil dilations in the critical region of 750 to 1250 ms after stimulus onset for the more difficult linguistic condition in all three experiments. Effects were similar for both eyes. Taken together these first results provide strong initial evidence that the ICA measure is sensitive to linguistically induced processing difficulty.

The first three experiments have shown that the ICA is sensitive to linguistic manipulations. Experiments 4 to 6 test whether the effects can be replicated in a dual task setting in which a driving task is performed simultaneously with a language comprehension experiment. We know from previous related work [15, 26] that the ICA is also sensitive to the driving task. We here attempt to replicate earlier results in terms of the latency between the ICA and the steering events, and analyze the effect of steering on the ICA in order to adequately account for both effects in our analyses.

The results of the single task experiments 1–3 and the dual task experiments 4–6 are remarkably consistent with one another, even though the modalities for the self-paced reading task and the listening while driving task were very different. The overall time course of the effects was also quite stable: with the exception of the semantic anomaly experiment in the dual task condition, we found a significant difference between linguistic conditions in all experiments for the time period of 750 to 1250ms after critical region onset, a time period that is also consistent with the ICA effect in the driving task (peak of correlation between stimulus and ICA).

Our final experiment tests whether the ICA can serve as a useful measure in experiments using the visual world paradigm. The visual world paradigm is popular in language research, because people’s locus of visual attention can offer us insight into the language interpretation and time course of processing. However, the visual world paradigm does not allow us to get any insight into how difficult it is for a subject to get to a specific interpretation. Here, we want to explore whether the ICA is sufficiently robust to changing fixation positions on the screen in order to allow us to use it to asses cognitive load in addition to visual attention when using the visual world paradigm.

The experiments presented here provide strong evidence that the ICA is sensitive to linguistic manipulations, showing significant effects in the expected direction on both eyes during the pre-defined time window of 750–1250ms post stimulus onset in all experiments except one (experiment 5), for which we found a significant effect 250ms earlier. In two of the experiments we also observed learning effects: in experiment 7 (visual world) and experiment 3 (relative clauses in self-paced reading), we found that the effect of the experimental manipulation was statistically significant in the first half of the experiment, but disappeared during the second half of the experiment.