FRONTIERS COPYRIGHT STATEMENT

© Copyright 2007-2016 Frontiers Media SA. All rights reserved.

All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA (“Frontiers”) or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.

The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers’ website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.

Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.

Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.

As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.

All copyright, and all rights therein, are protected by national and international copyright laws.

The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use.

ISSN 1664-8714
ISBN 978-2-88919-721-7
DOI 10.3389/978-2-88919-721-7

About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing.

All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world’s best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view.

By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area!

Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

TURNING THE MIND’S EYE INWARD: THE INTERPLAY BETWEEN SELECTIVE ATTENTION AND WORKING MEMORY

Topic Editors:

Elger Abrahamse, Ghent University, Belgium

Steve Majerus, Université de Liège, Belgium

Wim Fias, Ghent University, Belgium

Jean-Philippe van Dijck, Ghent University, Belgium

Historically, cognitive sciences have considered selective attention and working memory as largely separated cognitive functions. That is, selective attention as a concept is typically reserved for the processes that allow for the prioritization of specific sensory input, while working memory entails more central structures for maintaining (and operating on) temporary mental representations. However, over the last decades various observations have been reported that question such sharp distinction. Most importantly, information stored in working memory has been shown to modulate selective attention processing – and vice versa. At the theoretical level, these observations are paralleled by an increasingly dominant focus on working memory as (involving) the attended part of long-term memory, with some positions considering that working memory is equivalent to selective attention turned to long-term memory representations – or internal selective attention. This questions the existence of working memory as a dedicated cognitive function and raises the need for integrative accounts of working memory and attention. The next step will be to explore the precise implications of attentional accounts of WM for the understanding of specific aspects and characteristics of WM, such as serial order processing, its modality-specificity, its capacity limitations, its relation with executive functions, as well as the nature of attentional mechanisms involved. This research topic in Frontiers in Human Neuroscience aims at bringing together the latest insights and findings about the interplay between working memory and selective attention.

Citation: Abrahamse, E., Majerus S., Fias W., van Dijck, J-P., eds. (2016). Turning the Mind’s Eye Inward: The Interplay Between Selective Attention and Working Memory. Lausanne: Frontiers Media. doi: 10.3389/978-2-88919-721-7

Table of Contents

Editorial: Turning the Mind’s Eye Inward: The Interplay Between Selective Attention and Working Memory

Elger Abrahamse, Steve Majerus, Wim Fias and Jean-Philippe van Dijck

Attentional and non-attentional systems in the maintenance of verbal information in working memory: the executive and phonological loops

Valérie Camos and Pierre Barrouillet

A common short-term memory retrieval rate may describe many cognitive procedures

Evie Vergauwe and Nelson Cowan

Perceptual salience affects the contents of working memory during free-recollection of objects from natural scenes

Tiziana Pedale and Valerio Santangelo

The time course of protecting a visual memory representation from perceptual interference

Dirk van Moorselaar, Eren Gunseli, Jan Theeuwes and Christian N. L. Olivers

Flexibility of representational states in working memory

Nahid Zokaei, Shen Ning, Sanjay Manohar, Eva Feredoes and Masud Husain

Finding the answer in space: the mental whiteboard hypothesis on serial order in working memory

Elger Abrahamse, Jean-Philippe van Dijck, Steve Majerus and Wim Fias

Spatial coding of ordinal information in short- and long-term memory

Véronique Ginsburg and Wim Gevers

Better target detection in the presence of collinear flankers under high working memory load

Jan W. De Fockert and Jaclyn Leiser

Symbiosis of executive and selective attention in working memory

André Vandierendonck

Resource-sharing between internal maintenance and external selection modulates attentional capture by working memory content

Anastasia Kiyonaga and Tobias Egner

A multisensory perspective of working memory

Michel Quak, Raquel Elea London and Durk Talsma

Children with low working memory and children with ADHD: same or different?

Joni Holmes, Kerry A. Hilton, Maurice Place, Tracy P. Alloway, Julian G. Elliott and Susan E. Gathercole

Temporal dynamics of attentional selection in adult male carriers of the fragile X premutation allele and adult controls

Ling M. Wong, Flora Tassone, Susan M. Rivera and Tony J. Simon

How do selective attentional processes contribute to maintenance and recall in children’s working memory capacity?

Hannah E. Roome, John N. Towse and Chris Jarrold

Fixation duration surpasses pupil size as a measure of memory load in free viewing

Radha Nila Meghanathan, Cees van Leeuwen and Andrey R. Nikolae

The influences of working memory representations on long-range regression in text reading: an eye-tracking study

Teppei Tanaka, Masashi Sugimoto, Yuki Tanida and Satoru Saito

EDITORIAL
published: 10 November 2015
doi: 10.3389/fnhum.2015.00616

Editorial: Turning the Mind's Eye Inward: The Interplay Between Selective Attention and Working Memory

Elger Abrahamse[1][*], Steve Majerus[2], Wim Fias[1] and Jean-Philippe van Dijck[1]

1Department of Experimental Psychology, University of Ghent, Ghent, Belgium

2Department of Psychology, University of Liège, Liège, Belgium

Edited and reviewed by: Hauke R. Heekeren, Freie Universität Berlin, Germany

* Correspondence: Elger Abrahamse, elger.abrahamse@ugent.be

Received: 01 August 2015
Accepted: 26 October 2015
Published: 10 November 2015

Citation: Abrahamse E, Majerus S, Fias W and van Dijck J-P (2015) Editorial: Turning the Mind's Eye Inward: The Interplay Between Selective Attention and Working Memory. Front. Hum. Neurosci. 9:616. doi: 10.3389/fnhum.2015.00616

Keywords: working memory, selective attention, short-term memory, long-term memory, executive function

Working memory refers to an intriguing and essential problem in cognitive science: How does the brain succeed in maintaining information for certain duration and perform mental operations on the stored information? An important avenue in tackling this issue stems from the relatively well-defined domain of selective attention. The role of selective attention in working memory has become increasingly dominant over the last decades as information stored in working memory has been shown to modulate (external) selective attention processing—and vice versa (for reviews see e.g., Awh and Jonides, 2001; Kiyonaga and Egner, 2013). Indeed, whereas attentional components in traditional working memory models typically relate to executive mechanisms assumed to control and supervise dedicated working memory buffers (e.g., Baddeley and Hitch, 1974; Baddeley, 2000), more recent models envisage working memory as directly and specifically emerging from selective attention turned to long-term memory representations (e.g., Cowan, 1999; Oberauer, 2009).

Despite the prominent role that selective attentional models of working memory play in current cognitive science, there is still much work to do in determining the precise implications of these models for specific aspects and characteristics of working memory—in both healthy and neuropsychological populations. The current Research Topic in Frontiers in Human Neuroscience aims to contribute to this challenge and offers novel empirical observations, state-of-the-art reviews, and intriguing theoretical proposals that illuminate the interplay between selective attention and working memory. In general, these contributions can be categorized into three broad classes of studies:

First, the majority of the papers involve empirical or theoretical efforts directly aimed at a better understanding of the interplay between selective attention and working memory. Camos and Barrouillet (2014) review the literature and identify attentional and non-attentional mechanisms that contribute to the maintenance of information in verbal working memory. Vergauwe and Cowan (2014) conclude on the basis of a review that attentional search within working memory happens at a high-speed processing rate of about 35–40 ms per item, and propose that this rate reflects the involvement of gamma oscillations in the brain. Pedale and Santangelo (2015) investigated to what degree sensory saliency of the item information during encoding modulates the current contents of WM during recollection. van Moorselaar et al. (2015) focus on the reverse relationship and investigate the time course in which information in working memory is protected against interfering visual input. Zokaei et al. (2014) tackle an outstanding question about the representational state of items in working memory that are not prioritized by attentional focusing. Abrahamse et al. (2014) hypothesize that serial order coding in verbal working memory appeals to the spatial attention system and thereby can be synthesized with selective attentional models on working memory in general. Ginsburg and Gevers (2015) conclude from their experiments that serial order in working memory and long-term memory both drive spatial effects but differentially so. De Fockert and Leiser (2014) tackle the relationship between working memory functioning and executive attention and wonder whether the lack of available working memory resources can in some situations enhance information processing. By reviewing the literature, Vandierendonck (2014) elaborates on the link between executive attention and selective attention and proposes a symbiotic model in which their close relationship is stressed. Relatedly, Kiyonaga and Egner (2014) investigate to what degree interactions between external attention and attention in working memory depend on shared attentional resources. Quak et al. (2015) extend these issues by emphasizing the need for a multisensory approach to working memory, and list a number of potential starting points.

Second, the study of the interplay between selective attention and working memory is not only fruitful for increasing our understanding of the cognitive underpinnings of working memory, it also helps us to achieve a better understanding of specific cognitive and behavioral difficulties across various types of populations. Holmes et al. (2014) demonstrate this to be the case for children with ADHD and children with low working memory capacity. In a similar vein Wong et al. (2015) show that this approach is also helpful to describe the subtle cognitive characteristics associated with specific genetic variations, such as the fragile X premutation. Roome et al. (2014) propose and show how the bisectioning of working memory into various (attentional and other) components can help to better understand the (normal) development of working memory.

Third, the current Research Topic also contains studies that extrapolate insights from the interplay between selective attention and working memory to other domains. For example, Meghanathan et al. (2015) investigate whether the availability of working memory resources can be predicted from oculometric parameters, such as eye fixation duration and pupil size. Moreover, Tanaka et al. (2014) use eye-tracking methodology to investigate to what degree individual differences in working memory capacity are predictive of word relocation processes during reading.

Overall, we are convinced that the studies in the current Research Topic provide much food for thought, as well as inspiration to keep up the empirical work on the interplay between selective attention and working memory. We would explicitly like to thank all the authors for their contributions as well as the reviewers for their critical reading.

ACKNOWLEDGMENTS

EA was supported by Research Foundation Flanders under contract number 12C4715N. SM was supported by grants F.R.S.-FNRS N°1.5.056.10 (Fund for Scientific Research FNRS, Belgium), and ARC 12/17/01 REST (Université de Liège). WF was supported by the Ghent University Multidisciplinary Research Partnership “The integrative neuroscience of behavioral control,” and a GOA grant of Ghent University. SM and WF were supported by PAI-IUAP P7/11 (Belgian Science Policy).

REFERENCES

Abrahamse, E., van Dijck, J.-P., Majerus, S., and Fias, W. (2014). Finding the answer in space: the mental whiteboard hypothesis on serial order in working memory. Front. Hum. Neurosci. 8:932. doi: 10.3389/fnhum.2014.00932

Awh, E., and Jonides, J. (2001). Overlapping mechanisms of attention and spatial working memory. Trends Cogn. Sci. 5, 119–126. doi: 10.1016/S1364-6613(00)01593-X

Baddeley, A. (2000). The episodic buffer: a new component of working memory? Trends Cogn. Sci. 4, 417–423. doi: 10.1016/S1364-6613(00)01538-2

Baddeley, A. D., and Hitch, G. (1974). “Recent advances in learning and motivation,” in Working Memory, Vol. 8, ed G. A. Bower (New York, NY: Academic Press), 647–667.

Camos, V., and Barrouillet, P. (2014). Attentional and non-attentional systems in the maintenance of verbal information in working memory: the executive and phonological loops. Front. Hum. Neurosci. 8:900. doi: 10.3389/fnhum.2014.00900

Cowan, N. (1999). “An embedded-processes model of working memory,” in Models of Working Memory: Mechanisms of Active Maintenance and Executive Control, eds A. Miyake and P. Shah (New York, NY: Cambridge University Press), 62–101.

De Fockert, J. W., and Leiser, J. (2014). Better target detection in the presence of collinear flankers under high working memory load. Front. Hum. Neurosci. 8:821. doi: 10.3389/fnhum.2014.00821

Ginsburg, V., and Gevers, W. (2015). Spatial coding of ordinal information in short- and long-term memory. Front. Hum. Neurosci. 9:8. doi: 10.3389/fnhum.2015.00008

Holmes, J., Hilton, K. A., Place, M., Alloway, T. P., Elliott, J. G., and Gathercole, S. E. (2014). Children with low working memory and children with ADHD: same or different? Front. Hum. Neurosci. 8:976. doi: 10.3389/fnhum.2014.00976

Kiyonaga, A., and Egner, T. (2013). Working memory as internal attention: toward an integrative account of internal and external selection processes. Psychon. Bull. Rev. 20, 228–242. doi: 10.3758/s13423-012-0359-y

Kiyonaga, A., and Egner, T. (2014). Resource-sharing between internal maintenance and external selection modulates attentional capture by working memory content. Front. Hum. Neurosci. 8:670. doi: 10.3389/fnhum.2014.00670

Meghanathan, R. N., van Leeuwen, C., and Nikolaev, A. R. (2015). Fixation duration surpasses pupil size as a measure of memory load in free viewing. Front. Hum. Neurosci. 8:1063. doi: 10.3389/fnhum.2014.01063

Oberauer, K. (2009). “Design for a working memory,” in Psychology of Learning and Motivation: Advances in Research and Theory, Vol. 51, ed B. H. Ross (San Diego, CA: Elsevier Academic Press Inc), 45–100.

Pedale, T., and Santangelo, V. (2015). Perceptual salience affects the contents of working memory during free-recollection of objects from natural scenes. Front. Hum. Neurosci. 9:60. doi: 10.3389/fnhum.2015.00060

Quak, M., London, R. E., and Talsma, D. (2015). A multisensory perspective of working memory. Front. Hum. Neurosci. 9:197. doi: 10.3389/fnhum.2015.00197

Roome, H. E., Towse, J. N., and Jarrold, C. (2014). How do selective attentional processes contribute to maintenance and recall in children's working memory capacity? Front. Hum. Neurosci. 8:1011. doi: 10.3389/fnhum.2014.01011

Tanaka, T., Sugimoto, M., Tanida, Y., and Saito, S. (2014). The influences of working memory representations on long-range regression in text reading: an eye-tracking study. Front. Hum. Neurosci. 8:765. doi: 10.3389/fnhum.2014.00765

Vandierendonck, A. (2014). Symbiosis of executive and selective attention in working memory. Front. Hum. Neurosci. 8:588. doi: 10.3389/fnhum.2014.00588

van Moorselaar, D., Gunseli, E., Theeuwes, J., and Olivers, C. N. L. (2015). The time course of protecting a visual memory representation from perceptual interference. Front. Hum. Neurosci. 8:1053. doi: 10.3389/fnhum.2014.01053

Vergauwe, E., and Cowan, N. (2014). A common short-term memory retrieval rate may describe many cognitive procedures. Front. Hum. Neurosci. 8:126. doi: 10.3389/fnhum.2014.00126

Wong, L. M., Tassone, F., Rivera, S. M., and Simon, T. J. (2015). Temporal dynamics of attentional selection in adult male carriers of the fragile X premutation allele and adult controls. Front. Hum. Neurosci. 9:37. doi: 10.3389/fnhum.2015.00037

Zokaei, N., Ning, S., Manohar, S., Feredoes, E., and Husain, M. (2014). Flexibility of representational states in working memory. Front. Hum. Neurosci. 8:853. doi: 10.3389/fnhum.2014.0085

Conflict of Interest Statement: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Abrahamse, Majerus, Fias and van Dijck. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

REVIEW ARTICLE
published: 07 November 2014
doi: 10.3389/fnhum.2014.00900

Attentional and non-attentional systems in the maintenance of verbal information in working memory: the executive and phonological loops

Valérie Camos[1]* and Pierre Barrouillet[2]

1Laboratory of Cognitive Development, Fribourg Center for Cognition, Département de Psychologie, Université de Fribourg, Fribourg, Switzerland

2Developmental Cognitive Psychology, Faculté de Psychologie et des Sciences l’Education, Université de Genève, Genève, Switzerland

* Correspondence: Valérie Camos, Laboratory of Cognitive Development, Fribourg Center for Cognition, Département de Psychologie, Université de Fribourg, Rue de Faucigny 2, CH-1700 Fribourg, Switzerland e-mail: valerie.camos@unifr.ch

Edited by:
Steve Majerus, Université de Liège, Belgium

Reviewed by:
Nelson Cowan, University of Missouri, USA
Arnaud Szmalec, Université Catholique de Louvain, Belgium

Working memory is the structure devoted to the maintenance of information at short term during concurrent processing activities. In this respect, the question regarding the nature of the mechanisms and systems fulfilling this maintenance function is of particular importance and has received various responses in the recent past. In the time-based resource-sharing (TBRS) model, we suggest that only two systems sustain the maintenance of information at the short term, counteracting the deleterious effect of temporal decay and interference. A non-attentional mechanism of verbal rehearsal, similar to the one described by Baddeley in the phonological loop model, uses language processes to reactivate phonological memory traces. Besides this domain-specific mechanism, an executive loop allows the reconstruction of memory traces through an attention-based mechanism of refreshing. The present paper reviews evidence of the involvement of these two independent systems in the maintenance of verbal memory items.

Keywords: working memory, attention, phonological loop, executive loop, rehearsal, refreshing

INTRODUCTION

Working memory is a system dedicated to the storage and maintenance of information. This is a central system that allows us to interpret and comprehend our environment and ourselves by constructing transient representations. These representations are built using our internal and external states, and they can be conceived as mental models, as in Johnson-Laird’s (1983) theory. The role of working memory is to maintain these representations in face of decay and interference in order to avoid their loss, as well as to transform them for actions in accordance with our goals.

In the past decade, we have proposed a new model of working memory, named the Time-Based Resource-Sharing (TBRS) model (Barrouillet et al., 2004; see Barrouillet and Camos, 2012, 2015, for reviews). The TBRS model aims at accounting for complex span-related phenomena and more generally for working memory structure and functioning. Specifically, the TBRS model enlightens the role of attention in working memory by introducing the idea that attention is involved in the maintenance as well as in the processing of information. Moreover, our model allows an understanding of how attention is shared in a time-based manner between processing and maintenance, which are the two functions of working memory. Nevertheless, the TBRS model does not imply that attention is always involved in the maintenance activity of information in working memory. The aim of this review is to show that attentional and non-attentional systems subserve storage in working memory. Interestingly, these two types of systems are implicated in the maintenance of verbal information. After presenting these two maintenance systems within the framework of the TBRS model, we review evidence of the independence of these two systems, of their joint use to store verbal information, of the ability for adults to make adaptive choice between these systems, and of the impact that using one or the other system has on recall performance. We end our review with the presentation of some brain imaging data showing that different brain networks are distinguishable and would sustain each of these two maintenance systems.

THE MAINTENANCE OF VERBAL INFORMATION IN THE TBRS MODEL

According to the TBRS model, two systems can be involved in the maintenance of verbal information (Figure 1). The first system we call the executive loop includes an episodic buffer and a procedural system. Similar to the episodic buffer in the multi-component model (Baddeley, 2000), we assume that working memory representations are stored in a buffer in which they suffer from temporal decay and interference, and consequently must be reconstructed or reactivated to permit any processing. Like in Adaptive Control of Thought—Rational (ACT-R) model (Anderson, 1993, 2007), the procedural system reads this representation and, depending on the currently active goal, maintains or updates its content by firing the appropriate production rule. The current goal can also command a switch to another of the working memory representations held in the episodic buffer. When the representations need to be maintained, their reconstruction is achieved through attentional refreshing which requires attention. This idea of an attention-based mechanism of maintenance is inspired by Cowan’s suggestion that memory items can be reactivated by a scanning process (Cowan, 1992; Vergauwe and Cowan, in press) or by the recirculation of items through the focus of attention (Cowan, 1995). However, although the TBRS model focuses on the role of attentional processes in working memory, the maintenance of verbal information could also be achieved through an articulatory rehearsal process as Baddeley described in his model of the phonological loop (Baddeley, 1986; Baddeley and Logie, 1999). Thus, in the TBRS model, the executive loop and the phonological loop are the two loops in charge of verbal maintenance. The functioning of the phonological loop had been extensively studied in the past 50 years (e.g., Baddeley, 2007). We now briefly outline the well-known effects emerging from its functioning, which will be discussed through this review as well. The phonological loop is involved in the storage of verbal information in a phonological format. As a consequence, the storage of phonologically similar words leads to more confusion than of dissimilar words. This effect is named the phonological similarity effect. To actively maintain memory traces in the phonological loop, memory items are subvocally rehearsed through processes shared by language production. The existence of such a verbal maintenance mechanism had two consequences. First, a concurrent articulation using similar language processes could block or at least impede the subvocal rehearsal of memory items and impair working memory recall. Second, lists of short words are better recalled than lists of long words, because in a fixed duration, the former take a shorter time to articulate and would consequently benefit from more rehearsals than the latter, increasing the probability to be recalled. This effect is named the word length effect.

FIGURE 1. The Time-Based Resource-Sharing model architecture (Figure 6.1 in Barrouillet and Camos, 2015, p. 118).

Besides the phonological loop, the TBRS model describes another loop, the executive loop. Its functioning is based on four main proposals. First, the model assumes that both the processing and the maintenance of information within the executive loop rely on the same limited resource, which is attention. Because attention is a limited resource, it has to be shared between processing and storage. The second assumption is that many of the elementary cognitive steps involved in both processing and maintenance can only take place one at a time. When the executive loop is occupied by some processing episode, it is not available for the maintenance of memory items. The same idea is captured by the concept of the central bottleneck, developed by Pashler (1998), according to which central processes like response selection can only take place one at a time in such a way that the subsequent processes are postponed. Another way to express the same idea is to assume that the size of the focus of attention is limited to only one element with the consequence that attention could only select one item of knowledge at a time for the next cognitive operation (McElree and Dosher, 1989; Garavan, 1998; McElree, 1998, 2001; Oberauer, 2002, 2005). Thus, processing and storage activities could not take place in parallel, but alternate in occupying the central bottleneck. However, according to the third assumption of the TBRS model, as soon as attention is switched away, or in other words as soon as the focus of attention leaves the memory traces, their activation suffers from a time-related decay. Thus, in working memory span tasks, the memory traces of the to-be-maintained items decline when attention is occupied by the processing of distractors. Redirecting the focus of attention on the memory traces results in their refreshment before complete disappearance. Finally, the fourth assumption is that, due to the limitation of attention to only one element at a time and the time-related decay of memory traces outside the focus of attention, the sharing of attention is achieved through a rapid and incessant process of switching of this focus from processing to maintenance. As it can be suspected, most tasks typically do not induce a continuous capture of attention, and thus attention can be diverted from time to time, even for short periods of time, towards other thoughts and brought back to the current activity. This continuous switching of attention must be considered as a basic mental process underlying our phenomenal experience of thinking, which permits the coherence and cohesion of our mental life beyond the succession of changing thoughts. This conception in turn delineates a conception of cognitive load (CL). According to the TBRS framework, CL is the proportion of time during which tasks capture attention, thus impeding maintenance activities that require the executive loop. Within this framework, the CL is defined as: CL = Duration of attentional capture/Total time allowed to perform the task. It is important to note that even simple activities such as reading digits or response selection tasks can efficiently block attention for prolonged periods of time if they are performed under time constraints.

The TBRS model and more specifically its conception of CL were verified in several studies (e.g., Barrouillet et al., 2004, 2007). For example, Barrouillet et al. (2011) asked adult participants to perform complex span tasks in which they had to maintain series of letters, digits or words of ascending length while performing various concurrent tasks. These concurrent tasks were a location or parity judgment tasks, a stroop task (classic and numerical versions), or a memory updating task (the 2-back task). Although these tasks differed in their nature, they are all well-known for capturing attention because they involve some response selection, retrieval, updating, or inhibition of information. All of these processes are considered executive functions and thus are highly attention-demanding. The mean response time when the answer was correct was used to estimate the duration of attentional capture of these tasks, and to compute their respective CL. As depicted in Figure 2, the number of memory items that can be maintained (expressed in mean span) while performing a concurrent task is a direct function of the CL of this task. More specifically, the mean span decreases linearly with the increase in CL, independently of the nature of the tasks and of the attention-demanding processes they involve. Several other studies replicated this finding with different types of memoranda (verbal, visual, or spatial), with various distracting activities, and in adults as well as in children and adolescents (Barrouillet and Camos, 2015, for a review). Such a finding strengthens the assumption made by the TBRS model that memory items can be maintained by the executive loop through attentional refreshing, which competes for attention with other attention-demanding processes required by working memory span tasks.

FIGURE 2. Mean span (i.e., the mean number of maintained items) as a function of the cognitive load (Barrouillet et al., 2011).

Although the effect of CL on recall performance appears even under concurrent articulation, one can wonder if the attentional refreshing is really different from the subvocal rehearsal described in Baddeley’s (1986) multi-component model. Using a reading span task in which participants had to maintain words while reading sentences presented in successive segments, Hudjetz and Oberauer (2007) manipulated the reading instructions in such a way that subvocal rehearsal was more or less impeded. For this purpose, participants had to read the sentences either continuously or at their own pace, the former condition strongly impeding the use of subvocal rehearsal. Moreover, the CL of the reading task was increased by increasing the pace of presentation of the segments to be read, with a fast pace shortening the time to perform the task and reducing the availability of attention for maintenance activities relative to a slow pace. If the maintenance of words relies only on attentional refreshing, recall performance should not be affected by the type of reading (i.e., continuously or at one’s own pace). Instead, only the pace of presentation would affect recall performance, with poorer recall at a fast than a slow pace. On the contrary, if the maintenance is achieved through subvocal rehearsal, the pace of presentation and the type of reading should interact. That is, when reading at their own pace, a slow pace of presentation would give participants more time to rehearse words. Conversely, the continuous reading would make rehearsal more difficult and the beneficial effect of the longer presentation at the slow pace would disappear. Although the results revealed a significant effect of both factors, the lack of interaction between pace of presentation and reading instructions contradicts the idea that maintenance relies exclusively on subvocal rehearsal. These results showed that a maintenance mechanism different from subvocal rehearsal is implicated in the maintenance of memoranda in working memory. Moreover, they gave the first support to the dissociation between attentional refreshing and subvocal rehearsal and, by extension, between the phonological loop and the executive loop.

To summarize, the TBRS model suggests the existence of two distinct systems of maintenance in working memory. One, the executive loop, is a domain-general system in which the attentional refreshing maintains any type of information (e.g., verbal, visual, multimodal) in an episodic buffer. The other, the phonological loop, is a domain-specific system dedicated to the maintenance of verbal information under phonological code and that does not require attention, at least after a brief initial setup period (Naveh-Benjamin and Jonides, 1984). Thus, these two systems differ on the type of representations they process as well as on their maintenance mechanism, and as a consequence on the implication of attention. These differences lead to several predictions. The existence of two systems implies they can be used independently from each other to maintain information, but also that they can be used jointly. Moreover, it can be suggested that participants could favor one or the other system of maintenance depending on some constraints or instructions. Finally, the well-known effects of phonological similarity and word length, specific to the phonological nature of the representations, should emerge when the phonological loop is involved in maintenance, but should not be affected by any variation in the implication of the executive loop. In the following, we present some experimental evidence supporting these different predictions.

THE INDEPENDENCE OF ATTENTIONAL AND NON-ATTENTIONAL MAINTENANCE SYSTEMS

In a first study including four different experiments, Camos et al. (2009) tested the independence of the two loops hypothesized by the TBRS model. Using complex span tasks, the authors varied the opportunity of using one of the two loops while the other was impeded. For example, the phonological loop was impeded by a concurrent articulation whereas the availability of the executive loop was varied by introducing a more or less attention-demanding distracting task. As expected, the manipulation of the availability of one system while the other was impeded resulted in a reduction of recall performance, although participants were able to recall a decent number of verbal memory items. These findings confirmed the idea that both the executive and the phonological loops are able to maintain verbal information in working memory.

In further experiments, the interplay of these two systems was assessed by orthogonally manipulating the availability of attention and of articulatory processes. For example in Camos et al.’s (2009) Experiment 4, participants performed a distracting task that varied in attentional demand either silently or aloud. In a complex span task, each memory item was followed by a series of six digits successively displayed on screen. Participants had either to press the space bar when “5” appeared on screen (i.e., low-demanding detection task) or to verify if the 3rd and the 6th digits were the sum of the two previously presented digits (i.e., high-demanding verification task). These two experiments replicated the previous findings: impeding one or the other loop led to a reduction in recall performance. More interestingly, the effect of concurrent articulation was additive to the effect of the attentional demand resulting from the processing component of the complex span tasks (Figure 3). These results suggest that the phonological loop and the executive loop are two independent mechanisms involved in the maintenance of verbal information.

FIGURE 3. Mean span according to the task (detection vs. verification of additions), the type of reading (silent vs. aloud) and the pace in Camos et al.’s (2009) Experiment 4.

Camos et al.’s (2009) results are in line with the architecture proposed by the TBRS model with two distinct loops involved in the maintenance of verbal information. The existence of a second system, over and beyond the phonological loop and its rehearsal mechanism was initially mentioned by Baddeley and Hitch (1974), although never formally implemented in Baddeley’s multi-component model. This idea regularly reappeared within the multi-component model to explain how participants were able to maintain a substantial number of verbal items even under concurrent articulation (e.g., Vallar and Baddeley, 1982; Salamé and Baddeley, 1986; Hitch et al., 1989, 1993; Towse et al., 1998). Although the idea of different mechanisms intervening in the maintenance of verbal information could be found before in the literature, Camos et al.’s (2009) study provided the first empirical evidence of the independence of two maintenance systems, i.e., the executive and phonological loops.

THE JOINT USE OF THE ATTENTIONAL AND NON-ATTENTIONAL SYSTEMS

The existence of the two independent systems logically implies that they can be jointly used to maintain verbal items. Camos et al. (2009) provided some evidence of this joint use, as the introduction of concurrent articulation or concurrent attentional demand have an additive effect on recall performance, thereby supporting the idea that the two loops can act in conjunction. Another way to show the joint use of the phonological and executive loops was to examine the effects that the maintenance of verbal memory items has on concurrent processing activities.

Within the executive loop, when processing and storage are performed concurrently, processing episodes are postponed by maintenance activities in the same way that maintenance activities are postponed by processing. We have seen that the postponement of maintenance activities results in memory loss and thus reduced recall performance. However, the postponement of processing by maintenance activities should have a negligible effect on processing accuracy as long as the stimuli to be processed remain available in the environment. For example, this is the case in complex span tasks in which distractors remain on screen until the participant gives her response. Conversely, this postponement should appear in response times. Because attentional maintenance in the executive loop proceeds in a cumulative fashion, starting from the first list item and proceeding in forward order until the end (McCabe, 2008), this postponement should linearly increase with the number of memory items to be maintained. By contrast, when memory items are maintained within the phonological loop, such a postponement should not be observed.

To test this new set of predictions, Vergauwe et al. (2014) developed a new paradigm. They used a Brown-Peterson paradigm in which participants had to maintain a list of items for further recall and to perform an intervening task over a fixed retention interval prior to recall. However, participants were instructed to perform this intervening activity in such a way that, while trying to achieve the best performance in this task, they should not forget the memoranda. For example, in one experiment, participants were presented with series of 0 to 7 letters to be remembered, and asked during a 12-s retention interval to judge the parity of as many numbers as they can by pressing keys, each key press displaying a new number on screen. We assumed that to minimize the risk of forgetting and achieve a perfect recall of the memoranda, participants should refresh all of them before each processing episode.

In this experiment, to be sure that verbal information was only maintained through the executive loop, participants repeated “badibu” during the retention interval while completing the parity judgment task by pressing keys. Maintaining an increasing amount of verbal memoranda under concurrent articulation slowed the responses in the concurrent task (Figure 4, Experiment 3). The fact that refreshing memory items postpones concurrent processing activities indicates that the two activities compete for a general-purpose system. Contrary to attentional refreshing, the maintenance of verbal items through subvocal rehearsal requires very little attentional demand. Releasing the constraint to repeat “badibu” during the retention interval should allow participants to maintain as many verbal items as they can through subvocal rehearsal. As a consequence, attention would be available for performing the concurrent task without any postponement. This was the aim of Vergauwe et al.’s (2014) Experiment 4, in which participants had to maintain series of 0 to 7 letters while performing the parity judgment task, but contrary to the previous experiment, without any concurrent articulation. As illustrated in Figure 4, this experiment led to a very different pattern of results compared with Vergauwe et al.’s (2014) Experiment 3. Whereas the processing times steadily increased with the memory load under concurrent articulation, no such increase was observed in Experiment 4 till a load of four letters. As predicted, the slope remained nearly flat from 0 to 4 letters. This absence of postponement also contrasted with the increase in response times observed when memory load exceeded four letters. From 4 to 6 letters to be maintained, the slope was akin to the slope observed under concurrent articulation. Moreover, this slope gives an estimate of the speed of the refreshing process (around 50 ms per item). Such a value fits well with the neurophysiological explanation of short-term memory limitation by Lisman and Idiart (1995) in which each item would be stored in gamma oscillation subcycles (about 40 Hz) within a theta neuron network oscillation (see Luck and Vogel, 1998 for a similar account, and Vergauwe and Cowan, 2014, for discussion). These findings support the proposal made by the TBRS model of two distinct systems of verbal maintenance, with a verbal-specific system able to maintain up to four letters without any interference with a concurrent attention-demanding task, and an attentional system that competes with a concurrent task.

FIGURE 4. Mean processing times in ms for letters as a function of memory load depending on the presence (Letters + Parity + AS, Experiment 3) or absence (Letters + Parity, Experiment 4) of articulatory suppression (AS) in Vergauwe et al. (2014).

Apart from confirming the sequential functioning of working memory postulated by the TBRS model, these results shed light on the structure of working memory, confirming that there is a domain-general attentional system that is able to maintain verbal information and another non-attentional system that corresponds to the phonological loop in Baddeley’s (1986) theory.

THE ADAPTIVE CHOICE BETWEEN AN ATTENTIONAL AND A NON-ATTENTIONAL SYSTEMS

Another logical consequence of the existence of two independent systems is that adults should be able to favor one of them according to the constraints of the task or following instructions. To test the hypothesis that young adults can chose adaptively between the phonological loop and the executive loop, Camos et al. (2011) used a complex span paradigm in which the processing component was either a choice reaction time (CRT) task or a less demanding simple reaction time (SRT) task, and the memoranda were lists of six phonologically similar or dissimilar words. The rationale was that when the concurrent task is less demanding (i.e., SRT), attention would be available for maintenance through refreshing. Accordingly, participants should favor the executive loop because it likely enables the maintenance of non-phonological representations of the memory items. Such a mode of maintenance would reduce the confusability of the representations of the memoranda when they are phonologically similar words. By contrast, under a high attentional demand, participants should revert to the phonological loop, which requires less attention (Naveh-Benjamin and Jonides, 1984). Because subvocal rehearsal relies on the maintenance of phonological representations, recall performance in this case should suffer from the phonological similarity of some of the lists.

As depicted in Figure 5 for the “no instruction” condition, when participants performed the CRT task, their recall was better for lists of phonologically dissimilar than similar words, replicating the phonological similarity effect. However, when participants performed the SRT task as a concurrent task, the phonological similarity effect disappeared. As predicted, the emergence of the phonological similarity effect depended on the attentional demand of the concurrent activity. The attention-demanding CRT task led participants to rely on subvocal rehearsal for maintenance, and the phonological similarity effect occurred. By contrast, the non-demanding SRT task allowed for the use of the executive loop, and thus the phonological characteristics of the memoranda did not affect recall. To verify that the change of pattern concerning the phonological similarity effect was due to a change in maintenance loop, two other groups of participants were instructed to perform the same complex span task while using either subvocal rehearsal or attentional refreshing to maintain series of words. As depicted in Figure 5 for rehearsal and refreshing instructions, whereas an increase in the attentional demand of the concurrent task led to reduced recall performance in both experiments, the occurrence of the phonological similarity effect depended on the instructions. Whatever the amount of attention available, the phonological similarity effect appeared when participants were instructed to use rehearsal. On the contrary, under refreshing instruction, the phonological characteristics of the lists to be maintained never affected recall performance. Thus, Camos et al. (2011) showed that the use of the two maintenance loops is adaptive and flexible, young adults being able to favor one of the loops according to its relative effectiveness or instructions. Moreover, these findings suggest that the phonological loop maintains verbal information under phonological representations as indexed by the emergence of a phonological similarity effect, and the executive loop acts probably on richer memory traces involving a variety of features, making recall performance immune to the phonological similarity of the memoranda in the lists.

FIGURE 5. Mean percentage of correct recall as a function of the phonological similarity of the memory words in a list (similar vs. dissimilar), the type of concurrent processing (SRT vs. CRT) and the maintenance mechanism participants were instructed to use in Camos et al. (2011).

THE ATTENTIONAL AND NON-ATTENTIONAL SYSTEMS HAVE DIFFERENT IMPACT ON RECALL

The aforementioned studies presented have made it clear that two systems of maintenance for verbal information exist, and have stressed their independence. Moreover, the TBRS model predicts that the use of one or the other of the two systems should have different effects on recall performance. Whereas any increase in CL of the processing component should impede the maintenance activities of the executive loop and thus reduce recall performance, the use of phonological loop should make recall susceptible to well-known effects specific to its functioning, i.e., the phonological similarity effect and the word length effect.

To test these hypotheses, two studies orthogonally manipulated the attentional demand of the concurrent task and the availability of subvocal rehearsal (Camos et al., 2013; Mora and Camos, 2013). For this purpose, four complex span tasks were compared in which participants maintained series of either phonologically similar or dissimilar words, or series of short or long words (Figure 6). To vary the concurrent attentional demand, participants either performed a concurrent location judgment task or had nothing to do. The availability of subvocal rehearsal was varied either by allowing participants to remain silent through the concurrent task or by asking them to repeat the word “oui” (“yes” in French) at the rhythm of beeps heard in headphones. Both studies confirmed the predictions of the TBRS model.

FIGURE 6. The four conditions used in Camos et al. (2013) and in Mora and Camos (2013) in which each memory item was followed by an unfilled delay, by the repetitive utterance of “oui”, of a location judgment task, or the combination of the same location judgment task with the utterance of “oui”.

The addition of a concurrent task or of a concurrent articulation resulted in reduced recall performance in both studies (Figures 7, 8). These studies also replicated the well-known phonological similarity effect with dissimilar word lists being better recalled than the similar word lists (Figure 7), as well as the word length effect with better recall performance for short than long words (Figure 8). However, these effects did not interact with the presence vs. absence of an attentional-demanding concurrent task, but disappeared under concurrent articulation. A similar disappearance of these effects was already reported in simple span tasks when the memory items were visually presented, and when a concurrent articulation occurred during the encoding of the memoranda (e.g., Coltheart, 1999; Fallon et al., 1999; Baddeley and Larsen, 2007). The present studies showed that blocking articulatory processes during maintenance could lead to the same disappearance of the phonological similarity and the word length effects as impeding the encoding processes. Thus, the impact of the phonological characteristics of the memory items depended on the use of the verbal-specific system, whereas recall performance was immune from these effects under the use of the executive loop.

FIGURE 7. Variation of the phonological similarity effect as a function of the four conditions used by Camos et al. (2013).

FIGURE 8. Variation of the word length effect as a function of the four conditions used by Mora and Camos (2013).

DISTINCT BRAIN NETWORKS FOR THE ATTENTIONAL AND NON-ATTENTIONAL MAINTENANCE SYSTEMS

A vast amount of behavioral evidence supporting the distinction between a domain-general attentional system and a specialized non-attentional system was collected within the framework of the TBRS model. The literature brings further support to this distinction when examining the neural implementation of these two systems, and more specifically of their respective maintenance mechanisms, i.e., the attentional refreshing and the subvocal rehearsal.

Gruber (2001) observed that the brain network sustaining phonological storage is not uniquely localized, but depends on the possibility to rely or not on subvocal rehearsal. When a silent articulatory suppression prevents rehearsal, enhanced activity is observed in anterior prefrontal and inferior parietal brain areas. By contrast, when such suppression is relaxed, working memory performance activates Broca’s area and cortex along the left intraparietal sulcus. In other words, the non-articulatory maintenance of verbal information produced activation in a different network compared to subvocal rehearsal (see also Gruber and von Cramon, 2003, and Trost and Gruber, 2012, for similar findings). While these findings support the existence of two distinct systems of verbal maintenance, Raye et al. (2002, 2007) provided direct evidence of the dissociation between subvocal rehearsal and attentional refreshing, showing that they are neurally distinguishable processes. These authors showed an increased activation of the dorsolateral prefrontal cortex [DLPFC, Brodmann’s area (BA) 9] when young adults were instructed to refresh words (i.e., they were instructed to think briefly to the words) compared to repeating or reading them silently, or to simply press a button. Moreover, the results further distinguished attentional refreshing from subvocal rehearsal in revealing that Broca’s area (ventrolateral prefrontal cortex, VLPFC, BA 44) was selectively activated in a rehearsal condition (Raye et al., 2007). A similar finding was reported by Smith and Jonides (1999), suggesting that the use of subvocal rehearsal strategy relies on the activation of the VLPFC. Raye et al. (2007) concluded that the VLPFC reflects subvocal rehearsal of phonological information, while the DLPFC is assumed to reflect attention to various types of information (e.g., its activation did not differ between verbal and non-verbal information, Johnson et al., 2005). These neuroimagery data fit nicely with the TBRS model and the existence of two distinct systems. Indeed, whereas the Broca’s area is a specialized structure dedicated to language, the DLPFC is more broadly involved in executive control (D’Esposito et al., 1995). This neurological distinction between a specialized peripheral structure and an executive central structure echoes the differentiation introduced by the TBRS model between the phonological loop and the executive loop.

DIFFERENCES AND COMMONALTIES WITH OTHER WORKING MEMORY MODELS

While the TBRS model proposes the existence of two distinct systems involved in the maintenance of verbal information at short term, in this section we examine other theoretical frameworks that either suggest similar systems or conversely advance alternative proposals. The model presenting the most obvious similarity with the TBRS model is the last version of the multi-component model (Baddeley, 2012). The multi-component model of working memory distinguishes a central system from a domain-specific system for verbal information (Baddeley, 1986, 2000; Baddeley and Logie, 1999). For the domain-specific system, both models propose that verbal information is maintained in a phonological store by verbal rehearsal through a phonological loop. Moreover, the executive loop described by the TBRS model includes an episodic buffer in which cross-domain representations are constructed and maintained as suggested by Baddeley et al. (2010). Besides these similarities, the TBRS theory departs from the multi-component approach primarily in the structure and functioning of the central component. Whereas its structure and functioning of the central executive remains underspecified in Baddeley’s theory, we have suggested that it can be conceived as an executive loop connecting the episodic buffer with a procedural system. As a consequence and contrary to Baddeley’s modal model, the central system in the TBRS model is in charge of both processing and storage activities that compete for a common supply. Nevertheless, as noted above, in several works framed within the multicomponent model, Baddeley and collaborators have suggested another system that would supplement the phonological loop when overloaded (e.g., Baddeley and Hitch, 1974; Salamé and Baddeley, 1986). However, the TBRS model specifies the main constraints of this central system that are due to the sequential functioning of the executive loop and the ephemeral nature of working memory representations. As explained above, these two characteristics are at the root of the relationship between storage capacity and concurrent attentional demand, as seen in Figure 2.

The existence of such a conflict between processing and storage is also predicted by Cowan’s (1999, 2005) embedded-processes model. Cowan assumes that working memory can be conceived as the temporarily activated portion of long-term memory, with a subset of this activated memory that corresponds to the focus of attention. The focus of attention is limited to three to five chunks of information (see Luck and Vogel, 1997, for similar estimate in visuospatial domain), and is controlled by automatic orienting responses to changes in the environment as well as voluntary effort directed by the central executive towards current goals. A consequence of this working memory structure is that the focus of attention is involved in both the retention of information and in processing activities. Congruent with our distinction between an executive and phonological loops, Cowan also stresses that the capacity of the focus of attention appears clearly when verbal rehearsal is prevented, suggesting that maintenance of verbal information through attentional focusing and verbal rehearsal must be distinguished.

Other theories assume that working memory is the activated part of long-term memory, such as the concentric model proposed by Oberauer (2002) who distinguishes, within this activated part, a region of direct access containing about four items. Among them, one item is selected by the focus of attention for processing (see also McElree, 1998, 2001). Within the TBRS model, it can be conceived that the single item within the focus of attention would correspond to the working memory representation currently processed by the executive loop, whereas the region of direct access corresponds to the representations held in the episodic buffer. Consequently, the diverging conceptions about the size of the focus of attention (i.e., either one or four items according to Oberauer or Cowan, respectively) can be reconciled by considering that the sequential functioning of the executive loop requires that several representations are almost simultaneously present to the mind (i.e., in the episodic buffer), while only one representation is currently refreshed (cf. Figure 1). However, neither Cowan’s nor Oberauer’s theories stress the importance of temporal factors as the TBRS model does.

All of the previously highlighted theories assume that working memory is concerned with those processes that require attention (see also Lovett et al., 1999). Engle’s model made this point very clear in distinguishing working memory from short-term memory by the implication of controlled attention in working memory tasks (Engle et al., 1999; Engle and Kane, 2004). This model suggests that working memory is mainly involved on those activities that need executive control to maintain goal-relevant information under conditions of interference or competition. The TBRS model departs partially from this view, because it does not limit the attention involved in working memory to the controlled or executive attention. Of course, working memory is important to maintain information in an active state and to solve the conflicts resulting from interference and activation of prepotent but inappropriate responses as Engle and Kane argue (e.g., Rosen and Engle, 1998; Kane and Engle, 2003; Unsworth et al., 2004; Bunting, 2006). However, attention is also important for simple activations of knowledge from long-term memory, as suggested by theories like ACT-R (Anderson, 1993). As reported previously, even the identification of material as simple as digits or of a location can capture attention for a sizable amount of time that it leads to a reduction of recall performance.

More recently, Unsworth and Engle (2007a,b) have also suggested that working memory comprises two functionally different components, although rather different from the two loops described in the TBRS model. A first component, referred to as primary memory, actively maintains information over the short term. According to Unsworth et al. (2010), this component is conceptually similar to Cowan’s focus of attention. The second component, referred to as secondary memory, is needed to retrieve information that can not be maintained in primary memory when its capacity is exhausted or its content is displaced by irrelevant distracters. In this case, retrieval from secondary memory would rely on a cue-dependent mechanism. In the case of complex span tasks as used in the experiments reported above, memory items would be first stored in primary memory, but quickly displaced into secondary memory by the processing activity. At recall, the majority of the items would have to be retrieved from secondary memory through strategic search. By contrast, in simple span tasks, items could be held in primary memory, at least when list lengths are small (i.e., up to four items). With longer lists, items would be initially held in primary memory but some of them would be displaced into secondary memory.

Although Unsworth and Engle (2007a) did not explicitly claim it, it could be imagined that what they call primary memory corresponds to the phonological loop because they focused on verbal memory. Indeed, Vergauwe et al. (2014) observed that this phonological loop can hold up to four items, and when this capacity is exhausted, the executive loop works as a back-up system to refresh working memory representations. This option was evoked by Jarrold et al. (2010) who suggested that primary memory capacity could be supported by rehearsal. However, our review has made it clear that the phonological loop can maintain verbal items without interfering with attention demanding activities, demonstrating that it is distinct from the focus of attention. As a consequence, the phonological loop can not stand as the primary memory described by Unsworth and Engle (2007a). Another possibility would be to consider that primary memory consists of the assembly of the phonological and the executive loops, with secondary memory corresponding to long-term memory. However, we have seen that when participants were free to use verbal rehearsal for maintaining letters, (i.e., when both the phonological and the executive loops could be used for maintenance purpose), they were able to maintain up to six letters while performing a distracting task, which is beyond the expected primary memory capacity.

Finally, it is noticeable that the TBRS model is inspired by the ACT-R architecture (Anderson, 1993, 2007; Anderson and Lebière, 1998) and its procedural system. Within ACT-R, an imaginal module is used for storing intermediate information necessary for performing tasks and is comparable with the focus of attention in Cowan’s (2005) theory or with Baddeley’s episodic buffer. This imaginal module or problem state resource would be limited to only one coherent chunk of information with three or four slots and would create interference when it is requested by more than one task (Borst et al., 2010). Though our proposals are akin to these conceptions, some differences remain. Indeed, Borst et al. (2010) specify that not all tasks require the use of the imaginal module, for example when no intermediary results need to be stored or when all the necessary information is present in the world. Our conception differs because we assume that any process requiring an executive function involves the executive loop. Indeed, our results make clear that a simple response selection associated with the location of a stimulus on screen, or the direct retrieval of parity information that does not require any intermediary result, compete with concurrent maintenance of information within the episodic buffer. This suggests a more general and central representational role for the episodic buffer and the executive loop in our model than for the imaginal module postulated by ACT-R.

CONCLUSION

In summary, the two maintenance loops described in the TBRS model, the executive loop and the phonological loop, are two distinct and independent systems. Because of this independence, they can be jointly used to maintain verbal information. Moreover, adults can chose to favor the use of one or the other system, either intentionally when instructed to do so or adaptively because the use of the executive loop could reduce the confusion for phonologically similar material. However, the use of one or the other loop is not without consequence. Each of these systems of maintenance induces a different pattern of recall performance. The use of the phonological loop makes recall sensitive to the phonological characteristics of the material to be maintained, whereas the phonological nature of the memory items does not affect recall performance under the use of the executive loop. This does not imply that the executive loop is a “better” system of maintenance that should always be favored for verbal information. Because attentional refreshing is more attention-demanding than subvocal rehearsal, the former is very sensitive to the availability of attention and the presence of concurrent attention demanding task. Finally, brain imaging studies reported distinct neural structures supporting the separation of these two loops.

ACKNOWLEDGMENTS

This paper was presented at the International Conference on Working Memory (Cambridge, July 2014) in celebration of the Baddeley and Hitch (1974)’s 40th anniversary. This research was supported by grants from the Swiss National Science Foundation N° 100014_137860 to Valérie Camos and N° 100014_152550 to Pierre Barrouillet. We thank Vanessa Loaiza for her careful reading of this manuscript.

REFERENCES

Anderson, J. R. (1993). Rules of the Mind. Hillsdale, NJ: Lawrence Erlbaum Associates.

Anderson, J. R. (2007). How Can the Human Mind Occur in the Physical Universe? New York, NY: Oxford University Press.

Anderson, J. R., and Lebière, C. (1998). The Atomic Components of Thought. Mawhaw, NJ: Lawrence Erlbaum Associates.

Baddeley, A. D. (1986). Working Memory. Oxford: Clarendon Press.

Baddeley, A. D. (2000). The episodic buffer: a new component of working memory? Trends Cogn. Sci. 4, 417–423. doi: 10.1016/s1364-6613(00)01538-2

Baddeley, A. D. (2007). Working Memory, Thought and Action. Oxford: Oxford University Press.

Baddeley, A. D. (2012). Working memory: theories, models and controversies. Ann. Rev. Psychol. 63, 1–29. doi: 10.1146/annurev-psych-120710-100422

Baddeley, A. D., Allen, R. J., and Hitch, G. J. (2010). Investigating the episodic buffer. Psychol. Belg. 50, 223–243. doi: 10.5334/pb-50-3-4-223

Baddeley, A. D., and Hitch, G. J. (1974). “Working memory,” in Recent Advances in Learning and Motivation, ed G. A. Bower (New York: Academic Press), 647–667.

Baddeley, A. D., and Larsen, J. D. (2007). The phonological loop unmasked? A comment on the evidence for a “perceptual-gestural” alternative. Q. J. Exp. Psychol. (Hove) 60, 497–504. doi: 10.1080/17470210601147572

Baddeley, A. D., and Logie, R. H. (1999). “Working memory: the multiple-component model,” in Models of Working Memory: Mechanisms of Active Maintenance and Executive Control, eds A. Miyake and P. Shah (Cambridge: Cambridge University Press), 28–61.

Barrouillet, P., Bernardin, S., and Camos, V. (2004). Time constraints and resource-sharing in adults’ working memory spans. J. Exp. Psychol. Gen. 133, 83–100. doi: 10.1037/0096-3445.133.1.83

Barrouillet, P., Bernardin, S., Portrat, S., Vergauwe, E., and Camos, V. (2007). Time and cognitive load in working memory. J. Exp. Psychol. Learn. Mem. Cog. 33, 570–585. doi: 10.1037/0278-7393.33.3.570

Barrouillet, P., and Camos, V. (2012). As time goes by: temporal constraints in working memory. Curr. Dir. Psychol. Sci. 21, 413–419. doi: 10.1177/0963721412459513

Barrouillet, P., and Camos, V. (2015). Working Memory: Loss and Reconstruction. Hove: Psychology Press.

Barrouillet, P., Portrat, S., and Camos, V. (2011). On the law relating processing and storage in working memory. Psychol. Rev. 118, 175–192. doi: 10.1037/a0022324

Borst, J. P., Taatgen, N. A., and van Rijn, H. (2010). The problem state: a cognitive bottleneck in multitasking. J. Exp. Psychol. Learn. Mem. Cog. 36, 363–382. doi: 10.1037/a0018106

Bunting, M. F. (2006). Proactive interference and item similarity in working memory. J. Exp. Psychol. Learn. Mem. Cog. 32, 183–196. doi: 10.1037/0278-7393.32.2.183

Camos, V., Lagner, P., and Barrouillet, P. (2009). Two maintenance mechanisms of verbal information in working memory. J. Mem. Lang. 61, 457–469. doi: 10.1016/j.jml.2009.06.002

Camos, V., Mora, G., and Barrouillet, P. (2013). Phonological similarity effect in complex span task. Q. J. Exp. Psychol. 66, 1927–1950. doi: 10.1080/17470218.2013.768275

Camos, V., Mora, G., and Oberauer, K. (2011). Adaptive choice between articulatory rehearsal and attentional refreshing in verbal working memory. Mem. Cognit. 39, 231–244. doi: 10.3758/s13421-010-0011-x

Coltheart, V. (1999). Comparing short-term memory and memory for rapidly presented visual stimuli. Int. J. Psychol. 34, 293–300. doi: 10.1080/002075999399594

Cowan, N. (1992). Verbal memory span and the timing of spoken recall. J. Mem. Lang. 31, 668–684. doi: 10.1016/0749-596x(92)90034-u

Cowan, N. (1995). Attention and Memory: An Integrated Framework. New York, NY: Oxford University Press.

Cowan, N. (1999). “An embedded-process model of working memory,” in Models of Working Memory: Mechanisms of Active Maintenance and Executive Control, eds A. Miyake and P. Shah (Cambridge: Cambridge University Press), 62–101.

Cowan, N. (2005). Working Memory Capacity. Hove: Psychology Press.

D’Esposito, M., Detre, J. A., Alsop, D. C., Shin, R. K., Atlas, S., and Grossman, M. (1995). The neural basis of the central executive system of working memory. Nature 378, 279–281. doi: 10.1038/378279a0

Engle, R. W., and Kane, M. J. (2004). “Executive attention, working memory capacity and a two-factor theory of cognitive control,” in The Psychology of Learning and Motivation (Vol. 44), ed B. Ross (New York: Elsevier), 145–199.

Engle, R. W., Kane, M. J., and Tuholski, S. W. (1999). “Individual differences in working memory capacity and what they tell us about controlled attention, general fluid intelligence and functions of the prefrontal cortex,” in Models of Working Memory: Mechanisms of Active Maintenance and Executive Control, eds A. Miyake and P. Shah (Cambridge: Cambridge University Press), 102–134.

Fallon, A. B., Groves, K., and Tehan, G. (1999). Phonological similarity and trace degradation in the serial recall task: when CAT helps RAT, but not MAN. Int. J. Psychol. 34, 301–307. doi: 10.1080/002075999399602

Garavan, H. (1998). Serial attention within working memory. Mem. Cognit. 26, 263–276. doi: 10.3758/bf03201138

Gruber, O. (2001). Effects of domain-specific interference on brain activation associated with verbal working memory task performance. Cereb. Cortex 11, 1047–1055. doi: 10.1093/cercor/11.11.1047

Gruber, O., and von Cramon, D. Y. (2003). The functional neuroanatomy of human working memory revisited: evidence from 3-T fMRI studies using classical domain-specific interference tasks. Neuroimage 19, 797–809. doi: 10.1016/s1053-8119(03)00089-2

Hitch, G. J., Halliday, M. S., and Littler, J. E. (1989). Item identification time, rehearsal rate and memory span in children. Q. J. Exp. Psychol. 41, 321–337. doi: 10.1080/14640748908402368

Hitch, G. J., Halliday, M. S., and Littler, J. E. (1993). Development of memory span for spoken words: the role of rehearsal and item identification processes. Br. J. Dev. Psychol. 11, 159–169. doi: 10.1111/j.2044-835x.1993.tb00595.x

Hudjetz, A., and Oberauer, K. (2007). The effects of processing time and processing rate on forgetting in working memory: testing four models of the complex span paradigm. Mem. Cognit. 35, 1675–1684. doi: 10.3758/bf03193501

Jarrold, C., Tam, H., Baddeley, A. D., and Harvey, C. E. (2010). The nature and the position of processing determines why forgetting occurs in working memory tasks. Psychon. Bull. Rev. 17, 772–777. doi: 10.3758/pbr.17.6.772

Johnson, M. K., Raye, C. L., Mitchell, K. J., Greene, E. J., Cunningham, W. A., and Sanislow, C. A. (2005). Using fMRI to investigate a component process of reflection: prefrontal correlates of refreshing a just activated representation. Cogn. Affect. Behav. Neurosci. 5, 39–361. doi: 10.3758/CABN.5.3.339

Johnson-Laird, P. N. (1983). Mental Models. Cambridge: Cambridge University Press.

Kane, M. J., and Engle, R. W. (2003). Working-memory capacity and the control of attention: the contributions of goal neglect, response competition and task set to stroop interference. J. Exp. Psychol. Gen. 132, 47–70. doi: 10.1037/0096-3445.132.1.47

Lisman, J. E., and Idiart, M. A. P. (1995). Storage of 7 +/− 2 short-term memories in oscillatory subcycles. Science 267, 1512–1515. doi: 10.1126/science.7878473

Lovett, M. C., Reder, L. M., and Lebière, C. (1999). “Modeling working memory in a unified architecture: an ACT-R perspective,” in Models of Working Memory: Mechanisms of Active Maintenance and Executive Control, eds A. Miyake and P. Shah (Cambridge: Cambridge University Press), 135–182.

Luck, S. J., and Vogel, E. K. (1997). The capacity of visual working memory for features and conjunctions. Nature 390, 279–281.

Luck, S. J., and Vogel, E. K. (1998). Response to visual and auditory working memory capacity by N. Cowan. Trends Cogn. Sci. 2, 78–80. doi: 10.1016/s1364-6613(98)01144-9

McCabe, D. P. (2008). The role of covert retrieval in working memory span tasks: evidence from delayed recall tests. J. Mem. Lang. 58, 480–494. doi: 10.1016/j.jml.2007.04.004

McElree, B. (1998). Attended and non-attended states in working memory: accessing categorized structures. J. Mem. Lang 38, 225–252. doi: 10.1006/jmla.1997.2545

McElree, B. (2001). Working memory and focal attention. J. Exp. Psychol. Learn. Mem. Cogn. 27, 817–835. doi: 10.1037//0278-7393.27.3.817

McElree, B., and Dosher, B. A. (1989). Serial position and set size in short-term memory: the time course of recognition. J. Exp. Psychol. Gen. 118, 346–373.

Mora, G., and Camos, V. (2013). Two systems of maintenance in verbal working memory: evidence from the word length effect. PLoS One 8:e70026. doi: 10.1371/journal.pone.0070026

Naveh-Benjamin, M., and Jonides, J. (1984). Maintenance rehearsal: a two-component analysis. J. Exp. Psychol. Learn. Mem. Cogn. 10, 369–385. doi: 10.1037//0278-7393.10.3.369

Oberauer, K. (2002). Access to information in working memory: exploring the focus of attention. J. Exp. Psychol. Learn. Mem. Cogn. 28, 411–421. doi: 10.1037//0278-7393.28.3.411

Oberauer, K. (2005). Control of the contents of working memory: a comparison of two paradigms and two age groups. J. Exp. Psychol. Learn. Mem. Cogn. 31, 714–728. doi: 10.1037/0278-7393.31.4.714

Pashler, H. (1998). The Psychology of Attention. Cambridge, MA: MIT Press.

Raye, C. L., Johnson, M. K., Mitchell, K. J., Greene, E. J., and Johnson, M. R. (2007). Refreshing: a minimal executive function. Cortex 43, 135–145. doi: 10.1016/s0010-9452(08)70451-9

Raye, C. L., Johnson, M. K., Mitchell, K. J., Reeder, J. A., and Greene, E. J. (2002). Neuroimaging a single thought: Dorsolateral PFC activity associated with refreshing just-activated information. Neuroimage 15, 447–453. doi: 10.1006/nimg.2001.0983

Rosen, V. M., and Engle, R. W. (1998). Working memory capacity and suppression. J. Mem. Lang. 39, 418–436. doi: 10.1006/jmla.1998.2590

Salamé, P., and Baddeley, A. D. (1986). Phonological factors in STM: similarity and the unattended speech effect. B. Psychonomic Soc. 24, 263–265. doi: 10.3758/bf03330135

Smith, E. E., and Jonides, J. (1999). Storage and executive processes in the frontal lobes. Science 283, 1657–1661. doi: 10.1126/science.283.5408.1657

Towse, J. N., Hitch, G. J., and Hutton, U. (1998). A reevaluation of working memory capacity in children. J. Mem. Lang. 39, 195–217. doi: 10.1006/jmla.1998.2574

Trost, S., and Gruber, O. (2012). Evidence for a double dissociation of articulatory rehearsal and non-articulatory maintenance of phonological information in human verbal working memory. Neuropsychobiology 65, 133–140. doi: 10.1159/000332335

Unsworth, N., and Engle, R. W. (2007a). On the division of short-term and working memory: an examination of simple and complex span and their relation to higher order abilities. Psychol. Bull. 133, 1038–1066. doi: 10.1037/0033-2909.133.6.1038

Unsworth, N., and Engle, R. W. (2007b). The nature of individual differences in working memory capacity: active maintenance in primary memory and controlled search from secondary memory. Psychol. Rev. 114, 104–132. doi: 10.1037/0033-295x.114.1.104

Unsworth, N., Schrock, J. C., and Engle, R. W. (2004). Working memory capacity and the antisaccade task: individual differences in voluntary saccade control. J. Exp. Psychol. Learn. Mem. Cogn. 30, 1302–1321. doi: 10.1037/0278-7393.30.6.1302

Unsworth, N., Spillers, G. J., and Brewer, G. A. (2010). The contributions of primary and secondary memory to working memory capacity: an individual differences analysis of immediate free recall. J. Exp. Psychol. Learn. Mem. Cogn. 36, 240–247. doi: 10.1037/a0017739

Vallar, G., and Baddeley, A. D. (1982). Short-term forgetting and the articulatory loop. Q. J. Exp. Psychol. 34, 53–60. doi: 10.1080/14640748208400857

Vergauwe, E., Camos, V., and Barrouillet, P. (2014). The impact of storage on processing: implications for structure and functioning of working memory. J. Exp. Psychol. Learn. Mem. Cogn. 40, 1072–1095. doi: 10.1037/a0035779

Vergauwe, E., and Cowan, N. (2014). A common short-term memory retrieval rate may describe many cognitive procedures. Front. Hum. Neurosci. 8:126. doi: 10.3389/fnhum.2014.00126

Vergauwe, E., and Cowan, N. (in press). Attending to items in working memory: evidence that refreshing and memory search are closely related. Psychon. B. Rev. doi: 10.3758/s13423-014-0755-6

Conflict of Interest Statement: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Received: 26 July 2014; Accepted: 21 October 2014; Published online: 07 November 2014.

Citation: Camos V and Barrouillet P (2014) Attentional and non-attentional systems in the maintenance of verbal information in working memory: the executive and phonological loops. Front. Hum. Neurosci. 8:900. doi: 10.3389/fnhum.2014.00900

This article was submitted to the journal Frontiers in Human Neuroscience.

Copyright © 2014 Camos and Barrouillet. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

PERSPECTIVE ARTICLE
published: 07 March 2014
doi: 10.3389/fnhum.2014.00126

A common short-term memory retrieval rate may describe many cognitive procedures

Evie Vergauwe and Nelson Cowan*

Department of Psychological Sciences, University of Missouri, Columbia, MO, USA

* Correspondence: Nelson Cowan, Department of Psychological Sciences, University of Missouri, 210 McAlester Hall, Columbia, MO 65211, USA e-mail: cowann@missouri.edu

Edited by:
Steve Majerus, Université de Liège, Belgium

Reviewed by:
Tobias Egner, Duke University, USA
Jason M. Chein, Temple University, USA

We examine the relationship between response speed and the number of items in short-term memory (STM) in four different paradigms and find evidence for a similar high-speed processing rate of about 25–30 items per second (∼35–40 ms/item). We propose that the similarity of the processing rates across paradigms reflects the operation of a very basic covert memory process, high-speed retrieval, that is involved in both the search for information in STM and the reactivation or refreshing of information that keeps it in STM. We link this process to a specific pattern of rhythmic, repetitive neural activity in the brain (gamma oscillations). This proposal generates ideas for research and calls for an integrative approach that combines neuroscientific measures with behavioral cognitive techniques.

Keywords: short-term memory, working memory, attention, retrieval, refreshing, memory search

An important feature of human information processing is short-term memory (STM), the ability to retain a small amount of information in a highly accessible state for a short time. The capacity of STM is limited to a certain number of items, and a key issue in cognitive psychology is the reason why STM is limited. Here we suggest that, over the last 40–50 years, at least four different paradigms have been developed that provide insights into the temporal properties of STM. Despite the wide variety of paradigms, we observed an intriguing similarity in a high-speed processing rate of about 25–30 items per second, which can be inferred from the relationship between response speed and memory load. We propose that the similarity of the processing rates across paradigms may reflect a basic covert memory process (i.e., a memory process that is inferred from the pattern of recall performance across certain conditions, rather than being directly observable), high-speed retrieval, which can be used for either recognition of a probe item or reactivation (refreshing) of an item for the sake of maintenance. We also link this process to recent developments in the neuroscientific literature and discuss implications for future research.

THE RELATIONSHIP BETWEEN RESPONSE SPEED AND MEMORY LOAD IN FOUR PARADIGMS

After the seminal article of Miller (1956) on STM capacity limitations, human STM research mainly investigated the determinants of failure of STM by focusing on accuracy and error patterns in simple memory tasks. In the late 1960s, however, a complementary approach became increasingly popular. This approach consisted of studying how much time participants need in order to succeed in simple memory tasks. Specifically, Saul Sternberg studied how much time participants needed in order to indicate whether a probe item was present in a small set of memorized elements (Sternberg, 1966, 1969a). The rationale was that, if the information in memory is needed to select the appropriate response, then the time taken to give that response will reveal something about the process by which one is searching in memory for that information. In order to explore the timing of memory search, Sternberg proposed what we term the Sternberg Item-Recognition paradigm (Figure 1A). Although it is still the standard paradigm to investigate memory search rates, at least three other paradigms can be identified as providing insights into the temporal properties of STM (Figures 1B–D); all show a positive relation between the number of items to be retained in STM (memory load) and the time it takes to respond to a probe item (response latency). Figure 2A provides an overview of what, based on our review, seem to be necessary boundary conditions that must be met to observe a clear positive relation between memory load and response latency. In what follows, only studies that met these conditions are reported and, when interpreting the observed common processing rate, we will explicitly address the role of these boundary conditions.

FIGURE 1. Schematic presentation of four paradigms providing insights in the relation between response speed and memory load. In the example, participants are presented with three letters to be maintained: K, B and N. The hand symbol together with the hourglass refers to a response given by pressing a button for which the speed is the variable of interest here. In (A) the Sternberg Item-Recognition paradigm, we examined speed of response to probe as a function of the number of memory items; in (B) through (D) we examined speed of response to processing items as a function of the concurrent number of items in memory; (B) Brown-Peterson Pre-Load paradigm; (C) Complex Span paradigm; (D) Psychological Refractory Period paradigm.

FIGURE 2. (A) Schematic presentation of cognitive interpretation of the observed processing rate together with the boundary conditions (in black) that must be met to observe a clear relationship between memory load and response latency. Two different levels in STM are presented: a central level and a peripheral level. For verbal stimuli, the peripheral level offers an alternative maintenance mechanism (articulatory rehearsal), the use of which should be minimized when examining STM load for continuously presented items. (B) Estimates of STM retrieval slope for three kinds of verbal materials based on (1) the Sternberg Item-Recognition paradigm (gray bars), (2) the Brown-Peterson Pre-Load paradigm (red bar, second from the bottom), (3) the Complex Span paradigm (blue bar, last in the Words cluster), and (4) the Psychological Refractory Period (yellow bar, bottom). For the Sternberg Item-Recognition paradigm, the figure only includes studies that provided the information necessary to calculate 95% confidence intervals (represented by error bars). For the other paradigms, the unweighted average across studies mentioned in the text is presented.

For the sake of comparison, in the following review, we only included data of experiments that used simple verbal stimuli to be memorized (digits, letters and words), using healthy young adults as participants. We only included studies that provide the information necessary to examine a particular key index of the rate of retrieving information from STM, the slope of the function that relates response latency to STM load. Consequently, we only considered studies that included at least two different levels of memory load and that either reported the slope of the relation of interest or reported response latency for each memory load condition so that we could estimate the slope (averaged across positive and negative responses). Despite the fact that the four paradigms differ quite substantially in their methodology (see Figure 1), we identified a similar processing rate across them.

THE STERNBERG ITEM-RECOGNITION PARADIGM

The variable of interest is the speed with which participants decide whether the probe is a member of set of items held in STM by pressing, as quickly as possible without making errors (Figure 1A). It is assumed that this decision requires people to scan through the content of STM to compare the probe with each item in memory. Delay of the response is interpreted as the operation of this time-consuming covert memory search. The classical finding is that response times increase linearly with the size of the memory set with a slope of about 35–40 ms per additional item in memory (Sternberg, 1966, 1969a). The slope of this function is assumed to reflect the time it takes to retrieve a single item from STM. These classic findings of Sternberg launched a very productive line of investigation on memory search in cognitive science, with an overwhelming number of studies testing the original paradigm and variants of it. Because of the limited space here, the included studies using this paradigm were limited to the visual modality for presentation of both memory items and test items. On average, response latency increased at a rate of 37 ms per additional item held in memory.[1]

THE BROWN-PETERSON PRE-LOAD PARADIGM

In the paradigm developed by Brown (1958) and Peterson and Peterson (1959), a few stimuli to be remembered are followed by a processing task that is different enough to avoid material-specific interference, but challenging enough to prevent attention to the memoranda or rehearsal of them (Figure 1B). The main finding was that memory is lost rapidly across about 30 s. The variable of interest here, though, is processing speed on the concurrent processing task that precedes recall. Slowing down has been shown in several studies comparing response speed under concurrent memory load with response speed without a concurrent load (e.g., Shulman and Greenberg, 1971; Baddeley and Hitch, 1974; Logan, 1978). It is assumed that, during the retention interval filled with processing, people engage in storage-related activities. When processing and storage both rely on attentional resources, storage-related activities are expected to postpone concurrent processing activities. Methodological details can be found in Footnote 2.[2] Vergauwe et al. (2014) found that response latency increased linearly at a rate of 43 ms per additional item held in memory.

THE COMPLEX SPAN PARADIGM

In this paradigm, the presentation of items to be remembered is interleaved with items to be processed (Figure 1C). The purpose was originally to assess the capability of working memory under the assumption that storage and processing share a common cognitive resource so that both of them must be engaged in order for capability to be assessed (e.g., Daneman and Carpenter, 1980). The variable of interest for the present purposes is processing speed on the concurrent processing task. Several studies have shown longer response latencies in later processing phases (high memory load), compared to the first processing phase (low memory load; e.g., Friedman and Miyake, 2004; Chen and Cowan, 2009). As for the Brown-Peterson pre-load paradigm, the underlying assumption is that slower processing reflects resource-sharing between attention-demanding processing and storage activities. Methodological details can be found in Footnote 2. Jarrold et al. (2011) found linear trends across the successive processing phases showing that response latency increases at an average rate of 37 ms per additional item held in memory (41 ms in Experiment 1 and 33 ms in Experiment 2).

Another potential variable of interest, but one that requires further work, is the time it takes to retrieve the next item to be recalled. Cowan (1992) measured the timing of spoken recall for simple digit span in children and proposed that each inter-word pause reflects a process of search through working memory to find the next digit to be recalled. Subsequent work (Cowan et al., 1998) showed that the inter-word pauses for correctly-recalled lists did increase in approximately a linear fashion with increasing list length, in children in first grade (84 ms/item), third grade (58 ms/item), and fifth grade (25 ms/item). In adults, further work is needed to establish the scanning rate. One might worry that verbal rehearsal processes would play a role, though a relation between the spoken recall rate and search rates based on the scanning paradigm was demonstrated by Cowan et al. (1998) and by Hulme et al. (1999). In complex span, presumably rehearsal processes have been interrupted by the processing task. Recall in these tasks, however, might involve more than a simple search, for example an attempt to use the processing task as a context to retrieve the list items. Thus, Cowan et al. (2003) noted that inter-word pauses in the responses lasted 4–10 times longer than in simple span.

PSYCHOLOGICAL REFRACTORY PERIOD PARADIGM

This paradigm (Welford, 1952; Pashler, 1994) usually combines two processing tasks requiring two responses in succession on a single trial. The original point was to explore processing demands by studying how the processing for the first response delayed the second response. In the task variants of interest here, memory demands are combined with processing demands. After the memory set is presented, at various stimulus-onset asynchronies (SOAs), a single stimulus pertaining to the processing task is presented, to which a speeded response is required (Figure 1D). Some of these studies also manipulated the size of the memory set, which makes them of particular interest here. The finding of interest here is that the single speeded response took longer as more items were held in memory concurrently (e.g., Jolicoeur and Dell’Acqua, 1998; Stevanovski and Jolicoeur, 2007). Again, the underlying assumption is that processing and storage interfere with each other because they rely on a common attentional resource, resulting in slower processing. Methodological details can be found in Footnote 3.[3] Processing took about 46 ms longer per additional item in memory (32 and 60 ms in Stevanovski and Jolicoeur, 2007, in Experiments 2 and 3, respectively).

EMPIRICAL SUMMARY

We have identified a pattern that holds across four different paradigms: response speed slows down at a rate of about 30–40 ms per additional simple verbal item in memory (see Figure 2B). The similarity across the paradigms suggests strongly the existence of a high-speed processing rate in STM of about 25–30 items per second (the equivalent of 40–33 ms/item).

Previous studies have pointed out the similarity between the processing rates observed in the complex span paradigm and the Sternberg item-recognition paradigm (Jarrold et al., 2011), and between the rates observed in the Brown-Peterson pre-load paradigm and the Sternberg item-recognition paradigm (Vergauwe et al., 2014). The present contribution is to note the similarity of processing rate across a wider range of procedures, and to propose a cognitive interpretation of this high-speed processing rate, in the next section.

COGNITIVE INTERPRETATION OF HIGH-SPEED PROCESSING RATE IN HUMAN SHORT-TERM MEMORY (STM)

We interpret the identified processing rate as reflecting the operation of a very basic covert memory process, retrieval from STM. In this view, although information retrieval and maintenance are typically referred to as different stages in STM, they are proposed to rely on the same process. When responding to a probe in the Sternberg task, high-speed retrieval is used in the service of memory search. It brings items in the focus of attention so that one can check whether it matches the probe. The slope observed in this task reflects directly the use of high-speed retrieval. In the three remaining paradigms, high-speed retrieval is used in the service of memory maintenance; it brings items in the focus of attention so that the information gets reactivated or refreshed. When high-speed retrieval and concurrent processing share a common resource (attention), the use of high-speed retrieval influences concurrent processing speed so that response latency increases for each additional item that is maintained. Under the assumption that maintenance is accomplished through sequential reactivation of information in a cumulative fashion, starting from the first list item and proceeding in forward order until the end, the observed rate reflects the rate at which items are reactivated in STM. In the Sternberg task, it is assumed that the presentation of the probe initiates a complete cycle through STM. In the other paradigms described here, storage is combined with a self-paced processing task and the idea is that a complete cycle of refreshing is interpolated before attention-demanding processing takes place. Thus, provided that participants aim at performing well on the memory task, attention is first used for a complete cycle through STM before it is shifted to the next processing stimulus. It is possible, though, that the same assumption might not hold in tasks in which the processing task is to be performed at a predefined pace (i.e., computer-paced). In these tasks, every processing item is typically followed by a variable period of free time during which refreshing can take place in a continuous matter. If the process of refreshing is exhaustive in nature, one might expect that, upon the presentation of the next processing item, on average only half of the items in STM would still need to be reactivated. Slopes relating response times to memory load would then reflect the amount of time it takes to scan half of the number of items in STM.

A schematic presentation of our cognitive interpretation of the observed processing rate is shown in Figure 2A. Two different levels in STM are presented: (1) a central level that is domain-general in nature, closely related to attention, and (2) a peripheral level that is domain-specific in nature and independent from the central level. High-speed retrieval is used at the central level to bring information into the focus of attention.

Together with the observation of Cowan et al. (1998) that retrieval rate as measured in a search task correlates with memory span, the identification of a rapid retrieval rate across several paradigms is directly relevant to the long-standing debate regarding the nature of the severe capacity limit of STM. Theoretically, the capacity limit of STM might reflect the number of items that can be active simultaneously within a given time-window. If one assumes that there is a limited time-window within which the items need to be reactivated so that all of them can be retained, then the capacity limit of STM would depend on the retrieval rate with faster rates resulting in more items reactivated within the fixed time-window. A similar idea was proposed by Cavanagh (1972) who showed an inverse relation between STM span and memory search rate for different materials. The speed of retrieval in STM also indicates that STM functions in a way that is much more rapid and dynamic than most people would think. Importantly, we consider this rapid retrieval rate to be independent of the slower verbal rehearsal rate that relies on covert speech, even though both might serve the same goal of maintaining information in STM (see Cowan et al., 1998; Hulme et al., 1999; Camos et al., 2011).

Note that although the idea of a limited time-window implies the existence of time-based forgetting in STM, it is not incompatible with interference-based forgetting. When items are not reactivated in time, forgetting might occur either because memory traces have decayed or because newer representations have overwritten previous ones or have become confusable with the previous ones. The degree of confusability might then depend on the number of features that are shared between the representations in STM. Moreover, Ricker and Cowan (2014) have recently shown that the process of consolidation influences the observed rate of forgetting over time with more consolidation leading to slower rates of time-based forgetting. This finding indicates that the relationship between STM capacity, retrieval rate and decay rate might depend on the robustness of the trace. Also, the length of the critical time-window might differ between individuals and this possibility needs to be taken into consideration when focusing on the relation between high-speed retrieval and STM capacity across individuals.

BOUNDARY CONDITIONS

There are studies in which the slope of the relationship between response speed and memory load was substantially smaller than the proposed constant of about 37 ms per item in normal adults. For example, in a Sternberg task, Banks and Atkinson (1974) forced participants to respond so quickly that they made a lot of errors. A flatter slope may occur when speed is stressed at the expense of accuracy because participants base their response on a feeling of familiarity, which can occur for all items in parallel, rather than on a more time-consuming but accurate item-by-item memory search. Burrows and Okada (1975) showed that the Sternberg slope changes at the limits of STM with a shallow slope of 13 ms when considering memory loads ranging between 8 and 20 words, for which the only viable mechanism might be familiarity. In the processing times within complex span, but using viewing or reading times rather than simple reaction times (RT) slopes across memory loads vary considerably (e.g., Engle et al., 1992; Friedman and Miyake, 2004). Viewing or reading might be covertly interrupted for refreshing, evading measurement. There are also studies in which the slope of the relationship between response speed and memory load was larger than the proposed constant of about 37 ms per item. RT slopes across memory loads are considerably steeper (up to about 100 ms per item) in studies that use Sternberg-like tasks in which participants need to have access to serial order information in order to judge the probe correctly, as opposed to the typical Sternberg task in which access to item information is sufficient (e.g., Sternberg, 1969b; Ravizza et al., 2011; Majerus et al., 2012). Furthermore, studies in which a delay of several seconds was inserted between the presentation of the memory set and the presentation of the probe also reported somewhat steeper slopes (about 50–55 ms per item; e.g., Cairo et al., 2004; Chen and Desmond, 2005). Maintenance-related processes such as verbal rehearsal might take place during this delay and as such, influence the observed retrieval rate at the end of the trial. We suggest boundary conditions to observe a clear, positive relation between memory load and response latency, as presented in Figure 2A.

RELATING HIGH-SPEED RETRIEVAL IN SHORT-TERM MEMORY (STM) TO OSCILLATIONS IN THE BRAIN

Recent neuroscientific developments lead to a view of retrieval rate as governed by oscillations (rhythmic, repetitive neural activity; e.g., Lisman and Jensen, 2013). In the dual oscillation model of STM (Lisman and Idiart, 1995) it is proposed that the features of one item are active at the same time and are represented by a group of neurons that fire in the same gamma cycle (30–80 Hz). Next, the features of a second item are active at the same time and represented by the second gamma cycle within the same theta cycle (4–8 Hz). Lisman and Idiart (1995) linked the Sternberg slope to the duration of a gamma cycle. One item would be searched each time its gamma cycle of neural activity occurred. They also suggested that STM capacity limits could be determined by the number of gamma cycles that fit into one theta cycle. Given current uncertainties in these figures, this neural theory is reasonably compatible with a cognitive proposal by which STM capacity depends on the number of items that can be reactivated within a given time-window so that several items can be retained in a refreshed state simultaneously. Each gamma cycle would allow the refreshment of one item in STM. Our empirical retrieval rate of 37 ms/item would correspond to a gamma cycle of 27 Hz and would allow 3–6 items per theta cycle.

Thus, we propose to extend the view of Lisman and Idiart so that it encompasses our expanded function of high-speed retrieval. In this view, refreshing consists in the rapid reactivation of a limited number of items at a rate that reflects the length of one gamma cycle per item. In support of a link between STM maintenance and gamma oscillations, changes of oscillatory activity in the human gamma frequency band related to STM retention have been observed (e.g., Tallon-Baudry et al., 1998; Jokisch and Jensen, 2007; Meltzer et al., 2008) and Howard et al. (2003) showed that, in a Sternberg-type task, gamma power during retention was higher for larger memory sets. Furthermore, Roux et al. (2012) showed a relation between gamma-band activity and memory load in a left prefrontal area of the brain that has been associated with refreshing (e.g., Johnson et al., 2005). In this study, a number of red disks were displayed in different locations. After a short delay, a single red disk was shown and participants decided whether its location matched one of the study locations. An increase in gamma-band power between load 3 and load 6 was observed during the delay and this increase correlated with memory performance. Finally, Kamiński et al. (2011) found a negative correlation between individual’s STM performance and gamma cycle length. This is exactly the kind of relationship one would expect if STM capacity depends on the number of items that can be reactivated within a given time-window with each gamma cycle allowing the reactivation of one item in STM.

CONCLUSION AND OUTLOOK

The current proposal is novel in at least two ways. First, it proposes that the identified high-speed processing rate of about 27 items per second across four different procedures might reflect the operation of a very basic process of high-speed retrieval that serves both memory search and attention-based refreshing in STM. Thus, the attentional component of memory search and refreshing is proposed to be the same. This does not preclude the theoretical possibility that refreshing is equivalent to retrieval plus some additional operations; it only restricts these additional operations to a set of operations that do not require attention. Second, it proposes that this general process might be associated with gamma brain oscillations. We believe that our proposal has the potential of providing novel insights into the significant questions of how information is maintained in STM and why it’s capacity-limited. The proposal is based on a limited number of studies at this point and further research is needed, but the present proposal suggests several clear directions for further research.

Behavioral research should aim at testing the unique predictions that follow from our proposal. First, memory search and maintenance are proposed to rely on the same STM retrieval process. One direction is to look for interference patterns between both processes. The results of ongoing research of ours suggest that the memory search slope varies as a function of the time available to refresh memoranda. Another test of our proposal would be to compare the processing rates across the four paradigms in a within-participants design. This might also help us understand whether variations in the processing rate between procedures and materials are meaningful. Second, refreshing of a series of items is proposed to be enacted by consecutive gamma oscillations. Does the order of spontaneous refreshing follow the order of presentation? Can the distance between individual items in STM be described in terms of the number of gamma cycles that separate them? When a set of multiple items is successfully chunked into a few chunks, can we observe a decrease in the number of gamma cycles one needs to run through in order to refresh the entire set? In addition, future research should aim at testing the universality of the identified retrieval process by searching whether a similar processing rate can be observed in other paradigms and by examining response time distribution data. Another remaining question is whether transferring new external information into STM would occur at the same rate. The results of some studies suggest a slower rate of consolidation of about 200–250 ms per item in the Brown-Peterson pre-load paradigm (e.g., Jarrold et al., 2011; Vergauwe et al., 2014). This rate matches the length of theta cycles which have been linked to encoding new information (e.g., Klimesch, 1999). Finally, neurophysiological and cognitive approaches should be integrated to examine whether the length of gamma cycles and retrieval rate are influenced by the same factors (experimental, individual, developmental, clinical), and whether externally induced changes in gamma frequency (e.g., through magnetic stimulation of the brain) affect STM speed and capacity.

ACKNOWLEDGMENTS

This research was supported by Swiss National Science Foundation Grant PA00P1_139604 to Evie Vergauwe and NIH Grant R01-HD21338 to Nelson Cowan.

FOOTNOTES

1^The average of 37 ms/item refers to an unweighted average across 8 slopes for digits (M = 36 ms/item), 5 slopes for words (M = 36 ms/item) and 13 slopes for letters (M = 38 ms/item). When only taking into account the studies that provide information to calculate 95% confidence intervals (i.e., the studies included in the lower panel of Figure 2), the unweighted average is 36 ms/item.

2^In the Brown-Peterson pre-load paradigm and the complex span paradigm, a difference is typically made between the RT for the individual’s first response in the processing phase, referred to as first processing times and the mean of all subsequent RTs in that processing phase, referred to as subsequent processing times. While longer first processing times have been attributed to the consolidation of memory traces, longer subsequent processing times are typically attributed to the maintenance of memory traces (e.g., Engle et al., 1992; Jarrold et al., 2011). Because our focus is on the maintenance process rather than on consolidation, the current manuscript only reports analyses that concern subsequent processing times. The slopes reported for the complex span paradigm concern an average across different list lengths (from 4 to 7 words).

3^We included only conditions in which participants are required to recall all of the information after this speeded response because, only in those conditions, participants are required to make the processing response while keeping the information active in memory. Furthermore, we only included studies in which the effect of memory load on response time did not depend on SOA because only in those studies, the effect can be interpreted as a cost related to maintaining information in STM rather than consolidating information into STM.

REFERENCES

Baddeley, A. D., and Hitch, G. (1974). “Recent advances in learning and motivation,” in Working Memory (Vol. 8), ed G. A. Bower (New York: Academic Press), 647–667.

Banks, W., and Atkinson, R. (1974). Accuracy and speed strategies in scanning active memory. Mem. Cognit. 2, 629–636. doi: 10.3758/bf03198131

Brown, H. L., and Kirsner, K. (1980). A within–subjects analysis of the relationship between memory span and processing rate in short–term memory. Cogn. Psychol. 12, 177–187. doi: 10.1016/0010-0285(80)90007-9

Brown, J. (1958). Some tests of the decay theories of immediate memory. Q. J. Exp. Psychol. 10, 12–21. doi: 10.1080/17470215808416249

Burrows, D., and Okada, R. (1971). Serial position effects in high-speed memory search. Percept. Psychophys. 10, 305–308. doi: 10.3758/bf03212831

Burrows, D., and Okada, R. (1975). Memory retrieval from long and short lists. Science 188, 1031–1033. doi: 10.1126/science.188.4192.1031

Cairo, T. A., Liddle, P. F., Woodward, T. S., and Ngan, E. T. C. (2004). The influence of working memory load on phase specific patterns of cortical activity. Brain Res. Cogn. Brain Res. 21, 377–387. doi: 10.1016/j.cogbrainres.2004.06.014

Camos, V., Mora, G., and Oberauer, K. (2011). Adaptive choice between articulatory rehearsal and attentional refreshing in verbal working memory. Mem. Cognit. 39, 231–244. doi: 10.3758/s13421-010-0011-x

Cavanagh, J. P. (1972). Relation between the immediate memory span and the memory search rate. Psychol. Rev. 79, 525–530. doi: 10.1037/h0033482

Chen, S. H. A., and Desmond, J. E. (2005). Temporal dynamics of cerebro-cerebellar network recruitment during a cognitive task. Neuropsychologia 43, 1227–1237. doi: 10.1016/j.neuropsychologia.2004.12.015

Chen, Z., and Cowan, N. (2009). How verbal memory loads consume attention. Mem. Cognit. 37, 829–836. doi: 10.3758/mc.37.6.829

Clifton, C., and Tash, J. (1973). Effect of syllabic word length on memory-search rate. J. Exp. Psychol. 99, 231–235. doi: 10.1037/h0034643

Cowan, N. (1992). Verbal memory span and the timing of spoken recall. J. Mem. Lang. 31, 668–684. doi: 10.1016/0749-596x(92)90034-u

Cowan, N., Towse, J. N., Hamilton, Z., Saults, J. S., Elliott, E. M., Lacey, J. F., et al. (2003). Children’s working-memory processes: a response-timing analysis. J. Exp. Psychol. Gen. 132, 113–132. doi: 10.1037/0096-3445.132.1.113

Cowan, N., Wood, N. L., Wood, P. K., Keller, T. A., Nugent, L. D., and Keller, C. V. (1998). Two separate verbal processing rates contributing to short-term memory span. J. Exp. Psychol. Gen. 127, 141–160. doi: 10.1037/0096-3445.127.2.141

Daneman, M., and Carpenter, P. A. (1980). Individual differences in working memory and reading. J. Verbal Learning Verbal Behav. 19, 450–466. doi: 10.1016/s0022-5371(80)90312-6

Engle, R. W., Cantor, J., and Carullo, J. J. (1992). Individual differences in working memory and comprehension: a test of four hypotheses. J. Exp. Psychol. Learn. Mem. Cogn. 18, 972–992. doi: 10.1037//0278-7393.18.5.972

Friedman, N. P., and Miyake, A. (2004). The reading span test and its predictive power for reading comprehension ability. J. Mem. Lang. 51, 136–158. doi: 10.1016/j.jml.2004.03.008

Howard, M. W., Rizzuto, D. S., Caplan, J. B., Madsen, J. R., Lisman, J., Aschenbrenner-Scheibe, R., et al. (2003). Gamma oscillations correlate with working memory load in humans. Cereb. Cortex 13, 1369–1374. doi: 10.1093/cercor/bhg084

Hulme, C., Newton, P., Cowan, N., Stuart, G., and Brown, G. (1999). Think before you speak: pauses, memory search and trace redintegration processes in verbal memory span. J. Exp. Psychol. Learn. Mem. Cogn. 25, 447–463. doi: 10.1037/0278-7393.25.2.447

Jarrold, C., Tam, H., Baddeley, A. D., and Harvey, C. E. (2011). How does processing affect storage in working memory tasks? Evidence for both domain-general and domain-specific effects. J. Exp. Psychol. Learn. Mem. Cogn. 37, 688–705. doi: 10.1037/a0022527

Johnson, M. K., Raye, C. L., Mitchell, K. J., Greene, E. J., Cunningham, W. A., and Sanislow, C. A. (2005). Using fMRI to investigate a component process of reflection: prefrontal correlates of refreshing a just-activated representation. Cogn. Affect. Behav. Neurosci. 5, 339–361. doi: 10.3758/CABN.5.3.339

Jokisch, D., and Jensen, O. (2007). Modulation of gamma and alpha activity during a working memory task engaging the dorsal or ventral stream. J. Neurosci. 27, 3244–3251. doi: 10.1523/jneurosci.5399-06.2007

Jolicoeur, P., and Dell’Acqua, R. (1998). The demonstration of short-term consolidation. Cogn. Psychol. 36, 138–202. doi: 10.1006/cogp.1998.0684

Kamiński, J., Brzezicka, A., and Wrobel, A. (2011). Short-term memory capacity (7 ± 2) predicted by theta to gamma cycle length ratio. Neurobiol. Learn. Mem. 95, 19–23. doi: 10.1016/j.nlm.2010.10.001

Klimesch, W. (1999). EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain Res. Brain Res. Rev. 29, 169–195. doi: 10.1016/s0165-0173(98)00056-3

Lisman, J. E., and Idiart, M. A. (1995). Storage of 7 +/−2 short–term memories in oscillatory subcycles. Science 267, 1512–1515. doi: 10.1126/science.7878473

Lisman, J. E., and Jensen, O. (2013). The theta-gamma neural code. Neuron 77, 1002–1016. doi: 10.1016/j.neuron.2013.03.007

Logan, G. D. (1978). Attention in character classification: evidence for the automaticity of component stages. J. Exp. Psychol. Gen. 107, 32–63. doi: 10.1037/0096-3445.107.1.32

Majerus, S., Attout, L., D’Argembeau, A., Degueldre, C., Fias, W., Maquet, P., et al. (2012). Attention supports verbal short-term memory via competition between dorsal and ventral attention networks. Cereb. Cortex 22, 1086–1097. doi: 10.1093/cercor/bhr174

Meltzer, J. A., Zaveri, H. P., Goncharova, I. I., Distasio, M. M., Papademetris, X., Spencer, S. S., et al. (2008). Effects of working memory load on oscillatory power in human intracranial EEG. Cereb. Cortex 18, 1843–1855. doi: 10.1093/cercor/bhm213

Miller, G. A. (1956). The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychol. Rev. 63, 81–97. doi: 10.1037/h0043158

Pashler, H. (1994). Dual–task interference in simple tasks: data and theory. Psychol. Bull. 116, 220–244. doi: 10.1037//0033-2909.116.2.220

Peterson, L. R., and Peterson, M. J. (1959). Short-term retention of individual verbal items. J. Exp. Psychol. 58, 193–198. doi: 10.1037/h0049234

Puckett, J. M., and Kausler, D. H. (1984). Individual differences and models of memory span: a role for memory search rate? J. Exp. Psychol. Learn. Mem. Cogn. 10, 72–82. doi: 10.1037/0278-7393.10.1.72

Ravizza, S. M., Hazeltine, E., Ruiz, S., and Zhu, D. C. (2011). Left TPJ activity in verbal working memory: implications for storage- and senory-specific models of short term memory. Neuroimage 55, 1836–1846. doi: 10.1016/j.neuroimage.2010.12.021

Ricker, T. J., and Cowan, N. (2014). Differences in presentations methods in working memory procedures: a matter of working memory consolidation. J. Exp. Psychol. Learn. Mem. Cogn. 40, 417–428. doi: 10.1037/a0034301

Roux, F., Wibral, M., Mohr, H. M., Singer, W., and Uhlhaas, P. J. (2012). Gamma-band activity in human prefrontal cortex codes for the number of relevant items maintained in working memory. J. Neurosci. 32, 12411–12420. doi: 10.1523/jneurosci.0421-12.2012

Shulman, H. G., and Greenberg, S. N. (1971). Perceptual deficit due to division of attention between memory and perception. J. Exp. Psychol. 88, 171–176. doi: 10.1037/h0030879

Sternberg, S. (1966). High speed scanning in human memory. Science 153, 652–654. doi: 10.1126/science.153.3736.652

Sternberg, S. (1969a). Memory scanning: mental processes revealed by reaction-time experiments. Am. Sci. 4, 421–457.

Sternberg, S. (1969b). Two invariances in retrieval of contextual information from memory. Paper presented at the Eastern Psychological Association meeting, April 1969. Also, Bell Telephone Laboratories Memorandum MM69 1221–1229.

Stevanovski, B., and Jolicoeur, P. (2007). Visual short-term memory: central capacity limitations in short-term consolidation. Vis. Cogn. 15, 532–563. doi: 10.1080/13506280600871917

Tallon-Baudry, C., Bertrand, O., Peronnet, F., and Pernier, J. (1998). Induced gamma-band activity during the delay of a visual short-term memory task in humans. J. Neurosci. 18, 4244–4254.

Vergauwe, E., Camos, V., and Barrouillet, P. (2014). The effect of storage on processing: how is information maintained in working memory? J. Exp. Psychol. Learn. Mem. Cogn. doi: 10.1037/a0035779. [Epub ahead of print].

Welford, A. T. (1952). An apparatus for use in studying serial performance. Am. J. Psychol. 65, 91–97. doi: 10.2307/1418834

Conflict of Interest Statement: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Received: 10 December 2013; Accepted: 19 February 2014; Published online: 07 March 2014

Citation: Vergauwe E and Cowan N (2014) A common short-term memory retrieval rate may describe many cognitive procedures. Front. Hum. Neurosci. 8:126. doi: 10.3389/fnhum.2014.00126

Copyright © 2014 Vergauwe and Cowan. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

ORIGINAL RESEARCH ARTICLE
published: 17 February 2015
doi: 10.3389/fnhum.2015.00060

Perceptual salience affects the contents of working memory during free-recollection of objects from natural scenes

Tiziana Pedale1,2 and Valerio Santangelo2,3*

1Department of Psychology, Sapienza University of Rome, Rome, Italy

2Cognitive Neuroscience Group, Neuroimaging Laboratory, Santa Lucia Foundation, Rome, Italy

3Department of Philosophy, Social, Human and Educational Sciences, University of Perugia, Perugia, Italy

* Correspondence: Valerio Santangelo, Department of Philosophy, Social, Human and Educational Sciences, University of Perugia, Piazza G. Ermini, 1 06123 Perugia, Italy e-mail: valerio.santangelo@unipg.it

Edited by:
Jean-Philippe van Dijck, Ghent University, Belgium

Reviewed by:
Peter Klaver, University of Zurich, Switzerland
Michel Quak, Ghent University, Belgium

One of the most important issues in the study of cognition is to understand which are the factors determining internal representation of the external world. Previous literature has started to highlight the impact of low-level sensory features (indexed by saliency-maps) in driving attention selection, hence increasing the probability for objects presented in complex and natural scenes to be successfully encoded into working memory (WM) and then correctly remembered. Here we asked whether the probability of retrieving high-saliency objects modulates the overall contents of WM, by decreasing the probability of retrieving other, lower-saliency objects. We presented pictures of natural scenes for 4 s. After a retention period of 8 s, we asked participants to verbally report as many objects/details as possible of the previous scenes. We then computed how many times the objects located at either the peak of maximal or minimal saliency in the scene (as indexed by a saliency-map; Itti et al., 1998) were recollected by participants. Results showed that maximal-saliency objects were recollected more often and earlier in the stream of successfully reported items than minimal-saliency objects. This indicates that bottom-up sensory salience increases the recollection probability and facilitates the access to memory representation at retrieval, respectively. Moreover, recollection of the maximal- (but not the minimal-) saliency objects predicted the overall amount of successfully recollected objects: The higher the probability of having successfully reported the most-salient object in the scene, the lower the amount of recollected objects. These findings highlight that bottom-up sensory saliency modulates the current contents of WM during recollection of objects from natural scenes, most likely by reducing available resources to encode and then retrieve other (lower saliency) objects.

Keywords: visual, salience, working memory, capacity, free recollection, objects, natural scenes

INTRODUCTION

When we look at a complex scene for a small amount of time we will probably remember only some of the information that was included in the original scene. The possibility of remembering this information is strictly related to the chances of building an internal (memory) representation of the scene. Although internal representations are crucial for a number of high-level cognitive processes (e.g., Fuster, 2006), it is still not entirely clear why some objects in a scene have more chance than others to be stored in memory (see, for reviews, Gazzaley and Nobre, 2012; Kiyonaga and Egner, 2013). Previous literature provided evidence that highlight the key role played by low-level sensory features (i.e., line orientation, intensity contrast and color opponency, as indexed by saliency-maps; Itti et al., 1998) in biasing attention selection and working memory (WM) encoding (Stirk and Underwood, 2007; Fine and Minnery, 2009; Melcher and Piazza, 2011; Santangelo and Macaluso, 2013; Spotorno et al., 2013; see, for a recent review, Santangelo, 2015).

For instance, Fine and Minnery (2009) conducted a behavioral study in which they asked participants to remember the position of 3–5 target icons placed on a geographical map (encoding phase). After a retention interval, participants were asked to relocate the icons either on the map (50% of trials) or on a blank screen (50% of trials). Irrespective of the retrieval condition (map-on vs. map-off), Fine and Minnery found that the more salient an icon was (quantified using Itti et al., 1998, model), the more accurate subjects were in repositioning the icons. These findings provided initial evidence about the impact of low-level sensory features on the encoding of objects in WM. Consistent findings were also reported by Santangelo and Macaluso (2013) using a delayed match-to-sample task during viewing of natural scenes. During fMRI scanning, participants were presented with natural scenes for 4 s (encoding phase), which were followed by a retention interval of 8 s. After that, participants judged the location (same/different) of a target-object extracted from the initial scene. Santangelo and Macaluso found that retrieval accuracy increased along with object saliency at encoding, indicating that the probability of WM encoding was a function of sensory salience.

Overall, this literature consistently demonstrated that bottom-up sensory salience increases the probability of an object to be successfully selected, and then stored in memory. Interestingly, recent evidence suggests that the role of perceptual saliency might not only affect the storage of single objects (according to their specific saliency level), but the overall content of the WM representation. Melcher and Piazza (2011) reported a series of experiments in which they manipulated bottom-up sensory salience of simple stimuli. For the memory set, they presented displays including a variable number (i.e., a variable set size) of Gabor patches with different orientations for 200 ms. The saliency of one Gabor was manipulated by increasing its contrast and/or size. After a delay of 1000 ms a test Gabor was presented. Participants were asked to judge whether the orientation of the test Gabor was the same or different compared to the Gabor at the same location in the memory set. Melcher and Piazza found that memory performance for the most salient Gabor remained high, irrespective of increased set size, while memory performance dropped dramatically with set size when a non-salient item was tested. This finding was interpreted by Melcher and Piazza as evidence that the overall WM capacity was influenced by changes in the relative saliency of the items.

A similar conclusion was reached by Pooresmaeili et al. (2014). In each trial, they presented a tilted bar as a memory sample. Participants had to keep in mind the orientation of this bar for a following memory-based choice. In the next display, a bar with the same orientation as the sample bar and a bar with a different orientation were presented on either side of a central fixation point. On some trials, Pooresmaeili et al. manipulated the saliency of either the bar matching or not-matching with the sample bar (changing its color to red), while in the remaining trials all the bars were displayed in white (no salience condition). In Exp. 1, participants were asked to find the test bar that matched the orientation of the sample; while in Exp. 2, they had to find the non-matching bar. Pooresmaeili et al. reported that their participants chose a visually salient item more often when they looked for matching features and less often when they looked for a non-match, indicating that salient items are more likely to be identified as a match. Pooresmaeili et al. interpreted this finding in terms of capacity limitations during the test phase, in which the visually salient item is more likely to consume WM resources, with the effect to be erroneously identified as matching with the memory sample.

These studies provided intriguing evidence linking perceptual saliency to the modulation of available WM capacity. Here we further investigate this issue using more complex stimuli, i.e., pictures representing natural scenes. Natural scenes typically included multiple objects, which entail a high-level of stimulus competition during attention selection and access in memory (see, e.g., Henderson and Hollingworth, 1999; Henderson, 2003; Hollingworth, 2012). We therefore aim to assess whether WM capacity can be modulated by perceptual saliency (cf. Melcher and Piazza, 2011; Pooresmaeili et al., 2014) also when using complex and ecologically-valid material, i.e., complex and natural scenes. For this, we presented pictures of natural scenes for 4 s. After a retention period of 8 s, we asked participants to verbally report as many objects/details as possible of the previous scenes (i.e., a free recollection task; e.g., Standing, 1973). We then computed how many times the objects located at either the peak of maximal- or minimal-saliency in the scene (as indexed by a saliency-map; Itti et al., 1998) were recollected by participants. This procedure allowed us to compute two different indexes related to maximal- and minimal-saliency objects, namely “recollection probability” and “recollection position” (i.e., the probability of recollecting that object and its position in the stream of reported items, respectively). If the selection and storage of maximal-saliency objects is facilitated, we would expect higher recollection probability for maximal- compared to minimal-saliency objects. Similarly, if perceptual salience affects the access to scene representation, we would expect that maximal-saliency objects were recollected earlier than minimal-saliency objects.

These indexes (recollection probability and recollection position) were also used to assess the impact of bottom-up sensory saliency in affecting the contents of WM by means of two regression models, one for each saliency condition (maximal or minimal). The choice to use free recollection was motivated by the possibility of measuring WM capacity in a natural context, in terms of the “amount of recollected objects” within each scene. Accordingly, in the first regression model we assessed whether the probability of recollecting maximal-saliency objects (i.e., the recollection probability index) affected the contents of WM, i.e., the overall amount of information successfully reported by participants for each scene. We would expect that the higher the probability of encoding and then recollecting the maximal-saliency object, the more the decrease in the overall amount of reported information. This would indicate that bottom-up sensory saliency affects WM contents, with the storage of the most-salient object in the scene reducing the available resources to store and then recollect other—lower saliency—objects (cf. Melcher and Piazza, 2011). Within the same regression model we also assessed whether the position in which the maximal-saliency object was recollected (i.e., the recollection position index) affected the amount of successfully reported information. This would suggest that the impact of saliency on WM specifically arises during the access to the scene representation stored in WM: the earlier the maximal-saliency object is reported, the smaller the amount of recollected information, indicating that the access to the memory representation for the most-salient object in the scene decreases resource availability to report other—lower saliency—objects. By contrast, a null effect in this latter analysis (i.e., no impact of the recollection position index on the amount of recollected information) would be consistent with the notion that bottom-up saliency mainly affect the encoding—more than retrieval—of objects from natural scenes, in line with previous findings (Santangelo and Macaluso, 2013). Finally, the second regression model assessed the influence of recollection probability and recollection position on the amount of successfully reported information, but now specifically for minimal-saliency objects. We would expect no significant effects for this analysis, indicating that objects associated with low-levels of bottom-up saliency are not attentional capturing and then ineffective in modulating WM contents.

METHODS AND MATERIALS

PARTICIPANTS

Twenty healthy volunteers (9 males; mean age = 24.2 years, ranging from 21 to 34 years), students at the University of Perugia, participated in the study. They all gave written informed consent and were naïve to the main purpose of the study.

STIMULI AND TASK

The set of stimuli consisted of one hundred pictures depicting scenes of everyday life. These images were collected on the World Wide Web and had already been used by Santangelo and Macaluso (2013). The pictures included internal (e.g., a kitchen, a bathroom, etc.) and external scenes (e.g., a garden, a street, etc.), but no single-object photo or living things such as people or animals.

The task consisted in an encoding phase (4 s), a maintenance phase (8 s delay), and a recollection phase (time unlimited) (see Figure 1A). During the encoding phase, participants were presented with a picture, displayed at 18 × 12° of visual angle. Participants were required to memorize as many details as possible for later recollection. In fact, following the 8 s delay (blank screen), a display with the signal “start recollecting” was presented, and participants were asked to report verbally as many objects/details as possible of the previous scene. Participants were instructed to be as accurate as possible, taking all the time they needed (i.e., no time constraint in the recollection phase). When their recollection was over, participants pressed the space bar to move to the next trial. After an inter-trial interval of 1 s a new scene was presented. The order of trials was randomized across participants. Participants’ verbal responses were recorded with an external microphone and digitalized into .wav files.

FIGURE 1. (A) Diagram showing the sequence of events during one trial. The trial began with a picture presented for 4 s (encoding phase). A blank display was then shown for 8 s (delay phase), and was followed by a “start recollecting” signal. Participants had no time constraint to recollect as much objects/details as possible from the previous scene. When the recollection was over, participants pressed the space bar for the next trial. (B) Selection of target-objects corresponding to the point of maximal- (red line) or minimal-saliency (yellow line) in the scene.

DATA ANALYSIS

Each picture has been analyzed with the Saliency Toolbox 2.2,[1] which computes saliency maps using local discontinuities in line orientation, intensity contrast, and color opponency (Itti et al., 1998). Using the saliency map, we designated for each picture two “target” objects, corresponding either to the point of maximal-salience of the scene (i.e., the maximal-saliency object) or to the point of minimal-salience of the scene (i.e., the minimal-saliency object; see also Figure 1B). To avoid any ambiguity in selecting the maximal- or minimal-saliency objects within each scene, we excluded those objects (typically, large objects) located over more than one peak of saliency. In fact, it would be unclear in this case which value of saliency should be assigned to that object. This procedure therefore allowed us to be more confident about the contribution of saliency on object memory, computing retrieval performance (see below) associated with clear levels of saliency. Twenty-nine pictures of the initial set were excluded from further analyses, because it was impossible to select within these scenes objects located over one single peak of either maximal or minimal saliency. Importantly, in the final set of pictures there was a significant difference between the average saliency score for maximal- (2.36) and minimal-saliency (0.27) objects (t(70), p < 0.001). As a final constraint, we made sure that the size of target-objects did not significantly differ between maximal- and minimal-salience conditions (t(70); p = 0.526).

Participants’ verbal responses were tabulated into a datasheet. Objects in a scene were coded as successfully recollected only when correctly named. When the scene included a number of similar objects (e.g., several “cups” of different colors), an object was assigned as successfully recollected only when it was possible to establish univocally object/name relation (e.g., the recollection of a “green” or a “red” cup). Among the objects recollected by each participant within each scene we searched for the target-object, designated according to maximal- vs. minimal-level of perceptual saliency (see above).

For each scene and for each participant we computed whether the target-objects (at maximal- or minimal-saliency locations) were successfully recollected, and, if this was the case, what were their positions in the stream of recollected items. This procedure allowed us to compute the mean recollection probability (see Figure 2A) and the mean recollection position (see Figure 2B) of maximal- and minimal-saliency objects. The target-object position in the stream of recollected objects was scaled by the total amount of objects recollected for that scene by that participant (i.e., recollection position index = target-object position / total amount of recollected objects). This weighting procedure allowed us to compare more accurately the meaning of the different target positions among them: for instance, a target-object position of four when twelve objects were recollected has an entirely different meaning compared to when only four objects were recollected, i.e., among the first positions or the last position, respectively. This index varied between 0 and 1: the closer it was to 0, the more the target-object was recollected among the first positions; by contrast, the closer it was to 1, the more the target-object was recollected among the last positions.

FIGURE 2. (A) Bar graph showing a higher probability to recollect objects corresponding at the location of maximal (red bar) as compared to minimal (yellow bar) saliency in the scene. (B) Average position in the streams of recollected objects, indicating that objects corresponding at locations of maximal saliency were reported earlier than objects at minimal saliency. In both graphs, the error bars represents the standard error of the mean.

Finally, we computed the amount of recollected objects for each scene by each participant when either the maximal- or minimal-saliency object was successfully recollected (i.e., when the target-object was part of the internal memory representation of the scene). This amount was now scaled by the average amount recollected by all participants in that scene (i.e., recollected amount = number of objects recollected by that participant/average amount of objects recollected by all participants). Again, this weighting procedure allowed us to compare more accurately the meaning of each amount among them: for instance, to recollect six objects in a scene in which the whole group recollected an average of twelve objects is a poor performance, but recollecting six objects when the group recollected an average of five is an excellent performance. The closer this index was to 1, the closer the single subject performance was to the group average; the more this index was distant from 1, the more the performance was distant to the group average (i.e., poorer performance <1; better performance >1). Averaging across participants, we obtained the mean scaled amount of recollected objects for each single scene.

To assess the impact of perceptual saliency on WM contents we used two regression analyses, one for each saliency condition (maximal or minimal). Before the analysis we made sure that our data did not violate the assumption of homoscedasticity. In line with our predictions, we expected an effect only for the regression model related to the maximal-saliency condition, indicating that a high-level of bottom-up sensory salience predicts the overall amount of successfully recollected information. More specifically, the first regression analysis assessed whether the probability of having or not having recollected the maximal-saliency object (recollection probability index) and the access at retrieval to the stored representation (recollection position index) predicted the contents of WM, i.e., the scaled amount of recollected objects for that given scene. In this regression model we used the recollection probability and the recollection position as predictors, and the scaled amount of recollected objects as dependent variable. Importantly, this approach (i.e., using two predictors instead of carrying out separate regression models) has the advantage of estimating the particular influence of each predictor while controlling for the influence of the other predictors at the same time. The second regression model was analogous to the first model, but now including the indexes related to minimal-saliency objects (again, the recollection probability and position as predictors, and the scaled amount of recollected objects as dependent variable). The data were analyzed with SPSS 13.0 (Statistical Package for Social Science, SPSS Inc.).

RESULTS

Overall, participants reported a mean of 5.0 objects across the scenes, with marked differences related to the recollection probability of maximal- and minimal-saliency objects, as highlighted in Figure 2A. A two-tailed paired-samples t-test revealed a significant difference for our participants in the probability of recollecting objects according to their saliency level (t(19) = 15.4; p < 0.001), with maximal-saliency objects (0.66 ± 0.02) reported far more frequently than minimal-saliency objects (0.43 ± 0.02). Next, we analyzed whether perceptual saliency affected the position in which maximal- vs. minimal-saliency objects were recollected. As highlighted in Figure 2B, maximal-saliency objects (0.55 ± 0.01) were recollected earlier than minimal-saliency objects (0.65 ± 0.01; t(19) = −6.8; p < 0.001), indicating that maximal-salient objects are prioritized during the recollection phase.

The impact of perceptual saliency on free recollection of objects from natural scenes was further investigated by two regression analyses. These were used to establish whether increasing bottom-up sensory saliency at encoding (indexed by the probability to have successfully recollected maximal-saliency objects) and the specific access at retrieval to the stored representation of the scene (indexed by the recollection position) predicted the overall amount of successfully recollected objects (i.e., WM contents; see Melcher and Piazza, 2011). We found that, the current amount of successfully recollected information was modulated by perceptual saliency. The first regression model was significant (F(2,70) = 15.9, p < 0.001, R^2 = 0.319), and revealed a significant effect of the recollection probability index (β = −0.512, t = −4.7, p < 0.001; see Figure 3A) on the amount of recollected objects, but no effect of the recollection position index (β = 0.113, t = 1.0, p = 0.302; see Figure 3B).[2] Then, the higher the probability of recollecting the maximal-salient object, the more the decrease in the amount of successfully recollected information. By contrast, the recollection position of the maximal-salience objects did not significantly predict the amount of successfully recollected information, indicating that the specific access to scene representation at retrieval did not affect the current contents of WM. The second regression model was instead not significant (F(2,70) < 1, n.s., R^2 = 0.015), indicating that neither the recollection probability (β = −0.126, t = −1.0, p = 0.329; see Figure 3C) nor the recollection position (β = −0.017, t = −0.1, p = 0.893; see Figure 3D) of minimal-saliency objects significantly predicted the amount of successfully recollected information.

FIGURE 3. Scaled amount (N) of recollected objects when the maximal (panels A and B) or the minimal-saliency target (panels C and D) was successfully reported as a function of either the recollection probability (panels A and C) or the recollection position (B and D) of the target. Overall, these graphs indicate that the probability to have successfully recollected the maximal-saliency (MaxSal) object reduced the overall amount of recollected objects (cf. panel A).

DISCUSSION

The current study aimed to investigate whether low-level sensory features (i.e., bottom-up saliency) affected the probability of objects to be recollected from natural scenes, and, if so, whether the probability of recollecting maximal-saliency objects modulated the overall contents of WM. We presented pictures of natural scenes involving high-levels of competition among to-be-remembered objects. After an encoding phase of 4 s, and a retention phase of 8 s, we asked participants to verbally report as many objects as they could remember of the previous scene (i.e., a free-recollection task). It is worth noting here that free-recollection may suffer from potential limitations due to the involvement of other high-level cognitive functions, such as language (i.e., the requirement to “verbally” report the remembered objects). For instance, during the recollection phase a participant might fail to recall the “verbal label” (i.e., the name) corresponding to a specific object, with a consequent failure in verbally reporting that object, even though this was successfully stored into the internal memory representation. Crucially, however, here we did not make any specific assumption related to objects that were not recollected; by contrast, our analyses involved only those objects that were successfully recollected. Although we cannot assess whether objects that are not recollected are stored in memory or not, we must assume that a memory representation of the recollected objects exists. We therefore believe that the information related to successfully recollected objects in the current paradigm is reliable and can inform us about internal memory representation (or WM content) of those objects in the scene.

The current findings revealed that the probability of recollecting objects varied dramatically according to their saliency level at encoding (i.e., during scene viewing), with maximal-saliency objects reported far more often than minimal-saliency objects. We also found that the access to scene representation was facilitated for maximal-saliency objects, which were reported earlier in the stream of recollected items compared to minimal-saliency objects. Finally, we found that the probability to recollect maximal- (but not minimal-) saliency objects significantly predicted the overall amount of successfully recollected objects: the higher the probability of recollecting the maximal-saliency object, the lower the amount of recollected objects. Importantly, this effect was not significantly modulated by the current position of the target-object in the stream of reported items, indicating that this effect did not arise during access to the stored representation of the scene.

The increased memory performance (i.e., the recollection probability index) for maximal- vs. minimal-saliency objects is in line with the previous literature (Fine and Minnery, 2009; Santangelo and Macaluso, 2013). This effect is in agreement with the notion that bottom-up attention can modulate short-term memory, by increasing the likelihood of attentional “grabbing” items to be remembered later on (see, e.g., Schmidt et al., 2002; Botta et al., 2010). However, it is worth noting that here we use a more demanding WM task as compared to the previous literature (i.e., a free recollection task). Fine and Minnery (2009) used a task requiring a low-level of competition among the possible objects/targets, consisting on the encoding of only 3–5 items in each trial (i.e., not overloading WM capacity; Luck and Vogel, 2013). Santangelo and Macaluso (2013) used instead a task requiring a higher-lever of competition at encoding, presenting pictures of natural scenes (actually, the same as those used here), including a number of possible memory targets in each scene (i.e., a supra-span condition). However, Santangelo and Macaluso used at retrieval a visuo-spatial recognition test, presenting as memory target an object cut-out from the previous scene in the same or in a different position. This may have elicited responses simply based on a sense of “familiarity” with the scene. Here we use a more demanding WM task compared to this previous literature, that is a free recollection task (Craik and McDowd, 1987). As in all free recollection tasks (e.g., Lieberman and Culpepper, 1965), participants had no hints about the original scene (or—more generally—about the studied material), and they can only report what they had successfully encoded during scene viewing. The current finding therefore highlights that the saliency effect on memory performance is robust, revealing a prioritization on internal memory representation of maximal-saliency objects, over and above any sense of familiarity with the scene.

Bottom-up saliency not only increases the probability for an object to be recollected, but also speeds-up the access to the stored (memory) representation during the recollection phase. In fact, we found that maximal-saliency objects were recollected earlier than minimal-saliency objects in the stream of reported items. This prioritization effect at the retrieval phase is in line with recent findings reported by Pooresmaeili et al. (2014): they used a different paradigm wherein the salient/non-salient comparison was made at memory retrieval. Here we did not emphasize in any way one object above the others at encoding or retrieval (i.e., we used a “free” recollection task). Notwithstanding that, participants recollected the maximal-saliency object earlier than the minimal-saliency object. This might be interpreted in terms of a facilitated access (or a “prior entry”; see Spence and Parise, 2010) in the stored representation of the scene for objects located—during the encoding phase—at peaks of maximal-saliency.

Finally, the current findings highlight the role of perceptual saliency in affecting the overall number of objects successfully recollected from natural scenes. This finding might be interpreted in the light of the previous literature (Melcher and Piazza, 2011; see also Pooresmaeili et al., 2014), showing that bottom-up saliency affects the availability of WM resources, thus influencing its capacity. Accordingly, here we found that the higher the probability of reporting the most-salient object in the scene, the lower the overall amount of information successfully recollected in that scene. Although maximal-saliency objects are recollected on average earlier than minimal-saliency objects (see Discussion above and Figure 2B), the specific position of the maximal-saliency object in the stream of reported items did not affect the overall amount of recollected information (cf. the first regression model; see also red line in Figure 3B). In other words, the reduction in WM capacity was not modulated by the position of the target-object in the stream of recollected objects at retrieval. Although the interpretation of null effects has to be always very cautious, this finding (deserving further assessment in future research) seems to indicate that the decrease in the overall amount of successfully reported information did not arise during the attempt to access the information related to the target-object in the internal representation of the scene (see Pooresmaeili et al., 2014). By contrast, we suggest that during scene viewing (i.e., the encoding phase) the more an object is efficient to grab participants’ attention resources (according to its saliency level; see, e.g., Nardo et al., 2011, 2014), the less spared resources would be available to process other, lower-saliency, objects in the scene.

We acknowledge that the current task was not specifically designed to address the issue of whether the impact of perceptual saliency on WM contents arise at encoding or retrieval. In fact, we only collected WM performance at retrieval, without measuring any behavioral and/or physiological parameter during the encoding phase. Notwithstanding that, we note that our interpretation might be in good agreement with several models postulating an assignment of “attentional priorities” under conditions of high-levels of conflict/competitions among the stimuli (see, e.g., Desimone and Duncan, 1995; Itti and Koch, 2001; Pessoa, 2009). The assignment of attentional priorities might directly affect short-term memory representation (e.g., Bundesen, 1990). According to Bundesen et al. (2005, 2011), attention selection mechanisms would directly change the number of cortical neurons used to represent a given object, with the number of neurons increasing as a function of the task-relevance of the object itself. As a consequence, behaviorally important objects would have a high probability of winning the competition to be encoded and thus accessing an internal representation through the short-term memory system. The latter is conceived as a feedback mechanism that sustains activity in the neurons that have won the attentional selection/competition (see also Cowan, 1995, 2011, for a similar notion). Here we used a task in which all objects in the scene were equally task-relevant. In fact, we asked participants to freely report all objects they could remember. Notwithstanding that, we showed that objects corresponding to the point of maximal (vs. minimal) saliency in the scene were recollected with higher probability (reducing at the same time the overall amount of information successfully reported). This is consistent with the notion that visual saliency plays a key role in assigning attentional priorities (see, for reviews, Thompson and Bichot, 2005; Gottlieb, 2007). Speculatively, we interpret our findings within the framework of Bundesen et al.’s theory: the most-salient object would receive attentional resources and then encoding priority; this would lead to a higher recollection probability for the most-salient objects, but also to less attention resources for the other (lower saliency) objects in the scene, consistent with the reduction in the overall amount of information successfully recollected (cf. Figure 3A).

To conclude, the current study provided initial evidence that the processing of a maximal-saliency object in a natural scene is prioritized during formation of objects/scene memory traces and during later access to this stored representation. We found a reduction of the overall amount of successfully recollected information when maximal-salience objects entered the internal memory representation, thus having a higher chance to be recollected later on. We interpreted such a reduction as evidence that high-level perceptual saliency tends to exhaust attentional resources during the exploration of a natural and complex scene.

ACKNOWLEDGMENTS

We thank Alessandra Monti for data collection and data tabulation. We also thank Dr. Simon Grimble for language revision.

FOOTNOTES

1^http://www.saliencytoolbox.net/

2^Following the inspection of Figure 3A, one might argue that this regression analysis could be affected by an outlier data point (cf. the red dot in the upper-left corner). However, the regression model was still significant after removing this data point (F(2,69) = 12.4, p < 0.001, R = 0.269), showing again a significant effect of the recollection probability index (β = −0.506, t = −4.5, p < 0.001) on the amount of recollected objects, but no effect of the recollection position index (β = 0.036, t = 0.3, p = 0.748).

REFERENCES

Botta, F., Santangelo, V., Raffone, A., Olivetti Belardinelli, M., and Lupianez, J. (2010). Exogenous and endogenous spatial attention effects on visuo-spatial working memory. Q. J. Exp. Psychol. (Hove) 63, 1590–1602. doi: 10.1080/17470210903443836

Bundesen, C. (1990). A theory of visual attention. Psychol. Rev. 97, 523–547. doi: 10.1037/0033-295X.97.4.523

Bundesen, C., Habekost, T., and Kyllingsbæk, S. (2005). A neural theory of visual attention: bridging cognition and neurophysiology. Psychol. Rev. 112, 291–328. doi: 10.1037/0033-295x.112.2.291

Bundesen, C., Habekost, T., and Kyllingsbæk, S. (2011). A neural theory of visual attention and short-term memory (NTVA). Neuropsychologia 49, 1446–1457. doi: 10.1016/j.neuropsychologia.2010.12.006

Cowan, N. (1995). Attention and Memory: An integrated framework. Oxford Psychology Series No. 26. New York: Oxford University Press.

Cowan, N. (2011). The focus of attention as observed in visual working memory tasks: making sense of competing claims. Neuropsychologia 49, 1401–1406. doi: 10.1016/j.neuropsychologia.2011.01.035

Craik, F. I., and McDowd, J. M. (1987). Age differences in recall and recognition. J. Exp. Psychol. Learn. Mem. Cogn. 13, 474–479. doi: 10.1037/0278-7393.13.3.474

Desimone, R., and Duncan, J. (1995). Neural mechanisms of selective visual attention. Annu. Rev. Neurosci. 18, 193–222. doi: 10.1146/annurev.neuro.18.1.193

Fine, M. S., and Minnery, B. S. (2009). Visual salience affects performance in a working memory task. J. Neurosci. 29, 8016–8021. doi: 10.1523/jneurosci.5503-08.2009

Fuster, J. M. (2006). The cognit: a network model of cortical representation. Int. J. Psychophysiol. 60, 125–132. doi: 10.1016/j.ijpsycho.2005.12.015

Gazzaley, A., and Nobre, A. C. (2012). Top-down modulation: bridging selective attention and working memory. Trends Cogn. Sci. 16, 129–135. doi: 10.1016/j.tics.2011.11.014

Gottlieb, J. (2007). From thought to action: the parietal cortex as a bridge between perception, action and cognition. Neuron. 53, 9–16. doi: 10.1016/j.neuron.2006.12.009

Henderson, J. M. (2003). Human gaze control during real-world scene perception. Trends Cogn. Sci. 7, 498–504. doi: 10.1016/j.tics.2003.09.006

Henderson, J. M., and Hollingworth, A. (1999). High-level scene perception. Annu. Rev. Psychol. 50, 243–271. doi: 10.1146/annurev.psych.50.1.243

Hollingworth, A. (2012). Guidance of visual search by memory and knowledge. Nebr. Symp. Motiv. 59, 63–89. doi: 10.1007/978-1-4614-4794-8_4

Itti, L., and Koch, C. (2001). Computational modelling of visual attention. Nat. Rev. Neurosci. 2, 194–203. doi: 10.1038/35058500

Itti, L., Koch, C., and Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20, 1254–1259. doi: 10.1109/34.730558

Kiyonaga, A., and Egner, T. (2013). Working memory as internal attention: toward an integrative account of internal and external selection processes. Psychon. Bull. Rev. 20, 228–242. doi: 10.3758/s13423-012-0359-y

Lieberman, L. R., and Culpepper, J. T. (1965). Words versus objects: comparison of free verbal recall. Psychol. Reports. 17, 983–988. doi: 10.2466/pr0.1965.17.3.983

Luck, S. J., and Vogel, E. K. (2013). Visual working memory capacity: from psychophysics and neurobiology to individual differences. Trends Cogn. Sci. 17, 391–400. doi: 10.1016/j.tics.2013.06.006

Melcher, D., and Piazza, M. (2011). The role of attentional priority and saliency in determining capacity limits in enumeration and visual working memory. PLoS One 6:e29296. doi: 10.1371/journal.pone.0029296

Nardo, D., Santangelo, V., and Macaluso, E. (2011). Stimulus-driven orienting of visuo-spatial attention in complex dynamic environments. Neuron. 69, 1015–1028. doi: 10.1016/j.neuron.2011.02.020

Nardo, D., Santangelo, V., and Macaluso, E. (2014). Spatial orienting in complex audiovisual environments. Hum. Brain Mapp. 35, 1597–1614. doi: 10.1002/hbm.22276

Pessoa, L. (2009). How do emotion and motivation direct executive control? Trends Cogn. Sci. 13, 160–166. doi: 10.1016/j.tics.2009.01.006

Pooresmaeili, A., Bach, D. R., and Dolan, R. J. (2014). The effect of visual salience on memory-based choices. J. Neurophysiol. 111, 481–487. doi: 10.1152/jn.00068.2013

Santangelo, V. (2015). Forced to remember: when memory is biased by salient information. Behav. Brain Res. 283, 1–10. doi: 10.1016/j.bbr.2015.01.013.

Santangelo, V., and Macaluso, E. (2013). Visual salience improves spatial working memory via enhanced parieto-temporal functional connectivity. J. Neurosci. 33, 4110–4117. doi: 10.1523/jneurosci.4138-12.2013

Schmidt, B. K., Vogel, E. K., Woodman, G. F., and Luck, S. J. (2002). Voluntary and automatic attentional control of visual working memory. Percept. Psychophys. 64, 754–763. doi: 10.3758/bf03194742

Spence, C., and Parise, C. (2010). Prior-entry: a review. Conscious. Cogn. 19, 364–379. doi: 10.1016/j.concog.2009.12.001

Spotorno, S., Tatler, B. W., and Faure, S. (2013). Semantic versus perceptual salience in visual scenes: findings from change detection. Acta Psychol. 142, 168–176. doi: 10.1016/j.actpsy.2012.12.009

Standing, L. (1973). Learning 10000 pictures. Q. J Exp. Psychol. 25, 207–222. doi: 10.1080/14640747308400340

Stirk, J. A., and Underwood, G. (2007). Low-level visual saliency does not predict change detection in natural scenes. J. Vis. 7, 3.1–3.10. doi: 10.1167/7.10.3

Thompson, K. G., and Bichot, N. P. (2005). A visual salience map in the primate frontal eye field. Prog. Brain Res. 147, 251–262. doi: 10.1016/s0079-6123(04)47019-8

Conflict of Interest Statement: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Received: 30 July 2014; Accepted: 23 January 2015; Published online: 17 February 2015.

Citation: Pedale T and Santangelo V (2015) Perceptual salience affects the contents of working memory during free-recollection of objects from natural scenes. Front. Hum. Neurosci. 9:60. doi: 10.3389/fnhum.2015.00060

This article was submitted to the journal Frontiers in Human Neuroscience.

Copyright © 2015 Pedale and Santangelo. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

ORIGINAL RESEARCH ARTICLE
published: 13 January 2015
doi: 10.3389/fnhum.2014.01053

The time course of protecting a visual memory representation from perceptual interference

Dirk van Moorselaar*, Eren Gunseli, Jan Theeuwes and Christian N. L. Olivers

Department of Cognitive Psychology, VU University, Amsterdam, Netherlands

* Correspondence: Dirk van Moorselaar, Department of Cognitive Psychology, VU University, Van der Boechorststraat 1, 1081 BT Amsterdam, Netherlands e-mail: d.van.moorselaarl@vu.nl

Edited by:
Elger Abrahamse, Ghent University, Belgium

Reviewed by:
Mark Nieuwenstein, University of Groningen, Netherlands
Anastasia Kiyonaga, Duke University, USA
Alessandra S. Souza, University of Zürich, Switzerland

Cueing a remembered item during the delay of a visual memory task leads to enhanced recall of the cued item compared to when an item is not cued. This cueing benefit has been proposed to reflect attention within visual memory being shifted from a distributed mode to a focused mode, thus protecting the cued item against perceptual interference. Here we investigated the dynamics of building up this mnemonic protection against visual interference by systematically varying the stimulus onset asynchrony (SOA) between cue onset and a subsequent visual mask in an orientation memory task. Experiment 1 showed that a cue counteracted the deteriorating effect of pattern masks. Experiment 2 demonstrated that building up this protection is a continuous process that is completed in approximately half a second after cue onset. The similarities between shifting attention in perceptual and remembered space are discussed.

Keywords: visual working memory, attention, protection, retro-cue, time course

INTRODUCTION

Visual working memory and visual attention are separate constructs, but there is a functional overlap in their mechanisms (Awh and Jonides, 2001; Awh et al., 2006). A recent line of research involving spatial cues during the delay of visual memory tasks has demonstrated that focused attention can operate on mnemonic representations during maintenance, such that memory performance improves for cued items (Griffin and Nobre, 2003; Landman et al., 2003; Makovski and Jiang, 2007; Makovski et al., 2008; Sligte et al., 2008; Delvenne et al., 2010; Berryhill et al., 2012). The sensitivity of visual memory to attentional cues indicates that the mnemonic representations are not static, but that they can be modulated by top-down selective mechanisms.

Different mechanisms have been proposed to account for the cueing benefit. As the cue is presented after stimulus offset and there is thus no perceptual representation to enhance, Matsukura et al. (2007) proposed that cue effects are generated by a selective attention mechanism that protects the cued representation from degradation processes such as passive decay and inter-item interference. Consistent with such a protective mechanism, a cue has been found to enhance robustness to subsequent visual input from the test display (Landman et al., 2003; Makovski et al., 2008; Pertzov et al., 2013), or passively viewed images (Makovski and Jiang, 2007). It has also been argued that focused attention strengthens the binding between the content and the context of the cued representation (Kuo et al., 2011; Rerko and Oberauer, 2013). This increased binding then in turn facilitates retrieval of the content during memory test (which acts as the appropriate context).

Strengthening and protectionist mechanisms do not need to be mutually exclusive, and both types of account predict that cued items are less vulnerable to perceptual interference. The literature, however, has provided conflicting results on the effects of perceptual interference on memory performance, and how focused attention can counteract these effects. Whereas some have shown that distractions presented during maintenance impair memory performance, especially when they are of the same category as the memoranda (Dolcos et al., 2007; Zhang and Luck, 2008; Clapp et al., 2010), others have found that memory representations are insensitive to the effects of intervening masks (Irwin and Thomas, 2008; Pinto et al., 2013). Moreover, tests on the ability of a cue to counteract the effects of interference during maintenance have provided mixed results. Makovski and Jiang (2007) showed equal performance on cue trials with and without interference from passively viewed irrelevant stimuli presented between cue and test display. Pinto et al. (2013), however, found equal performance on cue trials with and without interference only when the interference was either presented in a different location (i.e., different hemifield), or contained different objects than the memoranda. In contrast, when the interference was displayed at the same location and consisted of the same objects of the to-be remembered information, performance on trials with and without interference started to diverge. Similarly mixed results have been observed with designs that incorporated interference manipulations in the interval between cue and test presentation. Whereas neither Hollingworth and Maxcey-Richard (2013) nor Rerko et al. (2014) observed a significant modulation of the cue benefit by an intervening task, Janczyk and Berryhill (2014) found a significant reduction of the cue effect when attention was shifted to another task before memory test.

The present study served two purposes. The first was to assess whether perceptual interference causes a visual memory to become less precise and/or causes it to be lost, and whether a cue can counteract such effects (Experiment 1). So far, studies that have demonstrated interference effects have shown that interference leads to an overall decline in memory performance, without specifying the nature of that decline. Second, we sought to investigate the temporal dynamics of building the cue-based protection of visual memories against perceptual interference (Experiment 2). There have been only a few studies looking at the effects of post-cue timing on memory performance, but they did not manipulate the amount of interference. Notably, Tanoue and Berryhill (2012) found that recall accuracy for cued items improved with time relative to non-cued items, with reliable benefits emerging 300 ms after the cue. Also, in a study by Pertzov et al. (2013) the cueing effect started to differentiate performance after about 300 ms. These studies show increased protection over time against interference caused by the test display. Such interference might be expected to occur because the test display is by definition relevant to the task, and thus the test items are processed to similar levels as the memoranda. In the present study, we are specifically interested whether cueing a memory shields it from irrelevant perceptual interference, and what the time course of this process is—something that cannot be assessed from these previous studies looking at the dynamics of cueing, as their designs did not include conditions of irrelevant intervening interference. The time course of protection against perceptual interference from a mask has been investigated before (Gegenfurtner and Sperling, 1993). In that study, performance on masked trials increased until it reached an asymptote at around 300 ms, suggesting that some time was required to protect the cued item against perceptual interference. However, their design did not include baseline conditions without a cue or a mask, making it difficult to assess the direct effect of the interfering mask. Moreover, as the stimuli were letters, verbal rehearsal might have aided memory performance. Finally, the experiments employed a discrete report memory test, which does not allow for a distinction between precision reduction vs. loss of memory. In the present study, we used a continuous report procedure which is a more sensitive memory measure and also allows performing a model fit to differentiate between two different aspect of maintenance, recall probability and precision of representations.

The basic procedure is illustrated in Figure 1. In all conditions participants were instructed to remember the orientations of three objects for a subsequent memory test. One of the memory items could then be highlighted by an endogenous cue. Furthermore between the memory and the probe display (and after the cue, if present) a pattern mask could appear. We argued that the cue would shield the memory from the mask, given sufficient time to consolidate the memory. Systematic manipulation of both cue (cue vs. no-cue) and mask (mask vs. no-mask) with variable delays then allowed us to assess at what moment in time the cue starts to have an effect and the time required for visual memory performance to stabilize upon cueing. Covertly directing attention to a location in the visual space in response to a central cue typically steadily increases between 100 and 400 ms after cue presentation (see Egeth and Yantis, 1997 for a review). A similar time-course may be observed in remembered space. Therefore, we expected performance to slowly diverge over time on masked trials with and without a cue, until performance on cued trials was indistinguishable for masked and unmasked trials.

FIGURE 1. Experiment 1: (A) Sequence of events in a trial of Experiment 1. If a cue was present the base of a triangle inside fixation pointed towards one of the memory locations. (B–D) In Experiment 2 the no cue/no mask condition was omitted and the stimulus onset asynchrony (SOA) between cue display and mask display was varied (100, 200, 350, 600 ms). Note that in cue/no-mask condition the SOA is actually a dummy SOA as only the fixation circle was presented at the same time as the pattern masks were presented in the mask conditions. The same logic applies to the no-cue/mask condition, as in this condition the SOA starts at the moment of offset of the retro-cue in the cue conditions.

EXPERIMENT 1: A CUE PROTECTS A MEMORY REPRESENTATION AGAINST PATTERN MASKING

The present study was designed to assess the time-course of protecting the cued representation within visual memory against perceptual interference. Therefore, it was essential that the chosen measure of memory performance would be sensitive to the effects of cues over time. Recently, new methods have been developed to assess the quality of a memory representation, which are based on a continuous feature and response space instead of a binary measure (Wilken and Ma, 2004; Bays and Husain, 2008; Zhang and Luck, 2008). This method results in an estimate of the quality of recall and it has been proposed that this measure can be used to track the deployment of resources over time (Bays et al., 2011). There is some inconsistency in the literature, however, whether cues enhance only the probability of successfully recalling an item, or also its representational quality. Pertzov et al. (2013) used a continuous report task in a paradigm that required participants to recall the orientation of previously presented bars. On some trials, a probabilistic cue pointed to one of the bars. The results showed that the average angular deviation of the reported orientation from the true orientation of the target increased over time on trials without a cue, whereas performance stabilized following a valid cue. However, since they did not provide a model fit (due to an insufficient number of trials) it is unclear whether this decreased memory performance can be attributed to a decline in precision, an increase in random responses, or both. Williams et al. (2013) demonstrated that a cue leads to more precise representations as well as a lower probability of dropping the relevant information from memory (Note that these results were obtained with directed forgetting cues rather than cues to maintain an item). In contrast, Murray et al. (2013) observed enhanced probability that the target was maintained in memory following a cue, but no effect on recall precision. However, the precision calculation of Murray et al. (2013) was based on accuracy measures obtained across change detection performances in varying degrees of change. In other words, rather than reporting the feature value of the representation from a continuous scale, participants responded to a discrete set of values. Therefore, we argue that the precision estimation in Murray et al. (2013) may not have been as sensitive as continuous recall report in the studies that observed an effect of cues on precision, and this might account for the lack of such an effect (but see Souza et al., 2014a). Consequently, in Experiment 1, we used a continuous report procedure to assess whether this method would be sensitive enough to pick up differences in representational quality in the present paradigm.

In the visual domain the time course of encoding is usually studied with a masking procedure, which halts the encoding into visual memory by overwriting preceding input. It is unclear, however, whether the same procedure would be effective for studying the time course of protecting a visual memory representation, as the literature is inconclusive on the effects of masking visual memory representations. As noted before, there is evidence both in favor and against the notion that (cued) memory representations are sensitive to the effects of masking. Most of these studies, however, did not incorporate a continuous recall measure and it is thus possible that the conflicting results are due to a binary measure being only partially sensitive to the quality of representations (Awh et al., 2007). Indeed, Zhang and Luck (2008) used a continuous recall measure in a masking design and found detrimental effects of the mask. Importantly, these effects were only apparent in the probability of recall, but not in precision. Thus, Experiment 1 served to establish whether the chosen method would be sensitive enough to pick up detrimental effects of the mask, and importantly, whether such effects could be counteracted by a cue. Experiment 2 was then specifically designed to study the time-course of building this cue based protection.

Recent work has shown that increasing the proportion of invalid cue trials reduces the retro-cue benefit Gunseli et al. (in press). Similarly, using forgetting cues rather than remembering cues, Williams and Woodman (2012) found that when a cue is not 100% valid, participants may not fully focus on the cued representation. Therefore, we used 100% valid cues to ensure that participants had every reason to focus attention on the cued representation.

METHOD

Participants

Twenty-four (13 females), aged 19–30 (M = 24), participated in exchange for course credit or a payment of €8 per hour. Our initial sample size contained twelve participants. However, one of the twelve participants showed a large effect completely opposite to that of the other participants As the pattern of results was consistent across the remaining participants, and since there was no objective reason to remove this outlier, we decided to test twelve more participants. All participants had normal or corrected-to-normal acuity and gave informed consent according to procedures approved by theScientific and Ethical Review Committee of the faculty of psychology and education of the VU University.

Apparatus, stimuli, procedure and design

A HP Compaq 8000 Elite computer running OpenSesame version 27.3 generated the stimuli on an Liyama Vision Master Pro 454 120 Hz screen and acquired the response data through the standard mouse. Participants were placed in a dimly lit room at a viewing distance of 70 cm. All stimuli were presented on a gray background (17 cd/m^2) and a small black fixation dot (0.19°) surrounded by a white circle marked central fixation throughout each trial.

The task incorporated a cueing paradigm developed by Griffin and Nobre (2003) with a continuous response measure. Each trial started with a 500 ms fixation display. A memory display and a test display were presented sequentially, discontinued by a 1400 ms retention interval (Figure 1A). The memory display was presented for 350 ms and contained three black bars (1.62° * 0.19°) located on the corners of a triangle that subtended either 4.4° or 4.6° from fixation (memory locations switched trial by trial). The orientation of each rectangle was chosen at random with the restriction that bars within the same trial differed by at least 15°. The test display contained a randomly oriented bar at the center of fixation and a cue indicating which location was being probed. Subjects were to indicate the precise orientation of the bar at the probed location by adjusting the mouse position. After a mouse response was made, the correct orientation was indicated by a white bar for 100 ms.

In addition, we presented 100 ms cue displays, 400 ms after offset of the memory displays. If a cue was present (50% of trials), one third of the fixation dot was filled with white lines such that the base of a white triangle within fixation marked one of the memory locations (cue trials). In the other half of the trials no cue was presented such that the cue display was identical to the fixation display (no-cue trials). The cue displays were followed by 100-ms interfering displays presented 400 ms after offset of the cue-display (masked trials; 50 % of trials). These interfering displays contained three pattern masks, each comprised of 6 black bars (0°–180° in steps of 30°), identical to the memory objects, centered on the memory locations of that specific trial. Thus, in total 4 different conditions were presented (cue/no-cue [2] × mask/no-mask [2]). Trials in the no-cue/no-mask condition were split in half, such that the retention interval was either 900 or 1400 ms. This timing manipulation allowed us to estimate the content of visual memory both at the time the mask was presented and after the full delay.

All participants completed 48 practice trials and 8 experimental blocks of 48 trials each. Each block consisted of 12 no-cue/no-mask, 12 no-cue/mask, 12 cue/mask and 12 cue/no-mask trials, randomly mixed, such that participants completed 96 trials in each condition. At the end of each block, feedback was given on average response error in the last block and overall in the whole experiment. Participants were encouraged to take a break between blocks.

Analysis

For completeness, we report a number of often-used dependent measures for each combination of participant and condition. First, for each trial, a raw measure of error was obtained by simply calculating the absolute angular deviation between the true orientation of the target and the orientation reported by the participant. The circular standard deviation (sd) of this error distribution was then calculated, which was taken as an overall measure of the quality of the memory representation (smaller values represent a better representational quality). Next, the data was fitted with a mixture model (Bays et al., 2009). This model decomposes a response distribution into three components: (1) a distribution of responses around the target, also referred to as the probability of recall, (2) distribution of responses around the non-target; and (3) a random response. The model also returns the concentration parameter K of the Von Mises distribution describing the response variability around the target and the non-target representation, which is interpreted to reflect the precision of the memory item. Raw error, sd, probability and precision estimates were each entered in repeated-measures ANOVAs and follow-up analyses were done with paired two-tailed t tests.

RESULTS AND DISCUSSION

Results from Experiment 1 are displayed in Table 1 and Figure 2. Raw error scores were entered in a repeated-measures ANOVA, with within-subjects factors cue (no-cue, cue) and mask (no-mask, mask). Note, however, that the no-cue/no-mask condition consisted of a short and a long interval condition, which made a fully crossed analysis impossible, so we present the analyses for these intervals separately. In addition, as performance did not significantly differ for these two intervals, (t(23) = 1.672, p = 0.11), we also present all analyses collapsed across interval.

Table 1. Experiment 1: Data columns represent mean of average deviation, mean precision (standard deviation of the error in participants’ responses), mean probability of reporting the cued item, mean probability of reporting an uncued item, the mean probability of random responses, and the response variability as described by the concentration parameter K of the Von Mises distribution.

FIGURE 2. Experiment 1: (A) Mean average deviation (B) standard deviation of error distributions; and (C) probability of recall estimates as a function of cue and distractor-type. Data for no-cue/no-mask condition is shown separately for the short (black square) and the long interval (black circle). Error bars in all figures are condition specific, within subject 95% CI’s (Morey, 2008).

As seen in Figure 2A, recall performance was better after a cue (F(1, 23) = 39.027, p < 0.001 for the short interval; F(1, 23) = 49.406, p < 0.001 for the long interval; and F(1,23) = 46.081, p < 0.001 for the collapsed data), while recall performance was impaired after presentation of the mask (F(1, 23) = 51.283, p < 0.001 for the short interval; F(1, 23) = 39.487, p < 0.001 for the long interval; F(1,23) = 53.563, p < 0.001 for the collapsed data). Important for the present purposes, the deteriorating effect of the mask was modulated by the cue, as illustrated by a significant cue × mask interaction at the short interval (F(1, 23) = 10.037, p = 0.004) and a close to significant interaction at the long interval (F(1, 23) = 4.015, p = 0.057). When the data were collapsed this interaction was significant (F(1,23) = 7.136, p < 0.02).

Without interference, recall performance improved following a cue relative to trials without a cue, at both long (t(23) = 5.327, p < 0.001) and short intervals (t(23) = 4.220, p < 0.001). The finding that mean deviation was larger in the short no-cue/no mask condition than in the cue/no-mask condition indicates that the effect of the cue cannot solely be attributed to a reduction in the duration of the retention interval. The mask also significantly affected recall performance. Without a cue, performance was worse following the mask, both at the short (t(23) = 5.856, p < 0.001) and the long intervals (t(23) = 4.337, p < 0.001). Importantly, as illustrated by the interaction, this deteriorating effect of the mask was modulated by the cue. Following the cue, recall performance was still impaired by the mask (t(23) = 2.469, p = 0.021), but the deteriorating effect of the mask (i.e., difference in mean deviation between mask and no-mask conditions) was significantly smaller on cue than on no-cue trials. Finally, mean deviation in the cue/mask condition was smaller than in the no-cue/no-mask condition (t(23) = 1.868, p = 0.075 for short interval; t(23) = 3.136, p = 0.005 for long interval).

To further investigate the source of these effects, first we analyzed the overall representational quality (i.e., sd of the error distribution) across conditions. As seen in Figure 2B, recall was better following a cue (F(1,23) = 48.440, p < 0.001 for the short interval; F(1,23) = 5.437, p = 0.029 for the long interval; F(1,23) = 58.394, p < 0.001 for the collapsed data) and it decreased after a mask (F(1,23) = 49.970, p < 0.001 for the short interval; F(1,23) = 6.911, p = 0.015 for the long interval; F(1,23) = 47.147, p < 0.001 for the collapsed data). As was also apparent from the mean deviations, masking impaired recall performance also after a cue (t(23) = 2.327, p = 0.029), but this effect of the mask was significantly smaller on cue than on no-cue trials (F(1, 23) = 12.339, p = 0.002 for the short interval; (F(1, 23) = 5.926, p = 0.023 for the long interval; F(1,23) = 9.610, p = 0.005 for the collapsed data).

Next we fitted the mixture model. Figure 2C shows the probability of recall for the probed item (i.e., the target). Consistent with the raw error and the sd analysis, in masked trials participants showed a decreased probability of reporting the target orientation (F(1, 23) = 5.720, p = 0.025 for the short interval; F(1, 23) = 8.109, p = 0.009 for the long interval; F(1,23) = 7.901, p = 0.01 for the collapsed data) and an increased probability of reporting the target orientation when it was cued (F(1, 23) = 17.626, p < 0.001 for the short interval; F(1, 23) = 19.674, p < 0.001 for the long interval; F(1,23) = 20.229, p < 0.001 for the collapsed data). Importantly, these main effects were accompanied by a trend towards an interaction (F(1, 23) = 3.284, p = 0.083 for the short interval; F(1, 23) = 3.495, p = 0.074 for the long interval; F(1,23) = 3.914, p = 0.06 for the collapsed data), reflecting a significant effect of the mask without a cue (t(23) = 2.233, p = 0.036 for the short interval; t(23) = 2.533, p = 0.019 for the long interval; t(23) = 2.591, p = 0.016 for the collapsed data), but no such effect when the mask was preceded by a cue (t = 0.776, p = 0.45).

The cue and mask effects on the probability of recall estimates were accompanied by a modulation of random responses. Random responses decreased following a cue (F(1, 23) = 11.414, p = 0.003 for the short interval; F(1, 23) = 11.379, p = 0.003 for the long interval; F(1,23) = 12.244, p = 0.002 for the collapsed data), and they increased following a mask (F(1, 23) = 5.398, p = 0.029 for the short interval; F(1, 23) = 8.549, p = 0.008 for the long interval; F(1,23) = 8.286, p = 0.008 for the collapsed data). However, the interaction failed to reach significance at the short interval (F = 1.350, p = 0.26) and was close to significance at the long interval (F(1,23) = 3.553, p = 0.072). Also, for the collapsed data there was no significant interaction (F(1,23) = 2.516, p = 0.126). In contrast, no such effects were observed in the probability of reporting an uncued item. Although it deserves noting that at the long retention interval there was a smaller probability of misreporting the wrong item in memory following a cue, a difference close to significance (F = 3.973, p = 0.058; all other F’s < 1.384, all other p’s > 0.251).

Finally, the overall cueing benefit and the overall deteriorating effect of the mask were also apparent in the precision estimates of the model output. Cueing resulted in more precise representations (F(1,23) = 7.610, p = 0.011 for the short interval; F(1,23) = 5.437, p = 0.029 for the long interval; F(1,23) = 7.261, p = 0.013 for the collapsed data) and masking resulted in less precise representations (F(1,23) = 7.617, p = 0.011 for the short interval; F(1,23) = 6.911, p = 0.015 for the long interval; F(1,23) = 8.798, p < 0.01 for the collapsed data). Interestingly, the cue × mask interaction that was present in all previous analyses, was completely absent for precision (all F’s < 0.026, all p’s > 0.87), indicating that a mask impaired the precision of a memory representation to the same extent with and without a cue.

Experiment 1 was conducted to assess whether or not memory representations are sensitive to the effects of visual masking and if so whether this deterioration can be counteracted by a cue. While some have found that memory representations are insensitive to the effects of intervening masks (e.g., Pinto et al., 2013), others have found that masks do impair memory performance (e.g., Gegenfurtner and Sperling, 1993). Most work on the effects of masking, however, used binary response measures, which precludes the possibility to specify the nature of the decline. Here we observed that the deteriorating effect of the mask can be attributed to a decline in the representational quality and an increase in random responses (but see Zhang and Luck, 2008, who only found an effect on probability of recall).

Also, it was found that cues improved the representational quality of the cued item and as in Williams et al. (2013) this effect was present in both model parameters. Important for the present purpose, the deteriorating effect of the mask was also modulated by the cue. Makovski and Jiang (2007) found no difference between cue trials with and without interference. Here however, consistent with Pinto et al. (2013), a mask that is related to the memory content and presented on the memory locations impaired memory performance, but to a lesser extent than without a cue. Interestingly, this modulation by the cue was completely absent in the estimates of precision and any effect was visible only in the probability of recall.

In Experiment 2 we set out to investigate the temporal dynamics of building up this cue-based protection against perceptual interference. Therefore, in Experiment 2 we used a similar set-up as in Experiment 1 and systematically varied the stimulus onset asynchrony (SOA) between the cue and the pattern mask, to evaluate the temporal dynamics of building up this protection.

EXPERIMENT 2: THE TIME-COURSE OF PROTECTION

Experiment 1 showed that attentional shifts induced by cues protect visual memory representations from perceptual interference. Experiment 2 was designed to investigate the time course of building up this protection. We included three different cueing conditions, and combined them with SOA: no-cue/mask, cue/mask and cue/no- mask trials. The comparison between no-cue/mask and cue/mask trials allowed us to assess the point in time the cue started to have an effect, whereas the comparison between cue/mask and cue/no-mask trials allowed us to assess the time required for the cue to reach full protection of the representation. In Experiment 1, the cue-based protection was apparent in the raw data (i.e., average deviation and sd) as well as the probability of recall. In Experiment 2, however, we had to reduce the number of trials per condition as a result of including multiple SOAs. Therefore, in Experiment 2 we only analyzed the sd of the response distribution as the model output for probability was already less strong for Experiment 1, and becomes even less reliable with the smaller number of trials used here.

METHOD

Participants

Seventeen young adults, aged 22–28 (M = 25), participated in exchange for course credit or a payment of €8 per hour. All had normal or corrected-to-normal acuity and gave informed consent according to procedures approved by the ethic commission of the VU University. One participant was excluded because she did not follow task instructions (i.e., used pen and paper during the experiment).

Apparatus, stimuli, procedure and design

The method was similar to Experiment 1 except for the following changes. Stimuli were presented on a Samsung SyncMaster 2233 120 Hz screen. The no-cue/no-mask condition was omitted as it was redundant for the purpose of Experiment 2. Moreover, to make the mask more effective, the 100 ms static mask was changed to a flickering mask: each mask display was presented three times for 100 ms discontinued by 50 ms fixation displays. Also, in all trials, the delay between the cue (cue/no-cue) and the interference displays (mask/no-mask) was varied systematically (Figures 1B–D). We chose four different SOAs ranging between 100 and 600 ms (100, 200, 350, 600). The SOA always referred to the interval between the onset of the cue display and the onset of the mask displays, even in conditions in which no cue or no mask was presented. This meant that the retention interval between memory offset and probe onset in all conditions varied between 1400 and 1900 ms depending on the selected SOA. Note that at the shortest SOA the mask was presented immediately after cue offset.

All participants completed 24 practice trials and 10 experimental blocks of 60 trials each. Each block consisted of 20 cue/mask, 20 cue/ no-mask and 20 no-cue/mask randomly mixed trials, with equal number of trials for each delay duration (five for each SOA following the cue presentation).

RESULTS AND DISCUSSION

Figure 3 shows the recall performance as a function of SOA and condition. For completeness, corresponding deviations are shown in Table 2. SDs and mean deviations were entered in a repeated measures ANOVA with within subjects factor SOA (100, 200, 350, 600) and condition (cue/no-mask, cue/mask, no-cue/mask). A Greenhouse-Geisser correction was applied in case of sphericity violations. There was no significant main effect for SOA (F = 0.68, p = 0.57 for sd; F = 1.012, p = 0.40 for mean deviation). There was a main effect for condition (F(1.4,20.17) = 38.635, p < 0.001 for sd; F(1.3,20.0) = 36.395, p < 0.001 for mean deviation), and a significant interaction (F(6,90) = 3.287, p = 0.006 for sd; F(6,90) = 3.776, p = 0.006 for mean deviation).

FIGURE 3. Experiment 2: Standard deviation of error distributions for all three conditions as a function of SOA.

Table 2. Experiment 2: Data represents mean of average deviation per condition across SOA’s.

Recall performance was lower than that observed in Experiment 1, arguably due to the more disruptive nature of the mask in Experiment 2. Across all SOA’s, recall performance was worst in the condition without a cue and with a mask (all t’s > 2.802, all p’s < 0.013 for sd; all t’s > 2.836, all p’s < 0.013 for mean deviation). Replicating findings from Experiment 1, the disruptive nature of the mask was counteracted by the cue. Results show that the cue based protection was completed within 350–600 ms following the cue, as the difference between cue/mask and cue/no-mask disappeared only at the longest SOA (t = 1.019, p = 0.32 for sd; t = 0.380, p = 0.71 for mean deviation). In contrast, at all shorter SOAs performance was significantly worse in the cue/mask condition than in the cue/no-mask condition (all t‘s > 2.462, all p’s < 0.026 for sd; all t‘s > 2.303, all p’s < 0.036 for mean deviation). Note that although cue based protection was not yet completed within the first 350 ms following the cue, the cue had a very rapid effect. Already at the 100 ms SOA performance in the cue/mask condition was better than in the no-cue/mask condition (t = 2.802, p = 0.013 for sd; t = 2.836, p = 0.013 for sd) and this effect increased over time (all t’s >3.038, all p’s < 0.008 for sd; all t’s > 2.913, all p’s < 0.011 for mean deviation). Figure 3 also suggests that incorporation of the cue slowly evolves over time. Indeed, when we analyzed the data for each condition separately, we observed a substantial linear trend (F(1,15) = 11.544, p = 0.004 for sd; F(1,15) = 12.165, p = 0.003 for mean deviation) for the cue/mask condition, but no such trend for the cue/no-mask baseline condition (F = 0.764, p = 0.52 for sd; F = 0.041, p = 0.84 for mean deviation) and if anything a weak trend in the opposite direction for the no-cue/mask condition (F = 3.45, p = 0.08 for sd; F = 3.464, p = 0.08 mean deviation). The observed linear trend in the cue/mask condition and no hint of such a trend in the cue/no-mask condition illustrates that the convergence of the two conditions at the longest SOA can be attributed to an increasing effectiveness of the cue over time. Thus, even though the cue had an immediate effect, optimal protection was only reached between 350 and 600 ms after cue offset.

GENERAL DISCUSSION

Previously, it has been proposed that cueing an item in retrospect shifts attention or memory resources from a distributed mode to a focused mode and thus protects the cued item against memory degradation (Matsukura et al., 2007) or perceptual interference (Lepsien and Nobre, 2007; Makovski and Jiang, 2007). The present study was conducted to investigate the time course of implementing this protection. For this purpose a cue design was combined with a masking procedure at various SOAs. This allowed us to measure both the time required to activate the cue and the necessary time to incorporate the cue. It was found that a cue stabilizes the representational quality of the cued item such that after sufficient time it is no longer sensitive to the effects of perceptual interference. In Experiment 1 it was found that 500 ms upon cue onset the cued representation was still sensitive to the deteriorating effect of the mask, although to a lesser extent than without a cue. In Experiment 2 600 ms following cue onset there was no longer an observable effect of the cue indicating that under the present conditions it took around 500–600 ms for the cue to be fully incorporated.

The observed cue benefits add to a conflicting literature on the effects of cues on the status of memory content. Although there is ample evidence that cues improve memory performance, the source of this performance benefit remains unclear. New modeling techniques have made it possible to divide memory performance into independent measures of capacity and resolution. These models, however, have provided conflicting results. On the one hand, there is evidence that cues improve the probability of recalling the cued object, but not its precision, suggesting independence between these two measures (Murray et al., 2013; Souza et al., 2014a). On the other hand Williams et al. (2013), observed cue benefits on both the probability of recall as well as on precision. Here, consistent with Williams et al. (2013), the cue benefit was apparent in both parameters.

Retro-cue benefits have been attributed to different mechanisms, one of which is protection against perceptual interference (Makovski and Jiang, 2007) or memory degradation (Matsukura et al., 2007). The cue benefit as observed here is in line with such an account. Recall performance improved on cue trials with and without a mask, suggesting that protection also operates during maintenance (e.g., protection against decay, inter-item interference). At the same time it was found that a cue improved recall performance in the face of interference above and beyond levels if there had been no interference (cue/mask vs. no-cue/no-mask). Although this result does certainly does not rule out a protection account, it is also consistent with an active ramping up of the cued representation either because the cued item is strengthened or because the non-cued items are removed from memory (e.g., Kuo et al., 2011; Souza et al., 2014b). Here, however, we cannot dissociate between these mechanisms, which do not need to be exclusive, as with the present paradigm the effects on the uncued items in memory remain unknown.

Another important aspect of the data is the observation that without a cue the visual memory representations were sensitive to the effects of perceptual interference by a pattern mask. As noted, there is some inconsistency in the literature on the effects of masking visual memory; whereas some studies observed an effect, other studies observed equal performance in conditions with and without a mask (Dolcos et al., 2007; Irwin and Thomas, 2008; Zhang and Luck, 2008; Clapp et al., 2010; Pinto et al., 2013). In the present study the deteriorating effect of the mask was evident in a measure of precision as well as in probability of recalling the cued item. Important for the present purposes, we showed that the deteriorating effect of the mask could be counteracted by a cue. Interestingly, when applying the mixture model in Experiment 1, the only factor that appeared to contribute to this beneficial effect of the cue was an improved probability of recall, while there were no effects whatsoever on precision. This suggests that the cue protects from the mask in an all or nothing fashion: Either the item is masked and therefore lost, or it survives the mask, and then does so intact—that is, as intact as if there was no mask.

In Experiment 2 it was found that around 600 ms following a cue, a mask no longer interfered with visual memory. The time-course of this attentional shift by the endogenous cue is remarkably similar to those observed in visual working memory and in the visual attention literature. As noted in the introduction both Tanoue and Berryhill (2012) and Pertzov et al. (2013) manipulated the post-cueing time and found that it took at least about 300 ms to find a significant effect for the cue on memory performance. Similar results have also been obtained when the cue is presented concurrently with the memoranda. Bays et al. (2011) instructed participants to remember the orientation of two bars. Before the items were masked, a white disk was briefly flashed at the location of one of the memory items, either simultaneously with memory onset or 1000 ms after stimulus onset. In both situations, when the cue was valid, a significant recall advantage for the cued item developed in the first 400 ms between cue-onset and mask presentation. Moreover, it has been found that transforming a visual memory into an attentional set also takes about 400 ms to be completed (Wilschut et al., 2013). Finally, in studies that systematically varied the SOA between a predictive cue and stimulus onset, performance has been found to increased steadily until it reaches a plateau at about 400 ms (see Egeth and Yantis, 1997 for a review). Here we find a similar, though somewhat longer time course to protect a visual memory representation against perceptual interference by a pattern mask. Together these data indicate that shifting attention in both perceptual and internal space is a relatively rapid process that is characterized by a monotonic rise until an asymptote is reached and that takes at least 300 ms to be completed.

In Experiment 2, we observed significant advantages for a memory item prioritized by an endogenous retro-cue across the range of selected SOAs. That is, a recall advantage was already apparent when pattern mask was presented immediately following the offset of the cue, at 100 ms SOA. Apparently, 100 ms is already sufficient to retrieve some undamaged information from the to be tested location. Importantly, this building up of a protection at the cued location against perceptual interference continued until a maximum protection was accomplished. Experiment 2 showed that between 350 and 600 ms performance became indistinguishable from unmasked conditions. In Experiment 1, however, 500 ms following cue onset, a mask still impaired memory performance. Thus, together the data from both Experiment 1 and 2 indicate that it took around 500 to 600 ms for the cue to be optimally implemented.

In conclusion, the present results indicate that retro-cueing a visual memory item counteracts the effects of perceptual interference on memory, and in particular leads to a better representational quality. Moreover, although initial beneficial effects of the retro-cue emerge quite rapidly, protection against perceptual interference steadily evolves over time and takes between 500 and 600 ms to be completed. This time-course is similar to what has been found before for predictive cues in both mnemonic and visual selection tasks.

REFERENCES

Awh, E., Barton, B., and Vogel, E. K. (2007). Visual working memory represents a fixed number of items regardless of complexity. Psychol. Sci. 18, 622–628. doi: 10.1111/j.1467-9280.2007.01949.x

Awh, E., and Jonides, J. (2001). Overlapping mechanisms of attention and spatial working memory. Trends Cogn. Sci. 5, 119–126. doi: 10.1016/s1364-6613(00)01593-x

Awh, E., Vogel, E., and Oh, S.-H. (2006). Interactions between attention and working memory. Neuroscience 139, 201–208. doi: 10.1016/j.neuroscience.2005.08.023

Bays, P. M., Catalao, R. F. G., and Husain, M. (2009). The precision of visual working memory is set by allocation of a shared resource. J. Vis. 9:7. doi: 10.1167/9.10.7

Bays, P. M., Gorgoraptis, N., Wee, N., Marshall, L., and Husain, M. (2011). Temporal dynamics of encoding, storage and reallocation of visual working memory. J. Vis. 11:6. doi: 10.1167/11.10.6

Bays, P. M., and Husain, M. (2008). Dynamic shifts of limited working memory resources in human vision. Science 321, 851–854. doi: 10.1126/science.1158023

Berryhill, M. E., Richmond, L. L., Shay, C. S., and Olson, I. R. (2012). Shifting attention among working memory representations: testing cue type, awareness and strategic control. Q. J. Exp. Psychol. 65, 426–438. doi: 10.1080/17470218.2011.604786

Clapp, W. C., Rubens, M. T., and Gazzaley, A. (2010). Mechanisms of working memory disruption by external interference. Cereb. Cortex 20, 859–872. doi: 10.1093/cercor/bhp150

Delvenne, J.-F., Cleeremans, A., and Laloyaux, C. (2010). Feature bindings are maintained in visual short-term memory without sustained focused attention. Exp. Psychol. 57, 108–116. doi: 10.1027/1618-3169/a000014

Dolcos, F., Miller, B., Kragel, P., Jha, A., and McCarthy, G. (2007). Regional brain differences in the effect of distraction during the delay interval of a working memory task. Brain Res. 1152, 171–181. doi: 10.1016/j.brainres.2007.03.059

Egeth, H. E., and Yantis, S. (1997). Visual attention: control, representation and time course. Annu. Rev. Psychol. 48, 269–297. doi: 10.1146/annurev.psych.48.1.269

Gegenfurtner, K. R., and Sperling, G. (1993). Information transfer in iconic memory experiments. J. Exp. Psychol. Hum. Percept. Perform. 19:845. doi: 10.1037//0096-1523.19.4.845

Griffin, I. C., and Nobre, A. C. (2003). Orienting attention to locations in internal representations. J. Cogn. Neurosci. 15, 1176–1194. doi: 10.1162/089892903322598139

Gunseli, E., van Moorselaar, D., Meeter, M., and Olivers, C. N. (in press). Strategic allocation of attention determines the fate of non-cued visual working memory representations. Psychon. Bull. Rev.

Hollingworth, A., and Maxcey-Richard, A. M. (2013). Selective maintenance in visual working memory does not require sustained visual attention. J. Exp. Psychol. Hum. Percept. Perform. 39, 1047–1058. doi: 10.1037/a0030238

Irwin, D. E., and Thomas, L. E. (2008). Visual sensory memory. Vis. Mem. 1, 9–43. doi: 10.1093/acprof:oso/9780195305487.003.0002

Janczyk, M., and Berryhill, M. E. (2014). Orienting attention in visual working memory requires central capacity: decreased retro-cue effects under dual-task conditions. Atten. Percept. Psychophys. 76, 715–724. doi: 10.3758/s13414-013-0615-x

Kuo, B.-C., Yeh, Y.-Y., Chen, A. J.-W., and D’Esposito, M. (2011). Functional connectivity during top-down modulation of visual short-term memory representations. Neuropsychologia 49, 1589–1596. doi: 10.1016/j.neuropsychologia.2010.12.043

Landman, R., Spekreijse, H., and Lamme, V. A. (2003). Large capacity storage of integrated objects before change blindness. Vision Res. 43, 149–164. doi: 10.1016/s0042-6989(02)00402-9

Lepsien, J., and Nobre, A. C. (2007). Attentional modulation of object representations in working memory. Cereb. Cortex 17, 2072–2083. doi: 10.1093/cercor/bhl116

Makovski, T., and Jiang, Y. V. (2007). Distributing versus focusing attention in visual short-term memory. Psychon. Bull. Rev. 14, 1072–1078. doi: 10.3758/bf03193093

Makovski, T., Sussman, R., and Jiang, Y. H. V. (2008). Orienting attention in visual working memory reduces interference from memory probes. J. Exp. Psychol. Learn. Mem. Cogn. 34, 369–380. doi: 10.1037/0278-7393.34.2.369

Matsukura, M., Luck, S. J., and Vecera, S. P. (2007). Attention effects during visual short-term memory maintenance: protection or prioritization? Percept. Psychophys. 69, 1422–1434. doi: 10.3758/bf03192957

Morey, R. D. (2008). Confidence intervals from normalized data: a correction to Cousineau (2005). Reason 4, 61–64.

Murray, A. M., Nobre, A. C., Clark, I. A., Cravo, A. M., and Stokes, M. G. (2013). Attention restores discrete items to visual short-term memory. Psychol. Sci. 24, 550–556. doi: 10.1177/0956797612457782

Pertzov, Y., Bays, P. M., Joseph, S., and Husain, M. (2013). Rapid forgetting prevented by retrospective attention cues. J. Exp. Psychol. Hum. Percept. Perform. 39, 1224–1231. doi: 10.1037/a0030947

Pinto, Y., Sligte, I. G., Shapiro, K. L., and Lamme, V. A. F. (2013). Fragile visual short-term memory is an object-based and location-specific store. Psychon. Bull. Rev. 20, 732–739. doi: 10.3758/s13423-013-0393-4

Rerko, L., and Oberauer, K. (2013). Focused, unfocused and defocused information in working memory. J. Exp. Psychol. Learn. Mem. Cogn. 39, 1075–1096. doi: 10.1037/a0031172

Rerko, L., Souza, A. S., and Oberauer, K. (2014). Retro-cue benefits in working memory without sustained focal attention. Mem. Cognit. 42, 1–17. doi: 10.3758/s13421-013-0392-8

Sligte, I. G., Scholte, H. S., and Lamme, V. A. F. (2008). Are there multiple visual short-term memory stores? PLoS One 3:e1699. doi: 10.1371/journal.pone.0001699

Souza, A. S., Rerko, L., Lin, H.-Y., and Oberauer, K. (2014a). Focused attention improves working memory: implications for flexible-resource and discrete-capacity models. Atten. Percept. Psychophys. 76, 2080–2102. doi: 10.3758/s13414-014-0687-2

Souza, A., Rerko, L., and Oberauer, K. (2014b). Unloading and reloading working memory: attending to one item frees capacity. J. Exp. Psychol. Hum. Percept. Perform. 40, 1237–1256. doi: 10.1037/a0036331

Tanoue, R. T., and Berryhill, M. E. (2012). The mental wormhole: internal attention shifts without regard for distance. Atten. Percept. Psychophys. 74, 1199–1215. doi: 10.3758/s13414-012-0305-0

Wilken, P., and Ma, W. J. (2004). A detection theory account of change detection. J. Vis. 4, 1120–1135. doi: 10.1167/4.12.11

Williams, M., Hong, S. W., Kang, M.-S., Carlisle, N. B., and Woodman, G. F. (2013). The benefit of forgetting. Psychon. Bull. Rev. 20, 348–355. doi: 10.3758/s13423-012-0354-3

Williams, M., and Woodman, G. F. (2012). Directed forgetting and directed remembering in visual working memory. J. Exp. Psychol. Learn. Mem. Cogn. 38, 1206–1220. doi: 10.1037/a0027389

Wilschut, A., Theeuwes, J., and Olivers, C. N. (2013). The time it takes to turn a memory into a template. J. Vis. 13:8. doi: 10.1167/13.3.8

Zhang, W. W., and Luck, S. J. (2008). Discrete fixed-resolution representations in visual working memory. Nature 453, U233–U235. doi: 10.1038/nature06860

Conflict of Interest Statement: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Received: 30 July 2014; Accepted: 17 December 2014; Published online: 13 January 2015.

Citation: van Moorselaar D, Gunseli E, Theeuwes J and Olivers CNL (2015) The time course of protecting a visual memory representation from perceptual interference. Front. Hum. Neurosci. 8:1053. doi: 10.3389/fnhum.2014.01053

This article was submitted to the journal Frontiers in Human Neuroscience.

Copyright © 2015 van Moorselaar, Gunseli, Theeuwes and Olivers. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

ORIGINAL RESEARCH ARTICLE
published: 06 November 2014
doi: 10.3389/fnhum.2014.00853

Flexibility of representational states in working memory

Nahid Zokaei1,[2]*, Shen Ning[1], Sanjay Manohar1,[2], Eva Feredoes[3] and Masud Husain1,[2]

1Department of Experimental Psychology, University of Oxford, Oxford, UK

2Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK

3School of Psychology and Clinical Language Sciences, University of Reading, Reading, UK

* Correspondence: Nahid Zokaei, Department of Experimental Psychology, University of Oxford, 9, South Parks Road, Oxford, OX1 3UD, UK e-mail: nahid.zokaei@psy.ox.ac.uk

Edited by:
Jean-Philippe Van Dijck, Ghent University, Belgium

Reviewed by:
Stephen Emrich, Brock University, Canada
Jyoti Mishra, University of California San Francisco, USA

The relationship between working memory (WM) and attention is a highly interdependent one, with evidence that attention determines the state in which items in WM are retained. Through focusing of attention, an item might be held in a more prioritized state, commonly termed as the focus of attention (FOA). The remaining items, although still retrievable, are considered to be in a different representational state. One means to bring an item into the FOA is to use retrospective cues (“retro-cues”) which direct attention to one of the objects retained in WM. Alternatively, an item can enter a privileged state once attention is directed towards it through bottom-up influences (e.g., recency effect) or by performing an action on one of the retained items (“incidental” cueing). In all these cases, the item in the FOA is recalled with better accuracy compared to the other items in WM. Far less is known about the nature of the other items in WM and whether they can be flexibly manipulated in and out of the FOA. We present data from three types of experiments as well as transcranial magnetic stimulation (TMS) to early visual cortex to manipulate the item inside FOA. Taken together, our results suggest that the context in which items are retained in WM matters. When an item remains behaviorally relevant, despite not being inside the FOA, re-focusing attention upon it can increase its recall precision. This suggests that a non-FOA item can be held in a state in which it can be later retrieved. However, if an item is rendered behaviorally unimportant because it is very unlikely to be probed, it cannot be brought back into the FOA, nor recalled with high precision. Under such conditions, some information appears to be irretrievably lost from WM. These findings, obtained from several different methods, demonstrate quite considerable flexibility with which items in WM can be represented depending upon context. They have important consequences for emerging state-dependent models of WM.

Keywords: working memory, attention, representational states, retro cueing, incidental cueing

INTRODUCTION

Working memory (WM) refers to the ability to hold and manipulate information in mind for brief periods of time (Baddeley, 2003). It has been proposed that not all information in WM is maintained in an equal state. For example, depending on task relevance such as likelihood of being probed, one item might require prioritization over others. These sorts of considerations have led some authors to argue that the state of WM representations might be determined by an interaction between long-term memory (LTM) and attention (e.g., Cowan, 1998; McElree, 1998; Oberauer, 2002).

While these models differ in how the capacity and nature of representations in various states are determined (see Larocque et al., 2014 for a comprehensive review), they all agree on the existence of least two distinct states. One of these states has attention focused on an item (or a subset of items), rendering it in a more prioritized state so that it can be accessed more readily, with higher accuracy and/or fidelity (e.g., Lepsien and Nobre, 2007; Pertzov et al., 2012; Zokaei et al., 2014). In line with the most prominent theoretical models, we will refer to these prioritized items as being inside a focus of attention (FOA; as defined by Cowan, 1998; Oberauer, 2002, 2009). The remaining items outside FOA—although still retrievable—are considered to be in a different representational state (Figure 1).

FIGURE 1. Schematic representation of items maintained in visual WM in two different representational states. Although multiple items can be maintained in WM (lower panel), one item (red object in top pane) might be held in a more prioritized state known as the “focus of attention” (FOA).

The distinction between the items inside and outside FOA has been demonstrated both empirically (Lewis-Peacock et al., 2011; LaRocque et al., 2013; Stokes et al., 2013) and theoretically, often described in terms of the activated LTM proportion of short term memory (e.g., Oberauer, 2002). A substantial amount of research has focused on the qualities of representations of the item in FOA, but far less is known about the nature of the other items. One important but outstanding question that we address in this study is whether items can flexibly move in and out of FOA.

MANIPULATION OF REPRESENTATIONAL STATES IN WM

Behaviorally, the item in FOA is identified as the item that is recalled faster and more accurately compared to other retained items (see Nee and Jonides, 2013; Larocque et al., 2014 for examples). There are various methods for manipulating representational states in WM that result in such a behavioral advantage. In this section, we review selected studies that have used various approaches to shed light on the nature of the item in FOA and as a consequence on the other, non-focused items.

Retro-cues shift the focus of attention in WM

A commonly used means to bring an item into FOA is the use of retrospective cues (“retro-cues”) presented during the retention interval, with the intention of directing attention to one specific item in WM. Specifically, a retro-cue will indicate to the participant which item is highly likely to be probed for recognition or recall. A consequence of retro cues is higher accuracy or precision of recall for the cued item compared to other uncued or invalidly cued items (Griffin and Nobre, 2003; Lepsien and Nobre, 2007; Makovsik and Jiang, 2007; Sligte et al., 2008, 2010; Astle et al., 2009; Lepsien et al., 2011; Berryhill et al., 2012; Pertzov et al., 2012).

Some studies have further attempted to characterize the neural underpinnings of the item in FOA achieved through retro-cueing (e.g., Lepsien and Nobre, 2007; Harrison and Tong, 2009; Lewis-Peacock et al., 2011; Nelissen et al., 2013; Larocque et al., 2014). In one such investigation, participants were presented with a sequence of two images to remember from two different categories of faces and scenes (Lepsien and Nobre, 2007). After a retention period they were shown a retro-cue indicating the relevant item for the upcoming memory probe. Event-related functional magnetic resonance imaging (fMRI) demonstrated that orienting attention to one of the items during WM maintenance modulated the activity in region of the brain involved in perception of the cued item: fusiform face area for faces and parahippocampal place area for scenes.

Using multi-voxel pattern analysis (MVPA; e.g., Haxby et al., 2001; Haynes and Rees, 2006) some investigators have reported stimulus-specific patterns in early visual areas during WM retention for retro-cued items. In a landmark study Harrison and Tong (2009) demonstrated that the orientation of the Gabor patch retro-cued during WM maintenance (from two retained orientations) could be decoded from the activity patterns in visual areas V1-V4. Nelissen et al. (2013) employed a similar retro-cueing WM task and showed that cued items could be accurately decoded from occipitotemporal cortex, but that decoding was at chance level for the un-cued items (Nelissen et al., 2013; See also Lewis-Peacock et al., 2011).

From these findings one can conclude that the retro-cued item is maintained in a different state compared to non-cued items, with its maintenance dependent, at least to some extent, on early visual cortical regions known to be involved in perception of the maintained information.

Recency: last item in the focus of attention

Many investigations of WM present items serially rather than simultaneously. It has been known for a long time that the last item in a sequence is better recalled than previous ones: the recency effect. Some authors argue that the most recent item is in fact automatically in the FOA. Evidence comes from behavioral studies demonstrating retrieval advantage for the most recent item compared to items presented earlier in the sequence (McElree and Dosher, 1989, 1993; Neath, 1993; Hay et al., 2007; Blalock and Clegg, 2010). Furthermore, this last item is recalled faster, with more accuracy and precision compared to all previous items, with the magnitude of recency effect dependent on the number of previous items in the sequence (Gorgoraptis et al., 2011; Zokaei et al., 2011). Thus, the well-known recency effect in the WM literature might be due to the obligatory assignment of the last item to the FOA.

Brain imaging techniques have shed light on possible dissociation in neural correlates for maintenance of the last item compared to earlier items in a sequence. Using fMRI, Nee and Jonides (2008) investigated the probed-evoked neural signal of the most recently presented word compared to words presented earlier in a sequence. The recognition of the last item was accompanied by increased activation in the inferior temporal cortex. Similar findings have been presented in later studies (Nee and Jonides, 2011, 2013). Furthermore, recognition of the last word in the memory sequence was accompanied by less hippocampal activation when compared to recognition of all previous items in a sequence leading to intriguing suggestion that non-focused items are maintained by the hippocampus (Oztekin et al., 2009, 2010). Hence, there is both behavioral and neural evidence for dissociation of the most recent item into WM compared to previous items.

In addition recent findings using transcranial magnetic stimulation (TMS) have provided causal evidence for different representational states in visual WM (Zokaei et al., 2014). Two random dot kinematograms (RDKs) moving in different directions were presented sequentially, with the aim of bringing the second RDK into FOA. Following a delay period, participants had to recall the direction of one of the two previously-presented RDKs. Transcranial magnetic stimulation was administered during the delay of the WM task, prior to the presentation of the probe, to motion sensitive area MT+ which was hypothesized to be maintaining the remembered motion directions (Bisley and Pasternak, 2000; Pasternak and Greenlee, 2005). Transcranial magnetic stimulation impaired recall precision of the motion direction in FOA and, crucially, conversely improved precision for the other non-focused direction (Zokaei et al., 2014). These findings provide some of the first causal evidence that there are at least two representational states, with the maintenance of only the item in FOA relying on an area involved in its perception.

Incidental-cueing brings an item into focus of attention

A new method for manipulating representational states in WM has recently been developed; incidental cueing. This approach is different to the retro-cueing approach, because the cue is not explicitly predictive of upcoming memory recall. Instead, the rationale is that once an item from those maintained in WM is used for a cognitive operation that is incidental or orthogonal to the memory requirements of the task, that item will automatically enter the FOA.

In the study by Zokaei et al. (2014) participants were once again presented with two RDKs now simultaneously, above and below a fixation cross, and in two different colors. Participants were required to remember the directions of motion of each of the RDKs. During the delay period, the fixation cross changed to the color of one of the maintained motion directions and participants indicated with a key press the location of the motion direction of that color in the memory array, i.e., whether it had been above or below the fixation cross. The color of the fixation cross was not informative of the upcoming memory probe, and the judgment about the location of one of the RDKs was completely orthogonal to the requirement to remember the direction of motion of the RDKs. Nevertheless, adding this incidental task, resulted in higher precision of recall of the motion direction that also matched the color of the fixation cross. Thus, this item appeared to be in FOA simply by virtue of being “incidentally” cued. In agreement with the results from the TMS experiment described above, TMS applied here during the delay period, after the incidental cue, also disrupted recall of the cued item, and improved it for the non-cued item.

A similar rationale was employed by Lewis-Peacock et al. (2011) who used a dual response and cueing WM task adapted from Oberauer (2005). Following a delay after presentation of the memory array, a cue appeared highlighting the first item to be probed. After the response, a second retro-cue appeared that indicated either the same item as that previously probed or alternatively cued the participant to switch to the other item in WM. For this second cue, only the task-relevant item (the cued item) could be successfully decoded from the fMRI BOLD signal, whereas the irrelevant (non-cued) item could not. This occurred despite no behavioral loss in performance when the second cue was different to the first (Lewis-Peacock et al., 2011, see LaRocque et al., 2013 for electroencephalography (EEG) analog).

The findings reported in this section provide evidence for the existence of a FOA within WM representations: an item can be held in a more privileged state with its recall more accurate and with higher precision compared to all other items maintained in WM. But what about the other items in WM? What happens to them and how are they stored?

THE FATE OF ITEMS OUTSIDE THE FOCUS OF ATTENTION IN WM

We now turn to investigations of how the items outside FOA are represented relative to those inside FOA. One hypothesis is that the item in FOA is protected from interference from un-cued items, with this protection coming at a cost for remembering un-cued items. Pertzov et al. (2012) simultaneously presented four oriented bars of different color and asked participants to remember their orientation. One of the bars was subsequently probed by its location or color. During the retention period, a retro-cue appeared indicating the item that was most likely to be later probed (70% validity). In the subsequent recall period, participants were asked to reproduce the exact orientation of one of the bars. Precision of recall was significantly worse for the invalidly cued trials (i.e., trials where one of the un-cued items were probed) compared to trials in which the probe was validly cued, and also to baseline trials in which there was no cue.

These results were interpreted as evidence for the validly cued item being in a state that was protected from interference by other items held in WM, with these other objects suffering from accelerated temporal decay. Similar findings have been reported by Lepsien and Nobre (2007) and Matsukura et al. (2007). In the latter study, participants were presented with two consecutive retro-cues, with the second cue 100% valid. The second cue could be same or different to the first. Two identical cues resulted in similar performance to a single valid retro-cue, while two different cues impaired performance. The authors explained their findings in terms of forgetting un-cued items due to either degradation or interference, alongside protection of the cued item.

Evidence for the degradation of un-cued items is, however, inconsistent. In WM change detection tasks, several studies report no significant impairment in recall of the un-cued information with two consecutive cues during the delay (Landman et al., 2003; Rerko and Oberauer, 2013). For example, Rerko and Oberauer (2013) presented participants with a memory array followed by either a single, two or three cues. The probed matched (50% trials) or mismatched the item that was cued last. There was no difference in accuracy between the cued and un-cued items.

The discrepancy in behavioral findings for memory of un-cued items may be explained by the degree of information conveyed by the cue. In tasks that show an effect on un-cued information (i.e., impaired performance in recalling those items) the cue carries predictive information indicating which will be the most relevant item—often the only one relevant item—for forthcoming memory recall. As a consequence, un-cued information is rendered behaviorally irrelevant because of the low probability that it will need to be accessed again. In effect, this therefore changes the task to WM for a single (cued) item only.

On the other hand, in studies that have failed to find an effect on un-cued items (i.e., recall/recognition accuracy is no different to cued items), participants did not need to attend to previous cues, but rather the last cue only (e.g., Landman et al., 2003; Rerko and Oberauer, 2013). Thus, only the final cue was informative with respect to the upcoming probe, and importantly with only 50% validity. It could therefore be argued that given the low amount of predictive information conveyed by the cue, it was rendered less effective compared to retro-cues with, for example, 70% validity. Moreover, in these studies the final cue was followed by a very short delay (<500 ms) prior to the presentation of the probe. But for a retro-cue to produce sufficient behavioral advantage, there should be at least 1 s between the presentation of the cue and the memory probe (Pertzov et al., 2012).

RECALL OF ITEMS NOT IN FOCUS OF ATTENTION

From the studies described above it is evident that research has mainly centered on the single, privileged item in FOA. As a consequence, far less is known about the nature of non-focused items, and whether these items can be brought back into FOA or are lost from memory. In light of previous findings, one can hypothesize that whether items can flexibly move in and out of the FOA might be highly dependent on their potential relevance to the task in hand. Thus, if a non-focused item has a high chance of being probed it might still be maintained in a state such that it can be retrieved with high quality.

We tested this hypothesis across four experiments in which we used a method for measuring WM performance that relies on participants to reproduce the exact qualities of the retained information, providing a measure of precision of recall (Gorgoraptis et al., 2011; Zokaei et al., 2011, 2013). Such sensitive measure of WM allows us to detect small changes in recall precision that may otherwise not be detectable with alternative measures. In the first two experiments we used cues that were orthogonal to the WM task at hand. We hypothesized that these cues would allow items to move flexibly in and out of the FOA, since they carry no information regarding the relevance of items in WM. On the other hand, in experiment 3 and 4, a retro-cue with 80% validity was used. Unlike the first two experiments, we predicted that in such a situation, non-focused (un-cued) items in WM might degrade to the extent that they cannot be brought back into FOA.

In our first two experiments we aimed to examine whether representational states of items in WM can flexibly change in situations in which all items in WM remain potentially, behaviorally relevant throughout the trial. In Experiment 1 we used two successive incidental cues while in Experiment 2, we used sequential presentation of items so that the last item was in FOA by virtue of recency. This was then followed by an incidental cue that could be same or different to the last item in the sequence. Note that this method of cueing always required participants to make a response, allowing us to confirm whether they attended to the cued item or not, rather than relying on participants to attend to the cue of their own volition, with no objective measure (e.g., Landman et al., 2003; Pertzov et al., 2012; Rerko and Oberauer, 2013).

EXPERIMENT 1: EFFECTS OF ONE OR TWO INCIDENTAL CUES

METHODS

Participants

Twenty healthy individuals (13 male) with an average age of 27 (range: 19–35) participated. All had normal or corrected to normal vision and reported normal color vision. They provided written consent to the procedure of the experiment, which was approved by the local ethics committee.

Stimuli

On each trial, two RDKs were presented below and above the fixation cross, subtending 10° of visual angle. Each RDK consisted of 50 dots (0.1° visual angle each), displayed within an invisible circular aperture (5.7° of visual angle). The color of the top RDK was chosen at random on each trial to be either green or red, with the lower RDK assigned the other color (red or green).

Dot lifetime and density were constant during RDK presentation with 100% coherent motion (constant speed of 4.5°/s). Motion direction for each RDK was selected from 0–360° with no angular separation between the two motion directions on each trial. A mask consisting of 5000 dots (50% red), covering the entire screen was presented immediately after RDK offset.

Stimuli were displayed on 14.1” display (resolution 800 × 600 pixels, refresh rate 60 Hz). Participants were seated approximately 60 cm from the monitor in a dimly lit room.

Procedure

A schematic representation of the task and different conditions is illustrated in Figure 2. Each trial started with a fixation cross (500 ms), followed by the presentation of two RDKs (the memory array for 300 ms) and mask (100 ms). In this experiment, cues could be presented at two different time points, referred to as 1st position and 2nd position cues.

FIGURE 2. Schematic and baseline corrected WM recall performance of trial events for Experiment 1. Two items were presented in the memory array. This was then followed by (A) an incidental cue in the first position; (B) an incidental cue in the second position; (C) two identical consecutive incidental cues or (D) two different consecutive incidental cues, before the presentation of the memory probe. Error bars are SEM.

In the baseline condition (1/9th of trials and not shown in the figure), the RDKs were simply followed by 7.2 s blank interval before presentation of probe stimuli. In 4/9th of trials, the stimuli were followed by a blank delay (1 s) and then the fixation cross briefly changed to either red or green (100 ms) which served as our method of incidental cueing. Participants had to indicate with a key press as accurately and as fast as possible the location of the RDK (above or below fixation cross) that was the same color as the fixation cross.

On half these trials (i.e., 2/9th of the overall trials), the cue was followed by a 6.1 s blank interval before the presentation of the probe (One cue, 1st condition—Figure 2A). On the remainder of the trials (2/9th of the overall number of trials), the memory array was followed by a 4.1 s delay, then the incidental cue (100 ms) and another delay (3 s) before the presentation of the probe (One cue, 2nd position—Figure 2B).

In the remaining trials (4/9th of the overall trials), the cue was followed by 3 s delay before the presentation of a second incidental cue which could be the same (Figure 2C) or different color (Figure 2D) to the first cue (Two cues condition). Participants were informed that the cue was orthogonal to the memory task.

The memory probe was a circle (5.7° visual angle in diameter) presented at the center of the screen with a line from the center positioned at a randomly-selected orientation. On trials with one incidental cue and those with two identical colored cues, the color of the probe was the same (50% of trials—congruent) or different (incongruent) to the color of the incidental cue(s).

In trials with two differently colored cues, on half the trials the probe was same color as the first cue while on the remaining the trials it was presented in the same color as the second cue. Using a mouse, participants adjusted the orientation of the line within the circle until it matched the direction of motion of the probed RDK. Accuracy of this matching procedure was emphasized over response time. Participants completed a practice block (30 trials) followed by seven blocks of 36 trials, different conditions randomly intermixed within a block.

Precision calculation

Recall error for the memory for motion task was calculated as the difference in response angle from target angle (i.e., the actual angle of the probed item). Recall precision was defined as the reciprocal of standard deviation of response error (Philipp, 2009). Note that precision is a measure of variability with higher precision corresponding to lower variability in memory. Due to small number of trials for all experiments reported here, we were unable to perform mixture modeling (Bays et al., 2009) to dissect our the sources of error resulting in a specific pattern of performance.

RESULTS

Participant performed the incidental cueing task with a high level of accuracy over all conditions (see Table 1 for mean accuracy and response times for all conditions). Trials in which there were incorrect responses to incidental cues were excluded from analysis.

Table 1. Mean accuracy and response times for the two cues for all conditions (accuracy in percentage and response times in ms).

We examined memory recall precision of motion direction for congruent vs. incongruent trials, corrected by performance in the baseline (no cue) condition in conditions where only one memory item was incidentally cued (i.e., the 1cue conditions, and two cues of the same color). There was an overall significant main effect of congruency, of whether the memory probe was about the same or different item to that specified by the previous incidental cue (F(1,19) = 11.06, p = 0.004). Thus recall precision was lower in the incongruent trials compared to congruent trials. This occurred in conditions when an incidental cue was presented in the first position (t(19) = 3.7, p = 0.001, Figure 2A), or second position (marginal significance, t(19) = 1.8, p = 0.084, Figure 2B) or when two cues of the same color were presented (t(19) = 2.27, p = 0.035, Figure 2C). The incongruency effect for an incidental cue in the second position was marginally significant (t(19) = 1.8, p = 0.084, Figure 2B). These findings indicate successful incidental cueing. Performing the incidental task (which is orthogonal to the motion direction memory task) made a systematic difference to recall precision on the memory task, depending on congruency of incidental cue to the RDK that was later probed.

Importantly, the cueing advantage was apparently driven by a decrease in recall precision on incongruent trials, compared to congruent trials. Compared to no-cue baseline condition (zero in figures), there were marginally significant decreases in recall precision on incongruent trials when one cue occupied the 1st position (t(19) = 1.89, p = 0.07, Figure 2A) and when two cues of the same color were presented (t(19) = 2.08, p = 0.051, Figure 2C). There was a significant drop in recall precision for incongruent trials when one cue occupied the 2nd position (t(19) = 2.28, p = 0.035, Figure 2B).

The critical condition, however, was when participants were presented with a sequence of two differently colored incidental cues. Note that in this case, we can consider the data in terms of whether the probe was the same as the item specified by the first incidental cue, or whether it was the same as the second cue. Prior to the second cue, one item would be considered to be in FOA. After it, however, another item might be brought into the FOA. This now becomes the “focused” item. Importantly recall precision for this item was neither significantly different to the no-cue/baseline (t(19) = 0.2, p = 0.85), nor to when a single cue was presented in 2nd position (t(19) = 0.3, p = 0.8; Figure 2D).

This finding shows that relatively high quality information regarding “other” items in WM—at least, similar to the baseline state—can be retrieved, in the context of a task where participants know that they might be asked to switch their attention between items in WM. This occurred despite the fact that such items were recalled with lower precision when they had not been brought into the privileged state of FOA by incidental cueing (Figures 2A–C).

Precision of recall for the “defocused” item in this condition, that is the item specified by the first cue and considered to be in FOA prior to the second cue was also no different than baseline (t(19) = 0.2, p = 0.83, Figure 2D) or to the item that was brought into FOA by the second cue (t(19) = 0.02, p = 0.99, Figure 2D). The lack of an effect on this item is discussed below.

DISCUSSION

These results demonstrate that in situations where items outside FOA remain potentially behaviorally relevant, they can be brought into FOA. The no cue, baseline condition used here allowed us to examine whether the effect of incidental cueing on WM performance was due to improvement of the item in FOA, or a degradation of the item outside it. The findings presented here show that the incidental cueing advantage can be explained specifically by a decrease in recall precision for the item outside FOA—rather than an advantage for the focused item—compared to baseline. So performing a task that requires consideration of the item specified by the incidental cue leads to a degradation of memory of the other item in WM, but no simultaneous boost to the cued item. Thus incidental cueing appears to operate in a very specific manner on the contents of WM.

A second important issue in these experiments is the fate of the de-focused item, that is the item that was inside FOA before the participant’s attention was switched to another item as the trial progresses. Here, recall precision for the defocused item was comparable to baseline, as well as focused condition—but not worse. This might not be predicted on the basis of attention being drawn away from this item and instead raises the possibility that the capacity of FOA is larger than 1 item.

EXPERIMENT 2: EFFECTS OF INCIDENTAL CUE ON RECENCY

In the second experiment we aimed to examine whether representational states of items in WM can flexibly change in situations in which all items in WM remain potentially, behaviorally relevant throughout the trial using a different method. Here, we used sequential presentation of items so that the last item was in FOA by virtue of recency. This was then followed by an incidental cue that could be same or different to the last item.

METHODS

Participants and stimuli

Seventeen healthy individuals (10 male) with an average age of 25 (range: 20–31) participated in Experiment 2. On each trial, two RDKs, consisting ether of red or green moving dots, were now presented sequentially at screen center (Figure 3A), subtending 10° of visual angle. Random dot kinematograms stimulus parameters were otherwise unchanged from Experiment 1.

FIGURE 3. (A) Schematic of trial events for Experiment 2. Following a sequential presentation of the memory array, the maintenance period was either blank or included an incidental cue before the presentation of the memory probe. (B) Recall precision for motion direction across conditions. Error bars are SEM.

Procedure

Each RDK was presented for 300 ms (+100 ms mask). There was 1000 ms blank delay between the two RDKs. In the baseline condition (not illustrated in the figure—1/3 of the overall trials), the 2nd RDK was then followed by an unfilled delay of 4.1 s. When cues were presented, the last RDK in the sequence was followed by a delay (1 s) before the presentation of the incidental cue (100 ms), which matched the first RDK in 50% of the trials (1/3 of the overall trials) and the last item for the remaining trials. In this experiment, unlike the previous one, participants had to indicate with a key press the serial position (first or second) of the RDK with the same color as the incidental cue. There was a 3 s delay, which was then followed by the presentation of the probe.

The probe was in the same color as the first RDK on half of the trials and in the remaining trials, it matched the last item in the sequence. This therefore resulted in four experimental conditions with a 2 × 2 design with color of incidental cue as the first factor and probe color as the second factor, both with two levels: color matching the first or second item in the sequence. Participants completed a practice block (30 trials) followed by seven blocks of 30 trials, different conditions randomly intermixed within a block.

RESULTS

Participant performed the incidental cueing task well, with mean accuracy of 95% (SD = 5.3) and mean response time of 878 ms (SD = 200 ms). Trials with incorrect response in the incidental cueing task were excluded from further analysis.

We next investigated the effect of incidental cueing on precision of recall of the item in FOA compared to other items in WM. In the baseline condition, we obtained the long-established recency effect: the last item was recalled with greater precision than the first. In trials when the first item—presumed not be in the FOA by the end of the sequence—was later cued by the incidental cue, there was a significant interaction between cueing conditions (baseline, cue 1st item, cue last item) and serial order (F(2,32) = 9.4, p = 0.001).

Comparison of the baseline, no-cue condition to incidentally cueing the 1st item, revealed a significant interaction between cueing condition and serial order (F(1,16) = 16.4, p = 0.001, Figure 3B.1). This was due to an increase in recall precision for the first item in the sequence (t(16) = 2.3, p = 0.035)—i.e., for the item now presumed to be in FOA—with a corresponding decrease in precision for the last, normally privileged, item (t(16) = 2.8, p = 0.012). This replicates our previous findings highlighting that in the context of an experiment where a non-privileged item in WM might be probed, it can be retrieved with relatively high precision after focusing attention upon it. In this experiment, unlike the first, we also found a cost in recall of the item that would otherwise be recalled with greater fidelity, here the last one in the sequence.

In trials where the last item in the sequence was cued, there was no significant interaction between cueing condition (cueing last item vs. baseline) and serial order of the probed item. The last item could not be recalled with any greater precision than in the baseline condition. As for the first item, although there was a modest increase in recall precision, this was not significant (t(16) = 1.8, p = 0.087, Figure 3B.2). It is possible that the small increase in recall precision for the first item was due to rehearsal of items once the last item was cued, before making a response regarding its serial position. Participants might go through the sequence in their mind, resulting in refocusing on the first item before focusing on the last item in the sequence. In fact, response times to incidental cues of the last item in the sequence was marginally longer compared to trials where the 1st item in the sequence is cued (t(16) = 1.9, p = 0.078).

DISCUSSION

The results from Experiments 1 and 2 confirm our general hypothesis that in situations where items outside FOA remain potentially behaviorally relevant, they can be brought into FOA. In Experiment 2, precision of memory decreased for the last item (i.e., the item presumed to be in FOA) in trials where the incidental cue directed attention to the first item, which was now recalled with greater precision than baseline (Figure 3B.1). This simultaneous cost vs. benefit effect is different to that observed in Experiment 1 where recall precision for the defocused item was comparable to baseline, as well as focused condition. However, it is important to note that the effects of Experiment 1 were overall weaker (perhaps due to a much longer delay between encoding and probing of memory than in Experiment 2: 7.2 s vs. 4.1); this could possibly influence the lack of any effect of the de-focused item in the previous experiment.

The effects observed in Experiment 2 might be explained by two different mechanisms. It is possible that the capacity of the FOA is limited, perhaps to just one item, so that in trials where attention shifts to focus on another item in memory, the previously privileged item is displaced from FOA. However, this hypothesis would be inconsistent with some previous observations. Specifically, Rerko and Oberauer (2013) demonstrated an opposite effect for the defocused item than that reported here, with its recall significantly better compared to other items in WM for shorter delay periods.

Alternatively, it is possible that the time for which an item might be elevated to FOA by means of some types of cueing is limited. This is supported by EEG findings measuring temporal changes in item representation over the course of a delay period. LaRocque et al. (2013) employed an identical design to that of Lewis-Peacock et al. (2011) with EEG, demonstrating that approximately 1.25 s after cueing the unfocused item, classification accuracy for the initially cued item is decreased to levels comparable to baseline.

RETRO-CUEING

If the item in FOA is disrupted by either having attention switch to focus on another item in WM (Experiments 1 and 2) or artificially through TMS to early sensory areas (Zokaei et al., 2014), the item outside FOA can improve to levels comparable to the previously privileged item. However, as discussed previously, this is dependent on context: in the previous experiments reported above, non-FOA items remained relevant to the task in hand. Therefore, the question that remains is what happens to the other items in WM when they are rendered largely irrelevant?

Previous literature on this remains controversial (e.g., Landman et al., 2003; Lepsien and Nobre, 2007; Matsukura et al., 2007; Rerko and Oberauer, 2013). Due to differences in methodology comparison across studies to make meaningful general conclusions is difficult. To compare both the behavioral influences of re-focusing attention to other items in WM as well the causal effects of TMS, we conducted two retro-cueing investigations that are closely matched in timing and stimuli to two of our previous studies.

In Experiment 3 we used two successive cues, with the first one being a retro-cue, informative about the nature of the upcoming probe (80% validity). To ensure that participants attended to this cue, the second cue was either a stay or a switch cue, with 20% of the trials a switch cue. Importantly, this second cue was 100% valid, so participants knew for sure which item in WM was going to be probed. We aimed to investigate whether people can bring back the previously rendered irrelevant item following a switch trial. This experiment otherwise matched our double incidental cueing paradigm (Experiment 1) with two successive cues in timing and stimuli.

In Experiment 4 we used a similar retro-cue paradigm to that previously employed by Zokaei et al. (2014) (their Experiment 2) with an 80% valid rather than an incidental cue. Transcranial magnetic stimulation was administered during WM maintenance following the presentation of the cue to examine the causal role of early visual areas in maintenance of item in different representational states achieved through retro-cues and importantly its influence on the un-cued items.

EXPERIMENT 3: STAY OR SWITCH FROM ITEM IN WM THAT WAS RETRO-CUED

METHODS

Participants

Seventeen healthy individuals (14 Male) with an average age of 28 (range: 22–34) participated in Experiment 3. All had normal or corrected to normal vision and reported normal color vision. Participants provided written consent to the procedure of the experiment, approved by the local ethics committee.

Procedure

In each trial, similar to Experiment 1, two RDKs were presented above and below the fixation cross (300 ms) followed by a mask (100 ms; see Figure 4A). This was followed by a blank delay of 1000 ms prior to the presentation of the retro-cue, which was in the form of fixation cross color change (100 ms). The fixation cross changed from black either to red or green (50% green) indicating the RDK that was more likely to be the cued later in the trial (80% validity for the next cue). This was then followed by a 3000 ms delay, before presentation of the second cue.

FIGURE 4. (A) Schematic of trial events for Experiment 3. Memory array is followed by a retro-cue with 80% validity regarding the 2nd cue. The second stay/switch cue was 100% valid for the upcoming probe. (B) Recall precision for the motion direction across conditions—baseline corrected. Error bars are SEM.

The second cue was 100% valid and was in the form of a word instructing participants to either “stay”, i.e., maintain the same colored RDK as cued before (80% of trials) or “switch”, i.e., switch to the other item WM (20% of trials). Note that in the vast majority of trials, therefore, the first retro-cue could be used to focus attention largely on one item held in WM.

Following a 3000 ms interval the item indicated by the second cue was probed and participants had to adjust the direction of the probe to match that of the cued item. These trials were intermixed with 30 baseline trials with no cueing during the delay but rather a long blank period (7200 ms) before presentation of the probe. Note that in this experiment, we used a switch/stay (second) cue in order to make sure participants attended to the first cue as no response to the that cue was required.

Participants completed a practice block (30 trials) followed by five blocks of 46 trials, different conditions randomly intermixed within a block and were informed of the validity of the cue prior to testing.

RESULTS

We first applied an ANOVA with cueing condition (valid, no-cue and invalid) as within-subject factors. There was a marginal effect of cue-type; F(2,30) = 3.15, p = 0.057. Furthermore, recall precision was significantly higher in (80%) stay compared to (20%) switch trials (t(14) = 2.7, p = 0.0.16; Figure 4B). With respect to the baseline (no-cue) condition, recall precision was not significantly different in stay trials (t(14) = 0.8, p = 0.4), but significantly decreased in switch trials (t(14) = 2.45, p = 0.027).

DISCUSSION

In this experiment the item that was initially retro-cued appeared to be held in a privileged state compared to the other item, which was effectively rendered largely irrelevant, because it was probed on only 20% of trials. These findings are consistent with the hypothesis that in situations where items outside FOA in WM are rendered largely irrelevant by the very low probability of being probed, they cannot subsequently be brought into FOA—at least to the level of precision of cued items—by having attention switched to them. The results are in line with previous experiments using informative cues to direct attention to one/subset of items in WM (e.g., Lepsien and Nobre, 2007; Matsukura et al., 2007; Pertzov et al., 2012). Next we used TMS to area MT+ in an attempt to disrupt the item in FOA.

EXPERIMENT 4: EFFECTS OF TMS TO AREA MT+

METHODS

Participants

Fifteen healthy individuals (8 Male) with average age of 25 (range: 18–32) participated in Experiment 4. All had normal or corrected to normal vision and reported normal color vision. Participants provided written consent to the procedure of the experiment, approved by the local ethics committee.

MT+ localization and TMS

A standard approach to MT+ localization using fMRI was applied (Huk et al., 2002). Left hemisphere clusters in the vicinity of MT+ (using anatomical guidelines described by Dumoulin et al., 2000) were identified in the native space of each participant and were overlaid onto their T1-weighted scan for a Brainsight frameless stereotaxy procedure (Rogue Research, Montreal, Canada). The participant’s scalp location of left MT+ was marked on their scalp for subsequent TMS.

Stimulation was delivered via Magtism Rapid^2 (The Magstim Company, Whitland, Wales, U.K.) using a 70-mm figure-eight coil. The coil handle pointed posteriorly rotated 45º, including a current approximately in the anterior or posterior direction. On each trial, 4 TMS pulses at 20 Hz were applied to left MT+ either at 60% (“high”, effective intensity) or at 24% (“low”, ineffective intensity) of maximum machine output. Low intensity trials were used to control for non-specific effects of TMS e.g., acoustic and tactile artifacts.

Procedure

Similar to Experiments 1 and 3, two RDKs were presented above and below fixation cross (300 ms) followed by a mask (100 ms; see Figure 5A) and blank delay (1000 ms). The fixation cross then changed briefly (similar to Experiment 3), out method of retro-cueing with 80% validity. This was then followed by a delay of 2600 ms before the administration of TMS. The TMS train lasted for 250 ms before presentation of the probe. The probe matched the color of the retro-cue in 80% of the trials (valid trials) while in the remaining 20% it did not (invalid trials). Trial sequence matched closely the timings of Zokaei et al. (2014) (their Experiment 2).

FIGURE 5. (A) Schematic of trial events for Experiment 4. Memory array is followed by a retro-cue with 80% validity. A 20 Hz four pulse TMS train was administered to MT+, followed by a delay and the presentation of the probe. (B) Recall precision for the motion direction across conditions. Error bars are SEM.

Participants completed a practice block (30 trials) followed by four blocks of 104 trials, different conditions randomly intermixed within a block.

RESULTS

In this experiment we compared high intensity (effective) TMS to low intensity (ineffective) TMS. Recall precision was significantly higher in valid (i.e., trials in which the probe matched the cue) compared to invalid trials (t(14) = 3.2, p = 0.006), indicating successful retro-cueing (Figure 5B).

We next assessed whether high vs. low intensity TMS affected recall precision. A two-way ANOVA with factors TMS intensity and trial type yielded a significant main effect of validity (F(1,14) = 9.5, p = 0.008) and TMS intensity (F(1,14) = 4.8, p = 0.045). The effect of TMS was due to a significant decrease in precision of memory after high intensity TMS on valid trials (t(14) = 2.4, p = 0.029, n.s after correcting for multiple comparisons), importantly with no change on invalid trials (t(14) = 0.7, n.s.). Thus the privileged item sustained a cost with high intensity TMS. Nevertheless, the behavioral advantage of validity remained, with memory precision being significantly higher for valid compared to invalid trials (t(14) = 2.3, p = 0.039, n.s after correcting for multiple comparisons).

DISCUSSION

The results from the TMS experiment are consistent with the hypothesis that in situations where items outside FOA in WM are rendered largely irrelevant by the very low probability of being probed, they cannot be brought into FOA by disrupting the item in FOA with TMS. Here, we used an analogous procedure to that in our previous published study (Zokaei et al., 2014) with the only distinction of having a retro-cue rather than an incidental cue. To the best of our knowledge, that report is one of the first to investigate the causal role of early visual areas in maintenance of the retro-cued item in WM demonstrating that the maintenance of the cued item, in line with previous imaging studies (Lepsien and Nobre, 2007; Harrison and Tong, 2009; Nelissen et al., 2013), relies to some extent on visual areas involved in its perception (Zokaei et al., 2014).

In the current experiment, TMS to MT+ did not have an effect on the other item retained in WM, unlike the previously observed result with an incidental cue (Zokaei et al., 2014). In the study reported here, there was no improvement in recall precision for the other item in memory presumably because high quality information regarding this item was lost because of the extremely low (20%) probability of being probed. It is, however, important to note that the smaller number of invalid relative to valid trials may also explain the lack of TMS effect on the invalid condition.

GENERAL DISCUSSION

The studies reported here sought to investigate the flexibility of representational states in WM: whether retained information can be flexibly moved in and out of the privileged state of FOA. Across four experiments, we explicitly and implicitly manipulated the state of two items in WM using different methods in order to place one in the FOA. Taken together, our results suggest that although the item in FOA is represented with higher recall precision compared to the non-privileged items in all experiments, the nature of the other items in WM crucially depends on the relevance of these items to the WM task. The context in which people are required to hold more than one item in WM makes a difference to how flexibly they can switch attention to non-focused items and improve their precision of recall. Also interesting to note is that the item in FOA was never recalled with greater precision than uncued items in the baseline condition. This suggests that the benefit conferred by the FOA is not necessarily due to straightforward increase in the precision with which the item is maintained compared to all other items. Indeed, this result fits with those of Gorgoraptis et al. (2011) in which recall precision of the last item in the sequence (i.e., the item in FOA) was similar to the precision of items when presented simultaneously regardless of memory set size. The implication is therefore that the FOA benefit may also depend on the context of the experiment.

In Experiment 1 we used either one or two consecutive incidental cues to explicitly bring the cued item into FOA: by requiring participants to make an action regarding a feature (in this case, location) of one of the retained items during WM delay period. The consequence of such incidental cueing is that this item, if probed later (congruent condition), will be recalled with higher precision than if the alternative item is probed (incongruent condition) (Zokaei et al., 2014). In Experiment 2 we used sequential presentation of the memory array followed by an incidental cue (in this case regarding, serial order), with the assumption that the last item in the sequence was in the privileged state by virtue of recency. Our working hypothesis, based on previous published studies discussed in the Section Introduction, is that the last cue in both these experiments places the cued item in the FOA. But importantly it carries no information regarding the relevance of the cued item for the upcoming WM task.

The results from these two studies demonstrated that in fact in situations where the item outside FOA remains potentially relevant to the task in hand, it can be brought back into the privileged state. Theoretically, these findings are in line with predictions made by the three-embedded components theory (Oberauer, 2002, 2009; Rerko and Oberauer, 2013) where no permanent detrimental effect on the other items in WM is predicted. In the context of this theoretical framework, the decrease in recall precision of the unfocused items might be due to direct interference from the item in FOA (e.g., Pertzov et al., 2012). Thus abolishing such interference, either through disruptive effects of TMS (Zokaei et al., 2014) or by re-focusing attention on non-privileged items (as in Experiments 1 and 2 here), can increase their WM precision to levels comparable to baseline.

If the items outside FOA are indeed maintained and can be brought back into FOA, then what are the mechanisms underlying their retention? A few studies have demonstrated a role of the hippocampus in maintenance of items outside FOA, with their retrieval accompanied by activation in MTL (Oztekin et al., 2009, 2010; Nee and Jonides, 2011). Consistent with such observations, patients with hippocampal sclerosis have impaired WM performance only for items presented earlier in the sequence, that is items outside the FOA (López-Frutos et al., 2014). Alternatively, it has been suggested that the maintenance of unfocused items is supported either through sustained rapid short-term synaptic plasticity (Mongillo et al., 2008; Buonomano and Maass, 2009; Stokes et al., 2013) or sustained neuronal firing in non-sensory regions (Fuster and Alexander, 1971; Goldman-Rakic, 1995). However, support for each of these accounts is sparse and the brain mechanisms by which an item is brought into FOA have yet to be elucidated.

Unlike in the first two experiments, in Experiments 3 and 4 we manipulated representational states using retro-cues, i.e., cues that appear to focus attention on the most relevant item in WM for the upcoming probe. In Experiment 3, we used two consecutive cues: the first one was a retro-cue, informing participants on the likely nature of the second cue with 80% validity. The second cue was 100% valid for the WM probe. Recall precision for the invalidly cued WM item (i.e., the item probed on only 20% of trials) was significantly lower compared to the validly cued one, despite having attention switch to this item with a second 100% valid retro-cue. Therefore, the irrelevant item in WM following a retro-cue could not be brought back into FOA, despite having attention focus on it via the second retro-cue. Thus some information was, to some extent, irretrievably lost from WM following the initial retro-cue.

In Experiment 4, the precision of memory for the other/irrelevant items did not improve following disruptive effects of TMS to the item in FOA, unlike previous studies using incidental cueing or sequential presentation of items (Zokaei et al., 2014). However, while the probability of being probed regarding the invalid item was extremely low (20%) in the TMS study we report here, in the incidental cueing study of Zokaei et al. (2014), there was a 50% probability of being probed on the uncued item.

In that report, the improvement of memory for the non-FOA item in WM following TMS was described in terms of weakening of the interference from the item in FOA on the non-privileged item. According to this hypothesis, TMS abolished the advantage of the item in FOA, leaving “baseline” WM performance intact and resulting in normalized performance for all items in WM. In accordance with such hypothesis, in the retro-cueing TMS study reported here (Experiment 4), “baseline” performance for the non-FOA items was decreased due to validity of the cue, prior to the administration of TMS. Abolishing the interference of item in the privileged state would therefore not be expected to improve the quality of the memory for the non-FOA item since it is maintained with low precision to begin with.

Findings from previous studies are not all consistent. Few have reported a cueing advantage (Lepsien and Nobre, 2007; Matsukura et al., 2007; Pertzov et al., 2012) while others have failed to observe one (Landman et al., 2003; Rerko and Oberauer, 2013). Current findings however can potentially explain such discrepancy in the literature if one considers the amount of information conveyed by the retro-cue regarding the likelihood of being probed on uncued items. In experiments that demonstrate a cueing advantage, the cue was valid, i.e., the cued item was later probed in majority of trials. On the other hand, in studies that have failed to report any effect of cueing, the cue was only 50% valid and hence no different to a no cue condition. In addition, some studies using two consecutive cues have no method of confirming whether participants attended to the cues or not (e.g., Landman et al., 2003; Rerko and Oberauer, 2013), unlike the methods used here. Together these factors may well contribute to the inconsistency in the literature.

Our findings, in light of previous literature, suggest that the magnitude of cueing is dependent on the amount of information conveyed by the cue—the context in which WM holds items. Future research might aim to directly test this hypothesis by varying the validity of the cue systematically, although as the frequency of invalid cues is reduced the number of trials required to generate sufficient data would increase massively. Furthermore, current advances in modeling the sources of error associated with a specific pattern of performance may shed light on possible mechanisms that an item gains its privileged state (Bays et al., 2009). Similar findings with memory set sizes larger than two are also essential in building a comprehensive understanding of flexibility of representational states in WM.

The findings reported here also have important implications for both theoretical and neural computational models of WM. Although some have proposed dynamic representational states of items during WM retention (Fujisawa et al., 2008; Barak et al., 2010; Pascanu and Jaeger, 2011; Stokes et al., 2013), most models assume static representations (Seung and Sompolinsky, 1993; Lisman et al., 1998; Compte et al., 2000; Durstewitz et al., 2000; Mongillo et al., 2008; Dempere-Marco et al., 2012; Wei et al., 2012). Our results provide support for remarkable flexibility of representations in WM dependent upon context, and its close relationship to directed attention. Moreover, state-models of WM do not often make the distinction between other/non-privileged items in WM in different contexts, assuming a similar fate for all these items irrespective of how they have achieved their state. Our results provide strong empirical support for differential effects of cue validity—context of the WM task—on the nature of items outside FOA. Given the compelling evidence provided here, both computational and theoretical models of WM might profitably take into account dynamic representational states in WM when several items are retained.

ACKNOWLEDGMENTS

Research was supported by the Wellcome Trust (grant number: 098282).

REFERENCES

Astle, D. E., Scerif, G., Kuo, B.-C., and Nobre, A. C. (2009). Spatial selection of features within perceived and remembered objects. Front. Hum. Neurosci. 3:6. doi: 10.3389/neuro.09.006.2009

Baddeley, A. (2003). Working memory: looking back and looking forward. Nat. Rev. Neurosci. 4, 829–839. doi: 10.1038/nrn1201

Barak, O., Tsodyks, M., and Romo, R. (2010). Neuronal population coding of parametric working memory. J. Neurosci. 30, 9424–9430. doi: 10.1523/jneurosci.1875-10.2010

Bays, P. M., Catalao, R. F. G., and Husain, M. (2009). The precision of visual working memory is set by allocation of a shared resource. J. Vis. 9, 7.1–7.11. doi: 10.1167/9.10.7

Berryhill, M. E., Richmond, L. L., Shay, C. S., and Olson, I. R. (2012). Shifting attention among working memory representations: testing cue type, awareness and strategic control. Q. J. Exp. Psychol. (Hove) 65, 426–438. doi: 10.1080/17470218.2011.604786

Bisley, J. W., and Pasternak, T. (2000). The multiple roles of visual cortical areas MT/MST in remembering the direction of visual motion. Cereb. Cortex 10, 1053–1065. doi: 10.1093/cercor/10.11.1053

Blalock, L. D., and Clegg, B. A. (2010). Encoding and representation of simultaneous and sequential arrays in visuospatial working memory. Q. J. Exp. Psychol. (Hove) 63, 856–862. doi: 10.1080/17470211003690680

Buonomano, D. V., and Maass, W. (2009). State-dependent computations: spatiotemporal processing in cortical networks. Nat. Rev. Neurosci. 10, 113–125. doi: 10.1038/nrn2558

Compte, A., Brunel, N., Goldman-Rakic, P. S., and Wang, X. J. (2000). Synaptic mechanisms and network dynamics underlying spatial working memory in a cortical network model. Cereb. Cortex 10, 910–923. doi: 10.1093/cercor/10.9.910

Cowan, N. (1998). Attention and Memory: An Integrated Framework. NY: OUP.

Dempere-Marco, L., Melcher, D. P., and Deco, G. (2012). Effective visual working memory capacity: an emergent effect from the neural dynamics in an attractor network. PLoS One 7:e42719. doi: 10.1371/journal.pone.0042719

Dumoulin, S. O., Bittar, R. G., Kabani, N. J., Baker, C. L. Jr., Le Goualher, G., Bruce Pike, G., et al. (2000). A new anatomical landmark for reliable identification of human area V5/MT: a quantitative analysis of sulcal patterning. Cereb. Cortex 10, 454–463. doi: 10.1093/cercor/10.5.454

Durstewitz, D., Seamans, J. K., and Sejnowski, T. J. (2000). Neurocomputational models of working memory. Nat. Neurosci. 3, 1184–1191. doi: 10.1038/81460

Fujisawa, S., Amarasingham, A., Harrison, M. T., and Buzsáki, G. (2008). Behavior-dependent short-term assembly dynamics in the medial prefrontal cortex. Nat. Neurosci. 11, 823–833. doi: 10.1038/nn.2134

Fuster, J. M., and Alexander, G. E. (1971). Neuron activity related to short-term memory. Science 173, 652–654. doi: 10.1126/science.173.3997.652

Goldman-Rakic, P. S. (1995). Cellular basis of working memory. Neuron 14, 477–485. doi: 10.1016/0896-6273(95)90304-6

Gorgoraptis, N., Catalao, R. F. G., Bays, P. M., and Husain, M. (2011). Dynamic updating of working memory resources for visual objects. J. Neurosci. 31, 8502–8511. doi: 10.1523/jneurosci.0208-11.2011

Griffin, I. C., and Nobre, A. C. (2003). Orienting attention to locations in internal representations. J. Cogn. Neurosci. 15, 1176–1194. doi: 10.1162/089892903322598139

Harrison, S. A., and Tong, F. (2009). Decoding reveals the contents of visual working memory in early visual areas. Nature 458, 632–635. doi: 10.1038/nature07832

Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L., and Pietrini, P. (2001). Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425–2430. doi: 10.1126/science.1063736

Hay, D. C., Smyth, M. M., Hitch, G. J., and Horton, N. J. (2007). Serial position effects in short-term visual memory: a SIMPLE explanation? Mem. Cognit. 35, 176–190. Available online at: http://eprints.lancs.ac.uk/11411/. [Accessed on July 8, 2011]. doi: 10.3758/bf03195953

Haynes, J.-D., and Rees, G. (2006). Decoding mental states from brain activity in humans. Nat. Rev. Neurosci. 7, 523–534. doi: 10.1038/nrn1931

Huk, A. C., Dougherty, R. F., and Heeger, D. J. (2002). Retinotopy and functional subdivision of human areas MT and MST. J. Neurosci. 22, 7195–7205.

Landman, R., Spekreijse, H., and Lamme, V. A. F. (2003). Large capacity storage of integrated objects before change blindness. Vision Res. 43, 149–164. doi: 10.1016/s0042-6989(02)00402-9

LaRocque, J. J., Lewis-Peacock, J. A., Drysdale, A. T., Oberauer, K., and Postle, B. R. (2013). Decoding attended information in short-term memory: an EEG study. J. Cogn. Neurosci. 25, 127–142. doi: 10.1162/jocn_a_00305

Larocque, J. J., Lewis-Peacock, J. A., and Postle, B. R. (2014). Multiple neural states of representation in short-term memory? It’s a matter of attention. Front. Hum. Neurosci. 8:5. doi: 10.3389/fnhum.2014.00005

Lepsien, J., and Nobre, A. C. (2007). Attentional modulation of object representations in working memory. Cereb. Cortex 17, 2072–2083. doi: 10.1093/cercor/bhl116

Lepsien, J., Thornton, I., and Nobre, A. C. (2011). Modulation of working-memory maintenance by directed attention. Neuropsychologia 49, 1569–1577. doi: 10.1016/j.neuropsychologia.2011.03.011

Lewis-Peacock, J. A., Drysdale, A. T., Oberauer, K., and Postle, B. R. (2011). Neural evidence for a distinction between short-term memory and the focus of attention. J. Cogn. Neurosci. 24, 61–79. doi: 10.1162/jocn_a_00140

Lisman, J. E., Fellous, J.-M., and Wang, X.-J. (1998). A role for NMDA-receptor channels in working memory. Nat. Neurosci. 1, 273–275. doi: 10.1038/1086

López-Frutos, J. M., Poch, C., García-Morales, I., Ruiz-Vargas, J. M., and Campo, P. (2014). Working memory retrieval differences between medial temporal lobe epilepsy patients and controls: a three memory layer approach. Brain Cogn. 84, 90–96. doi: 10.1016/j.bandc.2013.11.004

Makovsik, T., and Jiang, Y. V. (2007). Distributing versus focusing attention in visual short-term memory. Psychon. Bull. Rev. 14, 1072–1078. doi: 10.3758/bf03193093

Matsukura, M., Luck, S. J., and Vecera, S. P. (2007). Attention effects during visual short-term memory maintenance: protection or prioritization? Percept. Psychophys. 69, 1422–1434. doi: 10.3758/BF03192957

McElree, B. (1998). Attended and non-attended states in working memory: accessing categorized structures. J. Mem. Lang. 38, 225–252. doi: 10.1006/jmla.1997.2545

McElree, B., and Dosher, B. A. (1989). Serial position and set size in short-term memory: the time course of recognition. J. Exp. Psychol. Gen. 118, 346–373. doi: 10.1037/0096-3445.118.4.346

McElree, B., and Dosher, B. A. (1993). Serial retrieval processes in the recovery of order information. J. Exp. Psychol. Gen. 122, 291–315. doi: 10.1037/0096-3445.122.3.291

Mongillo, G., Barak, O., and Tsodyks, M. (2008). Synaptic theory of working memory. Science 319, 1543–1546. doi: 10.1126/science.1150769

Neath, I. (1993). Distinctiveness and serial position effects in recognition. Mem. Cognit. 21, 689–698. doi: 10.3758/bf03197199

Nee, D. E., and Jonides, J. (2008). Neural correlates of access to short-term memory. Proc. Natl. Acad. Sci. U S A 105, 14228–14233. doi: 10.1073/pnas.0802081105

Nee, D. E., and Jonides, J. (2011). Dissociable contributions of prefrontal cortex and the hippocampus to short-term memory: evidence for a 3-state model of memory. Neuroimage 54, 1540–1548. doi: 10.1016/j.neuroimage.2010.09.002

Nee, D. E., and Jonides, J. (2013). Trisecting representational states in short-term memory. Front. Hum. Neurosci. 7:796. Available online at: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3840432/ [Accessed on April 3, 2014]. doi: 10.3389/fnhum.2013.00796

Nelissen, N., Stokes, M., Nobre, A. C., and Rushworth, M. F. S. (2013). Frontal and parietal cortical interactions with distributed visual representations during selective attention and action selection. J. Neurosci. 33, 16443–16458. doi: 10.1523/jneurosci.2625-13.2013

Oberauer, K. (2002). Access to information in working memory: exploring the focus of attention. J. Exp. Psychol. Learn. Mem. Cogn. 28, 411–421. doi: 10.1037//0278-7393.28.3.411

Oberauer, K. (2005). Binding and inhibition in working memory: individual and age differences in short-term recognition. J. Exp. Psychol. Gen. 134, 368–387. doi: 10.1037/0096-3445.134.3.368

Oberauer, K. (2009). “Chapter 2 design for a working memory,” in Psychology of Learning and Motivation, ed B. H. Ross (USA: Academic Press), 45–100. Available online at: http://www.sciencedirect.com/science/article/pii/S007974210951002X. [Accessed on May 10, 2013].

Oztekin, I., Davachi, L., and McElree, B. (2010). Are representations in working memory distinct from representations in long-term memory? Neural evidence in support of a single store. Psychol. Sci. 21, 1123–1133. doi: 10.1177/0956797610376651

Oztekin, I., McElree, B., Staresina, B. P., and Davachi, L. (2009). Working memory retrieval: contributions of the left prefrontal cortex, the left posterior parietal cortex and the hippocampus. J. Cogn. Neurosci. 21, 581–593. doi: 10.1162/jocn.2008.21016

Pascanu, R., and Jaeger, H. (2011). A neurodynamical model for working memory. Neural Netw. 24, 199–207. doi: 10.1016/j.neunet.2010.10.003

Pasternak, T., and Greenlee, M. W. (2005). Working memory in primate sensory systems. Nat. Rev. Neurosci. 6, 97–107. doi: 10.1038/nrn1603

Pertzov, Y., Bays, P. M., Joseph, S., and Husain, M. (2012). Rapid forgetting prevented by retrospective attention cues. J. Exp. Psychol. Hum. Percept. Perform. 39, 1224–1231. doi: 10.1037/a0030947

Philipp, B. (2009). CircStat: a MATLAB toolbox for circular statistics. J. Stat. Softw.

Rerko, L., and Oberauer, K. (2013). Focused, unfocused and defocused information in working memory. J. Exp. Psychol. Learn. Mem. Cogn. 39, 1075–1096. doi: 10.1037/a0031172

Seung, H. S., and Sompolinsky, H. (1993). Simple models for reading neuronal population codes. Proc. Natl. Acad. Sci. U S A 90, 10749–10753. doi: 10.1073/pnas.90.22.10749

Sligte, I. G., Scholte, H. S., and Lamme, V. A. F. (2008). Are there multiple visual short-term memory stores? PLoS One 3:e1699. doi: 10.1371/journal.pone.0001699

Sligte, I. G., Vandenbroucke, A. R. E., Scholte, H. S., and Lamme, V. A. F. (2010). Detailed sensory memory, sloppy working memory. Front. Psychol. 1:175. doi: 10.3389/fpsyg.2010.00175

Stokes, M. G., Kusunoki, M., Sigala, N., Nili, H., Gaffan, D., and Duncan, J. (2013). Dynamic coding for cognitive control in prefrontal cortex. Neuron 78, 364–375. doi: 10.1016/j.neuron.2013.01.039

Wei, Z., Wang, X.-J., and Wang, D.-H. (2012). From distributed resources to limited slots in multiple-item working memory: a spiking network model with normalization. J. Neurosci. 32, 11228–11240. doi: 10.1523/jneurosci.0735-12.2012

Zokaei, N., Gorgoraptis, N., Bahrami, B., Bays, P. M., and Husain, M. (2011). Precision of working memory for visual motion sequences and transparent motion surfaces. J. Vis. 11:2. doi: 10.1167/11.14.2

Zokaei, N., Heider, M., and Husain, M. (2013). Attention is required for maintenance of feature binding in visual working memory. Q. J. Exp. Psychol. (Hove) 67, 1191–1213. doi: 10.1080/17470218.2013.852232

Zokaei, N., Manohar, S., Husain, M., and Feredoes, E. (2014). Causal evidence for a privileged working memory state in early visual cortex. J. Neurosci. 34, 158–162. doi: 10.1523/jneurosci.2899-13.2014

Conflict of Interest Statement: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Received: 28 July 2014; Accepted: 03 October 2014; Published online: 06 November 2014.

Citation: Zokaei N, Ning S, Manohar S, Feredoes E and Husain M (2014) Flexibility of representational states in working memory. Front. Hum. Neurosci. 8:853. doi: 10.3389/fnhum.2014.00853

This article was submitted to the journal Frontiers in Human Neuroscience.

Copyright © 2014 Zokaei, Ning, Manohar, Feredoes and Husain. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

HYPOTHESIS AND THEORY ARTICLE
published: 25 November 2014
doi: 10.3389/fnhum.2014.00932

Finding the answer in space: the mental whiteboard hypothesis on serial order in working memory

Elger Abrahamse[1]*†, Jean-Philippe van Dijck[1]†, Steve Majerus2,[3] and Wim Fias[1]

1Department of Experimental Psychology, Faculty of Psychology and Educational Sciences, Ghent University, Ghent, Belgium

2Department of Psychology, Cognition and Behavior, University of Liège, Ghent, Belgium

3Fund for Scientific Research FNRS, Belgium

* Correspondence: Elger Abrahamse, Department of Experimental Psychology, Faculty of Psychology and Educational Sciences, Ghent University, Henri Dunantlaan 2, 9000 Ghent, Belgium e-mail: elger.abrahamse@ugent.be

[†]These authors have contributed equally to this paper and should be considered joint first authors.

Edited by:
Lynne E. Bernstein, George Washington University, USA

Reviewed by:
Nelson Cowan, University of Missouri, USA
Valerie Camos, Université de Fribourg, Switzerland

Various prominent models on serial order coding in working memory (WM) build on the notion that serial order is achieved by binding the various items to-be-maintained to fixed position markers. Despite being relatively successful in accounting for empirical observations and some recent neuro-imaging support, these models were largely formulated on theoretical grounds and few specifications have been provided with respect to the cognitive and/or neural nature of these position markers. Here we outline a hypothesis on a novel candidate mechanism to substantiate the notion of serial position markers. Specifically, we propose that serial order WM is grounded in the spatial attention system: (I) The position markers that provide multi-item WM with a serial context should be understood as coordinates within an internal, spatially defined system; (II) internal spatial attention is involved in searching through the resulting serial order representation; and (III) retrieval corresponds to selection by spatial attention. We sketch the available empirical support and discuss how the hypothesis may provide a parsimonious framework from which to understand a broad range of observations across behavioral, neural and neuropsychological domains. Finally, we pinpoint what we believe are major questions for future research inspired by the hypothesis.

Keywords: serial order, working memory, space, positional models, position marker, spatial attention, hypothesis

SERIAL ORDER WORKING MEMORY AND SPATIAL PROCESSING

Working memory (WM) is a fundamental cognitive function and refers to the brief maintenance of information in an active and accessible state such that operations can be performed on it. It is considered to be crucial for major cognitive skills like language, reasoning and learning, not in the least for its core feature of maintaining serial order across multiple items (e.g., Baddeley, 2012). Without the ability to maintain serial order across items in WM, for example, it would be a tremendously effortful job to calculate the overall price of your purchases in a shop, to dial the phone number of a friend, to make yourself a decent sandwich or to construct a line of reasoning. In this paper we present a novel hypothesis on the nature of serial order WM.

The study on how serial order is coded within WM has a strong empirical tradition (Ebbinghaus, 1885 [1964]; Lashley, 1951; Sternberg, 1967; for a review see Marshuetz, 2005) and several theoretical models have been described. Broadly speaking, these models can be divided into two classes: associative chaining and position marker models. Associative chaining was one of the earliest approaches (e.g., Ebbinghaus, 1885 [1964]), the basic underlying idea being that serial order derives from associations between successive items and that each item acts as a retrieval cue for the next item (e.g., Lashley, 1951). Chaining models have been very efficient in explaining a hallmark observation in serial order recall performance: the gradual increase in RT when retrieving order information later in the memorized sequence (e.g., Sternberg, 1967). However, several objections to chaining models can be identified as well. Chaining models have difficulties in explaining the error patterns that are typically observed in serial order retrieval. For example, because it is assumed that order is encoded by contiguous associations between items, recall should fail for items following an erroneous recall. This is not typically observed. In addition, chaining models also have difficulties to explain distance-effects observed in WM (i.e., the observation that it is more difficult to determine the serial order for serially nearby compared to more distant items; Marshuetz et al., 2000; Attout et al., 2014). For (mainly) these reasons, theorists gradually rejected the hypothesis that chaining plays a crucial role in serial order memory (e.g., Henson et al., 1996; Farrell and Lewandowsky, 2002; Burgess and Hitch, 2006) and shifted towards the currently dominant position marker models.

Position marker models build on the idea that serial order coding in WM is achieved by binding the various items to-be-maintained to specific position markers (e.g., begin vs. end items, Henson, 1998; encoding strength, Page and Norris, 1998; oscillatory response, Brown et al., 2000; rank codes, Botvinick and Watanabe, 2007) and that serial order retrieval is achieved by recalling this conjunction. Despite being relatively successful in accounting for empirical observations and despite recent neuro-imaging support for the existence of position markers (Kalm and Norris, 2014), these models are still largely formulated on theoretical grounds and few specifications have been provided with respect to the cognitive and/or neural nature of position markers (but see below on Botvinick and Watanabe, 2007). Here we hypothesize on a novel candidate mechanism to substantiate the notion of serial position markers by relating it to the spatial attention system.

THE MENTAL WHITEBOARD HYPOTHESIS ON SERIAL ORDER WM

Before we present our hypothesis we start by outlining the empirical background to it, which involves interactions between serial order and spatial processing. First, van Dijck and Fias (2011) asked participants to maintain in WM a series of fruit and vegetable names in the order of presentation, with item presentation always centrally on the screen. It was observed that retrieving early items from this WM sequence facilitated a left hand response, while later items facilitated a right hand response. Importantly, using such non-spatial and centrally presented material prevented any confounding with involvement of spatial processing related to stimulus presentation. This study was the first to indicate a close link between serial order in WM and spatial processing.

In a second step, we confirmed that internal selective attention is intrinsically involved in searching through the serial order representation in WM. From the notion that there is direct interfacing between internal and external selective attention (Downing, 2000; Awh and Jonides, 2001; Corbetta and Shulman, 2002; Nobre et al., 2006; Johnson et al., 2013; Kiyonaga and Egner, 2013; Van der Lubbe et al., 2014), we combined a similar WM manipulation as described above with the well-known Posner cuing paradigm—typically used to study (external) spatial selective attention (Posner et al., 1982). In the Posner paradigm, it has been shown that an attention cue (for example a centrally presented arrow) presented shortly before a to-be-detected dot appears left or right on the screen, facilitates performance when it cues the subsequent dot location validly, but impairs performance when it cues the opposite location. We replaced the arrow cues by items (i.e., numbers) that were maintained in serial order WM, and observed that processing of later (in time) items of the WM sequence directed attention more to the right than earlier items within that sequence (van Dijck et al., 2013). This finding indicates that shifting attention within the internal space for serial order coding can be measured with external attention tools due to their direct interfacing. In a follow-up study, van Dijck et al. (2014) further replicated these findings and extended it to letters as the WM items. The latter was important in order to show the generalization of the mechanism.

Finally, the link between serial order WM and spatial processing was recently shown to also hold in the opposite direction. More specifically, retrieval from serial order WM was found to be facilitated (or hindered) by task-irrelevant, exogenous spatial attention cues (De Belder et al., in revision): exogenously directing attention to the left (right) facilitated retrieval of items early (late) in a WM sequence. Together with the work by van Dijck et al. (2013, 2014) this supports the bidirectionality of these effects, and further strengthens the notion that space is intrinsically involved in serial order WM.

Based on these findings we propose the hypothesis that serial order WM is grounded in the spatial attention system: (I) The position markers that provide multi-item WM with a serial context should be understood as coordinates within an internal, spatially defined system; (II) internal spatial attention is involved in searching through the resulting serial order representation; and (III) retrieval corresponds to selection by spatial attention. To illustrate our hypothesis, one may use a simple analog of a whiteboard: Whenever we need to remember a series of items in a specific order, we represent and maintain these items—for example, from left to right—on a “mental whiteboard”, in strong analogy to writing the items down on a physical whiteboard for later consultation. In fact, this comes very close to what was proposed—but not further specified—by Oberauer (2009, p. 53) who suggested that a “spatial medium of representation [is used] as a projection screen for relations on nonspatial dimensions”—such as serial order. Moreover, like external consultation itself involves moving the putative searchlight of attention (Crick, 1984) across the whiteboard, search and retrieval processes in serial WM are based on (selection by) internal spatial attention. We refer to our hypothesis as the mental whiteboard hypothesis.

In principal, the internal space that is used for serial order coding allows for flexibility: Coordinates along any well-arranged and orderly continuum (e.g., left to right; top to bottom; et cetera) may be recruited as best fits the task at hand. For example, in our work reported above, the task involved horizontally arranged stimuli and/or responses, favoring left to right encoding of serial order. With vertical task arrangements, however, top to bottom encoding may be probed (e.g., Dutta and Nairne, 1993; see below). Crucially, when the context does not cue spatial coding otherwise, we assume that it spontaneously occurs from left to right on the basis of the typically observed leftward bias in spatial processing (Jewell and McCourt, 2000; Della Sala et al., 2010) and/or a shaping by reading and writing direction (Whorf, 1956; Maass and Russo, 2003; Spalek and Hammad, 2005; Bonato et al., 2012). Especially reading direction has been shown to be an important force in shaping mental representations of space across related literatures (Zebian, 2005; Shaki and Fischer, 2008; Shaki et al., 2009; Bonato et al., 2012).

POSITION MARKING ON THE MENTAL WHITEBOARD

What does the mental whiteboard hypothesis offer to existing position marker models?

The primary value lies in filling in (some of) the blanks of the precise nature of position markers (i.e., coordinates in internal space) and the corresponding search (i.e., spatial attention) and retrieval processes (i.e., selection by spatial attention). Moreover, as the ground rules of computationally tested position marker models do not change by substantiating markers in terms of spatial coordinates, hallmark serial order observations (Marshuetz, 2005) can still be explained in our mental whiteboard hypothesis—again grounded in spatial attention systems. For example, the serial position effect (i.e., gradual increases in response times for items further in the sequence; cf. Sternberg, 1967) can be directly related to the directional consistency of attentional search—typically from left to right—through the spatially defined mental whiteboard representations. Additionally, the distance effect (see above) may be related to the observation that in external space processing, discrimination between two stimuli is more difficult when they are positioned at nearby as compared to further locations (e.g., Cave and Zimmerman, 1997; Bahcall and Kowler, 1999). This observation has been assigned to spatial attention interference, and a potential equivalent of this phenomenon in internal space may explain the distance effect in serial order. Indeed, a similar (attentional) interference explanation may hold for so-called transposition errors (e.g., Caramazza, 1996); that is, the observation that errors in serial recall often involve switches between serially nearby items. Finally, let us address the observed asymmetry in performance on backward and forward recall (Thomas et al., 2003). Specifically, response time patterns differ between a condition in which a memorized lists of words needs to be reproduced from start to end, and a condition in which the latter occurs from end to start. This type of observation may well be related to attentional search processes. As there is a strong attentional bias towards shifting from left to right (Jewell and McCourt, 2000; Spalek and Hammad, 2005; Della Sala et al., 2010)—which may itself be linked to reading direction (Spalek and Hammad, 2005)—the above mentioned differences in response time patterns may reflect differences in the experience-based development of attentional scanning and its fluency (with more skilled and/or controlled scanning from left to right) and/or continuous tendencies to abort right to left shifting. Future scrutiny of this type of explanations based on the workings of the (internal) spatial attentional system will help to confirm or falsify our mental whiteboard hypothesis.

Interpreting the notion of position markers as coordinates in an internal space may also help to explain why serial-recall sometimes seems to rely on both chaining and positional mechanisms (e.g., Serra and Nairne, 2000; Kahana et al., 2010; Kahana, 2012; Solway et al., 2012). Specifically, spatial coordinates that code for serial order may over time (i.e., when the WM representation is sufficiently long maintained) become associated with each other on the basis of spatial contiguity, and these associations may be responsible for chaining-like effects—without positing chaining as the fundamental mechanism underlying serial order.

A final asset of the mental whiteboard hypothesis is that it allows for convergence between general perspectives on WM and serial order. Over the last decades, WM is increasingly conceived as emanating directly from interactions between attention systems and long-term memory (LTM), with attention prioritizing the processing of specific pieces of information available in LTM (Cowan, 1999, 2001; Engle and Kane, 2004; Postle, 2006; Oberauer, 2009; Gazzaley and Nobre, 2012). Hence, in strong analogy to selecting specific information or locations in the outside world (i.e., external selective attention), it is assumed that we can attentionally search for, select and maintain in an active state information that is stored in the LTM systems of our brains (Chun et al., 2011; Kiyonaga and Egner, 2013); WM can be said to be equivalent to internal selective attention (Kiyonaga and Egner, 2013). In a broader sense, this attention-based perspective relates closely also to resource-based accounts of WM, such as the time-based resource sharing model of WM (Barrouillet and Camos, 2007, 2012; see also Ma et al., 2014). Critically, little effort has been made so far to conceptually embrace serial order in these general, attention-based perspectives on WM. At the same time, most of the serial order models do not make explicit their links to selective attention. The mental whiteboard hypothesis provides a candidate mechanism to close this conceptual gap between general WM perspectives and the specific notion of position markers. This is important because the success of the general, attention-based perspective on WM will—among others—critically depend on efforts to be reconciled with serial order models.

What are the alternatives to “finding the answer in space”?

Within the context of our work showing interactions between spatial and serial order processing, there may be two primary, alternative candidate mechanisms to support the notion of position markers. First, Botvinick and Watanabe (2007) proposed to build serial order on existing rank representations in the brain. Specifically, serially presented items can be tagged to fixed rank codes to maintain position—for example, a first item is tagged to the representation of “1” in the brain, a second item to “2”, etcetera (cf. Marshuetz, 2005). These order tags, then, may subsequently drive spatial processing in line with the Spatial Numerical Association of Response Codes or SNARC effect (Dehaene et al., 1993). The SNARC effect involves the robust finding that small numbers are faster responded to with a left hand response, while larger numbers are faster responded to with a right hand response—indicating a link between numbers and space (Dehaene et al., 1993). Currently, this alternative cannot be refuted, but we believe there are some indications in favor of our mental whiteboard hypothesis. In our work on serial order WM we often employed number stimuli, and we systematically observed that number magnitude—often despite its main effect on behavior—did no longer interact itself with spatial processing when casted in a WM sequence (van Dijck et al., 2013, 2014). Additionally, there may be relevant information in exploring serial order and space interactions across the vertical axis. Specifically, whereas number magnitude has been shown to map onto space in a bottom (small) to top (large) direction (e.g., Gevers et al., 2006), Dutta and Nairne (1993) observed a top to bottom organization for serial order WM.

A second alternative mechanism may be based on coding serial order through temporal stamps (e.g., Brown et al., 2000, 2007), with the spatial interactions being a by-effect of the so-called mental timeline (Bonato et al., 2012). Again, such alternative cannot be refuted at the current stance, but tentative indications exist for the claim that space subtends time in serial order coding. Most importantly, van Dijck et al. (unpublished work) show that the impact of serial order on spatial processing reverses when items are serially presented from right to left on the screen. If interactions with space were driven by temporal stamps, then such reversal would not be expected, while it fits well with a flexible system of serial order coding in space. Moreover, van Dijck and Fias (2011) showed that the impact of serial order on space was not due to overall reaction time—and thus elapsed time per se—as would be predicted from codes rooted in the temporal domain.

Hence, our mental whiteboard hypothesis builds on the notion that space underlies serial order WM without mediation by either temporal or numerical processes, but future efforts are definitely required to firmly ground this choice.

DIVERSE BEHAVIORAL OBSERVATIONS

The studies by van Dijck et al. (2013, 2014; De Belder et al., in revision) demonstrate a clear link between serial order WM and spatial processing along the horizontal axis. Although serial order WM has rarely been linked to spatial processing, the mental whiteboard hypothesis speaks to our imagination: we remember a series of items from left to right on an imaginary bow. This link may help to explain, reinterpret and/or parsimoniously integrate various previous observations.

From the notion that serial order is coded within a spatial coordinate system, a logical next question concerns how spatially coded serial order can be combined with other types of spatial encoding. For example, in dance there are tightly integrated temporal and spatial action sequences (i.e., when to perform which movement) that need to be learned and performed—how does the brain achieve this? An answer to this question may be searched in existing literature showing that coding in, maintenance of, and rapid switching between multiple spatial templates is indeed possible (e.g., Derdikman and Moser, 2010; Miles et al., 2011; Nitz, 2012).

At this moment we cannot make any claims about (dis)similarities between spatially coded serial order, and encoding of (external) space per se. However, there may be some relevant findings in the literature. From the notion that these build on related spatial systems, a strong prediction would be that interference results from simultaneous encoding and maintaining of serial order and (random) “other” spatial sequences. Hence, if we assume that order information is spontaneously coded along a spatial continuum (e.g., from left to right), then requiring participants to maintain at the same time a sequence of to be remembered information presented on external locations that do not follow this continuum (but rather inhabit space in a random manner), should produce interference because both tax spatial attention in a non-synchronous manner. In fact, various previous studies tentatively suggest that this is indeed the case.

For example, Gmeindl et al. (2011) administered both verbal and visuospatial sequence-memory tasks to participants: They were shown a short sequence of numbers (verbal) or locations (visuospatial) on the screen, and the goal in each trial was to reproduce as many target items as possible either in the same order (same order condition) as presentation, or in any order (no-order condition). Performance was enhanced in the no-order condition—and this was especially the case for the visuospatial sequences. In a second experiment, Gmeindl et al. again presented verbal or visuospatial sequences to participants. However, now each target sequence was immediately followed by a test sequence of the same kind, and the goal was to decide whether the first and second sequences were identical or not. The second sequence was either identical, contained the same items in a different serial order, or contained a novel item that replaced one of the items of the sequence. It was observed that participants failed to detect changes in serial order between target and test sequences more frequently for visuospatial than for verbal sequences. These findings fit the idea of interference between serial order and (external) spatial coding.

The results by Gmeindl et al. (2011) were more recently corroborated by Delogu et al. (2012). Participants were serially presented with five items (environmental sounds or pictures), and each item was presented at a different location. After this sequence was presented, they were asked to recall a specific item either at the location from which it was presented or at its serial order position within the overall sequence. When participants were instructed to maintain both types of information (i.e., it was not predictable which type of information was required to recall), it was observed for both the auditory and the visual sequences that serial order recall was hindered by the simultaneous encoding of item location, whereas the recall of item location was unaffected by the simultaneous encoding of serial order. This demonstrates again that interference may arise from simultaneous processing of serial order and spatial information, but the asymmetry also provides some information on the possible development of this intrinsic link—with external location information possibly being “prioritized” over serial order coding.

Whereas the findings of Gmeindl et al. (2011) were proposed to indicate domain-specific serial order processes, Delogu et al. (2012) actually interpreted their findings as supporting domain-generality. The latter would indeed be in line with various other studies suggesting domain-generality (Jones et al., 1995; Smyth, 1996; Depoorter and Vandierendonck, 2009), which are discussed in more detail below. For now we would like to point out that the here proposed link between serial order WM and spatial processing could parsimoniously account for these observations together—and this should be further explored in future research.

Another interesting issue is the link between spatially defined serial order representations, and reading and writing. In our culture, the latter develops from left to right and may have contributed to the shaping of spontaneous direction of serial order coding. In fact, a priori this was the main reason for predicting spatial coding along the left-right dimension in our previous studies (van Dijck and Fias, 2011; van Dijck et al., 2013). The notion provides some interesting additional hypotheses. For example, one could test a population of participants who read from right to left such as Palestinians or Iranians (Shaki et al., 2009), predicting to find similar but reversed interactions between serial order and spatial processing. Moreover, besides reading from left to right, we also read from top of the page to the bottom. Hence, it may be predicted that serial order WM may also interact with spatial processing along the vertical axis: later items in a WM sequence may trigger attention to be progressively shifted towards the lower regions of a(n) (internal or external) space. Dutta and Nairne (1993) provided some tentative support for this prediction. They presented participants with pairs of items (i.e., shapes), each of which occurred either first or second in time and above or below a fixation point. When both these temporal and spatial dimensions were task-relevant, recall performance was best for sequences in which first and second items were mapped respectively on the top and bottom locations (i.e., congruence between the serial order and spatial domains). In addition to the horizontal axis, this already hints towards a similar link between serial order WM and spatial coding along the vertical axis.

We recently provided more direct support for interactions between serial order processing and vertical processing using a paradigm described by Kirsner and Brown (1981). In their study, Kirsner and Brown (1981) presented participants on each trial with a series of two centrally presented digits which were shortly followed by two digits that were simultaneously flashed, one to the left and one to the right of fixation. Participants had to perform a detection task in which they were required to press a key whenever one of the two lateral digits (the target) matched either one of the earlier central digits–thus requiring the maintenance of these earlier digits in WM. Responses were fastest either when a left-side target matched the first presented digit, or when a right-side target matched the second presented digit. Even though this study was not framed as linking serial WM and spatial processing, its results may well be explained as such—providing an alternative to the authors’ original explanation in terms of hemispheric differences. Using this paradigm we show in a recent, unpublished study that serial order processing can interact with spatial processing both across the left-right and top-bottom dimension (Abrahamse, Acar, Fias and van Dijck, in preparation), indicating flexibility in the configuration of spatial coordinates (i.e., position markers) used to maintain serial order. The link to reading is an interesting avenue for future research.

A final issue that we would like to refer to here involves recent support for our hypothesis from research on primates (Adachi, 2014). Elsewhere, we have outlined how our hypothesis on serial order coding in a spatially defined system can account for both the SNARC (van Dijck and Fias, 2011) and attentional SNARC effects (van Dijck et al., 2014). Hence, we postulated that, when performing a typical (attentional) SNARC task, participants may soon form a mental representation in WM that includes the items (e.g., numbers) that occur during the experiment. These items are maintained in WM on the basis of a particular ordinal coding that is probably not the order with which they were presented in during the experiment (as this is not consistent throughout the experiment), but rather follows by default the canonical order implied by number magnitude. As such, serial order effects on response or attention processes may align with magnitude—without magnitude providing the spatial codes itself. Recently, a SNARC-like effect was observed in primates that fits our mental whiteboard hypothesis (Adachi, 2014). Specifically, primates were trained to search for number symbols (i.e., 1–9) within a squared matrix of locations on a screen and touch each of them in a fixed, learned order. Importantly, even though they had no experience with numbers as coding for magnitude (i.e., numbers were meaningless symbols for the primates), it was found that they were faster for items early in the sequence when they were presented left (as compared to right), and vice versa for items later in the sequence. This indicates that, in strong comparison to the human research described above, the left to right spatial coding of serial order information can also be observed in primates (Adachi, 2014, see also Drucker and Brannon, 2014). Interestingly, this finding also informs us that while reading may have contributed to shaping (spontaneous) serial order coding in space in humans, it certainly cannot be the single determinant of these processes.

Overall, even though not unequivocally providing empirical support for our mental whiteboard hypothesis on serial order, the rather heterogeneous set of behavioral studies outlined in this section can be parsimoniously integrated within this single hypothesis.

NEURAL SUBSTRATE OF SERIAL ORDER WM

A major challenge with respect to our here proposed mental whiteboard hypothesis concerns (the search for) its neural substrate. While direct investigation is yet to be reported, we would like to discuss two areas that jump out as viable candidates to support this type of coding: the hippocampus and the intraparietal sulcus (IPS).

Intraparietal sulcus

The IPS is an area at the lateral surface of parietal cortex. Without claiming its unique and/or sole contribution, specific parts of the IPS have been consistently linked to each of the crucial features that relate to the mental whiteboard hypothesis of serial order WM: verbal (and spatial) STM, serial order processing, and (reorientation of) spatial selective attention. Specifically, whereas more posterior and middle segments of the IPS are systematically involved in selective attention (and the integration of top-down and bottom-up attentional systems), the anterior IPS—especially of the right hemisphere—has been related to serial order coding. This suggests that the IPS provides a neural hub that drives the interactions between serial order and spatial attention. We will elaborate on these issues below and discuss how each of the three crucial features has been related to the others.

Inspired by the multi-component model of Baddeley and Hitch (1974), neuro-imaging studies initially aimed at finding the neural substrates of a dedicated verbal short-term storage system (cf. the phonological loop) that is relatively independent from general attentional processes and other storage systems. This search has not been very successful, as no site seems to respond to these criteria (Buchsbaum and D’Esposito, 2008). Rather, neuro-imaging studies appeared to strengthen the attention-based account of WM. More specifically, the IPS was shown to be sensitive to changes in verbal (Becker et al., 1999; Ravizza et al., 2004; Todd and Marois, 2004; Todd et al., 2005; Majerus et al., 2012) and visuospatial STM load (Nystrom et al., 2000; Majerus et al., 2010), in line with a domain-general (attentional) process that serves both verbal and visuospatial WM (Majerus et al., 2010, 2014; Cowan et al., 2011). Majerus et al. (2014), for example, used a machine-learning algorithm to determine the extent to which common neural patterns characterize WM retention in the verbal and visual modality. They found between-task prediction of the amount of WM load during the retention interval in regions of the dorsal attentional network (posterior parietal and superior frontal cortices), providing novel evidence for common, attention-based neural patterns underlying verbal and visual WM.

The link between IPS and selective spatial attention per se has received support though across other studies. Gillebert et al. (2011) reported on two patients that suffered from rare isolated IPS lesions (left posterior IPS vs. right horizontal segment of IPS), and showed its critical contribution to spatial selective attention as these patients were impaired (as compared to controls) on. This neuropsychological evidence confirms earlier indications from neuro-imaging work on the parietal cortex, and especially the IPS (e.g., Corbetta and Shulman, 2002; Hung et al., 2005; Molenberghs et al., 2007, 2008; Vandenberghe and Gillebert, 2009; Silk et al., 2010). Interestingly, Macaluso and Patria (2007) observed IPS activation for attentional reorienting along both the horizontal and vertical axes, while Pavani et al. (2002) showed IPS activation for moving sounds along both horizontal and vertical axes. These observations align with the analog of moving internal attention horizontally and vertically across an internal space.

Activation in IPS has thus been systematically linked both to verbal WM and spatial attention. For our current purposes, this becomes especially interesting in the light of additional neuro-imaging work suggesting that anterior IPS activation subtends serial order coding (Henson et al., 2000; Marshuetz et al., 2000; Majerus et al., 2006). For example, in a functional connectivity study by Majerus et al. (2006), participants performed a verbal WM task that probed recognition for either word identity or word order. They observed consistently stronger right IPS activation for order than for identity information. Left IPS was activated for both types of information, but crucially showed functional connectivity to right anterior IPS only for order information but not for identity information. As such, it could be suggested that (especially right) IPS provides the spatial template to code for serial order information—but possibly the link to verbal items is provided through left IPS. Recent evidence, however, points to the direct involvement of also the left IPS in serial order WM. Using fMRI, Attout et al. (2014) determined the degree of neural overlap between the serial order WM distance effects and numerical tasks and observed bilateral IPS activation. Hence, strong hemispheric lateralization of specific functions—if existing in the first place—is yet to be convincingly demonstrated. The link between IPS and serial order is further supported by single-unit recordings in primates (Nieder et al., 2006), as neurons in IPS have been observed to respond selectively to what Botvinick and Watanabe (2007) referred to as rank—which when combined with item information constitutes serial order. Finally, a tentative link to serial order can also be derived from a study by Sakai et al. (2002), who observed right IPS involvement in the learning of finger movement sequences—with motor responses being signaled through spatial stimuli.

Hence, in addition to the already reported roles of IPS in providing an integration zone for multisensory information (Macaluso and Driver, 2005; Anderson et al., 2010), for stimulus-driven and voluntary attentional control (Anderson et al., 2008; Geng and Mangun, 2009), and for temporal orienting and task-specific cortical areas (Davranche et al., 2011), we here propose that IPS is involved in the integration between serial order coding of information in WM and the spatial attention system. Moreover, the systematic links of IPS to both serial order and spatial selective attention supports the hypothesis that serial order maintenance is spatially defined. Future work is needed in order to reveal the full network that underlies serial order WM and internal attention.

Hippocampus

The second area that may be proposed as relevant for spatially defined serial order coding is the hippocampus. The hippocampus is traditionally linked to LTM. Specifically, it has been proposed to be involved in the creation and consolidation of episodic memory traces, with a particular focus on spatial information and navigation (e.g., Ekstrom et al., 2003). However, the exclusive link to LTM has recently been reconsidered, with various authors claiming that hippocampus also is involved—one way or another—in short-term and/or WM (Jensen and Lisman, 2005; von Allmen et al., 2013). Most interestingly for current purposes, it has been proposed that hippocampal theta and gamma oscillations together provide a system for serial order coding in WM (Jensen and Lisman, 2005). Specifically, a group of cells representing a single item is proposed to fire on each theta cycle, but only in a given gamma subcycle—thereby providing a short-term buffer for multiple items with a single item being reactivated for each of the four to eight gamma cycles that are nested within one theta cycle. Even though support from human subjects is mounting (Lisman and Jensen, 2013), the primary evidence for this hypothesis derives from rat studies (Jensen and Lisman, 2005). Interestingly, in rats these exact same hippocampal frequency bands have also been proposed to form a mechanism through which rats maintain a spatial configuration in WM in order to support spatial navigation (e.g., Buzsáki, 2005; Tort et al., 2009). Though premature, this link is suggestive of a perspective on serial order such as we have outlined in this paper, linking serial order to spatial coding. As such, our mental whiteboard hypothesis may be a specific example of what Buzsáki and Moser (2013) recently theorized, namely that “mechanisms of memory and planning have evolved from mechanisms of navigation in the physical world” (p. 130).

Hence, whereas direct investigation of the neural substrate underlying the mental whiteboard hypothesis of serial order is yet to emerge, there are already some venues to be derived from existing neuroscientific literature that may guide future investigation in this domain.

SERIAL ORDER WORKING MEMORY IN (CLINICAL) NEUROPSYCHOLOGY

Above we provided direct and/or indirect empirical support for the here hypothesized link between serial order WM and spatial processing at the behavioral and neural levels. At the level of (clinical) neuropsychology, this link has not been extensively explored as yet. Still, we here outline a number of studies on dyslexia and hemi-neglect that together will demonstrate the viability of our hypothesis in parsimoniously accounting for a rich set of findings in this domain, too.

In developmental dyslexia, deficits have been observed across separate studies in both serial order WM and in spatial attention. Specifically, Martinez Perez et al. (2012) showed verbal serial order WM impairments in dyslectic children, which could not be fully reduced to impaired phonological processing. Further supporting this idea, Hachmann et al. (2014) demonstrated that WM for serial order (and not for item information) for both verbal and non-verbal information is impaired in dyslexia, whereas Franceschini et al. (2012) showed that tests of visual spatial attention in preschoolers predicts future reading acquisition. The latter study supports a causal role of visual spatial attention in dyslexia, though further research is needed to rule out the alternative explanation that these observations can be attributed to efficient use of external spatial attention in the scanning of the document to-be-read–thus lacking a direct link to WM (or internal attention).

The notion that the link between serial order WM and spatial attention is involved in reading and spelling also receives (tentative) support from neglect dyslexia. Neglect dyslexia is a disorder commonly observed in hemi-spatial neglect patients who suffer from damage to the left or right parietal lobe. Their difficulties are characterized by reading and spelling errors on the contra-lesional side of words, suggesting that words—representations that involve serially coded graphemes—are spatially represented (Caramazza and Hillis, 1990). Although this disorder is typically associated with deficits in visuospatial attention per se, patients have actually been described who do not show neglect in tasks other than reading (e.g., Costello and Warrington, 1987; Katz and Sevush, 1989; Cubelli et al., 1991). So far, a comprehensive explanation for this dissociation is lacking, but detailed investigation of one such a patient suggests that neglect dyslexia can also be attributed to deficient coding of (abstract) ordinal position (with graded activation over the different positions), when tasks involve orthographic representations (Petrich et al., 2007). A potential explanation that integrates the currently available observations could be that for accurate writing and spelling, both WM and spatial attention are involved: all graphemes are serially represented in a spatial format in WM, and spatial attention is involved when retrieving this information during the writing or spelling process.

Along the same line of reasoning, additional indications for the link between serial order coding in WM and space can be found within the neglect literature. When neglect patients are asked to indicate the midpoint of a numerical interval they keep in mind, they tend to over-estimate the midpoint (e.g., when asked to indicate the midpoint of the interval 1–9 they may respond 7 instead of 5) as if they ignore the small numbers of the interval (e.g., Zorzi et al., 2002). This bias is typically considered as evidence for the long-term representation of numbers taking the form of a mental number line (MNL). However, more recent observations exist that more closely fit with our hypothesis linking serial order WM to spatial processing. Similar to neglect dyslexia, consistent double dissociations between number interval neglect and neglect within the perceptual space have been observed (e.g., Doricchi et al., 2005) suggesting the involvement of additional cognitive processes which may be the involvement of (serial order) WM. Indeed, Doricchi et al. (2009) showed that rightward deviations in number interval bisection in (right-brain-damaged) neglect patients are correlated both with spatial WM (i.e., Corsi block span) and verbal WM deficits (i.e., digit span). This may be explained by the fact that all these tasks share a common serial order component. Second, van Dijck et al. (2011) reported indications from a left-brain-damaged neglect patient for the notion that (serial) verbal WM supports numerical tasks that are typically linked to a spatial representation of numerical magnitudes. Again, these findings can be easily explained within the here-proposed mental whiteboard hypothesis. After all, for accurate number interval bisection, an ordered series on information (numbers) needs to be maintained, upon which controlled attentional processes operate to obtain a correct response (see Fias et al., 2011, for a detailed elaboration on this idea).

As with the behavioral support outlined above, we realize that each separate finding here can be easily accounted for in various ways; however, we believe that “the bigger picture” that is derived from these findings can be parsimoniously accounted for by the notion that serial order coding occurs within a spatially defined medium.

DIRECTING FUTURE RESEARCH

In order to contribute to future empirical efforts in this domain, we here close the paper with a number of challenges that we feel are important to further substantiate the mental whiteboard hypothesis.

BEHAVIORAL LEVEL

At the behavioral level, various empirical questions deserve to guide future research efforts in order to further specify the mental whiteboard hypothesis. First, we need to explore whether the internal spatial code that is derived at the moment of WM retrieval (and which we measure in our paradigms by its interaction with external spatial cues) derives from the direction of the last attention shift between two locations in internal space—or from the location within the overall internal space where attention is focused on that moment. Hence, will shifting attention from the last to the second last item in a long WM sequence provide a right spatial code (as attention is moving through the right-side of space) or a left spatial code (as the specific shift is leftwards from the last to second last item)?

Second, it will need to be explored to what extent the interaction between serial order and spatial attention is modality-independent: Can it also be observed in the visuospatial domain? This remains to be tested, for example, in a design similar to what we employed in the studies by van Dijck et al. (2013, 2014). There are already some indications for the notion that serial order WM per se is domain-general. At the neural level, for example, we already noted above that substantial overlap exists in brain areas that underlie verbal and visuospatial WM tasks (Becker et al., 1999; Nystrom et al., 2000; Ravizza et al., 2004; Majerus et al., 2010, 2014). Additionally, at the behavioral level we want to refer to a study by Depoorter and Vandierendonck (2009). They employed a dual-task methodology in which a secondary short-term memory task (verbal vs. visuospatial items; order vs. item memory task) was performed in the retention interval (i.e., the time between presentation and recall) of another, primary short-term memory task. The most important finding for current purposes was that recall performance on the primary task was impaired when both the primary and the secondary tasks involved an order component, irrespective of the modality of the stimulus materials. This is behavioral support for a domain-general serial order WM, in line with earlier suggestions along this line based on similarities between observations for verbal and spatial serial memory (Jones et al., 1995; Smyth, 1996; for a review see Hurlstone et al., 2014).

Third, in our studies on the link between serial order and spatial processing (van Dijck and Fias, 2011; van Dijck et al., 2013, 2014) we capitalized on the left-right dimension. However, as already described above, the mental whiteboard hypothesis postulates flexibility in the precise spatial configuration that is employed—with the only restriction being that it entails a well-arranged, orderly continuum. Along this line, the study by Dutta and Nairne (1993) already tells us that the top-bottom dimension can also be involved. As the precise spatial coding and subsequent trajectory of search through WM content may strongly depend on task context, future research is needed to clarify exactly how task context and representation interact.

Fourth, as noted above, WM maintenance is believed to ultimately lead to consolidation in LTM. As such, interference at the level of WM would be predicted to affect the formation of long-term serial representations. One major paradigm to explore this type of representations is the serial reaction time (SRT) task (Nissen and Bullemer, 1987; Abrahamse et al., 2010). In this task, participants respond one by one to the locations of a series of stimuli by pressing spatially corresponding keys on the key-board. Without them being aware of this, the series of stimuli actually consists of a fixed sequence of stimuli (e.g., with a length of ten stimuli) that is repeated over and over again. Despite lacking awareness of this sequential manipulation, decreases in response times and/or error percentages over time indicate (implicit) sequence learning. Shin and Ivry (2002) have explored if and to what extent spatial (a fixed sequence across four different locations) and temporal (a fixed sequence across four response-to-stimulus intervals) sequences can be learned simultaneously. They observed that spatial and temporal sequences were only learned simultaneously when they were perfectly correlated (i.e., both had equal sequence length of items). When uncorrelated, only the spatial sequence was learned. This type of finding could tentatively be explained by assuming shared systems for both the coding of spatial locations and the coding of ordinal response-to-stimulus interval (i.e., ranked from short to long).

Fifth, at what stage(s) is spatial processing involved in serial order WM? If we take the analog of the whiteboard then it would be more or less implicitly assumed that encoding, maintenance and retrieval (searching across the whiteboard) are all driven by spatial processing. Indeed, this also makes most sense from a logical perspective: why would spatial attention be involved in retrieving information that was not initially coded and maintained in a spatial format? However, this is not yet exclusively supported by available studies (Kirsner and Brown, 1981; van Dijck et al., 2013, 2014). Recent evidence is suggestive for spatial processing during retrieval. Ginsburg et al. (2014) asked participants to memorize a sequence of five numbers in correct serial order and presented during the retention interval numbers that were part of the sequence as well as other numbers for classification (parity judgment or magnitude comparison). Only when the participants were instructed to limit the classification task to numbers from the WM sequence (i.e., verifying its presence in WM) an interaction between the side of response (left or right) and position in the sequence was observed. When the instructions were to classify all numbers presented during the retention interval (no explicit WM retrieval is needed) no such interaction was observed. The roles of encoding and maintenance remain even more elusive up to now. Indirect evidence for the latter has recently been provided by Fischer-Baum and Benjamin (2014) who showed that the recall of serial order information was more accurate when during the encoding phase, the WM items progressed from left to right compared to situation where they progressed in a right to left fashion. Still, future research is required to sort out the involvement of spatial processing across the various stages.

Sixth, what is the role of chunking in the studies by van Dijck et al. (2013, 2014) discussed above? This is an important issue as these studies are the primary behavioral support for the hypothesis that serial order WM is grounded in spatial attention. In all these studies brief sequences were used of only two to four single items each. As this falls within the range of WM limitations for most people, there is no way to determine if these items were chunked together (in sets of either two or four items) or whether each item was represented separately. It is thus an open question if spatial attention is involved merely when searching within a single chunk, or whether the same processes apply to searching across separate items and/or chunks. This needs to be explored in future studies.

NEURO-IMAGING AND NEUROPSYCHOLOGICAL LEVEL

At the level of neuro-imaging it will be crucial to further outline the role of the IPS in serial order coding. As mentioned above, earlier work has indicated a link between verbal serial order and selective attention, but it did not allow for making any claims about the precise nature of selective attention involvement; hence, whether it concerns spatial selective attention formerly still remains to be explored. Possibly, future research can combine the standard Posner cuing paradigm with the adapted version employed by van Dijck et al. (2013), and explore overlap in brain areas.

Our review above points to two areas—the hippocampus and the IPS—whose involvement in serial order and/or spatial processing is fairly well established. From the mental whiteboard hypothesis, the next questions are how and/or when exactly each of these areas is involved, and how they are functionally linked together. It could be speculated that (spatially defined) serial order coding per se is tightly linked to hippocampal mechanisms, whereas the IPS contributes controlled and visuospatial imagery processes that underlie the attentional search across serial order representations. This hypothesis could be a starting point for future empirical work in this domain.

Finally, at the level of clinical neuropsychology, definite support for the hypothesis that serial order WM is grounded in spatial attention systems is currently lacking. At the moment, contributions of spatial attention to (verbal) serial order (impairments), and vice versa, are hardly explored in (clinical) neuropsychology, and we here call for a more rigorous exploration of this interesting link. Not so much with an eye on supporting our hypothesis, but primarily for the purpose of understanding neurological problems and future opportunities for rehabilitation in especially neglect patients, dyslexia and individuals with impaired verbal WM. For example, it has recently become clear that in dyslexia specific selective deficits in serial order processing exist (Trecy et al., 2013), and our hypothesis may provide a promising avenue for rehabilitation in pointing at spatial attention as an underlying mechanism. It is a missed chance that attentional processing, and particularly spatial attention, is rarely explored in patients with verbal STM deficits—just like pure serial order is hardly tested in spatial neglect.

We would like to point out that our mental whiteboard hypothesis also has methodological implications for (clinical) neuropsychology. The Corsi block tapping task, which today is still one of the most used tasks in assessing visuospatial WM capacity in specific patient populations, involves the simultaneous maintenance of both location and serial order information. As such, it may tax two highly connected systems of external and internal spatial attention. In close resemblance of the studies mentioned in the previous section, then, spatial attention for location information may even interfere with spatial attention for serial order information. This interference is not desirable for a task that aims to provide a clean measure of visuospatial WM capacity. Moreover, the amount of interference may depend on the extent to which serial order and location information align with each other—either on the left-to-right or the top-to-bottom dimension–in a particular spatial block configuration, and the latter varies across studies. We believe that for this domain it is crucial to disentangle item and serial order information depending on the issue that one aims to tackle. Similar concerns may be relevant for other visuospatial WM tasks as well.

CONCLUSION

Over the last decades, the attention-based accounts of WM have become increasingly popular. The mental whiteboard hypothesis provides a promising avenue for incorporating serial order within these accounts by grounding it in the spatial attention system. As such, it adds to the overall plausibility of an attention-based WM account. At the same time, this hypothesis provides a candidate mechanism to substantiate the cognitive nature of position markers, a major but currently underspecified concept in serial order models. Still, the mental whiteboard hypothesis currently remains underspecified, and future research is needed to turn this hypothesis into a solid theoretical account.

ACKNOWLEDGMENTS

Elger Abrahamse was supported by Research Foundation Flanders under contract number 12C4715N. Steve Majerus was supported by grants F.R.S.-FNRS N°1.5.056.10 (Fund for Scientific Research FNRS, Belgium), and ARC 12/17/01 REST (Université de Liège). Wim Fias was supported by the Ghent University Multidisciplinary Research Partnership “The integrative neuroscience of behavioral control,” and a GOA grant of Ghent University. Steve Majerus and Wim Fias were supported by PAI-IUAP P7/11 (Belgian Science Policy).

REFERENCES

Abrahamse, E. L., Jiménez, L., Verwey, W. B., and Clegg, B. A. (2010). Representing serial action and perception. Psychon. Bull. Rev. 17, 603–623. doi: 10.3758/PBR.17.5.603

Adachi, I. (2014). Spontaneous spatial mapping of learned sequence in chimpanzees: evidence for a SNARC-like effect. PLoS One 9:e90373. doi: 10.1371/journal.pone.0090373

Anderson, J. S., Ferguson, M. A., Lopez-Larson, M., and Yurgelun-Todd, D. (2010). Topographic maps of multisensory attention. Proc. Natl. Acad. Sci. U S A 107, 20110–20114. doi: 10.1073/pnas.1011616107

Anderson, E. J., Husain, M., and Sumner, P. (2008). Human intraparietal sulcus (IPS) and competition between exogenous and endogenous saccade plans. Neuroimage 40, 838–851. doi: 10.1016/j.neuroimage.2007.10.046

Attout, L., Fias, W., Salmon, E., and Majerus, S. (2014). Common neural substrates for ordinal representation in short-term memory, numerical and alphabetical cognition. PLoS One 9:e92049. doi: 10.1371/journal.pone.0092049

Awh, E., and Jonides, J. (2001). Overlapping mechanisms of attention and spatial working memory. Trends Cogn. Sci. 5, 119–126. doi: 10.1016/s1364-6613(00)01593-x

Baddeley, A. (2012). Working memory: theories, models and controversies. Annu. Rev. Psychol. 63, 1–29. doi: 10.1146/annurev-psych-120710-100422

Baddeley, A. D., and Hitch, G. (1974). “Working memory,” in The Psychology of Learning and Motivation (Vol. 8), ed G. H. Bower (New York: Academic Press), 47–89.

Bahcall, D. O., and Kowler, E. (1999). Attentional interference at small spatial separations. Vision Res. 39, 71–86. doi: 10.1016/s0042-6989(98)00090-x

Barrouillet, P., and Camos, V. (2007). “The time-based resource-sharing model of working memory,” in The Cognitive Neuroscience of Working Memory, eds N. Osaka, R. H. Logie and M. D’Esposito (Oxford: Oxford University Press), 59–80.

Barrouillet, P., and Camos, V. (2012). As time goes by temporal constraints in working memory. Curr. Dir. Psychol. Sci. 21, 413–419. doi: 10.1177/0963721412459513

Becker, J. T., MacAndrew, D. K., and Fiez, J. A. (1999). A comment on the functional localization of the phonological storage subsystem of working memory. Brain Cogn. 41, 27–38. doi: 10.1006/brcg.1999.1094

Bonato, M., Zorzi, M., and Umiltà, C. (2012). When time is space: evidence for a mental time line. Neurosci. Biobehav. Rev. 36, 2257–2273. doi: 10.1016/j.neubiorev.2012.08.007

Botvinick, M., and Watanabe, T. (2007). From numerosity to ordinal rank: a gain-field model of serial order representation in cortical working memory. J. Neurosci. 27, 8636–8642. doi: 10.1523/jneurosci.2110-07.2007

Brown, G. D., Neath, I., and Chater, N. (2007). A temporal ratio model of memory. Psychol. Rev. 114, 539–576. doi: 10.1037/0033-295x.114.3.539

Brown, G. D. A., Preece, T., and Hulme, C. (2000). Oscillator-based memory for serial order. Psychol. Rev. 107, 127–181. doi: 10.1037//0033-295x.107.1.127

Buchsbaum, B., and D’Esposito, M. (2008). The search for the phonological store: from loop to convolution. J. Cogn. Neurosci. 20, 762–778. doi: 10.1162/jocn.2008.20501

Burgess, N., and Hitch, G. J. (2006). A revised model of short-term memory and long-term learning of verbal sequences. J. Mem. Lang. 55, 627–652. doi: 10.1016/j.jml.2006.08.005

Buzsáki, G. (2005). Theta rhythm of navigation: link between path integration and landmark navigation, episodic and semantic memory. Hippocampus 15, 827–840. doi: 10.1002/hipo.20113

Buzsáki, G., and Moser, E. I. (2013). Memory, navigation and theta rhythm in the hippocampal-entorhinal system. Nat. Neurosci. 16, 130–138. doi: 10.1038/nn.3304

Caramazza, A. (1996). The role of the graphemic buffer in reading. Cogn. Neuropsychol. 13, 673–698. doi: 10.1080/026432996381881

Caramazza, A., and Hillis, A. E. (1990). Where do semantic errors come from? Cortex 26, 95–122. doi: 10.1016/s0010-9452(13)80077-9

Cave, K. R., and Zimmerman, J. M. (1997). Flexibility in spatial attention before and after practice. Psychol. Sci. 8, 399–403. doi: 10.1111/j.1467-9280.1997.tb00433.x

Chun, M. M., Golomb, J. D., and Turk-Browne, N. B. (2011). A taxonomy of external and internal attention. Annu. Rev. Psychol. 62, 73–101. doi: 10.1146/annurev.psych.093008.100427

Corbetta, M., and Shulman, G. L. (2002). Control of goal-directed and stimulus-driven attention in the brain. Nat. Rev. Neurosci. 3, 201–215. doi: 10.1038/nrn755

Costello, A. D., and Warrington, E. K. (1987). The dissociation of visuospatial neglect and neglect dyslexia. J. Neurol. Neurosurg. Psychiatry 50, 1110–1116. doi: 10.1136/jnnp.50.9.1110

Cowan, N. (1999). “An embedded-processes model of working memory,” in Models of Working Memory: Mechanisms of Active Maintenance and Executive Control, eds A. Miyake and P. Shah (New York, NY: Cambridge University Press), 62–101.

Cowan, N. (2001). The magical number 4 in short-term memory: a reconsideration of mental storage capacity. Behav. Brain Sci. 24, 87–114; discussion 114–185. doi: 10.1017/s0140525x01003922

Cowan, N., Li, D., Moffitt, A., Becker, T. M., Martin, E. A., Saults, J. S., et al. (2011). A neural region of abstract working memory. J. Cogn. Neurosci. 23, 2852–2863. doi: 10.1162/jocn.2011.21625

Crick, F. (1984). Function of the thalamic reticular complex: the searchlight hypothesis. Proc. Natl. Acad. Sci. U S A 81, 4586–4590. doi: 10.1073/pnas.81.14.4586

Cubelli, R., Nichelli, P., Bonito, V., De Tanti, A., and Inzaghi, M. G. (1991). Different patterns of dissociation in unilateral spatial neglect. Brain Cogn. 15, 139–159. doi: 10.1016/0278-2626(91)90023-2

Davranche, K., Nazarian, B., Vidal, F., and Coull, J. (2011). Orienting attention in time activates left intraparietal sulcus for both perceptual and motor task goals. J. Cogn. Neurosci. 23, 3318–3330. doi: 10.1162/jocn_a_00030

Dehaene, S., Bossini, S., and Giraux, P. (1993). The mental representation of parity and number magnitude. J. Exp. Psychol. Gen. 122, 371–396. doi: 10.1037/0096-3445.122.3.371

Della Sala, S., Darling, S., and Logie, R. H. (2010). Items on the left are better remembered. Q. J. Exp. Psychol. (Hove) 63, 848–855. doi: 10.1080/17470211003690672

Delogu, F., Nijboer, T. C., and Postma, A. (2012). Binding “when” and “where” impairs temporal, but not spatial recall in auditory and visual working memory. Front. Psychol. 3:62. doi: 10.3389/fpsyg.2012.00062

Depoorter, A., and Vandierendonck, A. (2009). Evidence for modality-independent order coding in working memory. Q. J. Exp. Psychol. (Hove) 62, 531–549. doi: 10.1080/17470210801995002

Derdikman, D., and Moser, E. I. (2010). A manifold of spatial maps in the brain. Trends Cogn. Sci. 14, 561–569. doi: 10.1016/j.tics.2010.09.004

Doricchi, F., Guariglia, P., Gasparini, M., and Tomaiuolo, F. (2005). Dissociation between physical and mental number line bisection in right hemisphere brain damage. Nat. Neurosci. 8, 1663–1665. doi: 10.1038/nn1563

Doricchi, F., Merola, S., Aiello, M., Guariglia, P., Bruschini, M., Gevers, W., et al. (2009). Spatial orienting biases in the decimal numeral system. Curr. Biol. 19, 682–687. doi: 10.1016/j.cub.2009.02.059

Downing, P. E. (2000). Interactions between visual working memory and selective attention. Psychol. Sci. 11, 467–473. doi: 10.1111/1467-9280.00290

Drucker, C. B., and Brannon, E. M. (2014). Rhesus monkeys (Macaca mulatta) map number onto space. Cognition 132, 57–67. doi: 10.1016/j.cognition.2014.03.011

Dutta, A., and Nairne, J. S. (1993). The separability of space and time: dimensional interaction in the memory trace. Mem. Cognit. 21, 440–448. doi: 10.3758/bf03197175

Ebbinghaus, H. (1885 [1964]). Memory: A Contribution to Experimental Psychology. New York: Dover.

Ekstrom, A. D., Kahana, M. J., Caplan, J. B., Fields, T. A., Isham, E. A., Newman, E. L., et al. (2003). Cellular networks underlying human spatial navigation. Nature 425, 184–188. doi: 10.1038/nature01964

Engle, R. W., and Kane, M. J. (2004). “Executive attention, working memory capacity and a two-factor theory of cognitive control,” in The Psychology of Learning and Motivation (Vol. 44), ed B. Ross (NY: Elsevier), 145–199.

Farrell, S., and Lewandowsky, S. (2002). An endogenous distributed model of ordering in serial recall. Psychon. Bull. Rev. 9, 59–79. doi: 10.3758/bf03196257

Fias, W., van Dijck, J.-P., and Gevers, W. (2011). “How is space associated with number? The role of working memory,” in Space, Time and Number in the Brain—Searching for Evolutionary Foundations of Mathematical Thought: Attention and Performance Xxiv, eds S. Dehaene and E. Brannon (Amsterdam: Elsevier Science), 133–148.

Fischer-Baum, S., and Benjamin, A. S. (2014). Time, space and memory for order. Psychon. Bull. Rev. 21, 1263–1271. doi: 10.3758/s13423-014-0604-7

Franceschini, S., Gori, S., Ruffino, M., Pedrolli, K., and Facoetti, A. (2012). A causal link between visual spatial attention and reading acquisition. Curr. Biol. 22, 814–819. doi: 10.1016/j.cub.2012.03.013

Gazzaley, A., and Nobre, A. C. (2012). Top-down modulation: bridging selective attention and working memory. Trends Cogn. Sci. 16, 129–135. doi: 10.1016/j.tics.2011.11.014

Geng, J. J., and Mangun, G. R. (2009). Anterior intraparietal sulcus is sensitive to bottom-up attention driven by stimulus salience. J. Cogn. Neurosci. 21, 1584–1601. doi: 10.1162/jocn.2009.21103

Gevers, W., Lammertyn, J., Notebaert, W., Verguts, T., and Fias, W. (2006). Automatic response activation of implicit spatial information: evidence from the SNARC effect. Acta Psychol. (Amst) 122, 221–233. doi: 10.1016/j.actpsy.2005.11.004

Gillebert, C. R., Mantini, D., Thijs, V., Sunaert, S., Dupont, P., and Vandenberghe, R. (2011). Lesion evidence for the critical role of the intraparietal sulcus in spatial attention. Brain 134, 1694–1709. doi: 10.1093/brain/awr085

Ginsburg, V., van Dijck, J. P., Previtali, P., Fias, W., and Gevers, W. (2014). The impact of verbal working memory on number-space associations. J. Exp. Psychol. Learn. Mem. Cogn. 40, 976–986. doi: 10.1037/a0036378

Gmeindl, L., Walsh, M., and Courtney, S. M. (2011). Binding serial order to representations in working memory: a spatial/verbal dissociation. Mem. Cognit. 39, 37–46. doi: 10.3758/s13421-010-0012-9

Hachmann, W. M., Bogaerts, L., Szmalec, A., Woumans, E., Duyck, W., and Job, R. (2014). Short-term memory for order but not for item information is impaired in developmental dyslexia. Ann. Dyslexia 64, 121–136. doi: 10.1007/s11881-013-0089-5

Henson, R. N. A. (1998). Short-term memory for serial order: the start-end model. Cogn. Psychol. 36, 73–137. doi: 10.1006/cogp.1998.0685

Henson, R. N. A., Burgess, N., and Frith, C. D. (2000). Recoding, storage, rehearsal and grouping in verbal short-term memory: an fMRI study. Neuropsychologia 38, 426–440. doi: 10.1016/s0028-3932(99)00098-6

Henson, R. N. A., Norris, D. G., Page, M. P. A., and Baddeley, A. D. (1996). Unchained memory: error patterns rule out chaining models of immediate serial recall. Q. J. Exp. Psychol. 49A, 80–115. doi: 10.1080/713755612

Hung, J., Driver, J., and Walsh, V. (2005). Visual selection and posterior parietal cortex: effects of repetitive transcranial magnetic stimulation on partial report analyzed by Bundesen’s theory of visual attention. J. Neurosci. 25, 9602–9612. doi: 10.1523/jneurosci.0879-05.2005

Hurlstone, M. J., Hitch, G. J., and Baddeley, A. D. (2014). Memory for serial order across domains: an overview of the literature and directions for future research. Psychol. Bull. 140, 339–373. doi: 10.1037/a0034221

Jensen, O., and Lisman, J. E. (2005). Hippocampal sequence-encoding driven by a cortical multi-item working memory buffer. Trends Neurosci. 28, 67–72. doi: 10.1016/j.tins.2004.12.001

Jewell, G., and McCourt, M. E. (2000). Pseudoneglect: a review and meta-analysis of performance factors in line bisection tasks. Neuropsychologia 38, 93–110. doi: 10.1016/s0028-3932(99)00045-7

Johnson, M. R., Higgins, J. A., Norman, K. A., Sederberg, P. B., Smith, T. A., and Johnson, M. K. (2013). Foraging for thought an inhibition-of-return-like effect resulting from directing attention within working memory. Psychol. Sci. 24, 1104–1112. doi: 10.1177/0956797612466414

Jones, D., Farrand, P., Stuart, G., and Morris, N. (1995). Functional equivalence of verbal and spatial information in serial short-term memory. J. Exp. Psychol. Learn. Mem. Cogn. 21, 1008–1018. doi: 10.1037//0278-7393.21.4.1008

Kahana, M. J. (2012). Foundations of Human Memory. New York, NY: Oxford University Press.

Kahana, M. J., Mollison, M. V., and Addis, K. M. (2010). Positional cues in serial learning: the spin-list technique. Mem. Cognit. 38, 92–101. doi: 10.3758/MC.38.1.92

Kalm, K., and Norris, D. (2014). The representation of order information in auditory-verbal short-term memory. J. Neurosci. 34, 6879–6886. doi: 10.1523/JNEUROSCI.4104-13.2014

Katz, R. B., and Sevush, S. (1989). Positional dyslexia. Brain Lang. 37, 266–289. doi: 10.1016/0093-934x(89)90019-9

Kirsner, K., and Brown, H. (1981). Laterality and recency effects in working memory. Neuropsychologia 19, 249–261. doi: 10.1016/0028-3932(81)90109-3

Kiyonaga, A., and Egner, T. (2013). Working memory as internal attention: toward an integrative account of internal and external selection processes. Psychon. Bull. Rev. 20, 228–242. doi: 10.3758/s13423-012-0359-y

Lashley, K. S. (1951). “The problem of serial order in behavior,” in Cerebral Mechanisms in Behavior. The Hixon Symposium, ed L. A. Jeffress (New York: John Wiley and Sons, Inc.), 112–136.

Lisman, J. E., and Jensen, O. (2013). The θ-γ neural code. Neuron 77, 1002–1016. doi: 10.1016/j.neuron.2013.03.007

Ma, W. J., Husain, M., and Bays, P. M. (2014). Changing concepts of working memory. Nat. Neurosci. 17, 347–356. doi: 10.1038/nn.3655

Maass, A., and Russo, A. (2003). Directional bias in the mental representation of spatial events: nature or culture? Psychol. Sci. 14, 296–301. doi: 10.1111/1467-9280.14421

Macaluso, E., and Driver, J. (2005). Multisensory spatial interactions: a window onto functional integration in the human brain. Trends Neurosci. 28, 264–271. doi: 10.1016/j.tins.2005.03.008

Macaluso, E., and Patria, F. (2007). Spatial re-orienting of visual attention along the horizontal or the vertical axis. Exp. Brain Res. 180, 23–34. doi: 10.1007/s00221-006-0841-8

Majerus, S., Attout, L., D’Argembeau, A., Degueldre, C., Fias, W., Maquet, P., et al. (2012). Attention supports verbal short-term memory via competition between dorsal and ventral attention networks. Cereb. Cortex 22, 1086–1097. doi: 10.1093/cercor/bhr174

Majerus, S., Cowan, N., Péters, F., Van Calster, L., Phillips, C., and Schrouff, J. (2014). Cross-modal decoding of neural patterns associated with working memory: evidence for attention-based accounts of working memory. Cereb. Cortex doi: 10.1093/cercor/bhu189. [Epub ahead of print].

Majerus, S., D’Argembeau, A., Martinez Perez, T., Belayachi, S., Van der Linden, M., Collette, F., et al. (2010). The commonality of neural networks for verbal and visual short-term memory. J. Cogn. Neurosci. 22, 2570–2593. doi: 10.1162/jocn.2009.21378

Majerus, S., Poncelet, M., Van der Linden, M., Albouy, G., Salmon, E., Sterpenich, V., et al. (2006). The left intraparietal sulcus and verbal short-term memory: focus of attention or serial order? Neuroimage 32, 880–891. doi: 10.1016/j.neuroimage.2006.03.048

Marshuetz, C. (2005). Order information in working memory: an integrative review of evidence from brain and behavior. Psychol. Bull. 131, 323–339. doi: 10.1037/0033-2909.131.3.323

Marshuetz, C., Smith, E. E., Jonides, J., DeGutis, J., and Chenevert, T. L. (2000). Order information in working memory: fMRI evidence for parietal and prefrontal mechanisms. J. Cogn. Neurosci. 12, 130–144. doi: 10.1162/08989290051137459

Martinez Perez, T., Majerus, S., Mahot, A., and Poncelet, M. (2012). Evidence for a specific impairment of serial order short-term memory in dyslexic children. Dyslexia 18, 94–109. doi: 10.1002/dys.1438

Miles, L. K., Tan, L., Noble, G. D., Lumsden, J., and Macrae, C. N. (2011). Can a mind have two time lines? Exploring space-time mapping in Mandarin and English speakers. Psychon. Bull. Rev. 18, 598–604. doi: 10.3758/s13423-011-0068-y

Molenberghs, P., Gillebert, C., Peeters, R., and Vandenberghe, R. (2008). Convergence between lesion-symptom mapping and fmri of spatially selective attention in the intact brain. J. Neurosci. 28, 3359–3373. doi: 10.1523/JNEUROSCI.5247-07.2008

Molenberghs, P., Mesulam, M., Peeters, R., and Vandenberghe, R. (2007). Re-mapping attentional priorities: differential contribution of superior parietal lobule and intraparietal sulcus. Cereb. Cortex 17, 2703–2712. doi: 10.1093/cercor/bhl179

Nieder, A., Diester, I., and Tudusciuc, O. (2006). Temporal and spatial enumeration processes in the primate parietal cortex. Science 313, 1431–1435. doi: 10.1126/science.1130308

Nissen, M. J., and Bullemer, P. (1987). Attentional requirements of learning: evidence from performance measures. Cogn. Psychol. 19, 1–32. doi: 10.1016/0010-0285(87)90002-8

Nitz, D. A. (2012). Spaces within spaces: rat parietal cortex neurons register position across three reference frames. Nat. Neurosci. 15, 1365–1367. doi: 10.1038/nn.3213

Nobre, A. C., Rao, A., and Chelazzi, L. (2006). Selective attention to specific features within objects: behavioral and electrophysiological evidence. J. Cogn. Neurosci. 18, 539–561. doi: 10.1162/jocn.2006.18.4.539

Nystrom, L. E., Braver, T. S., Sabb, F. W., Delgado, M. R., Noll, D. C., and Cohen, J. D. (2000). Working memory for letters, shapes and locations: fMRI evidence against stimulus-based regional organization in human prefrontal cortex. Neuroimage 11, 424–446. doi: 10.1006/nimg.2000.0572

Oberauer, K. (2009). Design for a working memory. Psychol. Learn. Motiv. 51, 45–100. doi: 10.1016/S0079-7421(09)51002-X

Page, M. P. A., and Norris, D. (1998). The primacy model: a new model of immediate serial recall. Psychol. Rev. 105, 761–781. doi: 10.1037//0033-295x.105.4.761-781

Pavani, F., Macaluso, E., Warren, J. D., Driver, J., and Griffiths, T. D. (2002). A common cortical substrate activated by horizontal and vertical sound movement in the human brain. Curr. Biol. 12, 1584–1590. doi: 10.1016/s0960-9822(02)01143-0

Petrich, J. A. F., Greenwald, M. L., and Berndt, R. S. (2007). An investigation of attentional contributions to visual errors in right “neglect dyslexia”. Cortex 43, 1036–1046. doi: 10.1016/s0010-9452(08)70701-9

Posner, M. I., Cohen, Y., and Rafal, R. D. (1982). Neural systems control of spatial orienting. Philos. Trans. R. Soc. Lond. B Biol. Sci. 298, 187–198. doi: 10.1098/rstb.1982.0081

Postle, B. R. (2006). Working memory as an emergent property of the mind and brain. Neuroscience 139, 23–38. doi: 10.1016/j.neuroscience.2005.06.005

Ravizza, S. M., Delgado, M. R., Chein, J. M., Becker, J. T., and Fiez, J. A. (2004). Functional dissociations within the inferior parietal cortex in verbal working memory. Neuroimage 22, 562–573. doi: 10.1016/j.neuroimage.2004.01.039

Sakai, K., Ramnani, N., and Passingham, R. E. (2002). Learning of sequences of finger movements and timing: frontal lobe and action-oriented representation. J. Neurophysiol. 88, 2035–2046. doi: 10.1152/jn.00116.2002

Serra, M., and Nairne, J. S. (2000). Part—set cuing of order information: implications for associative theories of serial order memory. Mem. Cognit. 28, 847–855. doi: 10.3758/bf03198420

Shaki, S., and Fischer, M. H. (2008). Reading space into numbers-a cross-linguistic comparison of the SNARC effect. Cognition 108, 590–599. doi: 10.1016/j.cognition.2008.04.001

Shaki, S., Fischer, M. H., and Petrusic, W. M. (2009). Reading habits for both words and numbers contribute to the SNARC effect. Psychon. Bull. Rev. 16, 328–331. doi: 10.3758/PBR.16.2.328

Shin, J. C., and Ivry, R. B. (2002). Concurrent learning of temporal and spatial sequences. J. Exp. Psychol. Learn. Mem. Cogn. 28, 445–457. doi: 10.1037//0278-7393.28.3.445

Silk, T. J., Bellgrove, M. A., Wrafter, P., Mattingley, J. B., and Cunnington, R. (2010). Spatial working memory and spatial attention rely on common neural processes in the intraparietal sulcus. Neuroimage 53, 718–724. doi: 10.1016/j.neuroimage.2010.06.068

Smyth, M. M. (1996). Serial order in spatial immediate memory. Q. J. Exp. Psychol. A 49, 159–177. doi: 10.1080/713755615

Solway, A., Murdock, B. B., and Kahana, M. J. (2012). Positional and temporal clustering in serial order memory. Mem. Cognit. 40, 177–190. doi: 10.3758/s13421-011-0142-8

Spalek, T. M., and Hammad, S. (2005). The left-to-right bias in inhibition of return is due to the direction of reading. Psychol. Sci. 16, 15–18. doi: 10.1111/j.0956-7976.2005.00774.x

Sternberg, S. (1967). Retrieval of contextual information from memory. Psychon. Sci. 8, 55–56. doi: 10.3758/bf03330664

Thomas, J. G., Milner, H. R., and Haberlandt, K. F. (2003). Forward and backward recall different response time patterns, same retrieval order. Psychol. Sci. 14, 169–174. doi: 10.1111/1467-9280.01437

Todd, J. J., Fougnie, D., and Marois, R. (2005). Visual short-term memory load suppresses temporo-parietal junction activity and induces inattentional blindness. Psychol. Sci. 16, 965–972. doi: 10.1111/j.1467-9280.2005.01645.x

Todd, J. J., and Marois, R. (2004). Capacity limit of visual short-term memory in human posterior parietal cortex. Nature 428, 751–754. doi: 10.1038/nature02466

Tort, A. B., Komorowski, R. W., Manns, J. R., Kopell, N. J., and Eichenbaum, H. (2009). Theta-gamma coupling increases during the learning of item-context associations. Proc. Natl. Acad. Sci. U S A 106, 20942–20947. doi: 10.1073/pnas.0911331106

Trecy, M. P., Steve, M., and Martine, P. (2013). Impaired short-term memory for order in adults with dyslexia. Res. Dev. Disabil. 34, 2211–2223. doi: 10.1016/j.ridd.2013.04.005

Vandenberghe, R., and Gillebert, C. R. (2009). Parcellation of parietal cortex: convergence between lesion-symptom mapping and mapping of the intact functioning brain. Behav. Brain Res. 199, 171–182. doi: 10.1016/j.bbr.2008.12.005

Van der Lubbe, R. H., Bundt, C., and Abrahamse, E. L. (2014). Internal and external spatial attention examined with lateralized EEG power spectra. Brain Res. 1583, 179–192. doi: 10.1016/j.brainres.2014.08.007

van Dijck, J.-P., Abrahamse, E. L., Acar, F., Ketels, B., and Fias, W. (2014). A working memory account of the interaction between numbers and spatial attention. Q. J. Exp. Psychol. (Hove) 67, 1500–1513. doi: 10.1080/17470218.2014.903984

van Dijck, J.-P., Abrahamse, E. L., Majerus, S., and Fias, W. (2013). Spatial attention interacts with serial-order retrieval from verbal working memory. Psychol. Sci. 24, 1854–1859. doi: 10.1177/0956797613479610

van Dijck, J.-P., and Fias, W. (2011). A working memory account for spatial-numerical associations. Cognition 119, 114–119. doi: 10.1016/j.cognition.2010.12.013

van Dijck, J. P., Gevers, W., Lafosse, C., Doricchi, F., and Fias, W. (2011). Non-spatial neglect for the mental number line. Neuropsychologia 49, 2570–2583. doi: 10.1016/j.neuropsychologia.2011.05.005

von Allmen, D. Y., Wurmitzer, K., Martin, E., and Klaver, P. (2013). Neural activity in the hippocampus predicts individual visual short-term memory capacity. Hippocampus 23, 606–615. doi: 10.1002/hipo.22121

Whorf, B. L. (1956). Language, Thought and Reality: Selected Writings of Benjamin Lee Whorf. ed J. B. Carroll, Cambridge, MA: MIT Press.

Zebian, S. (2005). Linkages between number concepts, spatial thinking and directionality of writing: the SNARC effect and the reverse SNARC effect in English and Arabic monoliterates, biliterates and illiterate Arabic speakers. J. Cogn. Cult. 5, 165–190. doi: 10.1163/1568537054068660

Zorzi, M., Priftis, K., and Umiltà, C. (2002). Brain damage: neglect disrupts the mental number line. Nature 417, 138–139. doi: 10.1038/417138a

Conflict of Interest Statement: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Received: 09 September 2014; Accepted: 01 November 2014; Published online: 25 November 2014.

Citation: Abrahamse E, van Dijck J-P, Majerus S and Fias W (2014) Finding the answer in space: the mental whiteboard hypothesis on serial order in working memory. Front. Hum. Neurosci. 8:932. doi: 10.3389/fnhum.2014.00932

This article was submitted to the journal Frontiers in Human Neuroscience.

Copyright © 2014 Abrahamse, van Dijck, Majerus and Fias. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

ORIGINAL RESEARCH ARTICLE
published: 30 January 2015
doi: 10.3389/fnhum.2015.00008

Spatial coding of ordinal information in short- and long-term memory

Véronique Ginsburg* and Wim Gevers

Center for Research in Cognition and Neurosciences (CRCN), Faculté des Sciences Psychologiques et de l’Éducation, Action Bias & Control Group, Université Libre de Bruxelles (ULB), Brussels, Belgium

* Correspondence: Véronique Ginsburg, Center for Research in Cognition and Neurosciences (CRCN), Faculté des Sciences Psychologiques et de l’Éducation, Action Bias & Control Group, Université Libre de Bruxelles (ULB), 50 avenue Franklin Roosevelt – CP191, 1050 Brussels, Belgium e-mail: veronique.ginsburg@ulb.ac.be

Edited by:
Elger Abrahamse, Ghent University, Belgium

Reviewed by:
Mario Bonato, Ghent University, Belgium
Oliver Lindemann, University of Potsdam, Germany
Masami Ishihara, Tokyo Metropolitan University, Japan

The processing of numerical information induces a spatial response bias: Faster responses to small numbers with the left hand and faster responses to large numbers with the right hand. Most theories agree that long-term representations underlie this so called SNARC effect (Spatial Numerical Association of Response Codes; Dehaene et al., 1993). However, a spatial response bias was also observed with the activation of temporary position-space associations in working memory (ordinal position effect; van Dijck and Fias, 2011). Items belonging to the beginning of a memorized sequence are responded to faster with the left hand side while items at the end of the sequence are responded to faster with the right hand side. The theoretical possibility was put forward that the SNARC effect is an instance of the ordinal position effect, with the empirical consequence that the SNARC effect and the ordinal position effect cannot be observed simultaneously. In two experiments we falsify this claim by demonstrating that the SNARC effect and the ordinal position effect are not mutually exclusive. Consequently, this suggests that the SNARC effect and the ordinal position effect result from the activation of different representations. We conclude that spatial response biases can result from the activation of both pre-existing positions in long-term memory and from temporary space associations in working memory at the same time.

Keywords: numbers, space, SNARC effect, ordinal coding, working memory, long-term memory

INTRODUCTION

In the domain of numerical cognition it is well known that the processing of numbers and space is highly related. The first scientific articles, illustrating this link, date back to the 19th century. Galton published two papers in which he reported that some persons explicitly represent numbers in a spatial organized way, that he termed “number form” (Galton, 1880a,b). One of the most striking demonstrations of such a numbers-space association is the Spatial Numerical Association of Response Codes (SNARC) effect (Dehaene et al., 1993). This effect reveals an association between numerical magnitude and lateralized motor responses: participants respond faster to small numbers with the left hand side and to large numbers with the right hand side. The SNARC effect has been observed in multiple studies using different design, tasks settings and populations (see Wood et al., 2008). One of these tasks is the parity judgment task (Dehaene et al., 1993). In this task, participants have to classify target numbers as odd or even by pressing a left or a right response button. The SNARC effect is also observed when magnitude information has to be accessed more explicitly such as in a magnitude comparison task (Dehaene et al., 1990). In this task, participants have to judge whether a target number is smaller or larger than a reference number (for instance: 5) by pressing a left or a right-sided response button.

Different frameworks exist interpreting the SNARC effect: Some argue that the SNARC effect results from an association between the position of the number on a left-to-right (or right-to-left depending on reading habits) oriented mental number line (MNL) and the position of the response (Restle, 1970; Dehaene et al., 1993). Other researchers argue that the SNARC effect results from associations between magnitude concepts such as “small”, or “large” and spatial concepts such as “left” or “right”. For instance Proctor and Cho (2006) suggest that stimulus and response characteristics are coded as positive and negative polarities. The SNARC effect then results from the fact that small and left are coded as negative polarity and large and right as positive polarity. Similarly, Gevers et al. (2006) proposed a computational model that explains the SNARC effect assuming associations between concepts such as small-large and left-right. Despite the differences in interpretation of the SNARC effect, all of these accounts converge on the idea that long-term representations underlie the interactions between numbers and the side of response.

However, the association between numbers and space is more flexible than one could expect on the basis of such long-term associations. Early reports on the SNARC effect demonstrated that relative instead of absolute magnitude information is associated with response side (Dehaene et al., 1993; Fias et al., 1996). For example, the number 5 was responded to faster with the left hand when it was relatively small within the range (e.g., numbers ranged from 4 to 9), but the same number was responded to faster with the right hand when it was relatively large within the range (e.g., from 1 to 5). More recently, Ben Nathan et al. (2009) investigated deeper the range effect in a magnitude comparison task by changing the standard reference from trial to trial. They also observed that the SNARC effect was influenced by the relative instead of the absolute magnitude of digits. For example, the number “3” was associated with the left hand when the referent was “4”, but with the right hand when the referent was “2”. Additionally, Bächtold et al. (1998) demonstrated that the SNARC effect could be reversed by means of mental imagery. When subjects were asked to imagine numbers on a clock face, the SNARC effect reversed, because now small numbers occurred on the right side of the clock face and large numbers on the left side. Similarly, Shaki and Fischer (2008) showed that Russian-Hebrew bilinguals presented a normal SNARC effect if they had to read a Russian text (reading from left to right) just before the SNARC task but this effect was significantly reduced if they had to read a Hebrew text (reading from right to left) just before.

This high flexibility of the association between numbers and response side seems to indicate that number-space associations are built up during the task. This suggests that working memory could play an important role in the creation of these associations (e.g., Fias et al., 2011). In agreement with this view, several studies demonstrated that the availability of working memory resources is necessary, under certain circumstances, to observe number-space associations (Herrera et al., 2008; van Dijck et al., 2009). Following these observations, van Dijck and Fias (2011) more directly investigated the role of working memory in the creation of number-space associations. In their experiment, participants were asked to keep a sequence of five numbers in working memory (randomly chosen between 1 and 10). Subsequently, participants had to perform a parity judgment task, but only on numbers that belonged to the memorized sequence. This go-nogo procedure was used to ensure that the numbers had to be retrieved from working memory. Interestingly, using this paradigm, the researchers observed that lateralized responses were not associated with the magnitude of the numbers (no SNARC effect) but with the ordinal position of the numbers in the memorized sequence. Regardless of their magnitude, numbers from the beginning of the memorized sequence were responded to faster with the left hand side whereas numbers at the end of the sequence were responded to faster with the right hand side (e.g., from here on this observation will be termed the ordinal position effect). On the basis of this observation, van Dijck and Fias (2011) proposed a working memory account of the SNARC effect as an alternative to the long-term representation of numbers (van Dijck and Fias, 2011). It was more specifically suggested that the SNARC effect observed in typical situations (e.g., magnitude comparison, parity judgment) does not result from a long-term representation of numbers but rather from the creation of a task relevant ordinal sequence in working memory. This interpretation is in accordance with the fact that a SNARC-like effect also appears with non numerical stimuli with an ordinal structure such as letters of the alphabet (Gevers et al., 2003), overlearned new sequences (Van Opstal et al., 2009; Previtali et al., 2010) or musical tones (Lidji et al., 2007).

It is well known that working memory is not a unitary process. Most models on working memory functioning make a distinction between maintaining and retrieving information from working memory (Baddeley and Hitch, 1974; Oberauer, 2002). In a follow up study, Ginsburg et al. (2014) examined whether and how the type of processing (maintenance or retrieval) influenced the mapping between numbers and lateralized responses. They replicated the observations of van Dijck and Fias (2011): an ordinal position effect was observed when participants needed to retrieve memorized numbers during the classification phase (go-nogo procedure). In another task, the go-nogo procedure was removed from the design. Participants had to respond to all digits, both inside and outside the working memory sequence (respond-all procedure). This way, the memorized numbers were maintained for later recall while retrieval was not required during the classification of the numbers. Using this respond-all procedure the ordinal position effect was no longer observed while the SNARC effect remerged. These observations were explained by referring to the working memory model proposed by Oberauer (2002). In this model, a distinction is made between items that need to be maintained for later recall and items that need to be retrieved while being maintained. Maintained items would be represented as increased activations in long-term memory. The retrieval of these items, maintained in working memory, would result in the creation of new temporary bindings between these items. Oberauer (2010) suggested that these temporary bindings link the items with locations in mental space such as the ordinal position in a list. On the basis of this framework, Ginsburg et al. (2014) speculated that the SNARC effect would typically result from increased activations in long-term memory while the ordinal position effect results from the new bindings that are created between the retrieved items.

The current study was set up to investigate the relation between the SNARC effect and the ordinal position effect. In the theoretical working memory framework of van Dijck and Fias (2011), it was hypothesized that both effects are derived from temporary position-space associations. Following this hypothesis, the association between numbers and space is not long-term but created during task performance. For instance, when participants perform a simple numerical classification task (e.g., magnitude comparison task), the sequence of numbers is strategically activated in its canonical order to facilitate task execution (van Dijck et al., 2013). When, on the other hand, participants retrieve numbers from a newly memorized sequence (go-nogo procedure), it would be this new relevant sequence that is activated to perform the task. In other words, depending on the specific task-set, only one sequence is preferentially activated in working memory.

According to an alternative theoretical proposal (Ginsburg et al., 2014), the canonical representation of the numbers is automatically activated, whenever a numerical task has to be performed, regardless of the task-set. However, when a new memorized sequence becomes relevant to perform the task, an imbalance is created between the activation of the irrelevant canonical representation and the activation of the new relevant sequence. A natural consequence of this view is that the SNARC effect and the ordinal position effect could be observed simultaneously provided that a balance exists in the activation of both short- and long-term sequences. As stimulus material, numbers are ideal to investigate this point because they can easily be used as material in working memory while at the same time continue to be overlearned in long-term memory.

EXPERIMENT 1

According to the framework of Ginsburg et al. (2014), it can be argued that the SNARC effect is not observed in the go-nogo task used by van Dijck and Fias (2011) because there is an imbalance between the activation of the numerical long-term representations and the new memorized sequence. As suggested by Oberauer (2010, on p. 281): “the more active a representation, the easier it is to retrieve it”. So, the new memorized sequence should be more activated than the canonical representation because this information must be retrieved to perform the task correctly. In this experiment, to increase the activation of numerical long-term representations, participants were asked to respond to all numbers (inducer task: respond all paradigm) before performing the diagnostic go-nogo task.

MATERIALS AND METHODS

Participants

In total, 42 paid volunteers (on average 22.00 years (SD = 2.41); 25 females (four left handed) and 17 males (two left handed)) participated in this experiment. All participants were undergraduate students recruited via an announcement on Facebook. Participants received 10 euros as compensation for their participation. The ethical committee approved this study and participants received a debriefing after completing a single 60 min session. All participants were naive with respect to the purpose of the experiment.

Material

The experiment was performed using E-Prime 2 Professional Software (Psychology Software Tools). Participants were seated in a quiet room approximately 50 cm from a 17 inch LCD computer screen with a resolution of 1280 × 1024 pixels. The motor responses were collected via button presses on a response box. Each digit (approximately 1.37°) was presented on the computer screen in white color on a black background.

Stimuli and procedure

The experiment consisted of 40 different blocks: 20 inducer blocks and 20 diagnostic blocks. Two response mappings were introduced, a SNARC compatible response mapping (small numbers, left response—large numbers, right response) and a SNARC incompatible response mapping (small numbers, right response—large numbers, left response). Each participant started with 10 inducer blocks followed by 10 diagnostic blocks. After this, the response mapping was switched and the participant again performed 10 inducer followed by 10 diagnostic blocks. Which response mapping was performed first was counterbalanced across participants, but the same order was maintained in the first and the second half of the experiment.

Each block was divided in three subsequent phases: an encoding phase, a classification phase, and a control phase. During the encoding phase, five digits (randomly chosen in the range from 1 to 10) were successively presented during 1500 ms at the center of the screen (with an interval of 200 ms). Each digit could be presented only once in a sequence. The first digit of this sequence was preceded by a fixation cross during 500 ms. Participants were instructed to memorize this numerical sequence in the correct order.

During the classification phase, participants continued to keep the memorized numerical sequence in mind while performing a magnitude comparison task. Participants had to indicate whether the presented target number was small (range 1–5) or large (range 6–10). In each block, all numbers ranging from 1 to 10 were randomly presented twice with the restriction that the same number could not be repeated on consecutive trials. A trial consisted of a fixation cross (500 ms) followed by a target number. The response deadline was set to 1500 ms. After this deadline or after a response, the next trial was initiated, following an inter-trial interval of 1000 ms. During the inducer blocks participants had to perform the magnitude comparison task on every presented number (respond all paradigm). During the diagnostic blocks participants continued with the magnitude comparison task but now responded only to numbers that belonged to the memorized sequence (go-nogo paradigm).

During the last phase, the control phase, participants had to judge by pressing on a response button whether or not a new sequence of five digits (sequentially presented in the center of the screen during 1000 ms with an interval inter-stimuli of 200 ms) was the same sequence as the one kept in memory.

The non-corresponding sequences were composed by the same five numbers of the memorized sequence but, at a random location, the order between two adjacent numbers was reversed. This phase is important to assure that participants memorized correctly the numerical sequence during the classification phase. For this reason, if the participant responded erroneously to the control phase during the diagnostic task, the entire block was introduced again at the end of the experiment. To ensure that all participants performed exactly the same inducer task, we decided to not repeat a block of trials with an incorrect response to the control phase. Rather, participants were eliminated from analyses if they made more than three errors to memorized blocks in the control phase.

After each block participants had the opportunity to take a break and started the next block by pressing a response button. Concerning the training, before each mapping condition (20 blocks each), participants performed one block with the respond all procedure and one block with the go-nogo procedure to get used to the task.

RESULT AND DISCUSSION

Inducer task—respond all procedure

The data of one participant were removed from the analysis because he made more than three errors during the control phase. For the analyses, we took into account only blocks with an accurate control phase and correct trials with reaction times (RTs) larger than 250 ms. Only 0.1% of data points were discarded with this RT cutoff.

During the inducer task, participants performed correctly on average 8.7 blocks (SD = 1.38) on 10 blocks. During the magnitude comparison task, the average reaction time was 530.97 ms (SD = 72.91 ms) and the average number of errors was 3.65% (SD = 3.59). A sharp drop in performance was observed for the numbers 5 and 6 (mean of errors = 10.29%) compared to the other numbers (mean of errors = 1.99%). Classifying the numbers 5 and 6 as small or large seem particularly difficult because these numbers lie at the boundary of small and large categorizations. For this reason these numbers were removed from the analyses (e.g., Ginsburg et al., 2014). Nevertheless, a separate analysis with the numbers 5 and 6 included resulted in exactly the same pattern of results. Because in the inducer task participants responded to all numbers, a SNARC effect but no ordinal position effect was expected. To investigate the presence of this effect we used a repeated measure ANOVA with numerical magnitude (2: small numbers 1–4, large numbers 7–10) and response side (2: left, right) as within-subjects factors (recommended by Schwarz and Keus, 2004). No main effect was significant. However a SNARC effect was observed, indicated by a significant interaction between magnitude and response side (F(1,40) = 4.44, p < 0.05, = 0.08). Participants responded faster to small digits with the left-hand side (mean RT = 506.34 ms, SD = 11.17) than with the right-hand side (mean RT = 516.03 ms, SD = 11.96) while they responded faster to large digits with the right-hand side (mean RT = 506.34 ms, SD = 11.17) than with the left-hand side (mean RT = 524.55 ms, SD = 11.80). A separate ANOVA was conducted to investigate the presence of the ordinal position effect for the memorized numbers with ordinal position (5: from 1 to 5) and response side (2: left, right) as within-subjects factors. No ordinal position effect was observed (p = 0.35). These analyses were complemented with a regression approach described by Lorch and Myers (1990; see also Fias et al., 1996). This method consists of computing the difference in RTs (dRT; RT right hand minus RT left hand) for each number (from 1 to 10, except 5 and 6) or for each position in working memory (from 1 to 5) separately. Per subject, these values were entered in a regression analysis with number or position as predictor. A t-test was performed to evaluate whether the regression weights of the group deviated significantly from zero. The regression analysis confirmed the presence of the SNARC effect (dRT= −6.49 ms, t(40) = −2.16, p < 0.05) (Figure 1A) but no evidence was found for the presence of the ordinal position effect (t(40)= 0.06, p= 0.95) (Figure 1B).

FIGURE 1. Experiment 1: the SNARC effect and the ordinal position effect in the inducer and the diagnostic tasks. Observed data and regression line of Experiment 1, representing RT differences between right and left hand responses in function of the numerical magnitude (in the inducer task (A)—in the diagnostic task (C) and in function of the ordinal position of digits in working memory (in the inducer task (B)—in the diagnostic task (D). Positive values reflect faster left than right responses.

In sum, replicating previous work (Ginsburg et al., 2014), when information inside working memory is maintained but not retrieved during the classification, a SNARC effect but no ordinal position effect was observed. The question now becomes whether this pre-activation of the numerical canonical order in the inducer task has an effect on the presence of the SNARC effect in the diagnostic task.

Diagnostic task—go-nogo procedure

The same cut-off criteria were maintained during the diagnostic task as during the inducer task. No data points were discarded with the RT cutoff of 250 ms. During this task, participants made on average a bit more than one error to the control phase across 10 blocks (for both conditions: odd-left/even-right and odd-right/even-left). As such, participants required on average 11.15 blocks (SD = 1.90) to finish the task having performed 10 correct sequences. During the magnitude comparison task, the average reaction time was 696.43 ms (SD = 98.43 ms) and the average number of errors was 4.85% (SD = 2.21). For the same reasons as in the inducer task, analyses were performed on all numbers except the numbers 5 and 6. Again, as in the inducer blocks, separate analyses on the entire range of numbers showed the same results. During this task, participants classified numbers as small or large only if they belonged to the memorized sequence. Because all responded stimuli were inside the sequence, we used repeated measures ANOVA with numerical magnitude (2: small, 1–4; large, 7–10), ordinal position (5: from 1 to 5) and response side (2: left, right) as within-subjects factors to investigate the presence of the SNARC effect and the ordinal position effect for the elements inside working memory. A main effect of position was observed (F(4,160) = 12.28, p < 0.001, = 0.23). Average RTs per position increased gradually (652, 676, 681, 708, 705 for each position, respectively). A polynomial contrast confirmed the linear trends of these RTs (F(1,40) = 37.00, p < 0.001, = 0.48), suggesting a serial search strategy. No other main effect reached significance. Indicating the presence of the ordinal position effect, a significant interaction was observed between ordinal position and response side (F(4,160) = 4.89, p < 0.005, = 0.11). The interaction between numerical magnitude and response side, representing the SNARC effect, was not significant (F(1,40) = 0.28, p = 0.60, = 0.01). The triple interaction between numerical magnitude, ordinal position, and response side was also not significant (F(4,160) = 0.55, p = 0.70, = 0.01). The data were re-analyzed using a Lorch and Myers (1990) regression analysis. The dRTs decreased 16.18 ms per position (t(40) = −3.82, p < 0.001) (Figure 1D). This was not the case for the SNARC effect, where the slope did not differ from zero (t(40) = −0.75, p = 0.46) (Figure 1C).

Only an ordinal position effect was observed and no SNARC effect in the diagnostic task. Because the inducer task was presented blocked-wise before the diagnostic task, it is possible that its influence was limited to the first part of the diagnostic task. To investigate this possibility, the possible presence of the SNARC effect was investigated taking the time course of the diagnostic task into account. To this end, a repeated measures ANOVA was run with magnitude (2: < 5 or > 6), response side (2: left hand–right hand) and time (2: first five correct blocks, five last correct blocks) as within-subjects factors. This analysis revealed an interaction between time, numerical magnitude and response side (F(1,40) = 4.20, p < 0.05, = 0.10) (Figure 2). Even though the interaction was significant, planned comparisons indicated no significant SNARC effect, neither for the first (F(1,40) = 2.07, p = 0.16), nor for the second half (F(1,40) = 0.27, p = 0.61).

FIGURE 2. The interaction between time*magnitude*response in the diagnostic task of Experiment 1. Mean reaction times in the magnitude comparison task of the diagnostic part (Experiment 1) as function of the factors number magnitude (small–large), response side (left–right) and time (5 first blocks–5 last blocks).

A final repeated measures ANOVA with ordinal position (2: early, late), response side (2: left, right) and time (2: first five correct blocks, five last correct blocks) revealed a main effect of ordinal position (F(4, 160) = 15.53, p < 0.001, = 0.28) and an interaction between position and response side (F(4,160) = 4.52, p < 0.005, = 0.10) but no interaction of the ordinal position effect with time (p = 0.675).

As in previous studies (van Dijck and Fias, 2011; Ginsburg et al., 2014), when a go-nogo procedure was used, only an ordinal position effect but no SNARC effect was observed. Even though a SNARC effect was observed during the inducer phase, this seemed not to have a strong influence on the activation of the numerical canonical order during the diagnostic task. However, an exploratory analysis taking the influence of the time course of the diagnostic task into account revealed a triple interaction between magnitude, time and response side. This interaction suggests that the inducer task has an influence on the performance in the diagnostic task, but that this influence is limited in time.

EXPERIMENT 2

In Experiment 2, the possible co-occurrence of the SNARC effect and the ordinal position effect is further explored. Even though based on exploratory analyses, the results of the previous experiment suggest that the inducer task has a time limited influence on the SNARC effect in the diagnostic task. This time, a task-switching paradigm is used to ensure that participants activate the numerical canonical order throughout the entire experiment. To this end, participants switched randomly between respond-all and go-nogo blocks.

An extra advantage of the task-switching paradigm is that it enables to investigate whether the spatial associations causing the ordinal position effect are created during encoding or during retrieval. On each block, an instruction was given with the written words “IN” or “ALL” to inform participants whether they had to perform the go-nogo paradigm (IN) or the respond all paradigm (ALL). This instruction was presented directly before or directly after the encoding phase. The process of encoding can be qualitatively different if participants know beforehand what task to perform on the encoded information compared to the situation where this information is given only after encoding. Therefore, if type of encoding is important for the ordinal position effect, this manipulation could have an influence on the associations between lateralized responses and ordinal position in working memory. If on the other hand, the spatial associations responsible for the ordinal position effect are created during retrieval, no such difference is expected.

MATERIALS AND METHODS

Participants

In total, 52 paid volunteers (on average 22.19 years (SD = 2.33); 39 females (4 left handed) and 13 males (all right handed)) participated in this experiment. All participants were undergraduate students recruited via an announcement on Facebook. Participants received 8 euros as compensation for their participation. The ethical committee approved this study and participants received a debriefing after completing a single 40 min session. All participants were naive with respect to the purpose of the experiment.

Material, stimuli and procedure

In Experiment 2, we used exactly the same material, stimuli and procedure as in Experiment 1 with the exception that the inducer and the diagnostic tasks were randomly intermixed. For each block, an instruction was given with the written words “IN” or “ALL” to inform participants whether they had to perform the magnitude comparison task only on digits inside the memorized sequence (diagnostic task: go-nogo paradigm) or on all presented digits (inducer task: respond all paradigm), respectively. Half of the participants received these instructions before the encoding phase while the other half of the participants received these instructions after the encoding phase. The entire block was introduced again at the end of the experiment if the participant responded erroneously to the control phase of this block, regardless of the task (inducer or diagnostic tasks).

The response mapping was counterbalanced across participants. Half of the participants started with a SNARC compatible mapping while the other half started with a SNARC incompatible mapping. The response mapping was switched after 20 blocks (10 inducer and 10 diagnostic blocks).

RESULT AND DISCUSSION

Inducer task—respond all procedure

The data of one participant were removed from the analysis because he did not follow the task instructions and did not respond during the classification phase. Another participant was excluded because he made too many errors (more than 2.5 SDs above the mean of errors) during the inducer task. We took into account only blocks with an accurate control phase and correct trials with RTs larger than 250 ms (two data points were discarded with this cut-off). During this task, participants performed correctly on average 11.72 blocks (SD = 2.56) on 10 blocks. During the magnitude comparison task, the average reaction time was 513.51 ms (SD = 72.88 ms) and the average number of errors was 4.70% (SD = 2.63). For the same reason as in the previous experiment, analyses for both tasks (inducer and diagnostic) were performed on all numbers except the numbers 5 and 6. Again, as in Experiment 1, an extra analysis on the entire range of numbers showed the same pattern of results. As in the previous experiment, we investigated whether a SNARC effect was obtained in the inducer task by using a repeated measure ANOVA with numerical magnitude (2: small numbers 1–4, large numbers 7–10) and response side (2: left, right) as within-subjects factors and time of instruction (2: before-after the encoding phase) as the between subject factor. The time of instruction did not interact significantly with the other factors. Only the interaction between numerical magnitude and response side was significant (F(1,48) = 17.48, p < 0.001, = 0.27), indicating the presence of a SNARC effect. Indeed, participants responded faster to small digits with the left-hand side (mean RT = 484.93 ms, SD = 13.26) than with the right-hand side (mean RT = 500.02 ms, SD = 14.16) while they responded faster to large digits with the right-hand side (mean RT = 492.11 ms, SD = 11.16) than with the left-hand side (mean RT = 518.76 ms, SD = 12.03). Further, the presence of the ordinal position effect was investigated for memorized numbers using repeated measures ANOVA with ordinal position (5: from 1 to 5) and response side (2: left, right) as within-subjects factors and time of instruction (2: before-after the encoding phase) as between subjects factor. The interaction between ordinal position and response side, suggesting an ordinal position effect, was not significant (p = 0.56). The analyses with the regression approach confirmed the same results. Concerning the SNARC effect, the regression slope differed from zero (t(49) = −4.43, p < 0.001) (Figure 3A) but this was not the case for the ordinal position effect (t(49) = −1.32, p = 0.19) (Figure 3B). In sum, as in Experiment 1 the inducer task resulted in a SNARC effect but no ordinal position effect.

FIGURE 3. Experiment 2: the SNARC effect and the ordinal position effect in the inducer and the diagnostic tasks. Observed data and regression line of Experiment 1, representing RT differences between right and left hand responses in function of the numerical magnitude (in the inducer task (A)—in the diagnostic task (C) and in function of the ordinal position of digits in working memory (in the inducer task (B)—in the diagnostic task (D). Positive values reflect faster left than right responses.

Diagnostic task—go-nogo procedure

During this task, participants made on average a bit more than one error to the control phase across 10 blocks (for both conditions: small-left/large-right and small-right/large-left). As such, participants required on average 11.37 blocks (SD = 1.90) to finish the task having performed 10 correct sequences. During the magnitude comparison task, the average reaction time was 680.11 ms (SD = 90.88 ms) and the average number of errors was 5.72 % (SD = 3.22). Only one data point was discarded with the RT cut-off of 250 ms. Given that participants responded only to digits inside working memory, we used repeated measures ANOVA with numerical magnitude (2: small, 1–4; large, 7–10), ordinal position (5: from 1 to 5) and response side (2: left, right) as within-subjects factors and time of instruction (2: before-after the encoding phase) as the between subject factor. This ANOVA indicated a main effect of position (F(4,192) = 12.00, p < 0.001, = 0.20). Average RTs per position increased gradually (674, 686, 704, 722, 712 for each position, respectively). The linear trends of these RTs was confirmed by the polynomial contrast (F(1,48) = 45.66, p < 0.001, = 0.49), suggesting a serial search strategy. The analysis revealed a significant interaction between ordinal position and response side, reflecting the presence of an ordinal position effect (F(4,192) = 5.54, p < 0.001, = 0.10). Importantly, also the interaction between numerical magnitude and response side (indicative for the SNARC effect) was significant (F(1,48) = 6.73, p < 0.05, = 0.12). The triple interaction between numerical magnitude, ordinal position, and response side was not significant (F(4,192) = 1.68, p = 0.16, = 0.03). These results were confirmed using the regression analysis. The dRTs decreased 12.46 ms per ordinal position (t(49) = −3.42, p < 0.005) (Figure 3D) and 10.98 ms per numbers (t(49) = −3.22, p < 0.005) (Figure 3C), indicating the presence of an ordinal position effect and a SNARC effect.

One could ask whether the presence of both effects simultaneously in Experiment 2 was not due to averaging. Indeed, the possibility remains that the position effect was dissociated from the SNARC effect at the individual level but not at the group level. That is, some participants would show the SNARC effect only while other participants would show the ordinal position effect only. If this were the case, we should have observed a negative correlation between the ordinal position effect and the SNARC effect. However, this was not observed (r = 0.019, p = 0.89).

In sum, during the inducer task, as in Experiment 1, we observed the presence of a SNARC effect. During the diagnostic task, both an ordinal position effect and a SNARC effect were observed. The moment of instruction (before or after the encoding phase) had no influence on either the SNARC effect or the ordinal position effect.

DISCUSSION

In two experiments we investigated the relation between the SNARC effect and the ordinal position effect. The ordinal position effect is believed to result from the creation of temporary position-space associations. If the SNARC effect also results from the activation of these temporary position-space associations (e.g., van Dijck and Fias, 2011), then it logically follows that the SNARC effect and the ordinal position effect are mutually exclusive. If, on the other hand, the SNARC effect results from the activation of long-term semantic representations and not from the activation of temporary-space associations, then both effects can in principle be observed at the same time (e.g., Ginsburg et al., 2014). Both our experiments consisted of an inducer task and a diagnostic task. The goal of the inducer task was to activate the associations between lateralized responses and the canonical order of digits, resulting in the SNARC effect. In the diagnostic part we measured the influence of pre-activating the SNARC effect on the categorization of a newly memorized sequence of numbers. The inducer task and the diagnostic task were presented blocked-wise (Experiment 1) or randomly intermixed (Experiment 2).

The results of the inducer task were highly similar in both experiments and similar to the results obtained in a previous study (Ginsburg et al., 2014). That is, both in Experiment 1 and in Experiment 2 the diagnostic task resulted in a SNARC effect but no ordinal position effect. The results differed between both experiments on the diagnostic task. In Experiment 1, the inducer task and the diagnostic task were presented blocked-wise. Only an ordinal position effect but no SNARC effect was observed. Additional exploratory analyses on the diagnostic task revealed that the SNARC effect interacted with time. This interaction suggests that the inducer task did have an influence on the presence of the SNARC effect during the diagnostic task, but that this influence was limited in time. We followed up on this hypothesis in Experiment 2 by randomly inter-mixing the inducer and the diagnostic tasks. Intermixing both tasks assured that the activation of the associations resulting in the SNARC effect (inducer part) remained active throughout the entire experiment. Importantly, as a consequence of this manipulation, both the SNARC effect and the ordinal position effect were observed simultaneously.

The current results are consistent with the suggestion that the SNARC effect reflects spatial associations with pre-existing activated bindings whereas the ordinal position effect reflects spatial associations with new temporary bindings needed for the task (Ginsburg et al., 2014). A tentative interpretation of this idea is that, in most everyday life situations, numbers are presented in their canonical order. Activation of this canonical long-term representation would automatically take place when we have to deal with numbers (as in a magnitude comparison task or in a parity judgment task). This activation is a prerequisite to create spatial associations between this canonical order and response side (e.g., the SNARC effect). However, when retrieval of numbers belonging to a new memorized sequence is needed (go-nogo task), an imbalance is established between the activation of the irrelevant canonical order and the activation of the new, non canonical, relevant sequence. Because the retrieval of the new memorized sequence is needed to perform the task (go-nogo procedure), the latter would receive more activation, resulting in an ordinal position effect. In Experiment 1, because the inducer and the diagnostic tasks were performed separately, the activation of the canonical representation of numbers was still lower than the activation of the new memorized sequence during the diagnostic task. As a consequence, the influence of the inducer task was short-lived illustrated by the interaction between the SNARC effect and time during the diagnostic task. In the second experiment, on the other hand, the inducer and diagnostic tasks were randomly intermixed and the switch between tasks requires constant updating of the current task set (canonical order and newly memorized order) (Monsell, 2003). Because of this task switching, bindings between memorized items must be established strong enough to be armed against interference but loosely enough to be rapidly dismantled (Oberauer, 2010). As such, during diagnostic blocks, a competition likely takes place between the activation of the irrelevant sequence of numbers activated in long-term memory (e.g., the canonical order of numbers) and the relevant sequence of numbers (newly memorized sequence of numbers) in working memory that needs to be maintained and retrieved (for a similar reasoning, see Oberauer, 2009; Szmalec et al., 2011). As a result, the ordinal position effect and the SNARC effect were simultaneously observed. A final relevant observation is that the SNARC effect and the ordinal position effect did not interact. This lack of interaction is congruent with the idea, as proposed above, that the SNARC effect and the ordinal position effect reflect the activation of different representations (long-term pre-existing representations and temporary working memory representations, respectively). Firm replication of this observation is however needed as it concerns a reasoning on the basis of the absence of an effect.

Even if the SNARC effect and the ordinal position effect reflect the activation of different types of representations (digits in long-term memory and in working memory, respectively), a similar attention mechanism might underlie both types of representations. Indeed, Fischer et al. (2003) demonstrated that merely looking at numbers can cause a shift of attention to the left or to the right side of space, depending on the canonical position of the digit in the sequence. The presentation of small numbers (1 and 2) induced a spatial shift of attention to the left, while the opposite pattern was observed for large numbers (8 and 9). More recently, van Dijck et al. (2014) demonstrated that such spatial shifts of attention could also be induced by the ordinal position of an item in a newly memorized sequence. This was demonstrated by combining the go-nogo procedure used by van Dijck and Fias (2011) with the spatial attention paradigm of Fischer et al. (2003). Spatial lateralized targets had to be detected only if a centrally presented number (or letter) belonged to a memorized sequence. Spatial shifts of attention were observed in accordance with the position of the item in working memory. A right dot advantage emerged with the retrieval of an element at the end of the memorized sequence. Such observations illustrate the role of spatial attention in number-space associations and add supplementary evidence to the implication of working memory on those associations. Indeed, working memory has been conceptualized as an attention-based process (e.g., Cowan, 1999; Oberauer and Hein, 2012) in which attention is a mechanism allocated for selection (Allport, 1987). Within this perspective, we consider the impact of working memory and attention as the influence of one and the same process.

The idea that the ordinal position in activated long-term memory and in a new memorized sequence can induce spatial shifts of attention does not imply that those representations themselves are spatial in nature. Our results demonstrate that shifts of attention (or alternative causes of the response bias) are more likely to be induced during the retrieval of the item from the sequence during the classification phase. Indeed, in previous work it was demonstrated that the ordinal position effect is not observed if retrieval is not required by the task (Ginsburg et al., 2014). On top of this, we observed in the Experiment 2 that the moment of instruction (before or after encoding) had no influence on the ordinal position effect and the SNARC effect, neither during the diagnostic or the inducer tasks. This suggests that the mapping between response side and the item is created during the retrieval of the target from the sequence to which it belongs (long-term or short-term representations).

In conclusion, the main result of the current study is that the SNARC effect and the ordinal position effect are not mutually exclusive. Therefore, both effects do not seem to be the result of the same underlying representation. Spatial associations can be tied to both activated long-term representations and to temporary short-term representations simultaneously. Furthermore, it was also observed that the time of instruction had no influence on the ordinal position effect. This observation strengthens the initial claim (e.g., Ginsburg et al., 2014) that spatial associations responsible for the ordinal position effect are created during the retrieval of the item from working memory.

ACKNOWLEDGMENTS

Véronique Ginsburg is a research fellow at the Belgian Fonds National de la Recherche Scientifique (F.R.S.–FNRS). This work was supported by the FNRS, M.I.S. N°F.4512.12.

REFERENCES

Allport, D. A. (1987). “Selection for action: some behavioural and neurophysiological considerations of attention and action,” in Perspectives on Perception and Action, eds H. Heuer and D. F. Saunders (Hillsdale, NJ: Erlbaum), 395–419.

Bächtold, D., Baumüller, M., and Brugger, P. (1998). Stimulus-response compatibility in representational space. Neuropsychologia 36, 731–735. doi: 10.1016/s0028-3932(98)00002-5

Baddeley, A. D., and Hitch, G. J. (1974). “Working memory,” in The Psychology of Learning and Motivation, ed G. H. Bower (New York, N.Y.: Academic Press), 47–89.

Ben Nathan, M., Shaki, S., Salti, M., and Algom, D. (2009). Numbers and space: associations and dissociations. Psychon. Bull. Rev. 16, 578–582. doi: 10.3758/pbr.16.3.578

Cowan, N. (1999). “An embedded-processes model of working memory,” in Models of Working Memory: Mechanisms of Active Maintenance and Executive Control, eds A. Miyake and P. Shah (Cambridge, U.K.: Cambridge University Press), 62–101.

Dehaene, S., Bossini, S., and Giraux, P. (1993). The mental representation of parity and number magnitude. J. Exp. Psychol. Gen. 122, 371–396. doi: 10.1037/0096-3445.122.3.371

Dehaene, S., Dupoux, E., and Mehler, J. (1990). Is numerical comparison digital? Analogical and symbolic effects in two-digit number comparison. J. Exp. Psychol. Hum. Percept. Perform. 16, 626–641. doi: 10.1037//0096-1523.16.3.626

Fias, W., Brysbaert, M., Geypens, F., and d’Ydewalle, G. (1996). The importance of magnitude information in numerical processing: evidence from the SNARC effect. Math. Cogn. 2, 95–110. doi: 10.1080/135467996387552

Fias, W., van Dijck, J. P., and Gevers, W. (2011). “How is number associated with space? The role of working memory,” in Space, Time and Number in the Brain: Searching for the Foundations of Mathematical Thought, eds S. Dehaene and E. Brannon (Amsterdam: Elsevier), 133–148.

Fischer, M. H., Castel, A. D., Dodd, M. D., and Pratt, J. (2003). Perceiving numbers causes spatial shifts of attention. Nat. Neurosci. 6, 555–556. doi: 10.1038/nn1066

Galton, F. (1880a). Visualised numerals. Nature 21, 252–256. doi: 10.1038/021252a0

Galton, F. (1880b). Visualized numerals. Nature 21, 494–495. doi: 10.1038/021494e0

Gevers, W., Reynvoet, B., and Fias, W. (2003). The mental representation of ordinal sequences is spatially organized. Cognition 87, B87–B95. doi: 10.1016/s0010-0277(02)00234-2

Gevers, W., Verguts, T., Reynvoet, B., Caessens, B., and Fias, W. (2006). Numbers and space: a computational model of the SNARC effect. J. Exp. Psychol. Hum. Percept. Perform. 32, 32–44. doi: 10.1037/0096-1523.32.1.32

Ginsburg, V., van Dijck, J.-P., Previtali, P., Fias, W., and Gevers, W. (2014). The impact of verbal working memory on number-space associations. J. Exp. Psychol. Learn. Mem. Cogn. 40, 976–986. doi: 10.1037/a0036378

Herrera, A., Macizo, P., and Semenza, C. (2008). The role of working memory in the association between number magnitude and space. Acta Psychol. (Amst) 128, 225–237. doi: 10.1016/j.actpsy.2008.01.002

Lidji, P., Kolinsky, R., Lochy, A., and Morais, J. (2007). Spatial associations for musical stimuli: a piano in the head? J. Exp. Psychol. Hum. Percept. Perform. 33, 1189–1207. doi: 10.1037/0096-1523.33.5.1189

Lorch, R. F., and Myers, J. L. (1990). Regression analyses of repeated measures data in cognitive research. J. Exp. Psychol. Learn. Mem. Cogn. 16, 149–157. doi: 10.1037//0278-7393.16.1.149

Monsell, S. (2003). Task switching. Trends Cogn. Sci. 7, 134–140. doi: 10.1016/S1364-6613(03)00028-7

Oberauer, K. (2002). Access to information in working memory: exploring the focus of attention. J. Exp. Psychol. Learn. Mem. Cogn. 28, 411–421. doi: 10.1037//0278-7393.28.3.411

Oberauer, K. (2009). Design for a working memory. Psychol. Learn. Motiv. Adv. Res. Theory 51, 45–100. doi: 10.1016/S0079-7421(09)51002-X

Oberauer, K. (2010). Declarative and procedural working memory: common principles, common capacity limits? Psychol. Belg. 50, 277–308. doi: 10.5334/pb-50-3-4-277

Oberauer, K., and Hein, L. (2012). Attention to information in working memory. Curr. Dir. Psychol. Sci. 21, 164–169. doi: 10.1177/0963721412444727

Previtali, P., de Hevia, M. D., and Girelli, L. (2010). Placing order in space: the SNARC effect in serial learning. Exp. Brain Res. 201, 599–605. doi: 10.1007/s00221-009-2063-3

Proctor, R. W., and Cho, Y. S. (2006). Polarity correspondence: a general principle for performance of speeded binary classification tasks. Psychol. Bull. 132, 416–442. doi: 10.1037/0033-2909.132.3.416

Restle, F. (1970). Speed of adding and comparing numbers. J. Exp. Psychol. 83, 274–278. doi: 10.1037/h0028573

Schwarz, W., and Keus, I. M. (2004). Moving the eyes along the mental number line: comparing SNARC effects with saccadic and manual responses. Percept. Psychophys. 66, 651–664. doi: 10.3758/bf03194909

Shaki, S., and Fischer, M. H. (2008). Reading space into numbers—A cross-linguistic comparison of the SNARC effect. Cognition 108, 590–599. doi: 10.1016/j.cognition.2008.04.001

Szmalec, A., Verbruggen, F., Vandierendonck, A., and Kemps, E. (2011). Control of interference during working memory updating. J. Exp. Psychol. Hum. Percept. Perform. 37, 137–151. doi: 10.1037/a0020365

van Dijck, J.-P., Abrahamse Acar, F. E., Ketels, B., and Fias, W. (2014). A working memory account of the interaction between numbers and spatial attention. Q. J. Exp. Psychol. (Hove) 67, 1500–1513. doi: 10.1080/17470218.2014.903984

van Dijck, J.-P., and Fias, W. (2011). A working memory account for spatial numerical associations. Cognition 119, 114–119. doi: 10.1016/j.cognition.2010.12.013

van Dijck, J.-P., Gevers, W., and Fias, W. (2009). Numbers are associated with different types of spatial information depending on the task. Cognition 113, 248–253. doi: 10.1016/j.cognition.2009.08.005

van Dijck, J.-P., Ginsburg, V., Girelli, L., and Gevers, W. (2013). “Linking Numbers to space: from the mental number line towards a hybrid account,” in The Oxford Handbook of Numerical Cognition, eds R. Cohen Kadosh and A. Dowker (Oxford, UK: Oxford University Press).

Van Opstal, F., Fias, W., Peigneux, P., and Verguts, T. (2009). The neural representation of extensively trained ordered sequences. Neuroimage 47, 367–375. doi: 10.1016/j.neuroimage.2009.04.035

Wood, G., Willmes, K., Nuerk, H.-C., and Fischer, M. H. (2008). On the cognitive link between space and number: a meta-analysis of the SNARC effect. Psychol. Sci. Q. 50, 489–525.

Conflict of Interest Statement: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Received: 27 July 2014; Accepted: 07 January 2015; Published online: 30 January 2015.

Citation: Ginsburg V and Gevers W (2015) Spatial coding of ordinal information in short- and long-term memory. Front. Hum. Neurosci. 9:8. doi: 10.3389/fnhum.2015.00008

This article was submitted to the journal Frontiers in Human Neuroscience.

Copyright © 2015 Ginsburg and Gevers. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

ORIGINAL RESEARCH ARTICLE
published: 14 October 2014
doi: 10.3389/fnhum.2014.00821

Better target detection in the presence of collinear flankers under high working memory load

Jan W. De Fockert* and Jaclyn Leiser

Department of Psychology, Goldsmiths, University of London, London, UK

* Correspondence: Jan W. De Fockert, Department of Psychology, Goldsmiths, University of London, New Cross, SE14 6NW, London, UK e-mail: j.de-fockert@gold.ac.uk

Edited by:
Wim Fias, Ghent University, Belgium

Reviewed by:
Filip Van Opstal, Université Libre de Bruxelles, Belgium
Elkan Akyurek, University of Groningen, Netherlands

There are multiple ways in which working memory can influence selective attention. Aside from the content-specific effects of working memory on selective attention, whereby attention is more likely to be directed towards information that matches the contents of working memory, the mere level of load on working memory has also been shown to have an effect on selective attention. Specifically, high load on working memory is associated with increased processing of irrelevant information. In most demonstrations of the effect to-date, this has led to impaired target performance, leaving open the possibility that the effect partly reflects an increase in general task difficulty under high load. Here we show that working memory load can result in a performance gain when processing of distracting information aids target performance. The facilitation in the detection of a low-contrast Gabor stimulus in the presence of collinear flanking Gabors was greater when load on a concurrent working memory task was high, compared to low. This finding suggests that working memory can interact with selective attention at an early stage in visual processing.

Keywords: working memory, selective attention, target detection, Gabor stimuli, collinear flankers

INTRODUCTION

Over the past decade, evidence has accumulated for a close link between selective attention and working memory (e.g., Awh and Jonides, 2001; Chun, 2011; Gazzaley and Nobre, 2012). An often reported effect is that distractibility in selective attention is enhanced in the context of either high working memory load (e.g., Lavie et al., 2004) or low working memory capacity (e.g., Kane and Engle, 2003; Ahmed and de Fockert, 2012), suggesting that efficient selective attention to task-relevant information relies on the availability of working memory. For example, the interference produced by irrelevant distractor letters flanking a relevant target (Eriksen and Eriksen, 1974) becomes greater under high working memory load (Lavie et al., 2004). Similar modulations of distractor processing as a function of working memory load have been reported in a range of other selective attention tasks (e.g., Lavie and De Fockert, 2005; Pecchinenda and Heil, 2007; Pratt et al., 2011).

At which point in the visual pathway does working memory affect selective processing? The majority of previous work suggesting that high load on a working memory task is associated with greater processing of task-irrelevant information in vision has used response-competition tasks. On those tasks, the effect of working memory load could originate anywhere between initial visual processing of the distractors and final response selection. Certain findings suggest that the effect of working memory on distractor processing occurs relatively early. First, working memory affects processing of distractors that are not associated with a task response (Lavie and De Fockert, 2005; De Fockert and Wu, 2009). Second, load on working memory load increases the attentional blink (Akyürek et al., 2007), an effect that is associated with modulation of an early component in electrophysiology (Akyürek et al., 2010). Finally, neuroimaging work has shown that activity in object-specific areas of the visual cortex, associated with distractor processing, is greater under high working memory load (De Fockert et al., 2001; Kelley and Lavie, 2011). High working memory load can also delay the neural response to a visual target in low-level visual areas in occipital cortex (Scalf et al., 2011).

The first aim of the current study was to examine the role of working memory in early vision processing. We measured the modulation of detection of a low contrast visual stimulus by the presence of high contrast flankers (Polat and Sagi, 1993), an effect that is thought to originate from lateral interactions between neural assemblies in early visual cortex (Gilbert and Wiesel, 1989; Polat et al., 1998; Freeman et al., 2001). When a low contrast Gabor stimulus is presented at fixation, detection thresholds are lower when it is flanked by high contrast, spatially aligned Gabors. In the first study to describe the lateral interactions produced by collinear visual flankers (Polat and Sagi, 1993), a centrally presented target Gabor was flanked by two high contrast flankers at varying target-to-flanker separations. In order for the flankers to facilitate target detection, the flankers have to have the same orientation as the target, be spatially aligned with the target, and occur at a certain target-to-flanker separation (Polat and Sagi, 1993). When those conditions are met, detection thresholds are significantly lower in the presence (vs. absence) of the flankers.

Importantly, the facilitation of target processing by high contrast flankers occurs only when the flankers are attended (Freeman et al., 2001). When observers are presented with a low contrast Gabor target accompanied by four flankers, two of which are aligned with the target and two of which are not, and are made to attend to two of the flankers while ignoring the other two, facilitation effects only occur when the attended flankers are collinear with the target. In other words, although collinear flankers were always present, they only facilitated detection thresholds when they were attended (Freeman et al., 2001). This finding strongly suggests that the high contrast flankers need to receive some attention in order to produce the performance gain on target detection. In the current study, observers were told that the flankers were irrelevant to the task (as their presence was not predictive of the presence of the relevant target), and attention should be focused on a central target stimulus. We argued that observers would be less able to maintain attentional focus on the target, and be more likely to attend to the irrelevant flanking Gabors, when working memory was highly loaded, as the ability to maintain attentional focus on the target location is compromised under high working memory load (e.g., Lavie et al., 2004; De Fockert and Bremner, 2011). As a result, the facilitation effect from collinear flankers should be greater under high working memory load.

In addition to testing whether working memory can modulate the effect of distractors that are assumed to involve early visual processing, our study had a second aim. The prediction that target processing should gain from high working memory load when distractors facilitate target performance forms a key test of the idea that loading working memory increases distraction. The previous evidence has mostly shown performance impairments in selective attention following increases in working memory load, such as slower and less accurate target responses because of greater flanker interference (Lavie et al., 2004), or stronger attentional capture by salient distractor singletons (Lavie and De Fockert, 2005), leaving open the possibility that the effects of high working memory load on performance in attention tasks partly result from an increase in general task difficulty. By contrast, here we anticipated an improvement in target performance under high working memory load, as more attention to the flankers should aid target detection.

METHOD

PARTICIPANTS AND STIMULI

Thirteen people (mean age, 20 years 5 months) volunteered to participate in the experiment, which was approved by the Department Ethics Committee at Goldsmiths. Sample size was determined on the basis of previous published work with the dual-task paradigm used here, showing that a significant effect of working memory load on distractor processing is obtained with around 12 participants (e.g., Lavie et al., 2004). We tested an additional participant to prevent being underpowered following the exclusion of one participant with poor performance on the working memory task.

The experiment was run in a darkened testing cubicle on a PC running E-Prime (Schneider et al., 2002). Stimuli were displayed on a non-linearized CRT (Mitsubishi Diamond Plus 220) at a viewing distance of approximately 60 cm. Gabor stimuli for the visual detection task were generated using an online E-Prime script[1]. The Gabor stimuli were symmetrical, with a Gaussian envelope and a spatial frequency of seven half-cycles in horizontal orientation. For the target stimuli, stimulus contrast was 0.3% (low contrast), 0.5% (medium contrast), or 0.9% (high contrast). For the flanker stimuli, contrast was either 50% (flankers present) or 0% (flankers absent). Gabor stimuli had a wavelength subtending 0.95° of visual angle, and the diameter of each Gabor stimulus subtended 2.48°. Target stimuli were presented at fixation in the screen center. Flankers were presented left and right of the target at 2.58° target-distractor center-to-center separation (2.7 wavelengths, a target-flanker distance previously shown to produce robust facilitation effects; Polat and Sagi, 1993), so that the entire stimulus array subtended 7.63° horizontally.

For the working memory task, a memory set consisting of either one (low load) or six (high load) digits was presented in green at the start of each trial. The single digit in the low load condition was presented at fixation. The six digits in the high load conditions were presented in a vertical line centered at fixation (to prevent cueing attention to the Gabor flanker locations which may occur following a horizontally presented memory set). Each digit subtended 0.38° horizontally and 0.57° vertically, and the entire high load set subtended 5.25° vertically. The memory probe was a single green digit with a question mark, presented at fixation. All stimuli were presented on a gray background (RGB values, 128,128,128), see Figure 1 for example stimulus displays.

FIGURE 1. Example of a high working memory load trial. Low working memory load trials were the same, apart from the memory set which contained only one digit presented for 1000 ms. There were five target detection displays for each working memory trial (images not to scale, target contrast enhanced for display purposes).

PROCEDURE

Each trial began with a 1000 ms central fixation screen (a black plus sign), followed by the memory set. The memory set consisted of one digit presented for 1000 ms in the low working memory load condition, and six digits presented for 2500 ms in the high load condition. The presentation durations of the low and high memory load sets allowed participants to rehearse the set at least once before presentation of the target detection trials. Participants had to memorize the set until the end of the trial. Next, the visual detection displays were presented. Each working memory trial contained five detection trials (to increase the amount of detection data collected for each working memory trial). Detection displays were presented for 200 ms, followed by a response screen consisting of a black question mark presented 5.44° below fixation (to prevent it from masking the preceding target stimulus), that remained visible until a response had been recorded. Participants were asked to press one of two keys on the numerical keypad with their right hand to indicate whether they thought the visual target stimulus had been present (press <2>) or absent (press <0>). On half the trials, the target was present, with equal proportions of each target contrast condition. On the other half of the trials, the target was absent. Different detection trial conditions were presented within each working memory trial. Accuracy feedback was given in the form of a tone following an incorrect response. After the final detection trial, a single memory probe digit was presented, and the participant had to use the same two keys as for the detection task to indicate whether or not the probe had been present in the set for that trial. On half of the working memory trials, a “present” response was correct. A feedback tone was presented following an incorrect response to the memory probe. Participants first received 32 practice trials on the detection task alone, on which the Gabor stimuli were presented until response. Next, they received five combined working memory/detection trials with high load working memory, with the same presentation durations as in the experimental trials. Two experimental blocks were then presented, one with low and one with high working memory load (order counterbalanced between participants), each containing 32 working memory trials (160 detection trials per working memory load condition).

RESULTS

Data from one participant, whose accuracy on the working memory task was below chance under high working memory load, were excluded. For the remaining 12 participants, responses to the working memory probe were analyzed first, which confirmed that the manipulation was effective in loading working memory. Mean response times were faster in the low working memory load condition (M = 1260) compared to the high working memory load condition (M = 1562; t(11) = 2.61, SEM = 115.8, p < 0.025, d = 0.78). Mean accuracy rates were also higher in the low working memory load condition (M = 0.841) compared to the high working memory load condition (M = 0.815), although this difference was not significant (t < 1).

Next, the probability to correctly detect a present Gabor patch was analyzed in a three (target contrast: low, medium, high) by two (flanker condition: present, absent) by two (working memory load: low, high) fully within-subjects Analysis of Variance (ANOVA; see Figure 2A). Only trials on which the working memory response was correct were included in this analysis. Stimulus visibility was successfully manipulated, as shown by a main effect of target contrast, F(1,11) = 42.03, MSe = 0.049, p < 0.001, = 0.793. Targets were least likely to be deemed present when they had low contrast (M = 0.326) compared to medium contrast (M = 0.535; t(11) = 5.86, SEM = 0.036, p < 0.001, d = 1.74) and high contrast (M = 0.741; t(11) = 7.18, SEM = 0.058, p < 0.001, d = 2.09), and less likely to be deemed present when they had medium contrast compared to high contrast (t(11) = 5.26, SEM = 0.039, p < 0.001, d = 1.52). The flanker facilitation effect (Polat and Sagi, 1993) was replicated, as shown by a significant main effect of flanker condition, F(1,11) = 18.92, MSe = 0.126, p < 0.01, = 0.632. Targets were more likely to be deemed present when the flankers were present (M = 0.662) compared to absent (M = 0.405). There was no main effect of working memory load (F < 1), but crucially, there was a significant two-way interaction between working memory load and flanker condition, F(1,11) = 4.93, MSe = 0.020, p < 0.05, = 0.310 (see Figure 3A). Planned follow-up tests (Bonferroni corrected) showed that the presence of the flankers led to a greater improvement in detection rates under high working memory load (from 0.389 to 0.699 in flanker absent and present conditions, respectively, t(11) = 4.47, SEM = 0.069, p < 0.001, d = 1.32) than under low working memory load (from 0.421 to 0.626, t(11) = 3.57, SEM = 0.057, p < 0.01, d = 1.04). No other effects were significant.

FIGURE 2. Mean proportion correctly detected targets (A) and d-prime scores (B), as a function of target contrast, flanker presence, and working memory load. Error bars represent between-subject standard error of the mean.

FIGURE 3. The interaction between flanker presence and working memory load in mean proportion correctly detected targets (A) and d-prime scores (B). Error bars represent between-subject standard error of the mean. * p < 0.05.

To confirm that the pattern of results in the main analysis did not reflect changes in response bias, d-prime scores were also computed as a function of target contrast, flanker condition and working memory load (see Figure 2B). Data were excluded from the d-prime analysis for one participant, whose overall d-prime score constituted an outlier (mean d-prime 1.82 standard deviations, with one of the conditions over four standard deviations away from the group mean). The d-prime scores showed the same pattern as the detection rates, with a main effect of target contrast, F(1,10) = 19.16, MSe = 3.50, p < 0.001, = 0.657. D-prime scores were greater for high contrast targets (M = 2.08), compared to medium (M = 0.51, t(10) = 3.6, SEM = 0.437, p < 0.01, d = 1.14) and low contrast targets (M = −0.36, t(10) = 5.15, SEM = 0.473, p < 0.001, d = 1.69). D-prime was greater in medium, compared with low contrast targets, t(10) = 3.44, SEM = 0.252, p < 0.01, d = 1.14. D-prime scores were also greater when the flankers were present (M = 1.03) than when they were absent (M = 0.45), although the main effect of flanker condition was not significant (F(1,11) = 2.61, MSe = 4.25, p = 0.138, = 0.207). The main effect of working memory load was not significant (F < 1), but importantly, there was again a significant interaction between working memory load and flanker condition, F(1,10) = 5.44, MSe = 3.09, p < 0.05, = 0.353 (see Figure 3B). Planned follow-up tests (Bonferroni corrected) showed that under high working memory load, there was a significant difference in d-prime between flanker present (M = 1.42) and absent displays (M = 0.13, t(10) = 3.25, SEM = 0.398, p < 0.01, d = 0.98). Under low working memory load, d-prime scores were not significantly affected by the presence of the flankers (flankers present, M = 0.65; flankers absent, M = 0.78; t < 1). No other effects were significant.

DISCUSSION

Target detection accuracy was significantly better in the presence of peripheral collinear flankers, and more so when concurrent load on working memory was high. The enhanced effect of the irrelevant flankers on target detection performance when the working memory task was relatively difficult is in line with two pieces of previous evidence. First, only collinear flankers that receive attentional processing facilitate target visibility (Freeman et al., 2001). Second, the extent of attentional processing of non-target information depends on the availability of working memory during selective attention (Lavie et al., 2004). Together, these prior findings suggest that making working memory unavailable for selection by increasing the load on a working memory task should increase the processing of the collinear flankers, leading to better target detection. This is what we found.

Performance improvements under high working memory load should be obtainable in other flanker tasks, whenever the distracting information is compatible with the current target. Indeed, in a previous study that measured both facilitation and interference effects from irrelevant distractor letters under varying levels of working memory load (Lavie et al., 2004, Experiment 1), both facilitation and interference were increased under high working memory load. However, flanker compatibility effects are more commonly computed by contrasting performance on trials with compatible and incompatible flankers, making it impossible to distinguish facilitation from interference. Moreover, in many response-competition tasks, such as the flanker task and the Stroop task, facilitation and interference are not symmetrical, and interference effects are generally larger than facilitation effects (e.g., Lindsay and Jacoby, 1994). In those paradigms, interference effects may therefore be more likely than facilitation effects to be modulated by working memory load.

There are a few previous demonstrations that performance can benefit in conditions when working memory is relatively unavailable for selective attention. When performing a demanding visual task, such as comparing the sizes of two lines with very similar lengths, an unexpected visual stimulus presented close to fixation often remains undetected (Rock et al., 1992). Such inattentional blindness is reduced (i.e., detection of the unexpected item is better) under high working memory load, presumably because selective attention is less efficiently focused on the relevant lines (De Fockert and Bremner, 2011). Inattentional blindness, however, concerns information other than the hitherto task-relevant lines, and therefore the release from inattentional blindness by working memory load does not involve a change in target performance, like we found in the current study. Other work has found that loading working memory can aid target performance, not because the processing of task-compatible distractors becomes more likely under high load, like we found here, but because high load can lead to a reduction in distractor processing, as long as the content of the working memory task has greater overlap with the distractor than with the target (Kim et al., 2005; Park et al., 2007). The current findings are therefore the first demonstration that loading working memory can facilitate perception following greater attention to irrelevant information.

Previous work has shown that attention can enhance detection of the type of stimulus that was used here (e.g., Cameron et al., 2002; Pestilli and Carrasco, 2005). In the absence of the flankers, we might therefore have expected target detection to be better under low (vs. high) working memory load, assuming that attention would be better focused on the target under low working memory load. Although in the flanker absent conditions, performance was indeed somewhat better under low, compared to high working memory load both in terms of accurate detection (low load, M = 0.42; high load, M = 0.39) and d-prime (low load, M = 0.78; high load, M = 0.13), neither of these differences reached statistical significance. In other words, whereas our manipulation of working memory load had the predicted effect of modulating target detection in the presence of the flankers, high working memory load did not reliably impair target detection when the flankers were absent.

The finding that in the flanker absent conditions, target detection was similar under low and high working memory load is also important in order to eliminate an alternative (yet invalid) explanation of the better target detection in the presence of flankers under high load. Following standard practice in the literature using the combined working memory/selective attention paradigm, working memory load was manipulated in separate blocks. It could therefore be argued that participants were simply more attentive in the high load condition, which in turn led to their better target detection when flankers were present. If this were the case, however, target detection should also have been better under high (vs. low) load in the absence of the flankers, which was clearly not the case: if anything, target detection without flankers was somewhat better under low working memory load, which is in line with recent findings showing that the subjective visibility of a briefly presented number stimulus is reduced as load on a concurrent working memory task increases (De Loof et al., 2013).

The lateral interactions that produce the flanker facilitation effects shown here are thought to occur in primary visual cortex (Gilbert and Wiesel, 1989; Polat et al., 1998), so these findings suggest that working memory may affect the processing of task-irrelevant information at an early stage. Unlike previous demonstrations showing greater distractor processing in a context of high (vs. low) working memory load, where the effect may have been due to greater response competition (e.g., Lavie et al., 2004), the improved target detection observed under high working memory load in the current study is likely to reflect greater perception of the peripheral flankers. Together with other previous findings suggesting that working memory affects the perception of to-be-ignored distractors (Lavie and De Fockert, 2005; De Fockert and Wu, 2009; Kelley and Lavie, 2011), our results show that working memory can affect the prioritization of information at an early stage in visual processing. The finding that load on a working memory task that involved maintaining sets of digits interacted with processing of visual distractors in a simple target detection task suggests that the link between working memory and low-level visual processing is indirect in this case, in that it does not involve content-specific interactions between working memory and perception. Instead, working memory and perceptual processing are more likely to interact in terms of resource demands in this case, such that general working memory resources are required to sustain focused visual selection.

FOOTNOTES

1^http://vision.psy.unipd.it/equipment.htm

REFERENCES

Ahmed, L., and de Fockert, J. W. (2012). Focusing on attention: the effects of working memory capacity and load on selective attention. PLoS One 7:e43101. doi: 10.1371/journal.pone.0043101

Akyürek, E. G., Hommel, B., and Jolicoeur, P. (2007). Direct evidence for a role of working memory in the attentional blink. Mem. Cognit. 35, 621–627. doi: 10.3758/bf03193300

Akyürek, E. G., Leszczyński, M., and Schubö, A. (2010). The temporal locus of the interaction between working memory consolidation and the attentional blink. Psychophysiology 47, 1134–1141. doi: 10.1111/j.1469-8986.2010.01033.x

Awh, E., and Jonides, J. (2001). Overlapping mechanisms of attention and spatial working memory. Trends Cogn. Sci. 5, 119–126. doi: 10.1016/s1364-6613(00)01593-x

Cameron, E. L., Tai, J. C., and Carrasco, M. (2002). Covert attention affects the psychometric function of contrast sensitivity. Vision Res. 42, 949–967. doi: 10.1016/s0042-6989(02)00039-1

Chun, M. M. (2011). Visual working memory as visual attention sustained over time. Neuropsychologia 49, 1407–1409. doi: 10.1016/j.neuropsychologia.2011.01.029

De Fockert, J. W., and Bremner, A. (2011). Release of inattentional blindness by high working memory load: elucidating the relationship between working memory and selective attention. Cognition 121, 400–408. doi: 10.1016/j.cognition.2011.08.016

De Fockert, J. W., Rees, G., Frith, C., and Lavie, N. (2001). The role of working memory load in selective attention. Science 291, 1803–1806. doi: 10.1126/science.1056496

De Fockert, J. W., and Wu, S. (2009). High working memory load leads to more Ebbinghaus illusion. Eur. J. Cogn. Psychol. 21, 961–970. doi: 10.1080/09541440802689302

De Loof, E., Verguts, T., Fias, W., and Van Opstal, F. (2013). Opposite effects of working memory on subjective visibility and priming. J. Exp. Psychol. Learn. Mem. Cogn. 39, 1959–1965. doi: 10.1037/a0033093

Eriksen, B. A., and Eriksen, C. W. (1974). Effects of noise letters on the identification of a target letter in a nonsearch task. Percept. Psychophys. 16, 143–149. doi: 10.3758/bf03203267

Freeman, E., Sagi, D., and Driver, J. (2001). Lateral interactions between targets and flankers in low-level vision depend on attention to the flankers. Nat. Neurosci. 4, 1032–1036. doi: 10.1038/nn728

Gazzaley, A., and Nobre, A. C. (2012). Top-down modulation: bridging selective attention and working memory. Trends Cogn. Sci. 16, 129–135. doi: 10.1016/j.tics.2011.11.014

Gilbert, C. D., and Wiesel, T. N. (1989). Columnar specificity of intrinsic horizontal and corticocortical connections in cat visual cortex. J. Neurosci. 9, 2432–2442.

Kane, M. J., and Engle, R. W. (2003). Working-memory capacity and the control of attention: the contributions of goal neglect, response competition and task set to stroop interference. J. Exp. Psychol. Gen. 132, 47–70. doi: 10.1037/0096-3445.132.1.47

Kelley, T. A., and Lavie, N. (2011). Working memory load modulates distractor competition in primary visual cortex. Cereb. Cortex 21, 659–665. doi: 10.1093/cercor/bhq139

Kim, S.-Y., Kim, M.-S., and Chun, M. M. (2005). Concurrent working memory load can reduce distraction. Proc. Natl. Acad. Sci. U S A 102, 16524–16529. doi: 10.1073/pnas.0505454102

Lavie, N., and De Fockert, J. W. (2005). The role of working memory in attentional capture by feature singletons. Psychon. Bull. Rev. 12, 669–674. doi: 10.3758/bf03196756

Lavie, N., Hirst, A., De Fockert, J. W., and Viding, E. (2004). Load theory of selective attention and cognitive control. J. Exp. Psychol. Gen. 133, 339–354. doi: 10.1037/0096-3445.133.3.339

Lindsay, D. S., and Jacoby, L. L. (1994). Stroop process dissociations: the relationship between facilitation and interference. J. Exp. Psychol. Hum. Percept. Perform. 20, 219–234. doi: 10.1037//0096-1523.20.2.219

Park, S., Kim, M. S., and Chun, M. M. (2007). Concurrent working memory load can facilitate selective attention: evidence for specialized load. J. Exp. Psychol. Hum. Percept. Perform. 33, 1062–1075. doi: 10.1037/0096-1523.33.5.1062

Pecchinenda, A., and Heil, M. (2007). Role of working memory load on selective attention to affectively valent information. Eur. J. Cogn. Psychol. 19, 898–909. doi: 10.1080/09541440601095388

Pestilli, F., and Carrasco, M. (2005). Attention enhances contrast sensitivity at cued and impairs it at uncued locations. Vision Res. 45, 1867–1875. doi: 10.1016/j.visres.2005.01.019

Polat, U., Mizobe, K., Pettet, M., Kasamatsu, T., and Norcia, T. (1998). Collinear stimuli regulate visual responses depending on cell’s contrast threshold. Nature 391, 580–584.

Polat, U., and Sagi, D. (1993). Lateral interactions between spatial channels: suppression and facilitation revealed by lateral masking experiments. Vision Res. 33, 993–999. doi: 10.1016/0042-6989(93)90081-7

Pratt, N., Willoughby, A., and Swick, D. (2011). Effects of working memory load on visual selective attention: behavioral and electrophysiological evidence. Front. Hum. Neurosci. 5:57. doi: 10.3389/fnhum.2011.00057

Rock, I., Linnett, C. M., Grant, P., and Mack, A. (1992). Perception without attention: results of a new method. Cogn. Psychol. 24, 502–534. doi: 10.1016/0010-0285(92)90017-v

Scalf, P. E., Dux, P. E., and Marois, R. (2011). Working memory encoding delays top-down attention to visual cortex. J. Cogn. Neurosci. 23, 2593–2604. doi: 10.1162/jocn.2011.21621

Schneider, W., Eschman, A., and Zuccolotto, A. (2002). E-Prime User’s Guide. Pittsburgh: Psychology Software Tools.

Conflict of Interest Statement: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Received: 10 July 2014; Accepted: 25 September 2014; Published online: 14 October 2014.

Citation: De Fockert JW and Leiser J (2014) Better target detection in the presence of collinear flankers under high working memory load. Front. Hum. Neurosci. 8:821. doi: 10.3389/fnhum.2014.00821

This article was submitted to the journal Frontiers in Human Neuroscience.

Copyright © 2014 De Fockert and Leiser. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

HYPOTHESIS AND THEORY ARTICLE
published: 08 August 2014
doi: 10.3389/fnhum.2014.00588

Symbiosis of executive and selective attention in working memory

André Vandierendonck*

Department of Experimental Psychology, Ghent University, Ghent, Belgium

* Correspondence: André Vandierendonck, Department of Experimental Psychology, Ghent University, Henri Dunantlaan 2, B-9000 Gent, Belgium e-mail: andre.vandierendonck@ugent.be

Edited by:
Jean-Philippe Van Dijck, Ghent University, Belgium

Reviewed by:
Tobias Egner, Duke University, USA
Theodore Zanto, University of California, San Francisco, USA

The notion of working memory (WM) was introduced to account for the usage of short-term memory resources by other cognitive tasks such as reasoning, mental arithmetic, language comprehension, and many others. This collaboration between memory and other cognitive tasks can only be achieved by a dedicated WM system that controls task coordination. To that end, WM models include executive control. Nevertheless, other attention control systems may be involved in coordination of memory and cognitive tasks calling on memory resources. The present paper briefly reviews the evidence concerning the role of selective attention in WM activities. A model is proposed in which selective attention control is directly linked to the executive control part of the WM system. The model assumes that apart from storage of declarative information, the system also includes an executive WM module that represents the current task set. Control processes are automatically triggered when particular conditions in these modules are met. As each task set represents the parameter settings and the actions needed to achieve the task goal, it will depend on the specific settings and actions whether selective attention control will have to be shared among the active tasks. Only when such sharing is required, task performance will be affected by the capacity limits of the control system involved.

Keywords: working memory, selective attention, executive control, executive attention, task-set switching

INTRODUCTION

It seems self-evident that the likelihood of successfully recalling a previously observed sequence of events will be larger when attention was than when it was not selectively oriented towards these events. Nevertheless, theories of working memory (WM) have been focusing on executive attention rather than on other kinds of attention like orienting attention, which is a main constituent of selective attention tasks. Furthermore, the evidence in support of a role for orienting or selective attention in WM has not been settled, and it remains an open question whether selective attention is indeed an indispensable aspect of WM. The presumption that selective attention plays a central role in WM is certainly tempting. The present article scrutinizes this hypothesis by reviewing the behavioral evidence regarding interactions between selective and executive attention in tasks taxing WM. This review will be restricted to studies relevant to the interaction of both kinds of attention and will therefore not be completely representative. In preview, the paper will show that both kinds of attention have a symbiotic relationship within WM.

At this point, it seems appropriate to provide some delineation of the attention categories involved in the present article. The term selective attention typically refers to the kind of processing involved in orienting attention towards a specific set of entities or representations while ignoring others. Such processing may be driven either by more automatic or by more top-down controlled processing (Schneider and Shiffrin, 1977; Shiffrin and Schneider, 1977; Neuman, 1984). Executive attention, in contrast, refers to attention processing related to responding and task execution. Research on selective attention relies on a range of tasks that are known as selective attention tasks. However, rarely ever a task is process-pure and, in fact, many of the selective attention tasks also involve executive attention processes, as will become clear later in this article. In the literature, a range of terms, such as executive control, execution function, and cognitive control has been used interchangeably to refer to the kind of attention involved in control of action. In the present article no strict distinction will be made between these terms.

WORKING MEMORY: EXECUTIVE ATTENTION AND CONTROL

The concept of WM finds its origin in the context of research on short-term memory. Originally, the short-term memory system (STS) was believed to temporarily maintain incoming information before it is transferred to the (episodic) long-term memory system. These ideas were expressed in a range of short-term memory models published in the 1960s and 1970s; the most representative of these models was the so-called “modal model” proposed by Atkinson and Shiffrin (1968). Although in this conception the STS was essentially a (passive) store, it was also endowed with so-called control processes, such as coding and rehearsal. These control processes allow active restructuring of the information with the aim of increasing the memory system’s efficiency, and presumably require attention. Later on, the idea that the STS can also be used to temporarily maintain information in the service of other activities gradually gained ground. If control processes like rehearsal, coding, etc., are useful to improve long-term recall, these processes should also help short-term recall, as in looking up a phone number in the directory to make an immediate call (without the intention of remembering the phone number an hour or a day later). But no doubt, temporary storage of a piece of information can also be useful in other tasks, such as carrying or borrowing in mental arithmetic (e.g., Hitch, 1978; Imbo et al., 2007), or to maintain the premises into focus to reach a conclusion (e.g., Vandierendonck and De Vooght, 1997).

In fact, the consideration that short-term memory storage can be at the service of other tasks required a new approach to the study of temporary memory. For the study of STS, an examination of its performance in free recall was a suitable methodology. In contrast, an investigation of the properties of a memory system that provides services to other tasks, required a methodology in which the limiting capacities of the memory system can be measured in situations requiring different amounts of temporary storage in the service of another task. This resulted in two methodological changes. First, instead of free recall, serial recall became the standard method to measure the limits of the memory system because it is less vulnerable to familiarity-based recollections. Second, in order to examine memory-consumption by another task, a dual-task methodology was required. In such a methodology, typically a memory task is performed concurrently with another cognitive task that allegedly requires memory for its execution. On the basis of these methodological innovations, Baddeley and Hitch (1974) showed that the WM system had to provide flexible storage facilities supervised by a controlling agent; for this supervision they borrowed the notion of central executive from artificial intelligence research.

The framework introduced by Baddeley and Hitch (1974) can best be qualified as a multi-component WM. It originally consisted of two modality-specific slave systems (phonological loop and visuospatial WM) supervised by the central executive. This framework stimulated a large amount of research on WM. Apart from a range of new models on the storage facilities provided by the WM system (e.g., Logie, 1986, 1995; Hulme et al., 1991; Page and Norris, 1998, 2009; Burgess and Hitch, 1999, 2006; Brown et al., 2000), an approach to measure individual differences in WM capacity showed the importance of WM in processes such as reading (Daneman and Carpenter, 1980, 1983), counting (Case et al., 1982), mental arithmetic (Turner and Engle, 1989), and the relation of WM to intelligence (Engle et al., 1999). As research progressed and the number of dual-task studies increased, the call for fractionation of the central executive sounded louder (Baddeley, 1996a,b) and attempts to redefine this agent as a collection of executive functions (Miyake et al., 2000) or to recast the executive in terms of more basic executive processes (e.g., Szmalec et al., 2005; Vandierendonck et al., 2007, 2008) were published.

In all these studies, the role of attention was central. However, of the different attention networks distinguished by Posner (e.g., Posner and Petersen, 1990; Posner and Rothbart, 2007; Petersen and Posner, 2012) only the executive network is included in most present-day WM theories. This is completely consistent with the position that the role of the central executive corresponds with the supervisory attention model of Norman and Shallice (1986), which also basically requires the executive attention network to control actions. The question may be raised whether the WM system also calls on one or more of the other attention networks. In what follows, the utility of broadening the attention scope of theories of WM is further investigated. First, the breadth of the executive attention basis of WM theories is explored by a review of different attention tasks that are modulated by WM capacity. Next, attentional selectivity or orienting attention is considered, by briefly reviewing the evidence. Finally, an attempt is presented to integrate all these findings in a comprehensive view of the attentional basis of WM.

ATTENTION AND WORKING MEMORY

To test whether WM includes particular forms of attention, basically two methodologies can be used. On the one hand, it is possible to use the traditional dual-task method in which two tasks calling on a particular resource are performed concurrently. When performance on either or both tasks is impaired compared to a single-task execution of these tasks, it follows that they are competing for this particular resource. In contrast, when two tasks allegedly tax different resources, concurrent performance of these tasks is not expected to result in performance impairments. Thus, a double dissociation can be established (e.g., Klauer and Zhao, 2004). On the other hand, an individual differences approach can be applied by selecting a group of participants with a high and a group with a low working-memory capacity (often top and bottom 25% of the distribution) as measured by one of the many dedicated WM span tasks (e.g., Daneman and Carpenter, 1980, 1983; Turner and Engle, 1989). If the factor of WM capacity interacts with a difficulty variation on the other task in such a way that the low capacity group’s performance suffers more from the difficulty variation than that of the high capacity group, it follows that the second task requires more WM capacity.

In the present section, the focus is on a range of attention tasks that require orienting towards or selection of particular stimuli that also require participants to ignore irrelevant or previously relevant stimuli. These tasks are known to involve controlled attention. As this term is sometimes used as a synonym for executive control, it seems quite likely that these tasks call on WM or tax some common resources or processes. In what follows, mostly behavioral studies are considered, although occasionally ERP findings are discussed as well. A useful review of electrophysiological studies of the relationship between selective attention and WM can be found in Gazzaley and Nobre (2012).

STROOP TASK

Many attentional tasks require some form of control for their execution. Consider, for example, the Stroop (1935) task. In the standard form of the task, participants are shown words and are requested to name the color of the print. In incongruent trials, color words are presented shown in a color incongruent with the word meaning. Congruent trials consist of words in which the print color and the word name match. Sometimes, also neutral trials are shown in which the print color of a non-color word has to be named. In order to produce a correct answer, the relevant feature (print color) must be selected. In incongruent trials this is difficult because the irrelevant feature (the word meaning) is accessed automatically. The ensuing conflict must be resolved, which leads to slower and more error-prone responding. More particularly, the responses are slower than on congruent and neutral trials. Usually congruent and incongruent trials are mixed, and typically incongruent trials are slower when they are less frequent (MacLeod, 1991).

Several studies have shown that low-span participants show a larger Stroop interference effect (i.e., slower and more error-prone responding to incongruent than to congruent and neutral trials) than the high-span participants (Long and Prat, 2002; Kane and Engle, 2003; Kiefer et al., 2005; Meier and Kane, 2013). This difference is also modulated by the frequency of incongruent trials and the order in which blocks with few and many incongruent trials are presented. This is taken as evidence that high-span subjects are better able to keep the task goal active in WM (Kane and Engle, 2003; Morey et al., 2012).

In a series of experiments, Kim et al. (2005) varied the modality of the WM load. Thus they observed increased interference when the WM load and Stroop task were in the same modality (e.g., both verbal), no interference effect when the WM load was in a modality different from the Stroop task (e.g., verbal Stroop task with visuospatial WM load), and decreased interference when the WM load was in the same modality as the distracter of the Stroop task (e.g., both verbal). Other studies focused on modulation of post-conflict control. A study by Soutschek et al. (2013), for example, shows that a concurrent WM load modulates the post-conflict control. Over three experiments, different types of WM load were applied. When the WM task was an arithmetic updating task or an n-back task, but not when the WM task was a simple load task (recall a number of digits), the interaction of current trial congruency by previous trial congruency, which is a marker of post-conflict adaptation (Botvinick et al., 2001), was modulated by the WM load. In other words, the requirement to update WM contents depletes WM attentional resources to such an extent that it is no longer possible to perform control adjustments after an incongruent Stroop trial; simply maintaining a series of up to six digits does not have this effect.

FLANKER TASK

In the flanker task (Eriksen and Eriksen, 1974), participants are requested to categorize a central stimulus with a left or right key-press, while it is flanked by either compatible or incompatible stimuli. As an example, consider a central stimulus (left or right arrow) flanked by two stimuli on the left and two on the right; the flankers are also arrows, either all left pointing or all right pointing. When the flankers are compatible with the central stimulus (e.g., arrows pointing in the same direction), responses are faster than when the flankers are incompatible (Flanker Compatibility Effect, FCE). When stimulus and flankers are compatible they all favor the same response, but when they are incompatible they favor conflicting responses resulting in a slower response and a larger likelihood of an error. As in the Stroop task, also in the Flanker task, post-conflict adjustment has been observed (Botvinick et al., 2001).

Lavie et al. (2004) showed in a series of experiments that the FCE was more increased under a larger WM load. Pratt et al. (2011) compared flanker performance on an arrow-flanker task under single-task and dual-task conditions while recording early and late attention-sensitive event-related potentials (P1 and P300). In the dual-task condition, a memory load of 4 or 7 items (Sternberg task; Sternberg, 1966) was presented for later recall and during the retention interval a number of flanker trials were presented. The FCE was observed, and it was reduced under both load conditions. P300 amplitude to incompatible trials was also reduced under dual-task conditions. These findings suggest that under WM load it was more difficult to suppress interference from the incompatible flankers. The observation that P1 amplitude was reduced on all dual-task flanker (compatible and incompatible) trials showed that increased WM demands reduce top-down attentional control over early visual processing. A general FCE was also confirmed in a correlational study with structural equation modeling (Keye et al., 2009). This study also tested the role of WM in post-conflict adjustment, but could not confirm this role.

NEGATIVE PRIMING

When the presently relevant stimulus was present but irrelevant on the previous trial, it is said that the present stimulus is negatively primed. This results in a slower response to the relevant stimulus compared to a neutral situation where the stimulus was not present on the previous trial (Tipper, 1985; Tipper and Driver, 1988). Note that negative priming is the opposite of repetition priming where the previous and the current relevant stimulus are the same. Agreement about the mechanism behind negative priming is still lacking, but the competition between representations or processes linked to the previous (ignored event) and the present (relevant) event is part of most accounts. For that reason, it is likely that WM modulates negative priming.

This was confirmed in a study with negative priming in a letter-naming task under a range of conditions that varied the WM load from 0 to 4 words that were presented for later recall (Engle et al., 1995). Under loads 0–2, negative priming was present, but it became gradually smaller and disappeared completely from load 3 on. Because both the negative priming task and the WM load were verbal, it is possible that this result is due to a domain-specific interference. This was tested in another study that included both verbal and visuospatial WM loads (Conway et al., 1999). Two experiments used letter naming to investigate negative priming, combined with a WM load of 0–4 words in the first experiment and visuospatial WM load of 0–4 polygons in the second experiment. In addition, the participants were classified as low or high WM span on the basis of the operation span (OSPAN; Turner and Engle, 1989). Both experiments revealed the presence of negative priming, but this effect was only significant at load 0, irrespective of the type of WM load. It was also expected that the high-span participants would show more negative priming than the low-spans. The rationale for this expectation is that negative priming is the result of coping with interference and that high-span subjects are better able to handle interference. This expectation was also confirmed in the observation that only high-spans showed a negative priming effect at load 0, whereas low-spans showed no negative priming effect at all.

Long and Prat (2002) reported similar findings. One of their experiments concerning color word Stroop stimuli has been mentioned in the section of Stroop Tasks (above). In another experiment, they also used Stroop stimuli and compared neutral trials with incongruent trials, half of which were conflict trials (incongruent color word pairs) and the other half were negative priming trials where the previously ignored word was the color name of the present trial. In high-span participants, no naming latency difference was observed between neutral trials and conflict trials; this confirms the earlier reported findings about the Stroop interference effect. Negative priming (the RT difference between conflict trials and negative priming trials) was very high. In contrast, low spans showed a large Stroop interference effect and a small, but reliable negative priming effect. These findings corroborate earlier findings about Stroop interference and they are consistent with the findings reported by Conway and colleagues regarding negative priming (Engle et al., 1995; Conway et al., 1999).

It should also be noted that in contrast to identity-based negative priming as in all these reviewed studies, location-based negative priming is not affected by either a visuospatial or a verbal WM load (Kahan et al., 2013). Unfortunately, this study is not very convincing because the so-called memory load (of three items) was presented at a rate of 350 ms per item (300 ms on, 50 ms off), and the memory test did not require to recall or to recognize any of the items specifically; instead a judgment of frequency was asked (more even or odd numbers for the verbal load e.g.). The finding that such a load does not remove location-based negative priming does not seem to allow strong statements about the role of WM in location-based priming.

ATTENTIONAL BLINK

The attentional blink refers to an impaired ability to detect a second target during an interval of about 400–600 ms after detecting a first target (Raymond et al., 1992). The typical procedure for detecting the attentional blink consists of a rapid serial visual presentation (RSVP) of letters in which a letter is shown every 100 ms (the exact value varies slightly over studies). One letter is in a different color (e.g., white instead of black) and occurs as the first target (T1); on part of the trials a second target (e.g., X; T2) occurs in the same color as all the other letters and at the end of the series (usually about 20 letters), the question whether T2 was or was not present is to be answered. T2 is presented at various positions after T1. When presented on positions 2–7 after T1, the frequency of detecting T2 decreases; when presented immediately after T1, detection frequency is not impaired. There is a large literature on the attentional blink and many of the task parameters have been varied (rate of presentation, usage of additional targets, visual and verbal memory loads, etc.), sometimes leading to surprising outcomes.

Some studies have used a WM load while performing the RSVP-attentional blink task. The findings of these studies are somewhat variable, but some studies found no variation in the size of the attentional blink effect with increased memory load, although the memory load affected some performance aspects (e.g., Akyürek and Hommel, 2005, 2006). However, when participants had to judge whether T1 is part of the WM load, the attentional blink increased (Akyürek et al., 2007). Other studies measured WM capacity by means of the OSPAN and found that the attentional blink was decreased with higher WM capacity (Arnell et al., 2010) and that this was even the case when Raven’s standard progressive matrices scores were partialed out (Colzato et al., 2007).

STIMULUS-RESPONSE COMPATIBILITY

When stimulus features or dimensions overlap with response features or dimensions, stimulus-response compatibility (S-R compatibility) is bound to occur. Two types of S-R compatibility (see Kornblum et al., 1990) are of primary interest here, namely compatibility due to an overlap between the relevant stimulus and response dimensions (e.g., respond left to a left positioned or left-pointing stimulus) which is also known as S-R compatibility proper, and compatibility due to an overlap between an irrelevant stimulus dimension and the relevant response dimension. The Simon effect (e.g., Simon and Rudell, 1967) is an example of the latter: consider the request to respond with a left key-press to a red circle and to respond right to a green circle, responses will be faster if the red circle is positioned on the left side of the screen compared to when it is positioned on the right. Position on the screen is here irrelevant, but it affects responding. Both types of compatibility require action control, which is one of the typical expressions of executive control. Performance on such S-R compatibility tasks is therefore expected to be related to WM capacity or WM load. A few published studies are relevant to this issue, most of them concern the Simon effect.

There is a lot of variability in the methodologies used in these studies, which makes it difficult to extract a clear pattern of findings. Some studies report no or only a modest effect of a memory load on the Simon effect (Stins et al., 2004; Stürmer et al., 2005), whereas other studies found some effects (Zhao et al., 2010; Wühr and Biebl, 2011). It seems quite likely that the Simon effect is not very susceptible to WM load, especially as it seems rather easy to reverse the Simon effect (Notebaert et al., 2006). It is probably more interesting to follow the logic applied in studies of the Stroop effect and the FCE, and to look at conflict adaptation. Weldon et al. (2013) measured WM capacity in a Simon experiment. WM capacity was not related to performance on the Simon task, but a measure of the magnitude of the trial-by-trial conflict adaptation correlated negatively with WM capacity for low-span and near 0 for high-span participants.

INTERIM CONCLUSION

In this section, attention tasks were considered that involve both selection and control. A common theme among these tasks and the way they are performed is that in the selection of the relevant stimulus feature and consequently in performing the correct response, some form of conflict or competition between processes occurs that may cause erroneous and/or delayed responses. This is the case for the Stroop interference effect, the flanker compatibility effect, and the Simon effect. Incongruent or incompatible trials in each of these are based on a competition between irrelevant and relevant stimulus features or dimensions. In a particular respect, negative priming is similar, because a previously irrelevant stimulus becomes now relevant and as a consequence the action coupled to the stimulus has to be changed, creating a conflict between the old and the new action link. Only the attentional blink seems to be different, but it may be too early to draw conclusions on the underlying processes. In all these cases, attentional control is needed, and the evidence shows that the observed interference and its control (sometimes in a trial-by-trial conflict adjustment) is modulated by the individual’s amount of WM capacity and that increases of WM load modulate the observed effects. As a temporary conclusion, it may be said that all these forms of attention are mediated by WM or are at least calling on processes that are shared with WM.

WORKING MEMORY AND VISUAL SEARCH

In the present section, the focus is on selective attention tasks of which it is not clear that they involve executive attention. In particular, some forms of perceptual selectivity will be considered, such as attentional capture, visual (perceptual) search and environmental monitoring. In all these tasks, participants are given the instruction to search for a particular target. Usually this target is only briefly presented or described before the start of the search; therefore, it must be assumed that the searched-for object is active in WM.

ATTENTIONAL CAPTURE

Sometimes particular events stand out and capture attention so-to-speak automatically. For example, a single poppy in a lawn will be noticed immediately. Hence, searching for a singleton (stimulus with unique features) is rather easy, such as finding a red circle among green circles and squares. However, if the object of the search is to find the green square among green circles and one single red circle (irrelevant singleton), finding the target object may be hampered by the presence of the irrelevant singleton. The question is now considered whether such searches are mediated by WM.

Lavie and de Fockert (2005) used a search task where nine figures (circles and diamonds) were arranged in a circular lay out. All the figures were shown in red on a black background, except for an irrelevant color singleton (green circle) that was present on some trials. Each figure contained either a horizontal or a tilted line. The stimuli were presented for 200 ms and the requirement was to find the red circle among the red diamonds (and occasional green circle) and to decide on the orientation of the line. This task was performed either alone or in a dual-task situation with a verbal WM load (six digits). Search was slowed by the presence of the singleton, and this effect was augmented under load. This observation was further corroborated in an event-related fMRI study that showed that the presence of the singleton was associated with higher superior parietal activation (in line with a capture account) and higher frontal activity (Lavie and de Fockert, 2006). Behavioral singleton interference correlated with the frontal activity, and singleton interference was also higher under WM load.

Further specifications of the relationship between WM and attentional capture come from studies that used other types of WM load. One study used only a visual WM load and confirmed the finding that the presence of the WM load increased the singleton interference (Olivers et al., 2006). This study also observed that the effect was stronger when the irrelevant singleton overlapped with the WM load but only when difficult to verbalize pictures were used. A study of Burnham et al. (2014) explicitly tested the role of different WM components in attentional capture. They found that only tasks that tax visuospatial WM and executive control increased distracter interference, while a phonological WM load did not affect capture.

VISUAL SEARCH

It thus seems that in at least some visual search tasks, a WM load modulates performance. Can these observations be generalized to other types of visual or perceptual search? A study by Kane et al. (2006) examined the relationship between WM capacity and visual search. In three experiments the investigators selected high and low complex span participants to perform a series of visual search tasks in which speed of response and search errors were registered. In the first experiment, participants searched a letter F among distracters (efficient search with letters O as distracter; inefficient search with letters E as distracter); set size, presence of the target, and organization of the layout were varied. Although strong and reliable search RT slopes were observed, WM span did not affect performance. In the second experiment, a similar design was used in which participants searched either for a red vertical line among green vertical lines and red and green horizontal lines (conjunction search) or for an F among E and 90° tilted T’s (spatial configuration search) in different set sizes. Again, reliable RT slopes were observed, but WM span did not affect or modulate the findings. In the final experiment, participants had to detect an F (regular or mirrored) among regular and mirrored E’s and T’s tilted 90° forward or backward. All the symbols were presented on three concentric rings at eight equally spread positions over the rings. Search was to be performed in two different ways. In one condition, the participants were requested to perform a search of the middle ring starting at the top (12 o’clock) position and following the positions clockwise until they found the first F (regular or mirrored; there could be more than one F). In the other condition, search was not constrained but was aimed to find the F on the middle ring (there was only one F on this ring). The constrained search (command search) was used because it had been reported that such search requires volition and is much slower than standard search (Wolfe et al., 2000). The command search was indeed slower than the unconstrained search, but again WM span did not affect neither modulate performance. These findings led Kane et al. (2006) to conclude that the executive control function of WM does not “generalize to difficult attention tasks lacking the need to actively maintain goals to restrain prepotent responses or constrain attentional focus to particular stimuli or locations in space amid distractors” (p. 771).

Nevertheless, it is important not to overstate the scope of these findings. Indeed, the already reported attentional capture studies show that search performance is affected by a WM load, in a task that is not dramatically different from the ones used by Kane et al. (2006). These authors themselves, moreover, remark that several dual-task studies did report effects of WM load on visual attention tasks (e.g., Woodman and Luck, 2004). Anderson et al. (2008) report a dual-task study of efficient and inefficient search performance. Participants were presented 4 or 10 randomly rotated L’s in a circular arrangement with the request to decide whether a target X (efficient search) or T (inefficient search) was present or not. The search task was performed in isolation or within the retention interval of a WM task with either a low (3 items) or a high load (5 items). Inefficient search, but not efficient search, was affected by the size of the memory load. This was the case for a spatial WM load as well as for a verbal WM load. These findings clearly show that at least inefficient visual search calls on domain-general WM resources. Given that this study used a task that is quite similar to one of the tasks used by Kane et al. the possibility that the correlational methodology used by these authors may be less sensitive to detecting WM modulation in visual search.

Findings like these may strengthen the impression that the methodology used (correlational or dual-task) plays an important role. No doubt, there are important differences between these methodologies (e.g., Logie, 2011), and the possibility that the correlational methodology used by these authors may be less sensitive to detecting WM modulation in visual search should not be rejected on a priori grounds. Yet, small changes to the design may result in different findings. Sobel et al. (2007) made some changes to the conjunction search task used by Kane et al. (2006) in order to allow a distinction between bottom-up and top-down search mechanisms. They found that searches based on bottom-up processes were not related to WM capacity, but searches based on top-down processes were performed better by high-span than by low-span participants.

That small changes to the design may indeed affect the results was also shown in a more recent study of Poole and Kane (2009). They presented target location cues for 1–8 target positions either followed by a long (1500 ms) or a short (30 ms) interval before the (inefficient) search display was shown. They found that high-span participants identified targets (F or mirrored F) faster than low-span subjects, but only when distracters were present on non-target positions, and only with long cue-stimulus intervals. Thus it seems that individual differences in visual search performance are only related to individual differences in WM capacity when it is necessary to maintain the search focus over a longer period and when distracters at non-focused positions are present.

INPUT MONITORING

Another aspect of search behavior is found in situations where the environment is monitored or scanned for the occurrence of a particular event, this is also known as input monitoring. On the basis of a conceptual analysis, Vandierendonck (2000a,b) proposed that input monitoring could be one of the more basic processes underlying executive control. In order to test the role of input monitoring, it was assumed that events occurring randomly distributed over time required more input monitoring effort than events occurring in a fixed time schedule. The rationale for this was that a fixed time schedule may be handled by automatic processes, while for randomly occurring events the monitoring process must be continuously adapted. Deschuyteneer and Vandierendonck (2005) investigated mental arithmetic performance (simple sums) while concurrently and continuously another task had to be performed that varied the degree of input monitoring and the involvement of response selection. These two variations were crossed. The secondary task consisted of high or low tones that were presented at a fixed tempo (1 tone every 1200 ms) or in an unpredictable tempo (random alternation of 900 and 1500 ms). Each tone required a response. In the simple response condition, one single response was to be emitted as soon as a tone was presented; in the response selection condition, low and high tones were responded to each with their associated response. The answer time to the arithmetic sums was slowed when a response selection was required compared to the requirement to produce a simple response. In contrast, answer times were not significantly different for fixed and random schedules of tone presentation, indicating that input monitoring is not part of the attentional resources required to execute the arithmetic sums. As it has been shown before that such sums call on WM (Hitch, 1978; Lemaire et al., 1996) and more specifically, on the executive control system (De Rammelaere et al., 1999, 2001; De Rammelaere and Vandierendonck, 2001; Imbo et al., 2005), these findings do not corroborate the hypothesis that input monitoring is part of executive control. In a similar study with calculation of arithmetic products as the primary task, these findings were confirmed: concurrent response selection but not concurrent input monitoring affected performance on the arithmetic task.

The hypothesis that input monitoring is part of executive control was also tested with saccades as primary task. Several studies have shown that anti-saccades (eye-movements away from a peripheral stimulus) but not pro-saccades (eye-movements towards a peripheral stimulus) call on WM’s executive system (e.g., Roberts et al., 1994; Stuyven et al., 2000; Kane et al., 2001). Vandierendonck et al. (2008) compared pro-saccade and anti-saccade execution either in a single-task condition or in a dual-task condition with a concurrent and continuous tone response task. There were four dual-task conditions resulting from orthogonal parametric variations in input monitoring and response selection (fixed vs. random tone intervals and simple or choice reaction task). Both pro- and anti-saccades suffered from a non-specific dual-task cost, but more interestingly, neither input monitoring nor response selection played any role in pro-saccades which are generally believed to be triggered automatically (Hallett, 1978; Kristjánsson et al., 2001), whereas anti-saccades were not only slower when response selection was required in the tone response task, but also when the spacing of the tones was random rather than fixed. The latter finding supports the hypothesis that input monitoring is part of the attentional control loop. It may be the case, though, that input monitoring overlaps more with eye-movement control than with executive control deployed in mental calculation.

Summarizing the results on input monitoring, it appears that input monitoring calls on executive attention when controlled saccades but not when automatic saccades have to be performed. However, arithmetic performance (simple sums and products) does not seem to be disturbed by an increased demand to monitor input. Note however, that these studies tested executive control without imposing a WM load. The present evidence therefore remains indirect and evidence directly involving WM operations is needed for a more solid support for the role of input monitoring in the attentional subsystem of WM.

WHAT LINKS SELECTIVE ATTENTION TO EXECUTIVE CONTROL?

In balance, the evidence reviewed in the previous sections shows that in many cases selective attention tasks call on working memory, in particular on its executive attention control mechanism. However, in a number of situations selective attention operates without any executive demands (e.g., attentional capture, efficient visual search, …). The question that must be asked then is how working memory theory can account for these differences. In this section, the view is defended that selective attention taxes executive control depending on the characteristics of the task being executed. More specifically, the thesis will be developed that when a task (i.e., a goal-directed activity) is performed, a means-end representation is instantiated[1] in working memory, also known as a task set, and when this representation includes attention selectivity as a means to achieve the task goal, and only then, task executive will call on executive demands. The view defended by Kim et al. (2005) with pools of modality-specific resources selectively contributing to content-based interference is not in contradiction with the present development as both views continue to build on Baddeley (1986) view on working memory.

The motivation for developing this view is the consideration that executive control as defined in some WM models (e.g., Baddeley and Hitch, 1974) still has the characteristics of a homunculus, notwithstanding the efforts that have been made to fractionate the central executive (Baddeley, 1996a,b, 2000; Miyake et al., 2000; Vandierendonck et al., 2007). The view developed here tries to specify the executive control processes in such a way that these processes are triggered whenever the appropriate conditions are met, so that no other supervisory control system is needed to overview proper application of these processes.

Taking into account that a range of studies did not find any effects of a WM load on task switching (e.g., Logan, 2004, 2006; Kane et al., 2007; Kiesel et al., 2007), while there is evidence showing that task switching calls on executive control processes (Goschke, 2000; Miyake et al., 2000; Baddeley et al., 2001; Friedman et al., 2008; Liefooghe et al., 2008), it seems that task and task-set representation are not competing with memory tasks for storage, but are competing for control processes. Yet, execution of a task requires a task set that remains active until the task is finished or abolished. According to some authors, whenever a task set becomes active, it is retrieved from long-term memory, and maintained into an active state in WM (e.g., Mayr and Kliegl, 2000). However, if the task set occupies WM storage space, it is expected that an increased WM load would impair task-switching performance and would do so even more when the task sets are more complex. Neither of these seems to be the case. Hence, if the task set is maintained in WM without affecting task-switching performance, it must be assumed that the task set is maintained separately from regular WM contents. A possible solution is to assume that the task set is maintained in a dedicated WM system for execution-related information, an executive working memory module (eWM), whereas regular WM storage is maintained in a kind of declarative WM module (dWM; Oberauer, 2009; Vandierendonck, 2012) or an episodic buffer (Baddeley, 2000) linking phonological and visuospatial representations to each other and to their long-term memory representations.

Before executing a new task, the intention to do so is adopted. This entails activation of a goal representation in WM and the retrieval of the task set from long-term memory. The goal representation is established by instantiating a label (e.g., the task name) referring to the goal or the task in dWM, and the task set is retrieved from long-term memory and configured in eWM. There are several reasons for making the assumption that dWM contains a reference to the goal. First, research has shown that other information present in dWM has an effect on task switching performance; more particularly, supportive information enhances performance while distracting information impairs performance in a task switching context (Goschke, 2000; Arrington, 2008). It may be argued that is at variance with findings that a WM load does not impair task-switching performance. However, in the context of task execution (also a memory task), a goal is always present and would thus always be part of dWM. Second, procedural knowledge that matches the contents of dWM, including the goal label, will be selected for execution in order to achieve the goal.

The task set is loaded in eWM. A task set is a collection of task-execution parameters that specify and constrain the actions that can be taken to achieve the task goal (cf. Logan and Gordon, 2001). For the rather simple cognitive tasks as the ones considered here (situations requiring solving of a new problem or solving of a complex problem are not considered here, because in such cases it cannot be assumed that long-term memory contains a complete task set as the means to attain the goal are not yet known), the task set will contain a representation of actions that lead to goal achievement. Other parameters that may be set during the instantiation of the task set include orientation of attention towards the relevant stimulus sets, setting a response threshold (determining speed/accuracy trade-off), maybe also a response bias, response modality, etc. (see also Vandierendonck et al., 2010). In this context, it is important to note that every intentional activity has a goal (represented in dWM) and a task set (stored in eWM). This implies that an intention to memorize some events for usage or recall some time later has also such representations.

Instead of presenting here a complete representation of the model (see Vandierendonck, 2012, for a more extensive description), a few examples will be elaborated to clarify the operation of the model. First, the operation of the model is described in performing a simple short-term memory task with immediate serial recall. Next, the model activity is described in performing a similar memory task with delayed recall, where the retention interval is filled with the execution of another cognitive task, as in dual-task situations as used in complex WM span tasks (Daneman and Carpenter, 1980; Turner and Engle, 1989) or in experiments parameterizing the amount of cognitive load (e.g., Barrouillet et al., 2004). Finally, it will be explained how the model does or does not call on selective attentional processing.

IMMEDIATE SERIAL RECALL

As recall is intended, a task goal for intentionally encoding and maintaining the sequence of events is instantiated in dWM, and the corresponding task set is configured in eWM. This is a rather simple task set, as there are no stimulus categorization rules or response categories. Task-set parameters will be needed to specify which subset of stimuli must be encoded, whether to use verbal rehearsal, or to use memory refreshing, whether chunking must be attempted and how big the chunks should be. Once the task set is configured, an automatic process continuously checks contents of dWM and eWM; when a content matches a condition-action rule in procedural LTM or one of the active task-set rules in eWM, the rule may be activated and the action specified in the rule is executed. Because this process continuously checks WM contents for execution of production rules, it is called the executive loop (a similar mechanism plays in production-rule models such as adaptive character of thought (ACT); Anderson and Lebiere, 1998). Each time a new memorandum is presented (e.g., a consonant), it will be encoded in dWM and a rehearsal or refreshment process will be applied to the new and to the previously presented memoranda. If necessary and suitable, chunking of memoranda in dWM may be attempted. This will create a new chunk, a new entity in long-term memory that contains the elements in their recorded order (which occasionally may be incorrect), or retrieve an already existing chunk from long-term memory. An instantiation of the chunk is added to dWM replacing the constituting elements. The executive loop continues until the recall signal occurs (or until another goal takes over). From this time on, the task goal changes from encode-and-maintain to maintain-and-recall. The goal representation in dWM is updated and the new task set is stored in eWM. The executive loop continues to run, but now the conditions have changed such that attention is no longer applied to incoming stimuli, but instead a retrieval loop searches dWM for the oldest element from the episode. When the oldest event is found and it is a chunk, an unpacking process is started that finds the oldest element within the chunk. Thus the stored elements are retrieved one by one and sent to the speech production process. It should be noted that due to the capacity limitation inherent to dWM, the number of memoranda that can be recalled in the correct order is limited to about 4–5 when no chunking is invoked; with chunking the number can be increased and may result in an average of 7–9 items (3–4 chunks).

SERIAL RECALL WITH FILLED RETENTION INTERVAL

Now a more complex task involving dual tasking is considered. In this example, a series of letters is presented for serial recall after a short retention interval at the end of presentation. During this retention interval, another task, namely parity judgment of digits must be executed: a series of digits are presented according to a paced schedule and for each digit a fast manual response is required categorizing the digit as odd (left key press) or even (right key press). During, the letter presentation part, basically the same series of events are taking place as in the previous example (encoding, rehearsing/refreshing, chunking the presented consonants), but because of the filled delay interval, at the end of presentation of the memoranda, a stimulus is presented that announces the requirement to perform another task. Up to that point, the encode-and-maintain goal is active. Next, dual tasking becomes necessary, which implies that two goals must be served, namely a goal to maintain the sequence of consonants (maintenance) and a goal to perform the instructed task (e.g., parity judgment) in response to the stimulus or stimuli (digits) presented during this retention period. What actually happens according to the model is that the presentation of a digit will (re)activate the parity goal and the task set for categorizing the digits as odd (left key-press) or even (right key-press). Other parameters that can be set at task configuration may depend on the specific instructions given at the start of the experiment (e.g., response threshold on the basis of requirements for speed and/or accuracy, the output modality, such as which hands and/or fingers to use, etc.). Each time a digit is presented (e.g., 4) it is instantiated in dWM. The executive loop will allow activation of the association “4-even”; this will result in adding “even” to dWM. Next, the rule “even-right” will fire, adding “right” to dWM. All these dWM events (“parity”, “4”, “even”, “right”) will be bound and result in application of the corresponding task-set rule, which results in producing a right key-press. The binding is now released and the elements (“parity”, “4”, “even”, “right”) are no longer needed and lose their activation. The parity goal is attained, and control shifts to the other goal (maintenance). This takes some time (goal-switching cost), but once the maintenance goal and task set are reactivated, the stored sequence is being rehearsed or refreshed. This goes until a next digit is presented which triggers reactivation of the parity goal and task set, or until the recall signal occurs. At recall, the same processes do occur as described in the immediate recall example.

Because the intervening activity uses storage facilities of dWM, on average, retrieval of the sequence of memoranda will become less efficient. If no chunking is attempted and hence only dWM is used for storage, this will most likely be the case. However, to the extent efforts were made during presentation of the memoranda to chunk them and outsource some of the storage to long-term memory, the bigger the likelihood that recall will not suffer much. Because similar series of memoranda and task executions follow each other, traces of previous task executions will keep some activity level in WM so as to create interference between present and past relevant elements. To the extent that such interference occurs, performance decrements will most likely occur. No doubt, occasionally such interference may also affect goal representations, leading to a more important performance failure.

WHEN DOES SELECTIVE ATTENTION ENTER THE PICTURE?

Orientation of attention is encoded within the task set; it is one of the task-execution parameters/constraints. For example in configuring a task set for encoding and maintenance of information in an experimental context, the instructions usually specify where in the environment (e.g., center of the screen) which kind of memoranda (e.g., words, letters, etc.) will be presented. These instructions constitute ways to constrain the task set, which is encoded in the form of parameter settings. In other words, an intentional memorization task in principle always requires selective attention such as to encode the stimuli or stimulus aspects that must remembered and to ignore other ones. Therefore, if the participants comply with the instructions in the experiment, a WM load always includes selective attention.

Consider a few examples of a dual-task context. Suppose that there is a WM load and that during the retention interval a second task must be performed, for example, a visual search situation as was used by Anderson et al. (2008): during a WM load, either an efficient or an inefficient visual search must be performed. As already explained, a serial WM load engages selective attention in the memorization task set. The secondary task involves visual search. This requires a task set that also engages selective attention. Now, the search occurs in two variations. With efficient search (find X among L distracters), the X is so different from the distracters that it pops out. In fact, although the participant is prepared to selectively attend, the target is found automatically without effort and without in fact engaging controlled attention. In contrast, with inefficient search (find T among L), selective attention is needed and each element has to be checked whether it fits the target description. This involves control of the order in which positions are scanned and checking whether the target is present at that position. In comparison to efficient search, the action has to be controlled, and this also takes longer. This leaves less time to revert to the memory maintenance task to refresh the stored memoranda. For several reasons, then, it is to be expected that inefficient search interferes with WM whereas efficient search does not.

As another example, take a dual-task situation in which one task is intended to tax executive control, with less or more selective attention, but without memory load, and the other task involves less or more selective attention. In the review above, one such experiment compared performance on pro-saccade (eye movement towards a stimulus in peripheral view) and anti-saccade (eye movement away from peripheral stimulus) tasks, under a range of conditions meant to tax executive control processes (Vandierendonck et al., 2008). One contrast of interest is that between a task where an auditory stimulus either occurs at fixed intervals or occurs at variable intervals. This difference corresponds to situations with respectively low and high amounts of input monitoring. When the sounds occur at fixed intervals, a repetitive response can be programmed that more or less matches the stimulus rate. In contrast, when the sounds occur at random intervals, this is not possible and the aural input has to be scanned for the occurrence of a sound to which a response can then be programmed. In the latter case, the task set will engage controlled selective attention. However, only when the other task also engages controlled attention, impairment is expected to occur. In other words, with a pro-saccade task which is known to be performed automatically without any need for control processes (Hallett, 1978; Kristjánsson et al., 2001), the requirement for less or more input monitoring will make no difference, but in combination with an anti-saccade task, the presence of a high level of input monitoring creates a situations in which two tasks compete for the same resource, and it is expected that a higher level of input monitoring will make a difference. This is exactly what was observed in this study.

POTENTIAL LIMITATIONS

It can be argued that the view expressed in this modeling reduces executive control to one single executive function, namely task-set shifting. The model indeed assumes that the relevant task set is represented in a dedicated part of the working memory system (eWM) and that this representation together with the contents of dWM triggers the processes that result in goal achievement. Does this mean that processes engaged by other executive functions such as updating, inhibition, and others (Burgess, 1997; Rabbitt, 1997; Miyake et al., 2000) are excluded by the present view? They aren’t. For example, it is not at all clear that updating really is an executive function. Some authors have claimed that updating is not itself an executive function, but is rather a task demand, i.e., a requirement imposed by the task to continuously keep memory contents up-to-date (e.g., Szmalec et al., 2011). If memory updating is in fact a task demand, there is no doubt that a task-set representation can serve all what is needed. It seems evident that when the task is changed, completed or abolished, the related WM contents are no longer maintained in dWM. Besides, if the task set itself is no longer needed it will also be released from eWM. In other words, task changes result in an updating of the memory contents. A similar argument can be made for the executive function of inhibition. When particular memory contents are not useful to task execution, there is no task set that supports these contents and if they conflict with task execution, an inhibition process will be applied. Rather than defending a view based on bundles of processes as expressed in executive functions that themselves may easily develop into ill-defined agents or even homunculi, the present view attributes control to processes that are triggered when particular conditions are met, such as the presence of particular contents in dWM, the presence of a particular task set in eWM, and a knowledge base in (procedural) long-term memory that contains the appropriate rules that connect the conditions to actions or processes.

SIMILARITIES TO OTHER MODELING ATTEMPTS

The model presented here is not a completely unique effort. Building on the multicomponent WM model of Baddeley and et al. (e.g., Baddeley and Hitch, 1974; Baddeley, 2000), it borrowed the production-rule logic as used in the ACT model (Anderson and Lebiere, 1998). Like Baddeley’s episodic buffer, the declarative WM module’s function is not only concerned with maintaining information in an active state, it is also needed for binding some of the contents. The present modeling was also influenced by Barrouillet’s time-based resource sharing model (Barrouillet et al., 2004, 2007). Barrouillet’s model attributes impaired recall in dual-task situations to the fact that the central attentional resource has to be time-shared between memory refreshments and task execution. This sharing involves rapid switching of attention from memory to task and vice versa. In the present model, the dominant task set determines which activity or process can be deployed (e.g., memory refreshment vs. parity judgment, e.g.), and the longer the time spent on executing the parity task, the less opportunity is left over for memory refreshment. One difference with Barrouillet’s model is that the present model does not assume rapid switching, but rather assumes that there is a cost associated with switching between memory refreshment and execution of another task.

The distinction between declarative and executive WM modules is reminiscent of Oberauer (2009, 2010) distinction between declarative and procedural WM. There are a few important differences however between Oberauer’s procedural WM (pWM) module and the executive WM module in the present model. Whereas pWM is considered to be activated procedural LTM, and hence essentially contains one or more stimulus-response mappings, eWM is not the activated part of procedural LTM, but is instead an autonomous module containing task set information, including parameters specifying task execution. While in Oberauer’s view, dWM and pWM have the same structure and perform the same kind of actions be it on different contents, in the present view, there is no analogy in the operation of dWM and eWM; each module has its own function but their contents enter a collaborative process to achieve the present goal.

GENERAL DISCUSSION

The evidence regarding a role for WM in selective attention was reviewed and showed that very often, but not always, a WM load impaired performance in selective attention tasks. Similarly, correlational studies showed a relationship between individual WM capacity and selective attention performance in most but not all selective attention tasks. A model was presented that provides a basis for distinguishing selective attention tasks that do and those that do not interfere with WM. Basically, this model accounts for this difference by assuming that interference is bound to occur when both tasks (the WM task and the attention task) are prepared for attentional selectivity on condition the selective attention does not occur automatically.

While it would certainly be interesting to know whether WM (and its executive control mechanism) interact with other attention contexts than selective attention, the present paper did not discuss the interactions of executive control with the alerting network (Posner and Petersen, 1990). One reason for not following up this link concerns the difficulty to design experiments that are not contaminated by the presence of arousal, because the alerting network is intricately linked to arousal phenomena, while executive control in a dual-task context calls on effort which is also intricately linked to arousal.

Another attention mechanism that recently has gained interest concerns self-control (Petersen and Posner, 2012). An important part of research in recent years has indeed been concerned with aspects of self-control, volition, and the like. As shown by some studies on task switching, the WM system may provide an excellent substrate for self-control in the form of the phonological loop (Goschke, 2000; Baddeley et al., 2001; Emerson and Miyake, 2003; Miyake et al., 2004; Bryck and Mayr, 2005; Demanet et al., 2010; Vandierendonck et al., 2012). That there are links between executive control and self-control is beyond doubt, but should be the subject of a separate study.

FOOTNOTES

1^Although it is often assumed that working memory consists of activated long-term memory (e.g., Mayr and Kliegl, 2003; Oberauer, 2009), others conceive WM as a temporary working space that is linked to long-term memory but separate from it. In line with the latter conception, elements stored in WM are not simply activated; the present model assumes that an instance is encoded in WM that combines perceived and retrieved features from LTM into a single trace.

REFERENCES

Akyürek, E. G., and Hommel, B. (2005). Short-term memory and the attentional blink: capacity versus content. Mem. Cognit. 33, 654–663. doi: 10.3758/bf03195332

Akyürek, E. G., and Hommel, B. (2006). Memory operations in rapid serial visual presentation. Eur. J. Cogn. Psychol. 18, 520–536. doi: 10.1080/09541440500423160

Akyürek, E. G., Hommel, B., and Jolicoeur, P. (2007). Direct evidence for a role of working memory in the attentional blink. Mem. Cognit. 35, 621–627. doi: 10.3758/bf03193300

Anderson, E. J., Mannan, S. K., Rees, G., Sumner, P., and Kennard, C. (2008). A role for spatial and nonspatial working memory processes in visual search. Exp. Psychol. 55, 301–312. doi: 10.1027/1618-3169.55.5.301

Anderson, J. R., and Lebiere, C. (1998). The Atomic Components of Thought. New York: Lawrence Erlbaum Associates.

Arnell, K. M., Stokes, K. A., MacLean, M. H., and Gicante, C. (2010). Executive control processes of working memory predict attentional blink magnitude over and above storage capacity. Psychol. Res. 74, 1–11. doi: 10.1007/s00426-008-0200-4

Arrington, C. M. (2008). The effect of stimulus availability on task choice in voluntary task switching. Mem. Cognit. 36, 991–997. doi: 10.3758/mc.36.5.991

Atkinson, R. C., and Shiffrin, R. M. (1968). “Human memory: a proposed system and its control processes,” in The Psychology of Learning and Motivation (Vol. 2), eds K. W. Spence and J. T. Spence (New York: Academic Press), 89–195.

Baddeley, A. (1986). Working Memory. Oxford: Oxford University Press.

Baddeley, A. (1996a). Exploring the central executive. Q. J. Exp. Psychol. 49A, 5–28. doi: 10.1080/713755608

Baddeley, A. (1996b). The fractionation of working memory. Proc. Natl. Acad. Sci. U S A 93, 13468–13472. doi: 10.1073/pnas.93.24.13468

Baddeley, A. (2000). The episodic buffer: a new component of working memory? Trends Cogn. Sci. 4, 417–423. doi: 10.1016/s1364-6613(00)01538-2

Baddeley, A., Chincotta, D., and Adlam, A. (2001). Working memory and the control of action: evidence from task switching. J. Exp. Psychol. Gen. 130, 641–657. doi: 10.1037/0096-3445.130.4.641

Baddeley, A. D., and Hitch, G. (1974). “Working memory,” in The Psychology of Learning and Motivation (Vol. 8), ed G. H. Bower (New York: Academic Press), 47–89.

Barrouillet, P., Bemardin, S., Portrat, S., Vergauwe, E., and Camos, V. (2007). Time and cognitive load in working memory. J. Exp. Psychol. Learn. Mem. Cogn. 33, 570–585. doi: 10.1037/0278-7393.33.3.570

Barrouillet, P., Bernardin, S., and Camos, V. (2004). Time constraints and resource sharing in adults’ working memory spans. J. Exp. Psychol. Gen. 133, 83–100. doi: 10.1037/0096-3445.133.1.83

Botvinick, M. M., Braver, T. S., Barch, D. M., Carter, C. S., and Cohen, J. D. (2001). Conflict monitoring and cognitive control. Psychol. Rev. 108, 624–652. doi: 10.1037/0033-295X.108.3.624

Brown, G. D. A., Preece, T., and Hulme, C. (2000). Oscillator-based memory for serial order. Psychol. Rev. 107, 127–181. doi: 10.1037/0033-295x.107.1.127

Bryck, R. L., and Mayr, U. (2005). On the role of verbalization during task set selection: switching or serial order control? Mem. Cognit. 33, 611–623. doi: 10.3758/bf03195328

Burgess, N., and Hitch, G. J. (1999). Memory for serial order: a network model of the phonological loop and its timing. Psychol. Rev. 106, 551–581. doi: 10.1037/0033-295x.106.3.551

Burgess, N., and Hitch, G. J. (2006). A revised model of short-term memory and long-term learning of verbal sequences. J. Mem. Lang. 55, 627–652. doi: 10.1016/j.jml.2006.08.005

Burgess, P. W. (1997). “Theory and methodology in executive function research,” in Methodology of Frontal and Executive Function, ed P. Rabbitt (Hove: Psychology Press), 81–116.

Burnham, B. R., Sabia, M., and Langan, C. (2014). Components of working memory and visual selective attention. J. Exp. Psychol. Hum. Percept. Perform. 40, 391–403. doi: 10.1037/a0033753

Case, R., Kurland, D. M., and Goldberg, J. (1982). Operational efficiency and the growth of short-term-memory span. J. Exp. Child Psychol. 33, 386–404. doi: 10.1016/0022-0965(82)90054-6

Colzato, L. S., Spape, M. M. A., Pannebakker, M. M., and Hommel, B. (2007). Working memory and the attentional blink: blink size is predicted by individual differences in operation span. Psychon. Bull. Rev. 14, 1051–1057. doi: 10.3758/bf03193090

Conway, A. R. A., Tuholski, S. W., Shisler, R. J., and Engle, R. W. (1999). The effect of memory load on negative priming: an individual differences investigation. Mem. Cognit. 27, 1042–1050. doi: 10.3758/bf03201233

Daneman, M., and Carpenter, P. A. (1980). Individual differences in working memory. J. Verbal Learn. Verbal Behav. 19, 450–466. doi: 10.1016/S0022-5371(80)90312-6

Daneman, M., and Carpenter, P. A. (1983). Individual differences in integrating information between and within sentences. J. Exp. Psychol. Learn. Mem. Cogn. 9, 561–584. doi: 10.1037/0278-7393.9.4.561

De Rammelaere, S., and Vandierendonck, A. (2001). Are executive processes used to solve simple mental arithmetic production tasks? Curr. Psychol. Lett. 2, 79–89.

De Rammelaere, S., Stuyven, E., and Vandierendonck, A. (1999). The contribution of working memory resources in the verification of simple mental arithmetic sums. Psychol. Res. 62, 72–77. doi: 10.1007/s004260050041

De Rammelaere, S., Stuyven, E., and Vandierendonck, A. (2001). Verifying simple arithmetic sums and products: are the phonological loop and the central executive involved? Mem. Cognit. 29, 267–273. doi: 10.3758/bf03194920

Demanet, J., Verbruggen, F., Liefooghe, B., and Vandierendonck, A. (2010). Voluntary task switching under load: contribution of top-down and bottom-up factors in goal-directed behavior. Psychon. Bull. Rev. 17, 387–393. doi: 10.3758/PBR.17.3.387

Deschuyteneer, M., and Vandierendonck, A. (2005). Are ‘input monitoring’ and ‘response selection’ involved in solving simple mental additions. Eur. J. Cogn. Psychol. 17, 343–370. doi: 10.1080/09541440440000032

Emerson, M. J., and Miyake, A. (2003). The role of inner speech in task switching: a dual-task investigation. J. Mem. Lang. 48, 148–168. doi: 10.1016/s0749-596x(02)00511-9

Engle, R. W., Conway, A. R. A., Tuholski, S. W., and Shisler, R. J. (1995). A resource account of inhibition. Psychol. Sci. 6, 122–125. doi: 10.1111/j.1467-9280.1995.tb00318.x

Engle, R. W., Tuholski, S. W., Laughlin, J. E., and Conway, A. R. (1999). Working memory, short term memory and general fluid intelligence: a latent variable approach. J. Exp. Psychol. Gen. 128, 309–311. doi: 10.1037/0096-3445.128.3.309

Eriksen, B. A., and Eriksen, C. W. (1974). Effects of noise letters upon the identification of a target in a non-search task. Percept. Psychophys. 16, 143–149. doi: 10.3758/bf03203267

Friedman, N. P., Miyake, A., Young, S. E., DeFries, J. C., Corley, R. P., and Hewitt, J. K. (2008). Individual differences in executive functions are almost entirely genetic in origin. J. Exp. Psychol. Gen. 137, 201–225. doi: 10.1037/0096-3445.137.2.201

Gazzaley, A., and Nobre, A. C. (2012). Top-down modulation: bridging selective attention and working memory. Trends Cogn. Sci. 16, 129–135. doi: 10.1016/j.tics.2011.11.014

Goschke, T. (2000). “Intentional reconfiguration and involuntary persistence in task set switching,” in Control of Cognitive Processes: Attention and Performance XVIII, eds S. Monsell and J. S. Driver (Cambridge, MA: MIT Press), 331–355.

Hallett, P. E. (1978). Primary and secondary saccades to goals defined by instructions. Vision Res. 18, 1279–1296. doi: 10.1016/0042-6989(78)90218-3

Hitch, G. (1978). The role of short-term working memory in mental arithmetic. Cogn. Psychol. 10, 302–323. doi: 10.1016/0010-0285(78)90002-6

Hulme, C., Maughan, S., and Brown, G. D. A. (1991). Memory for familiar and unfamiliar words: evidence for a long-term memory contribution to short-term memory span. J. Mem. Lang. 30, 685–701. doi: 10.1016/0749-596x(91)90032-f

Imbo, I., De Rammelaere, S., and Vandierendonck, A. (2005). New insights in the role of working memory in carry and borrow operations. Psychol. Belg. 45, 101–121. doi: 10.5334/pb-45-2-101

Imbo, I., Vandierendonck, A., and De Rammelaere, S. (2007). The role of working memory in the carry operation of mental arithmetic: number and value of the carry. Q. J. Exp. Psychol. (Hove) 60, 708–731. doi: 10.1080/17470210600762447

Kahan, T. A., Oldak, V. A., and Lichtman, A. S. (2013). Working memory loads affect location-based negative priming differently than inhibition of return. J. Cogn. Psychol. 25, 473–492. doi: 10.1080/20445911.2013.789855

Kane, M. J., Bleckley, M. K., Conway, A. R. A., and Engle, R. W. (2001). A controlled-attention view of working-memory capacity. J. Exp. Psychol. Gen. 130, 169–183. doi: 10.1037/0096-3445.130.2.169

Kane, M. J., Conway, A. R. A., Hambrick, D. Z., and Engle, R. W. (2007). “Variation in working memory capacity as variation in executive attention and control,” in Variations in Working Memory, eds A. R. A. Conway, C. Jarrold, M. J. Kane, A. Miyake and J. N. Towse (New York: Oxford University Press), 21–48.

Kane, M. J., and Engle, R. W. (2003). Working-memory capacity and the control of attention: the contributions of goal neglect, response competition and task set to stroop interference. J. Exp. Psychol. Gen. 132, 47–70. doi: 10.1037/0096-3445.132.1.47

Kane, M. J., Poole, B. J., Tuholski, S. W., and Engle, R. W. (2006). Working memory capacity and the top-down control of visual search: exploring the boundaries of “executive attention”. J. Exp. Psychol. Learn. Mem. Cogn. 32, 749–777. doi: 10.1037/0278-7393.32.4.749

Keye, D., Wilhelm, O., Oberauer, K., and van Ravenzwaaij, D. (2009). Individual differences in conflict-monitoring: testing means and covariance hypothesis about the Simon and the Eriksen Flanker task. Psychol. Res. 73, 762–776. doi: 10.1007/s00426-008-0188-9

Kiefer, M., Ahlegian, M., and Spitzer, M. (2005). Working memory capacity, indirect semantic priming and stroop interference: pattern of interindividual prefrontal performance differences in healthy volunteers. Neuropsychology 19, 332–344. doi: 10.1037/0894-4105.19.3.332

Kiesel, A., Wendt, M., and Peters, A. (2007). Task switching: on the origin of response congruency effects. Psychol. Res. 71, 117–125. doi: 10.1007/s00426-005-0004-8

Kim, S. Y., Kim, M. S., and Chun, M. M. (2005). Concurrent working memory load can reduce distraction. Proc. Natl. Acad. Sci. U S A 102, 16524–16529. doi: 10.1073/pnas.0505454102

Klauer, K. C., and Zhao, Z. M. (2004). Double dissociations in visual and spatial short-term memory. J. Exp. Psychol. Gen. 133, 355–381. doi: 10.1037/0096-3445.133.3.355

Kornblum, S., Hasbroucq, T., and Osman, A. (1990). Dimensional overlap: cognitive basis for stimulus-response compatibility—a model and taxonomy. Psychol. Rev. 97, 253–270. doi: 10.1037//0033-295x.97.2.253

Kristjánsson, A., Chen, Y., and Nakayama, K. (2001). Less attention is more in the preparation of antisaccades, but not prosaccades. Nat. Neurosci. 4, 1037–1042. doi: 10.1038/nn723

Lavie, N., and de Fockert, J. (2005). The role of working memory in attentional capture. Psychon. Bull. Rev. 12, 669–674. doi: 10.3758/bf03196756

Lavie, N., and de Fockert, J. (2006). Frontal control of attentional capture in visual search. Vis. Cogn. 14, 863–876. doi: 10.1080/13506280500195953

Lavie, N., Hirst, A., de Fockert, J. W., and Viding, E. (2004). Load theory of selective attention and cognitive control. J. Exp. Psychol. Gen. 133, 339–354. doi: 10.1037/0096-3445.133.3.339

Lemaire, P., Abdi, H., and Fayol, M. (1996). The role of working memory resources in simple cognitive arithmetic. Eur. J. Cogn. Psychol. 8, 73–103. doi: 10.1080/095414496383211

Liefooghe, B., Barrouillet, P., Vandierendonck, A., and Camos, V. (2008). Working memory costs of task switching. J. Exp. Psychol. Learn. Mem. Cogn. 34, 478–494. doi: 10.1037/0278-7393.34.3.478

Logan, G. D. (2004). Working memory, task switching and executive control in the task span procedure. J. Exp. Psychol. Gen. 133, 218–236. doi: 10.1037/0096-3445.133.2.218

Logan, G. D. (2006). Out with the old, in with the new: more valid measures of switch cost and retrieval time in the task span procedure. Psychon. Bull. Rev. 13, 139–144. doi: 10.3758/bf03193825

Logan, G. D., and Gordon, R. D. (2001). Executive control of attention in dual-task situations. Psychol. Rev. 108, 393–434. doi: 10.1037/0033-295x.108.2.393

Logie, R. H. (1986). Visuo-spatial processes in working memory. Q. J. Exp. Psychol. 38A, 229–247. doi: 10.1080/14640748608401596

Logie, R. H. (1995). Visuo-Spatial Working Memory. Hillsdale, NJ: Lawrence Erlbaum Associates.

Logie, R. H. (2011). The functional organization and capacity limits of working memory. Curr. Dir. Psychol. Sci. 20, 240–245. doi: 10.1177/0963721411415340

Long, D. L., and Prat, C. S. (2002). Working memory and stroop interference: an individual differences investigation. Mem. Cognit. 30, 294–301. doi: 10.3758/bf03195290

MacLeod, C. M. (1991). Half a century of research on the stroop effect: an integrative review. Psychol. Bull. 109, 163–203. doi: 10.1037/0033-2909.109.2.163

Mayr, U., and Kliegl, R. (2000). Task-set switching and long-term memory retrieval. J. Exp. Psychol. Learn. Mem. Cogn. 26, 1124–1140. doi: 10.1037//0278-7393.26.5.1124

Mayr, U., and Kliegl, R. (2003). Differential effects of cue changes and task changes on task-set selection costs. J. Exp. Psychol. Learn. Mem. Cogn. 29, 362–372. doi: 10.1037/0278-7393.29.3.362

Meier, M. E., and Kane, M. J. (2013). Working memory capacity and stroop interference: global versus local indices of executive control. J. Exp. Psychol. Learn. Mem. Cogn. 39, 748–759. doi: 10.1037/a0029200

Miyake, A., Emerson, M. J., Padilla, F., and Ahn, J. C. (2004). Inner speech as a retrieval aid for task goals: the effects of cue type and articulatory suppression in the random task cuing paradigm. Acta Psychol. (Amst) 115, 123–142. doi: 10.1016/j.actpsy.2003.12.004

Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., and Wager, T. D. (2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: a latent variable analysis. Cogn. Psychol. 41, 49–100. doi: 10.1006/cogp.1999.0734

Morey, C. C., Elliott, E. M., Wiggers, J., Eaves, S. D., Shelton, J. T., and Mall, J. T. (2012). Goal-neglect links stroop interference with working memory capacity. Acta Psychol. (Amst) 141, 250–260. doi: 10.1016/j.actpsy.2012.05.013

Neuman, O. (1984). “Automatic processing: a review of recent findings and a plea for an old theory,” in Cognition and Motor Processes, eds W. Prinz and A. Sanders (Berlin: Springer), 255–293.

Norman, D. A., and Shallice, T. (1986). “Attention to action: willed and automatic control of behavior,” in Consciousness and Self-Regulation (Vol. 4), eds R. J. Davidson, G. E. Schwarts and D. Shapiro (New York: Plenum Press), 1–18.

Notebaert, W., Gevers, W., Verguts, T., and Fias, W. (2006). Shared spatial representations for numbers and space: the reversal of the snarc and the Simon effects. J. Exp. Psychol. Hum. Percept. Perform. 32, 1197–1207. doi: 10.1037/0096-1523.32.5.1197

Oberauer, K. (2009). “Design for a working memory,” in Psychology of Learning and Motivation: Advances in Research and Theory (Vol. 51), ed B. H. Ross (San Diego: Elsevier Academic Press Inc.), 45–100.

Oberauer, K. (2010). Declarative and procedural working memory: common principles, common capacity limits? Psychol. Belg. 50, 277–308. doi: 10.5334/pb-50-3-4-277

Olivers, C. N. L., Meijer, F., and Theeuwes, J. (2006). Feature-based memory-driven attentional capture: visual working memory content affects visual attention. J. Exp. Psychol. Hum. Percept. Perform. 32, 1243–1265. doi: 10.1037/0096-1523.32.5.1243

Page, M. P. A., and Norris, D. (1998). The primacy model: a new model of immediate serial recall. Psychol. Rev. 105, 761–781. doi: 10.1037/0033-295x.105.4.761-781

Page, M. P. A., and Norris, D. (2009). A model linking immediate serial recall, the Hebb repetition effect and the learning of phonological word forms. Philos. Trans. R. Soc. Lond. B Biol. Sci. 364, 3737–3753. doi: 10.1098/rstb.2009.0173

Petersen, S. E., and Posner, M. I. (2012). The attention system of the human brain: 20 years after. Annu. Rev. Neurosci. 35, 73–89. doi: 10.1146/annurev-neuro-062111-150525

Poole, B. J., and Kane, M. J. (2009). Working-memory capacity predicts the executive control of visual search among distractors: the influences of sustained and selective attention. Q. J. Exp. Psychol. (Hove) 62, 1430–1454. doi: 10.1080/17470210802479329

Posner, M. I., and Petersen, S. E. (1990). The attention system of the human brain. Annu. Rev. Neurosci. 13, 25–42. doi: 10.1146/annurev.neuro.13.1.25

Posner, M. I., and Rothbart, M. K. (2007). Research on attention networks as a model for the integration of psychological science. Annu. Rev. Psychol. 58, 1–23. doi: 10.1146/annurev.psych.58.110405.085516

Pratt, N., Willoughby, A., and Swick, D. (2011). Effects of working memory load on visual selective attention: behavioral and electrophysiological evidence. Front. Hum. Neurosci. 5:57. doi: 10.3389/fnhum.2011.00057

Rabbitt, P. (1997). “Introduction: methodologies and models in the study of executive function,” in Methodology of Frontal and Executive Function, ed P. Rabbitt (Hove: Psychology Press), 1–38.

Raymond, J. E., Shapiro, K. L., and Arnell, K. M. (1992). Temporary suppression of visual processing in an RSVP task: an attentional blink. J. Exp. Psychol. Hum. Percept. Perform. 18, 849–860. doi: 10.1037/0096-1523.18.3.849

Roberts, R. J. Jr., Hager, L. D., and Heron, C. (1994). Prefrontal cognitive processes: working memory and inhibition in the antisaccade task. J. Exp. Psychol. Gen. 123, 374–393. doi: 10.1037/0096-3445.123.4.374

Schneider, W., and Shiffrin, R. M. (1977). Controlled and automatic information processing. I. Detection, search and attention. Psychol. Rev. 84, 1–66. doi: 10.1037/0033-295x.84.1.1

Shiffrin, R. M., and Schneider, W. (1977). Controlled and automatic human information processing. II. Perceptual learning, autonomic attending and a general theory. Psychol. Rev. 84, 127–190. doi: 10.1037/0033-295x.84.2.127

Simon, J. R., and Rudell, A. P. (1967). Auditory S-R compatibility: the effect of an irrelevant cue on information processing. J. Appl. Psychol. 51, 300–304. doi: 10.1037/h0020586

Sobel, K. V., Gerrie, M. P., Poole, B. J., and Kane, M. J. (2007). Individual differences in working memory capacity and visual search: the roles of top-down and bottom-up processing. Psychon. Bull. Rev. 14, 840–845. doi: 10.3758/bf03194109

Soutschek, A., Strobach, T., and Schubert, T. (2013). Working memory demands modulate cognitive control in the stroop paradigm. Psychol. Res. 77, 333–347. doi: 10.1007/s00426-012-0429-9

Sternberg, S. (1966). High speed scanning in human memory. Science 153, 652–654. doi: 10.1126/science.153.3736.652

Stins, J. F., Vosse, S., Boomsma, D. I., and de Geus, E. J. C. (2004). On the role of working memory in response interference. Percept. Mot. Skills 99, 947–958. doi: 10.2466/pms.99.7.947-958

Stroop, J. R. (1935). Studies of inteference in serial verbal reactions. J. Exp. Psychol. 18, 643–662. doi: 10.1037/h0054651

Stürmer, B., Seiss, E., and Leuthold, H. (2005). Executive control in the Simon task: a dual-task examination of response priming and its suppression. Eur. J. Cogn. Psychol. 17, 590–618. doi: 10.1080/09541440540000077

Stuyven, E., Van der Goten, K., Vandierendonck, A., Claeys, K., and Crevits, L. (2000). Saccadic eye movements under conditions of cognitive load. Acta Psychol. 104, 69–85. doi: 10.1016/S0001-6918(99)00054-2

Szmalec, A., Vandierendonck, A., and Kemps, E. (2005). Response selection involves executive control: evidence from the selective interference paradigm. Mem. Cognit. 33, 531–541. doi: 10.3758/bf03193069

Szmalec, A., Verbruggen, F., Vandierendonck, A., and Kemps, E. (2011). Control of interference during working memory updating. J. Exp. Psychol. Hum. Percept. Perform. 37, 137–151. doi: 10.1037/a0020365

Tipper, S. P. (1985). The negative priming effect: inhibitory priming by ignored objects. Q. J. Exp. Psychol. A 37A, 571–590. doi: 10.1080/14640748508400920

Tipper, S. P., and Driver, J. (1988). Negative priming between pictures and words: evidence for semantic analysis of ignored stimuli. Mem. Cognit. 16, 64–70. doi: 10.3758/bf03197746

Turner, M. L., and Engle, R. W. (1989). Is working memory capacity task dependent? J. Mem. Lang. 28, 127–154. doi: 10.1016/0749-596x(89)90040-5

Vandierendonck, A. (2000a). Bias and processing capacity in generation of random time intervals. Cogn. Sci. Q. 1, 205–233. doi: 10.1016/S0001-6918(99)00054-2

Vandierendonck, A. (2000b). Is judgment of random time intervals biased and capacity limited? Psychol. Res. 63, 199–209. doi: 10.1007/pl00008179

Vandierendonck, A. (2012). Role of working memory in task switching. Psychol. Belg. 52, 229–253. doi: 10.5334/pb-52-2-3-229

Vandierendonck, A., and De Vooght, G. (1997). Working memory constraints on linear reasoning with spatial and temporal contents. Q. J. Exp. Psychol. A 50A, 803–820. doi: 10.1080/027249897391892

Vandierendonck, A., Demanet, J., Liefooghe, B., and Verbruggen, F. (2012). A chain-retrieval model for voluntary task switching. Cogn. Psychol. 65, 241–283. doi: 10.1016/j.cogpsych.2012.04.003

Vandierendonck, A., Deschuyteneer, M., Depoorter, A., and Drieghe, D. (2008). Input monitoring and response selection as components of executive control in prosaccades and antisaccades. Psychol. Res. 72, 1–11. doi: 10.1007/s00426-006-0078-y

Vandierendonck, A., Liefooghe, B., and Verbruggen, G. (2010). Task switching: interplay of reconfiguration and interference control. Psychol. Bull. 136, 601–626. doi: 10.1037/a0019791

Vandierendonck, A., Szmalec, A., Deschuyteneer, M., and Depoorter, A. (2007). “Towards a multicomponential view of executive control. The case of response selection,” in Working Memory: Behavioural and Neural Correlates, eds N. Osaka and R. Logie (Oxford: Oxford University Press), 247–259.

Weldon, R. B., Mushlin, H., Kim, B., and Sohn, M. H. (2013). The effect of working memory capacity on conflict monitoring. Acta Psychol. (Amst) 142, 6–14. doi: 10.1016/j.actpsy.2012.10.002

Wolfe, J. M., Alvarez, G. A., and Horowitz, T. S. (2000). Attention is fast but volition is slow. Nature 406:691. doi: 10.1038/35021132

Woodman, G. F., and Luck, S. J. (2004). Visual search is slowed when visuospatial working memory is occupied. Psychon. Bull. Rev. 11, 269–274. doi: 10.3758/bf03196569

Wühr, P., and Biebl, R. (2011). The role of working memory in spatial S-R correspondence effects. J. Exp. Psychol. Hum. Percept. Perform. 37, 442–454. doi: 10.1037/a0020563

Zhao, X. A., Chen, A. T., and West, R. (2010). The influence of working memory load on the Simon effect. Psychon. Bull. Rev. 17, 687–692. doi: 10.3758/PBR.17.5.687

Conflict of Interest Statement: The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The Guest Associate Editor Jean-Philippe van Dijck declares that, despite being affiliated to the same institution as author André Vandierendonck, the review process was handled objectively and no conflict of interest exists.

Received: 02 May 2014; Accepted: 15 July 2014; Published online: 08 August 2014.

Citation: Vandierendonck A (2014) Symbiosis of executive and selective attention in working memory. Front. Hum. Neurosci. 8:588. doi: 10.3389/fnhum.2014.00588

This article was submitted to the journal Frontiers in Human Neuroscience.

Copyright © 2014 Vandierendonck. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

ORIGINAL RESEARCH ARTICLE
published: 27 August 2014
doi: 10.3389/fnhum.2014.00670

Resource-sharing between internal maintenance and external selection modulates attentional capture by working memory content

Anastasia Kiyonaga[*] and Tobias Egner

Department of Psychology and Neuroscience and Center for Cognitive Neuroscience, Duke University, Durham, NC, USA

* Correspondence: Anastasia Kiyonaga, Department of Psychology and Neuroscience and Center for Cognitive Neuroscience, Duke University, 450 Research Drive, Levine Science Research Center, Room B203, Box 90999, Durham, NC 27708, USA e-mail: anastasia.kiyonaga@duke.edu

Edited by:
Elger Abrahamse, Ghent University, Belgium

Reviewed by:
Christian N. L. Olivers, Vrije Universiteit, Netherlands
Evie Vergauwe, University of Missouri-Columbia, USA

It is unclear why and under what circumstances working memory (WM) and attention interact. Here, we apply the logic of the time-based resource-sharing (TBRS) model of WM (e.g., Barrouillet et al., 2004) to explore the mixed findings of a separate, but related, literature that studies the guidance of visual attention by WM contents. Specifically, we hypothesize that the linkage between WM representations and visual attention is governed by a time-shared cognitive resource that alternately refreshes internal (WM) and selects external (visual attention) information. If this were the case, WM content should guide visual attention (involuntarily), but only when there is time for it to be refreshed in an internal focus of attention. To provide an initial test for this hypothesis, we examined whether the amount of unoccupied time during a WM delay could impact the magnitude of attentional capture by WM contents. Participants were presented with a series of visual search trials while they maintained a WM cue for a delayed-recognition test. WM cues could coincide with the search target, a distracter, or neither. We varied both the number of searches to be performed, and the amount of available time to perform them. Slowing of visual search by a WM matching distracter—and facilitation by a matching target—were curtailed when the delay was filled with fast-paced (refreshing-preventing) search trials, as was subsequent memory probe accuracy. WM content may, therefore, only capture visual attention when it can be refreshed, suggesting that internal (WM) and external attention demands reciprocally impact one another because they share a limited resource. The TBRS rationale can thus be applied in a novel context to explain why WM contents capture attention, and under what conditions that effect should be observed.

Keywords: working memory, attention, visual search, resource-sharing

WORKING MEMORY (SOMETIMES) BIASES VISUAL ATTENTION

Working memory (WM) typically describes the short-term maintenance and manipulation of internal information (i.e., no longer available to the senses), but the material being maintained in WM can impact the focusing of attention toward external stimuli. WM can be volitionally used to maintain current goals and guide where attention is allocated (Bundesen, 1990; Desimone and Duncan, 1995; Woodman and Chun, 2006), but items matching WM content can also involuntarily capture visual attention in an unrelated task (e.g., Soto et al., 2005; Olivers et al., 2006). The current study explores boundary conditions of this WM biasing, and bridges two separate literatures to test a mechanism that might explain how WM and attention are linked.

A high load on WM storage can impair performance of an attention-demanding task (e.g., De Fockert, 2001; Chen and Cowan, 2009), and items maintained in WM can guide eye movements and attention toward matching, but task-irrelevant, items when a visual search occurs during the WM delay interval (for reviews see Soto et al., 2008; Olivers et al., 2011). Visual search is typically speeded, for instance, if a memory-matching item coincides with a search target (a valid trial), and slowed if a memory-matching item coincides with a search distracter (Soto et al., 2005). Attention can be captured by external WM-matching items even when the WM content never cues the search target (Soto et al., 2005; Olivers et al., 2006), and no memory probe is given after the search array (Kiyonaga et al., 2012)—conditions under which there would be no incentive to voluntarily attend to the WM-match. Moreover, WM content can capture attention above and beyond highly perceptually salient and “pop-out” targets (Soto et al., 2006; Dowd and Mitroff, 2013), and even in a simple detection task with no competition for selection among stimuli (Hollingworth et al., 2013), suggesting that the link between WM representations and visual orienting is obligatory and has its impact early in the processing stream. These observations are consistent with the biased competition model of attention (Desimone and Duncan, 1995), wherein the active representation of an item being held in WM causes stimuli in the environment matching those maintained features to be preferentially attended (i.e., win the competition for selection).

There are also many instances, however, when WM and attention processes can occur simultaneously without impeding one another (e.g., we are generally able to rehearse our to-do list while also operating a car and following traffic signals). Accordingly, in some studies attention task performance has remained efficient despite concurrent WM storage demands (Woodman et al., 2001; Cocchini et al., 2002), and WM content has failed to involuntarily capture visual attention (Downing and Dodds, 2004; Houtkamp and Roelfsema, 2006; Woodman and Luck, 2007; Peters et al., 2009). Why might WM and attention demands impact one another in some situations but not others? These inconsistencies fuel an ongoing debate about the degree of overlap between the content and function of WM and attention, and experimental attempts to explain the conflicting results (Soto and Humphreys, 2008; Han and Kim, 2009; Olivers, 2009; Dombrowe et al., 2010; Zhang et al., 2010, 2011; Tsvetanov et al., 2012) have yet to establish a clear set of rules governing how and when WM will impact attention—and specifically when WM content will capture visual attention in an unrelated task. We postulate that the framework of a fruitful model of the relationship between WM storage and processing can be applied to potentially resolve these discrepancies.

RESOURCE-SHARING BETWEEN WORKING MEMORY AND ATTENTION

The time-based resource-sharing model (TBRS; Barrouillet et al., 2004, 2007, 2011) assumes that items are maintained in WM via a “refreshing” process that requires attention, and the same resource that is used to refresh WM is also used for attentional processing of external stimuli. TBRS quantifies the amount of information that can be maintained in WM, as a function of intervening attention or “processing” demands, by varying the time-consumption of processing events during WM maintenance—either by manipulating the number of stimuli to process or the demand level of individual processing tasks. In brief, increasing processing demands impairs WM storage.

While tests of the TBRS model have exclusively examined the impact of attention demands on WM storage (i.e., time-consuming processing limits WM maintenance), studies of WM biasing of selection (e.g., Downing, 2000; Soto et al., 2005; Olivers et al., 2006) have primarily examined the impact of WM content on the allocation of attention (i.e., information maintained in WM determines what gets processed). Together, however, the two literatures suggest that there is a reciprocal trade-off between WM and attention. Indeed, there is a growing consensus that WM can be thought of as attention oriented toward internal representations (Awh and Jonides, 2001; Postle, 2006; Chun, 2011; Gazzaley and Nobre, 2012; Kiyonaga and Egner, 2013). WM for an item can be improved by retrospectively focusing attention on it in response to a cue (i.e., “retro-cue benefit”; Griffin and Nobre, 2003), spatial attention is recruited to maintain WM representations (Awh et al., 1998; Nobre et al., 2004), WM representations can impact behavior just like visually attended stimuli (Kiyonaga and Egner, 2014), and WM and attention have been demonstrated to rely on the same neural mechanisms (Awh et al., 2000; Lepsien et al., 2005; Kuo et al., 2009; Ikkai and Curtis, 2011). Prominent embedded process theories of WM, further, describe activation and maintenance in WM as accomplished by directing an internal focus of attention at long-term memory representations (Cowan, 1988, 2001; Oberauer, 2009; Oberauer and Hein, 2012), and another recent theory (Olivers et al., 2011) suggests that WM content might capture attention only when it is held in that focus of attention.

Accordingly, we have recently proposed that internal (WM) and external (visual selection) prioritization processes share a common attention resource (i.e., the same focus of attention shifts between endogenously and exogenously activated representations), and this resource-sharing—akin to the time-shared refreshing mechanism described by TBRS—might explain why the contents of WM, sometimes unwittingly, impact the allocation of visual attention (Kiyonaga and Egner, 2013). In particular, if the internal maintenance of information occurs via refreshing, and this brief foregrounding places that information in the focus of attention (cf. Johnson et al., 2007; Chun and Johnson, 2011), this will activate the sensory neurons that are responsive to the features of the maintained representations, and sensitivity to matching items in the environment will consequently be increased (even if they are not immediately task-relevant). Here, we adapt the TBRS methodology to test this hypothesis.

As dictated by TBRS, if the focus of attention is continually oriented toward external stimuli for processing, internal refreshing should be hindered, and the magnitude of this hindrance will depend on the relative time-consumption of the external attention demand: more time-intensive demands to direct attention externally will result in less opportunity to refresh the WM content, and worse WM recognition performance. Crucial to our hypothesis, the obstruction of internal refreshing should keep internal content out of the focus of attention, thereby decreasing perceptual sensitivity for stimuli with matching features, which should in turn lead to an attenuated influence of WM content on the selection of external stimuli. In short, if we apply the logic of TBRS more broadly to the relationship between internally geared (e.g., WM content) and externally geared (e.g., visual search) attention, then one can potentially explain (a) why WM content captures attention in the first place (because the two share the same attention resource) and (b) when that capture of attention is likely to occur (when that resource has ample time to switch between internal refreshing and external selection, but not when refreshing is prevented).

EXPERIMENT 1

If our proposal were correct, we should be able to systematically modulate the degree to which internal WM content captures attention by experimentally manipulating the time required for externally-geared attentional processes. To test this possibility, we combined elements of the methods used to test both TBRS and the WM biasing of visual attention. We devised a delayed match-to-sample WM task wherein the time-consumption of delay-spanning attention demands was manipulated by independently varying both the number of intervening visual search processes to be performed, and the amount of available time to perform them (i.e., the rate of presentation). WM items could cue visual search targets or distracters (or neither). We predicted that under a low rate of external attention demands there should be opportunity to regularly refresh the internal representations, which should then influence the allocation of external attention (leading to capture of visual attention by WM content), and be remembered well—regardless of the total number of externally-geared attention processes. Under a high rate of external attention demands, however, refreshing of the internal content should be limited, thus it should be less able to guide external selection (leading to reduced capture of visual attention by WM content), and be remembered more poorly. This pattern of results would support the existence of a common cognitive resource to lie at the root of the relationship between WM representations and visual attention, and validate the applicability of the TBRS approach to study this relationship.

METHODS

Participants

Thirty-one volunteers (10 female, median age = 20) gave written informed consent and received course credit or $10.00 payment for their participation. The study was approved by the Duke University Institutional review board.

Stimuli and procedure

The experiment was programmed and presented using the Psychophysics Toolbox extensions (Brainard, 1997) for Matlab R2010a (Mathworks Inc., Natick, MA, USA) on a Dell Optiplex 960 computer. Stimuli were viewed from approximately 60 cm on an LCD monitor with a 60 Hz refresh rate and a screen resolution of 1280 × 1024 pixels. The task was composed of “mini-blocks,” each comprising a delayed match-to-sample WM test with a series of visual search trials during the delay (wherein WM items could reappear in the search array; cf. Soto et al., 2005). Each mini-block began with the presentation of a central fixation dot for 1500 ms, which was then replaced by a to-be-remembered colored shape cue for 1000 ms. The WM cues were randomly selected from a combination of five shapes (circle, square, triangle, diamond, and pentagon) and five colors (RGB values—red: 255, 0, 0; blue: 0, 0, 255; green: 0, 255, 0; yellow: 255, 255, 0; pink: 255, 0, 255), each subtending a visual angle of approximately 3.8° × 3.8°. The memory cue was then followed by a fixation display of either 250 or 750 ms (depending on condition), then a sequence of either two or four visual search trials, each displayed for 400 ms, and separated by a central fixation cross for either 250 or 750 ms (Figure 1). All stimuli were presented against a gray background (RGB: 128, 128, 128).

FIGURE 1. Structure of a single mini-block, and the four possible conditions of the WM delay-spanning visual search sequence in Experiment 1. Invalid, neutral, and valid search displays occurred in random order. Experiment 2 was identical except that search count varied between 4 and 6, and invalid, neutral, and valid search displays were constrained to occur equally often.

Each search display was composed of four lines surrounded by colored shapes—one within each quadrant of the screen at a quasi-random location from trial to trial. Three of the lines were vertical and one—the target—was tilted 25° to the left or right. The participants' task was to indicate the orientation of the one slanted line in each array. The surrounding shapes, each subtending approximately 3.8 × 3.8°, were randomly chosen from the same set as the memory cues, but constrained so that there were no color or shape matches among stimuli in a given search display, and no partial matches to the memory item for that mini-block (though there could be a complete match). Target locations and colored shape stimuli were randomly chosen, thus individual search trials could be valid (memory item reappears surrounding the search target), invalid (memory item reappears surrounding a distracter), or neutral (memory item does not reappear in the search display). After the visual search sequence, a single colored shape memory probe appeared at the center of the screen underneath a question mark for 1500 ms. The participants' task was to indicate whether this shape was the same or different from the original memory cue. The memory probe was always a whole or partial match to the memory cue (never different on both color and shape), and match and non-match probes occurred equally often and in random order.

The number of visual search trials between memory cue and probe (search count: two vs. four), and the duration of the inter-stimulus interval between search arrays (search pace: 250 ms fast vs. 750 ms slow) were combined to produce four different mini-block conditions, (total search sequence durations: two fast, 1.55 s; two slow, 3.05 s; four fast, 2.85 s; four slow, 5.35 s), which were presented in random order. This allowed us to look at the unique contributions to later memory of both the number of operations that needed to be performed during the WM delay, and the pace at which they occurred. Since the WM cue could be a valid, invalid, or neutral indicator of the search target, we were able to evaluate the impact of mini-block condition on the effect of WM cueing validity. After a practice block, participants completed eight experimental runs, each consisting of 24 mini-blocks (i.e., 48 mini-blocks per condition). The proportion of valid, invalid and neutral trials was unconstrained within a mini-block, but any given trial type occurred equally often across each mini-block condition. Across the entire experiment, one in three search trials were neutral (32 per two fast and two slow, 64 per four fast and four slow); of the remaining two thirds when the memory item reappeared in the search, it corresponded to a distractor three out of four times (invalid: 48 per two fast and two slow, 96 per four fast and four slow). Thus, one in six total search trials were valid (16 per two fast and two slow, 32 per four fast and four slow). We predicted that, if internal representations guide external selection when they are being refreshed, a fast search pace would dampen the influence of cue validity, and impair memory recognition, irrespective of search count.

RESULTS AND DISCUSSION

We performed 3 × 2 × 2 ANOVAs including the factors of WM cue validity (invalid, neutral, valid), search count (two vs. four), and search pace (fast vs. slow), on visual search performance measures. Measures included accuracy (% correct) and mean correct response time (RT; within 3 SD of an individual's mean) for trials when the memory probe was also correctly performed. Visual search accuracy was unaffected by WM-matching or mini-block condition, while RT was sensitive to WM cue validity, F(1, 29) = 6.83, p < 0.05, and search pace, F(1, 29) = 19.65, p < 0.001, though not to search count, p > 0.2. Participants were significantly slower to correctly discriminate the search target when a WM matching item appeared as a distracter (invalid) than when the memory item coincided with a target (valid) or did not reappear at all (neutral; Figure 2A). While participants were faster overall during the fast-paced conditions, this factor of pace also interacted with WM cue validity, F(1, 29) = 4.43, p < 0.05. Most importantly to our research question, the magnitude of the RT cost of an invalid visual search distracter (invalid RT-neutral RT) that was robust during slow-paced conditions, t(30) = 4.3, p < 0.001 was eliminated when the search trials had to be performed at a fast pace, t(30) = 0.8, p > 0.4 (Figure 2B).

FIGURE 2. Task performance in Experiment 1. (A) Response Times—collapsed across all mini-block conditions—when WM cues were matching visual search distracters (invalid), targets (valid), or did not reappear in the search display (neutral). (B) The magnitude of costs (slowing by invalid cues) and benefits (speeding by valid cues) to visual search as a function of the rate of search presentation. (C) Accuracy on memory probes for each mini-block condition.

We also performed 2 × 2 ANOVAs including the factors of search count (two vs. four), and search pace (fast vs. slow) on WM probe accuracy and RT. Accuracy on the delayed-recognition memory test was better when it was only preceded by two search trials (vs. four), F(1, 30) = 4.46, p < 0.05. While there was no main effect of search pace on probe accuracy, p > 0.8, the search pace and count did interact, F(1, 30) = 6.44, p < 0.05 (Figure 2C). Participants were equally accurate in the slow-paced conditions, whether they had performed two or four search trials, p > 0.9; in the fast-paced conditions, however, they were less accurate after four search trials (vs. two), t(30) = 3.32, p < 0.01. Memory probe accuracy was best overall during the two fast condition (which also resulted in the shortest delay between memory cue and probe), and worst during the four fast condition (even though this condition had a shorter delay between cue and probe than the four slow condition). Participants were also faster to correctly respond to the memory probe when they only had to complete two search trials (vs. four), F(1, 30) = 22.14, p < 0.001, and faster when they had to be completed at a fast pace (vs. slow), F(1, 30) = 4.29, p < 0.05, but there was no interaction between these factors, p > 0.4.

In support of the notion that WM content may capture visual attention when it is being rehearsed or refreshed, the slowing of visual search by a WM matching distracter was curtailed when the WM delay was filled with a series of fast-paced visual search trials—which would presumably occupy attention externally and hinder refreshing of internal WM content. This effect of validity on search speed was not sensitive to the total number of search trials that needed to be completed—only to the pace of their completion. In support of the interpretation that this attenuation of attentional capture is due to the blockage of refreshing, WM accuracy was also worst after performing four visual search trials at a fast pace. This memory impairment cannot be explained by just the time-related decay of WM representations, since probe accuracy was better after four search trials performed at a slow pace, which yielded a longer delay between WM cue and probe. This is consistent with the idea that the fast-paced visual search sequence monopolized attention resources, leaving little time to direct attention toward refreshing the WM content. As a consequence, the to-be-remembered items were less able to capture visual attention in the search.

We did, however, observe two unexpected results. First, there was no overall benefit to visual search of a memory-matching target. Our ability to detect a validity benefit may have been undermined by the small proportion (and total number) of these valid trials, which constituted only one sixth of all trials. Second, memory probe performance was best after two fast-paced visual search trials, counter to our expectation that the fast search pace would impair memory performance, regardless of the number of operations to be performed (cf. Barrouillet et al., 2011). However, this finding could be attributable to the fact that the delay between WM cue and probe was so short in this condition (1.55 s) that it may have been possible to maintain and identify the probe without needing to direct attention toward refreshing the WM representation. We addressed these unexpected results, and sought to confirm our interpretation of them, in Experiment 2.

EXPERIMENT 2

The conclusion that WM refreshing and visual search share a resource would be strengthened if we were also able to demonstrate that (1) the visual search benefits of a WM-matching target are attenuated when WM refreshing is hampered, and (2) fast-paced external attention demands can interfere with memory accuracy, regardless of the number of search operations to be performed. In Experiment 2, thus, we (1) altered the proportion of valid, invalid, and neutral visual search trials to increase the incidence of valid trials, and (2) increased the number of search trials to be performed in all mini-blocks, to prolong the minimum delay between WM cue and probe so that maintenance of the WM representation would require refreshing.

METHODS

Participants

Twenty volunteers (13 female, median age = 18.5) gave written informed consent and received course credit or $10.00 payment for their participation. The study was approved by the Duke University Institutional review board.

Stimuli and procedure

The experimental paradigm was identical to Experiment 1 with the exception of two important changes: (a) The proportion of valid, invalid, and neutral visual search trials was constrained so that they each occurred equally often (⅓ of all trials) and (b) the minimum number of visual search operations to perform was increased from two to four (thus raising the minimum time interval from WM cue to probe from 1.55 to 2.85 s), and the maximum number from four to six. In Experiment 2, consequently, participants completed either four or six search trials, at either a slow or fast pace, during each WM delay. Again, these were combined to produce four mini-block conditions (total search sequence durations: four fast, 2.85 s; four slow, 5.35 s; six fast, 4.15 s; six slow, 7.65 s), which were presented in random order. The participant's task was identical to that in Experiment 1: to indicate the orientation of the slanted line target in every visual search array, and to indicate whether a probe was a match or non-match to the WM cue presented prior to the visual search sequence. After a practice block, participants completed 6 experimental runs, each consisting of 24 mini-blocks (i.e., 36 mini-blocks per condition; 48 search trials of each validity type per four fast and four slow, 72 search trials of each validity type per six fast and six slow).

RESULTS AND DISCUSSION

As in Experiment 1, we performed 3 × 2 × 2 ANOVAs including the factors of WM cue validity (invalid, neutral, valid), search count (four vs. six), and search pace (fast vs. slow), on visual search accuracy and RT. Accuracy was best during slow-paced mini-blocks, F(1, 19) = 28.05, p < 0.001, and best with valid cueing (and worst with invalid WM cueing), F(1, 19) = 4.41, p < 0.05, and there was no effect of search count, nor any significant interactions between factors. Visual search RT was sensitive to both WM-matching and mini-block condition. Search performance was fastest with valid cueing, and slowest with invalid cueing, F(1, 19) = 7.67, p < 0.01 (Figure 3A). Relative to the neutral condition, search was marginally slowed by a WM-matching distractor, t(19) = 2.0, p < 0.06, and, critical to the goals of Experiment 2, expedited by a WM-matching target, t(19) = 2.6, p < 0.05. Search time was slightly slower when six search trials had to be completed (vs. four), F(1, 19) = 4.53, p < 0.05, and faster during the fast paced conditions (vs. slow), F(1, 19) = 14.59, p < 0.001. Search count interacted with pace, F(1,19) = 6.15, p < 0.05, in that the speed difference between four and six trial sequences was only evident in the slow-paced condition. Search count also interacted with WM cue validity, F(2,38) = 5.29, p < 0.01, in that the magnitude of the effect of validity was larger when there were fewer search trials to be performed. As observed in Experiment 1, the effect of WM validity also interacted with search pace, F(2,38) = 3.71, p < 0.05. Most importantly to the goals of Experiment 2, both the cost of a WM-matching distractor (invalid RT-neutral RT), t(19) = 1.5, p > 0.1, and the benefit of a WM-matching target (valid RT-neutral RT), t(19) = 0.5, p > 0.5, were eliminated during the fast pace search sequences (Figure 3B).

FIGURE 3. Task performance in Experiment 2. (A) Response Times—collapsed across all mini-block conditions—when WM cues were matching visual search distracters (invalid), targets (valid), or did not reappear in the search display (neutral). (B) The magnitude of costs (slowing by invalid cues) and benefits (speeding by valid cues) to visual search as a function of the rate of search presentation. (C) Accuracy on memory probes for each mini-block condition.

We also performed 2 × 2 ANOVAs including the factors of search count (two vs. four), and search pace (fast vs. slow), on WM probe accuracy and RT. Memory recognition speed was unaffected by mini-block condition. Unlike Experiment 1, memory accuracy was not significantly influenced by the number of searches that had been performed, but it was worse after fast-paced visual search sequences of either four or six trials, F(1, 19) = 12.41, p < 0.01 (Figure 3C). This effect of visual search pace on memory performance did interact, however, with the search count, F(1, 19) = 9.64, p < 0.01, in that the fast pace had a greater disruptive effect when a greater number of search trials had been completed. Thus, although TBRS would predict WM performance to be unaffected by the total number of processing events (as long as they occur at a constant pace), in the current task context, WM storage appears to be influenced by a combination of the absolute amount of processing that occurs between memory cue and probe (cf. Ricker and Cowan, 2010), as well as the amount of available time during the delay to refresh the WM content. One possibility is that the processing manipulation used here prevented WM refreshing to the point that the time-related decay proposed by TBRS (when items cannot be refreshed) had an accumulating impact on WM as more time elapsed (i.e., when six searches had to be performed rather than four). Alternatively, given that memory items could be fairly easily verbalized, perhaps a verbal rehearsal strategy was used to provide some degree of protection against the fast-paced search sequence at shorter delays (cf. Vergauwe et al., 2014).

In sum, both slowing of visual search by a WM-matching distracter and speeding by a WM-matching target were dampened when the WM delay-spanning search sequence was presented at a fast pace, and presumably prevented the WM content from being refreshed. Consequently, memory probe accuracy was also worse during these fast-paced, refreshing-preventing conditions. Thus, regardless of the number of operations to be performed during a WM delay, or the total duration of that delay, biasing of visual search was minimized when the delay was filled with an attention-demanding visual task that left little time for refreshing of WM content.

GENERAL DISCUSSION

SUMMARY

In two experiments, we observed that high frequency external stimulus processing demands can curb the extent to which visual selection favors items in the environment that match the contents of WM. Akin to the manner in which heightened perceptual load can limit the processing of perceptual distracters (Rees et al., 1997), here an increased external load modulated both the influence and retention of internal content. These data provide further evidence of a shared resource between internal (i.e., WM maintenance) and external (i.e., visual attention) selection processes, they implicate that resource as a source of visual capture by WM contents, and they likewise illuminate why WM contents might capture visual attention in some situations but not others. The results are consistent with earlier observations that time demands and cognitive load can limit the impact of WM on visual attention (Soto and Humphreys, 2008; Dombrowe et al., 2010; Dalvit and Eimer, 2011), and in keeping with the broader notion that WM is akin to internally-oriented attention (Postle, 2006; Chun, 2011; Gazzaley and Nobre, 2012; Kiyonaga and Egner, 2013). The TBRS approach—of manipulating processing demand to measure its impact on WM storage capacity—may thus offer a promising means of investigating the related phenomenon of WM biasing of attention.

ALTERNATIVE INTERPRETATIONS

Recent studies suggest that the delay following presentation of a WM cue may be used to consolidate the WM representation, and that different consolidation times can lead to different rates of forgetting (Ricker and Cowan, 2014; Vergauwe et al., 2014). Given that the duration of the interval between the WM cue and visual search sequence varied with trial condition in the current study, it may be possible that the fast-paced conditions provided less time to consolidate the WM item, which may have then had less impact on search performance. We point out, however, that we report visual search performance only for trials with a correct response on the memory probe; so, it is unlikely to be the case that WM cues failed to impact search simply because they were never consolidated/remembered in the first place. On average, memory probe performance was worst after fast-paced visual search sequences (but still well above chance), suggesting that our manipulation was effective at limiting WM refreshing. That overall impediment to refreshing also manifested itself as a failure for the WM content to capture visual attention under those fast-paced conditions, even when the WM content was ultimately correctly recalled. It is possible that when refreshing was impeded—but items still correctly remembered—WM representations were designated a different “accessory” status outside the focus of attention (cf. LaRocque et al., 2014), whereby they would be less likely to impact visual selection (cf. Olivers et al., 2011), but were able to be later reinstated into the focus of attention for WM retrieval (Kiyonaga et al., 2012).

A number of other alternatives—besides a simple refreshing impediment—might explain the increased loss of information from WM when more visual search trials were given at a faster pace. It could be argued, for instance, that the visual search manipulation used here does not place demands exclusively on visual attention per se, but also response selection and execution mechanisms that could interfere with WM maintenance more than the visual demands. Additionally, the possibility that directing attention to objects in visual search leads to their transfer into WM (Schmidt et al., 2002), or that tracking previously attended locations in visual search taxes WM (Castel et al., 2003), might both interfere with concurrent WM representations. The serial order in a box model (SOB; e.g., Oberauer and Lewandowsky, 2008; Lewandowsky et al., 2009; Oberauer et al., 2012), likewise, attributes the decay of WM to interference from distractors, which require time to be removed from the limited-capacity store. Considering our results within the SOB framework, we presume that the attention resource that is required to remove interference from WM is the same one that would otherwise be directed at maintaining the WM content; thus, the time-consumption of resolving interference would also dampen the biasing of visual attention toward WM-matching items. We would, therefore, argue that although by different means, each of these alternatives prevents a limited shared attention resource from being oriented toward the to-be-remembered content. That is, regardless of the specific barrier to directing attention at WM content, when such a barrier exists, that internal content will be prevented from guiding visual selection. A potentially informative means to further test the refreshing mechanism proposed herein would be to examine the magnitude of WM biasing across the delay period (i.e., on the first search display vs. the last in the sequence), in a task with greater trial numbers of each condition, to illuminate how the WM influence changes as time and processing demands accrue.

IMPLICATIONS

Competition between the domains of WM and attention has typically been revealed by maximizing demands to the point that the presumed shared resource is fully occupied, and processing in one or both domains consequently suffers (e.g., De Fockert, 2001; Barrouillet et al., 2004). Accordingly, we found that WM accuracy was impaired when the shared resource was tied up by requirements to direct attention externally. Within the unique task structure employed here, however, it was in the lower visual attention demand conditions—because the shared resource was available to refresh WM representations—that internal content was able to impact external attentional selection. This study thus marks a novel methodological approach to understanding the link between WM and visual selection. By applying the TBRS structure to the dual-task WM-visual search paradigm, we have found further evidence that the contents of both WM and attention are determined by a shared selection mechanism that alternates between internal and external domains, and is limited in the amount of information it can process in a given period of time. Most importantly, when this selection resource has ample time to regularly refresh internal WM content, that content will influence external selection, but when such refreshing is hindered external selection will be unaffected by internal WM representations.

While the findings reported here indicate that the biasing of visual attention by WM contents only occurs when task constraints allow refreshing of WM representations, however, it is not the case that such WM biasing is only prevented when external task demands are maximal; it has in fact been shown that external search targets can be effectively prioritized even in the absence of stringent time demands. Several studies, for instance, have failed to find an impact of WM content on visual attention when the visual search target changes from trial to trial (Downing and Dodds, 2004; Houtkamp and Roelfsema, 2006; Olivers, 2009; Peters et al., 2009). It could be the case that when the search target is variable, it is more demanding to maintain, and thus remains in the focus of attention where it will guide visual search above and beyond other internal material (cf. Woodman et al., 2007). When the WM content is known to be predictably harmful to the current task (Woodman and Luck, 2007; Carlisle and Woodman, 2011; Kiyonaga et al., 2012), top-down strategies might be applied to reallocate the WM content to an accessory status outside the focus of attention, and thereby limit its influence on visual attention. Other studies have varied the perceptual difficulty of the visual search (Han and Kim, 2009) or the time window for responding to the search (Dalvit and Eimer, 2011), and have explained the subsequent attenuation of WM biasing as a result of increased cognitive control under these conditions, but both findings are highly consistent with the premise of the present study: that maintenance of internal WM content and demands on externally-geared attention rely on a shared, limited resource, and WM content will not bias visual attention when that shared resource is not regularly directed toward refreshing the WM content in the focus of attention. Time-consuming external attention demands are one way of limiting WM refreshing, but any task attribute that limits the extent to which attention is directed at the WM content should interfere with both WM recall (though sometimes impacting speed or precision rather than accuracy) and WM biasing of selection of external stimuli. The current data thus demonstrate how and when internal representations capture attention, although the specific task setup reflects only one of a number of ways to probe the boundary conditions of this interaction.

The limits of this shared prioritization resource can furthermore be applied to explain a number of other observations that have been interpreted as indicating that WM and attention demands are discrete (e.g., Woodman et al., 2001; Hollingworth and Maxcey-Richard, 2012; Hollingworth and Hwang, 2013; Rerko et al., 2014). Specifically, the absence of interference between unrelated WM and attention demands would be expected if the shared selection mechanism was given enough time to comfortably alternate between tasks. One study, for instance, required participants to complete a visual search during the delay interval of a change detection task, and concluded that sustained attention was not required for selective maintenance in WM (Hollingworth and Maxcey-Richard, 2012). In this example, participants were given 2000 ms to complete the visual search, but they typically took less than half of that time to respond, indicating that there was ample time within the visual search window to redirect the focus of attention back to refresh the WM content. Absolutely unbroken sustained attention is therefore unnecessary to maintain WM, as long as there is some opportunity to occasionally refresh the WM content. The mechanism tested herein can thus be used to reconcile a collection of seemingly inconsistent findings.

CONCLUSIONS

A complete understanding of the prioritization between internal and external domains will require future examination of whether the boundaries of this mechanism are, for instance, in the rate of decay of internal representations (in the absence of refreshing), the efficiency of assimilating external information, or refreshing internal information (cf. Barrouillet and Camos, 2012), the speed of alternating between processes and domains, or the volume of the resource pool. If what have previously been considered distinct cognitive concepts (i.e., WM and attention) rely on a common resource, and that resource can be trained or enhanced (e.g., Anguera et al., 2013; Kundu et al., 2013), then a specific characterization of its underlying capacity can contribute to understanding and enhancement of information processing in multiple realms of cognition. These data suggest not only that WM and external attention are related, but that they are reciprocal because they both hinge on a common attention resource. Although the notion of reciprocity between WM and attention is not a brand new one, the bridging of methods previously employed to study distinct cognitive operations represents a new framework within which to study and understand the means by which we select and process information in all domains.

ACKNOWLEDGMENTS

This research was partially supported by National Institute of Mental Health Award R01MH087610 to Tobias Egner. We thank John Lucas for help with data collection and David Soto for comments.

REFERENCES

Anguera, J. A., Boccanfuso, J., Rintoul, J. L., Al-Hashimi, O., Faraji, F., Janowich, J., et al. (2013). Video game training enhances cognitive control in older adults. Nature 501, 97–101. doi: 10.1038/nature12486

Awh, E., Anllo-Vento, L., and Hillyard, S. A. (2000). The role of spatial selective attention in working memory for locations: evidence from event-related potentials. J. Cogn. Neurosci. 12, 840–847. doi: 10.1162/089892900562444

Awh, E., and Jonides, J. (2001). Overlapping mechanisms of attention and spatial working memory. Trends Cogn. Sci. 5, 119–126. doi: 10.1016/S1364-6613(00)01593-X

Awh, E., Jonides, J., and Reuter-Lorenz, P. A. (1998). Rehearsal in spatial working memory. J. Exp. Psychol. Hum Percept. Perform. 24, 780–790. doi: 10.1037/0096-1523.24.3.780

Barrouillet, P., Bernardin, S., and Camos, V. (2004). Time constraints and resource sharing in adults' working memory spans. J. Exp. Psychol. Gen. 133:83. doi: 10.1037/0096-3445.133.1.83

Barrouillet, P., Bernardin, S., Portrat, S., Vergauwe, E., and Camos, V. (2007). Time and cognitive load in working memory. J. Exp. Psychol. Learn. Mem. Cogn. 33, 570–585. doi: 10.1037/0278-7393.33.3.570

Barrouillet, P., and Camos, V. (2012). As time goes by temporal constraints in working memory. Curr. Dir. Psychol. Sci. 21, 413–419. doi: 10.1177/0963721412459513

Barrouillet, P., Portrat, S., and Camos, V. (2011). On the law relating processing to storage in working memory. Psychol. Rev. 118, 175. doi: 10.1037/a0022324

Brainard, D. H. (1997). The psychophysics toolbox. Spat. Vis. 10, 433–436. doi: 10.1163/156856897X00357

Bundesen, C. (1990). A theory of visual attention. Psychol. Rev. 97, 523–547. doi: 10.1037/0033-295X.97.4.523

Carlisle, N. B., and Woodman, G. F. (2011). Automatic and strategic effects in the guidance of attention by working memory representations. Acta Psychol. 137, 217–225. doi: 10.1016/j.actpsy.2010.06.012

Castel, A. D., Pratt, J., and Craik, F. I. M. (2003). The role of spatial working memory in inhibition of return: evidence from divided attention tasks. Percept. Psychophys. 65, 970–981. doi: 10.3758/BF03194827

Chen, Z., and Cowan, N. (2009). How verbal memory loads consume attention. Mem. Cognit. 37, 829–836. doi: 10.3758/MC.37.6.829

Chun, M. M. (2011). Visual working memory as visual attention sustained internally over time. Neuropsychologia 49, 1407–1409. doi: 10.1016/j.neuropsychologia.2011.01.029

Chun, M. M., and Johnson, M. K. (2011). Memory: enduring traces of perceptual and reflective attention. Neuron 72, 520–535. doi: 10.1016/j.neuron.2011.10.026

Cocchini, G., Logie, R., Sala, S., MacPherson, S., and Baddeley, A. (2002). Concurrent performance of two memory tasks: evidence for domain-specific working memory systems. Mem. Cognit. 30, 1086–1095. doi: 10.3758/BF03194326

Cowan, N. (1988). Evolving conceptions of memory storage, selective attention, and their mutual constraints within the human information-processing system. Psychol. Bull. 104, 163–191. doi: 10.1037/0033-2909.104.2.163

Cowan, N. (2001). The magical number 4 in short-term memory: a reconsideration of mental storage capacity. Behav. Brain Sci. 24, 87–114. doi: 10.1017/S0140525X01003922

Dalvit, S., and Eimer, M. (2011). Memory-driven attentional capture is modulated by temporal task demands. Vis. Cogn. 19, 145–153. doi: 10.1080/13506285.2010.543441

De Fockert, J. W. (2001). The role of working memory in visual selective attention. Science 291, 1803–1806. doi: 10.1126/science.1056496

Desimone, R., and Duncan, J. (1995). Neural mechanisms of selective visual attention. Annu. Rev. Neurosci. 18, 193–222. doi: 10.1146/annurev.ne.18.030195.001205

Dombrowe, I., Olivers, C. N. L., and Donk, M. (2010). The time course of working memory effects on visual attention. Vis. Cogn. 18, 1089–1112. doi: 10.1080/13506281003651146

Dowd, E. W., and Mitroff, S. R. (2013). Attentional guidance by working memory overrides salience cues in visual search. J. Exp. Psychol. Hum. Percept. Perform. 39, 1786–1796. doi: 10.1037/a0032548

Downing, P., and Dodds, C. (2004). Competition in visual working memory for control of search. Vis. Cogn. 11, 689–703. doi: 10.1080/13506280344000446

Downing, P. E. (2000). Interactions between visual working memory and selective attention. Psychol. Sci. 11, 467–473. doi: 10.1111/1467-9280.00290

Gazzaley, A., and Nobre, A. C. (2012). Top-down modulation: bridging selective attention and working memory. Trends Cogn. Sci. 16, 129–135. doi: 10.1016/j.tics.2011.11.014

Griffin, I. C., and Nobre, A. C. (2003). Orienting attention to locations in internal representations. J. Cogn. Neurosci. 15, 1176–1194. doi: 10.1162/089892903322598139

Han, S. W., and Kim, M. S. (2009). Do the contents of working memory capture attention? Yes, but cognitive control matters. J. Exp. Psychol. Hum. Percept. Perform. 35, 1292. doi: 10.1037/a0016452

Hollingworth, A., and Hwang, S. (2013). The relationship between visual working memory and attention: retention of precise colour information in the absence of effects on perceptual selection. Philos. Trans. R. Soc. B Biol. Sci. 368:20130061. doi: 10.1098/rstb.2013.0061

Hollingworth, A., Matsukura, M., and Luck, S. J. (2013). Visual working memory modulates rapid eye movements to simple onset targets. Psychol. Sci. 24, 790–796. doi: 10.1177/0956797612459767

Hollingworth, A., and Maxcey-Richard, A. M. (2012). Selective maintenance in visual working memory does not require sustained visual attention. J. Exp. Psychol. Hum. Percept. Perform. 39, 1047–1058. doi: 10.1037/a0030238

Houtkamp, R., and Roelfsema, P. R. (2006). The effect of items in working memory on the deployment of attention and the eyes during visual search. J. Exp. Psychol. Hum. Percept. Perform. 32:423. doi: 10.1037/0096-1523.32.2.423

Ikkai, A., and Curtis, C. E. (2011). Common neural mechanisms supporting spatial working memory, attention and motor intention. Neuropsychologia 49, 1428–1434. doi: 10.1016/j.neuropsychologia.2010.12.020

Johnson, M. R., Mitchell, K. J., Raye, C. L., D'Esposito, M., and Johnson, M. K. (2007). A brief thought can modulate activity in extrastriate visual areas: top-down effects of refreshing just-seen visual stimuli. Neuroimage 37, 290–299. doi: 10.1016/j.neuroimage.2007.05.017

Kiyonaga, A., and Egner, T. (2013). Working memory as internal attention: toward an integrative account of internal and external selection processes. Psychon. Bull. Rev. 20, 228–242. doi: 10.3758/s13423-012-0359-y

Kiyonaga, A., and Egner, T. (2014). The working memory Stroop effect: when internal representations clash with external stimuli. Psychol. Sci. 25, 1619–1629. doi: 10.1177/0956797614536739

Kiyonaga, A., Egner, T., and Soto, D. (2012). Cognitive control over working memory biases of selection. Psychon. Bull. Rev. 19, 639–646. doi: 10.3758/s13423-012-0253-7

Kundu, B., Sutterer, D. W., Emrich, S. M., and Postle, B. R. (2013). Strengthened effective connectivity underlies transfer of working memory training to tests of short-term memory and attention. J. Neurosci. 33, 8705–8715. doi: 10.1523/JNEUROSCI.5565-12.2013

Kuo, B.-C., Rao, A., Lepsien, J., and Nobre, A. C. (2009). Searching for targets within the spatial layout of visual short-term memory. J. Neurosci. 29, 8032–8038. doi: 10.1523/JNEUROSCI.0952-09.2009

LaRocque, J. J., Lewis-Peacock, J. A., and Postle, B. R. (2014). Multiple neural states of representation in short-term memory? It's a matter of attention. Front. Hum. Neurosci. 8:5. doi: 10.3389/fnhum.2014.00005

Lepsien, J., Griffin, I. C., Devlin, J. T., and Nobre, A. C. (2005). Directing spatial attention in mental representations: interactions between attentional orienting and working-memory load. Neuroimage 26, 733–743. doi: 10.1016/j.neuroimage.2005.02.026

Lewandowsky, S., Oberauer, K., and Brown, G. D. A. (2009). No temporal decay in verbal short-term memory. Trends Cogn. Sci. 13, 120–126. doi: 10.1016/j.tics.2008.12.003

Nobre, A. C., Coull, J. T., Maquet, P., Frith, C. D., Vandenberghe, R., and Mesulam, M. M. (2004). Orienting attention to locations in perceptual versus mental representations. J. Cogn. Neurosci. 16, 363–373. doi: 10.1162/089892904322926700

Oberauer, K. (2009). “Chapter 2 design for a working memory,” in Psychology of Learning and Motivation, Vol. 51, (Elsevier), 45–100. Available online at: http://linkinghub.elsevier.com/retrieve/pii/S007974210951002X

Oberauer, K., and Hein, L. (2012). Attention to information in working memory. Curr. Dir. Psychol. Sci. 21, 164–169. doi: 10.1177/0963721412444727

Oberauer, K., and Lewandowsky, S. (2008). Forgetting in immediate serial recall: decay, temporal distinctiveness, or interference? Psychol. Rev. 115, 544–576. doi: 10.1037/0033-295X.115.3.544

Oberauer, K., Lewandowsky, S., Farrell, S., Jarrold, C., and Greaves, M. (2012). Modeling working memory: an interference model of complex span. Psychon. Bull. Rev. 19, 779–819. doi: 10.3758/s13423-012-0272-4

Olivers, C. N. (2009). What drives memory-driven attentional capture? The effects of memory type, display type, and search type. J. Exp. Psychol. Hum. Percept. Perform. 35, 1275. doi: 10.1037/a0013896

Olivers, C. N., Meijer, F., and Theeuwes, J. (2006). Feature-based memory-driven attentional capture: visual working memory content affects visual attention. J. Exp. Psychol. Hum. Percept. Perform. 32:1243. doi: 10.1037/0096-1523.32.5.1243

Olivers, C. N. L., Peters, J., Houtkamp, R., and Roelfsema, P. R. (2011). Different states in visual working memory: when it guides attention and when it does not. Trends Cogn. Sci. 15, 327–334. doi: 10.1016/j.tics.2011.05.004

Peters, J. C., Goebel, R., and Roelfsema, P. R. (2009). Remembered but unused: the accessory items in working memory that do not guide attention. J. Cogn. Neurosci. 21, 1081–1091. doi: 10.1162/jocn.2009.21083

Postle, B. R. (2006). Working memory as an emergent property of the mind and brain. Neuroscience 139, 23–38. doi: 10.1016/j.neuroscience.2005.06.005

Rees, G., Frith, C. D., and Lavie, N. (1997). Modulating irrelevant motion perception by varying attentional load in an unrelated task. Science 278, 1616–1619. doi: 10.1126/science.278.5343.1616

Rerko, L., Souza, A. S., and Oberauer, K. (2014). Retro-cue benefits in working memory without sustained focal attention. Mem. Cognit. 42, 712–728. doi: 10.3758/s13421-013-0392-8

Ricker, T. J., and Cowan, N. (2010). Loss of visual working memory within seconds: the combined use of refreshable and non-refreshable features. J. Exp. Psychol. Learn. Mem. Cogn. 36, 1355–1368. doi: 10.1037/a0020356

Ricker, T. J., and Cowan, N. (2014). Differences between presentation methods in working memory procedures: a matter of working memory consolidation. J. Exp. Psychol. Learn. Mem. Cogn. 40, 417–428. doi: 10.1037/a0034301

Schmidt, B. K., Vogel, E. K., Woodman, G. F., and Luck, S. J. (2002). Voluntary and automatic attentional control of visual working memory. Percept. Psychophys. 64, 754–763. doi: 10.3758/BF03194742

Soto, D., Heinke, D., Humphreys, G. W., and Blanco, M. J. (2005). Early, involuntary top-down guidance of attention from working memory. J. Exp. Psychol. Hum. Percept. Perform. 31:248. doi: 10.1037/0096-1523.31.2.248

Soto, D., Hodsoll, J., Rotshtein, P., and Humphreys, G. W. (2008). Automatic guidance of attention from working memory. Trends Cogn. Sci. 12, 342–348. doi: 10.1016/j.tics.2008.05.007

Soto, D., and Humphreys, G. W. (2008). Stressing the mind: the effect of cognitive load and articulatory suppression on attentional guidance from working memory. Percept. Psychophys. 70, 924–934. doi: 10.3758/PP.70.5.924

Soto, D., Humphreys, G. W., and Heinke, D. (2006). Working memory can guide pop-out search. Vis. Res. 46, 1010–1018. doi: 10.1016/j.visres.2005.09.008

Tsvetanov, K. A., Arvanitis, T. N., and Humphreys, G. W. (2012). Dissociating effects of stimulus identity and load on working memory attentional guidance: lengthening encoding time eliminates the effect of load but not identity. Q. J. Exp. Psychol. 65, 1475–1483. doi: 10.1080/17470218.2012.694895

Vergauwe, E., Camos, V., and Barrouillet, P. (2014). The impact of storage on processing: how is information maintained in working memory? J. Exp. Psychol. Learn. Mem. Cogn. 40, 1072–1095. doi: 10.1037/a0035779

Woodman, G. F., and Chun, M. M. (2006). The role of working memory and long-term memory in visual search. Vis. Cogn. 14, 808–830. doi: 10.1080/13506280500197397

Woodman, G. F., and Luck, S. J. (2007). Do the contents of visual working memory automatically influence attentional selection during visual search? J. Exp. Psychol. Hum. Percept. Perform. 33:363. doi: 10.1037/0096-1523.33.2.363

Woodman, G. F., Luck, S. J., and Schall, J. D. (2007). The role of working memory representations in the control of attention. Cereb. Cortex 17, i118–i124. doi: 10.1093/cercor/bhm065

Woodman, G. F., Vogel, E. K., and Luck, S. J. (2001). Visual search remains efficient when visual working memory is full. Psychol. Sci. 12, 219–224. doi: 10.1111/1467-9280.00339

Zhang, B., Zhang, J. X., Huang, S., Kong, L., and Wang, S. (2011). Effects of load on the guidance of visual attention from working memory. Vision Res. 51, 2356–2361. doi: 10.1016/j.visres.2011.09.008

Zhang, B., Zhang, J. X., Kong, L., Huang, S., Yue, Z., and Wang, S. (2010). Guidance of visual attention from working memory contents depends on stimulus attributes. Neurosci. Lett. 486, 202–206. doi: 10.1016/j.neulet.2010.09.052

Conflict of Interest Statement: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Received: 11 June 2014; Accepted: 12 August 2014; Published online: 27 August 2014.

Citation: Kiyonaga A and Egner T (2014) Resource-sharing between internal maintenance and external selection modulates attentional capture by working memory content. Front. Hum. Neurosci. 8:670. doi: 10.3389/fnhum.2014.00670

This article was submitted to the journal Frontiers in Human Neuroscience.

Copyright © 2014 Kiyonaga and Egner. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

REVIEW
published: 21 April 2015
doi: 10.3389/fnhum.2015.00197

A multisensory perspective of working memory

Michel Quak*, Raquel Elea London and Durk Talsma

Department of Experimental Psychology, Ghent University, Ghent, Belgium

* Correspondence: Michel Quak, Department of Experimental Psychology, Ghent University, Henri Dunantlaan 2, 9000, Ghent, Belgium michel.quak@ugent.be

Edited by:
Steve Majerus, Université de Liège, Belgium

Reviewed by:
Marian Berryhill, University of Nevada, Reno, USA
Candice Coker Morey, University of Edinburgh, UK
Stephen Darling, Queen Margaret University, UK

Although our sensory experience is mostly multisensory in nature, research on working memory representations has focused mainly on examining the senses in isolation. Results from the multisensory processing literature make it clear that the senses interact on a more intimate manner than previously assumed. These interactions raise questions regarding the manner in which multisensory information is maintained in working memory. We discuss the current status of research on multisensory processing and the implications of these findings on our theoretical understanding of working memory. To do so, we focus on reviewing working memory research conducted from a multisensory perspective, and discuss the relation between working memory, attention, and multisensory processing in the context of the predictive coding framework. We argue that a multisensory approach to the study of working memory is indispensable to achieve a realistic understanding of how working memory processes maintain and manipulate information.

Keywords: working memory, multisensory processing, multisensory integration, attention, top-down, bottom-up

A Multisensory Perspective of Working Memory

In everyday life we experience a continuous stream of information that we perceive through sight, sound, smell, taste, and touch. Even though this experience is mostly multisensory, that is, we receive information from multiple senses simultaneously, psychological research has primarily focused on studying our senses in isolation. While we are beginning to understand how our senses interact at various stages of processing (for an overview see, e.g., Wallace et al., 1993; Beauchamp, 2005; Ghazanfar and Schroeder, 2006; Stein and Stanford, 2008; Klemen and Chambers, 2012) it is still heavily debated whether the higher-order mental representations that are derived from these sensory inputs still contain modality- specific information or not. For instance, in working memory, research has focused on resolving whether information is memorized in the form of separate, modality or domain specific representations (Baddeley and Hitch, 1974; Schneider and Detweiler, 1988), or as integrated representations (Atkinson and Shiffrin, 1968; Cowan, 2001).

Multisensory processing refers to the interaction of signals arriving nearly simultaneously from different sensory modalities. This implies that information from one modality can influence information processing in another modality. Information from different sensory modalities can also be combined into a single multisensory event, a process that is referred to as multisensory integration (Stein et al., 2010). In accordance with the suggestions of Stein et al. (2010) we will use the terms “modality-specific” or “cross-modal” when describing the properties of objects and “unisensory” or “multisensory” when referring to neural or behavioral processes associated with a single or multiple sensory modalities.

The aim of this paper is to discuss the current status of research on multisensory processing and the implications of these findings for our theoretical understanding of working memory. To do so, we will focus on reviewing working memory research conducted from a multisensory perspective. We will argue that a multisensory approach to the study of working memory is indispensable to achieve a realistic understanding of how working memory processes maintain and manipulate information.

Working Memory and the Multisensory Brain

In their seminal work, Atkinson and Shiffrin (1968) devised a model for the flow of information in human memory, which subsequently became known as the modal model. They suggested that environmental information is processed by various modality-specific sensory registers before it is combined into a single, modality-independent, or more formally amodal, percept and transferred into a short-term store. According to this view, the short-term store is an amodal, general-purpose mechanism. Atkinson and Shiffrin referred to this mechanism as “working memory”, as it was considered to be responsible for a variety of operations, such as the selection, manipulation, and rehearsal of the memorized items.

A few years later, Baddeley and Hitch (1974) proposed a multiple-component model of working memory where information is assumed to be stored in two domain-specific subsystems (the phonological loop and the visuo-spatial sketchpad) that are directed by a general control mechanism (the central executive). The phonological loop is responsible for short-term maintenance of speech-based and acoustic items. The visuo-spatial sketchpad maintains visually and/or spatially encoded items. In contrast to Atkinson and Shiffrin’s (1968) idea of a domain-independent (i.e., amodal) store, Baddeley and Hitch (1974) assume that information (e.g., verbal or spatial) is maintained in its corresponding domain-specific store.

Over the years it has become clear that information from different domains showed more interaction in working memory than one would expect from a strongly domain-specific perspective (e.g., Jiang et al., 2000; Logie et al., 2000; Prabhakaran et al., 2000). An episodic buffer was added to Baddeley and Hitch (1974) original working memory model to account for, amongst other things, the apparent interaction between phonological and visual processes (Baddeley, 2000). The episodic buffer can be conceived as an amodal storage component, which was estimated to hold up to four chunks of information. Additionally, it was proposed to act as a link between all the other working memory components described above. For this revised model, Baddeley (2000) suggested that the episodic buffer integrates memory traces that may originate from different senses into a coherent perceptual scene.

On the basis of several studies, Postle (2006) has proposed that the brain areas involved in sensory perception are also responsible for the short-term storage of sensory information. For instance, functional magnetic resonance imaging (fMRI) studies showed object-specific memorization effects for faces in the posterior fusiform gyrus (e.g., Druzgal and D’Esposito, 2003; Ranganath et al., 2004), an area considered to be vital for face recognition. Postle and D’Esposito (1999) found activity related to memorization of visual object location and depiction in ventral temporal and occipital visual brain areas. Similarly, event-related potential (ERP) modulations can be seen in posterior and occipital recording sites during short-term memorization of visual objects contralateral to the to-be-remembered objects (e.g., Klaver et al., 1999; Vogel and Machizawa, 2004). Such findings (for an overview see, Postle, 2006; D’Esposito and Postle, 2015) indicate that memorizing modality-specific sensory information involves the same brain areas as those involved in the initial sensory processing of that information. This idea is compatible with the classical view that integration of the senses would take place at a later stage of processing, after initial unisensory processing has taken place (see Talsma, 2015, for a discussion). Indeed, using neurophysiological methods with animals (e.g., Wallace et al., 1993; Fuster et al., 2000) and fMRI with humans (e.g., Calvert et al., 2000; Wright et al., 2003; Beauchamp et al., 2004) several higher-order brain areas have been identified that seem to be dedicated to integrating information from multiple unisensory sources. Brain areas typically regarded as multisensory in the human brain can for example be found in the lateral occipital-temporal cortex, such as the superior temporal sulcus (STS; Beauchamp, 2005).

An increasing number of studies now suggest, however, that multisensory processing can already take place in brain areas that were considered to be strictly unisensory (see for a review, Foxe and Schroeder, 2005; Macaluso and Driver, 2005). For example, Giard and Peronnet (1999) found multisensory ERP effects as early as 40 ms post-stimulus over occipital scalp areas, suggesting that multisensory interactions take place much earlier than previously assumed. Using fMRI, Foxe et al. (2002) showed integration related effects of auditory and somatosensory stimuli within a region of the auditory cortex previously thought to be unisensory. This brain area was more strongly activated by multisensory stimuli than what might be expected on the basis of a mere summation of either auditory or tactile stimulation alone. Likewise, Dionne et al. (2010) found increased BOLD signal in the right primary somatosensory cortex during a delayed sensory-to-motor task for cross-modal visual-somatosensory stimuli compared to modality-specific stimuli.

These findings also have implications for the memorization of multisensory information. If indeed, as Postle (2006) proposes, the brain areas responsible for perceptual processing are the same as those involved in memorization, and if multisensory effects can already be observed in the primary sensory cortices, then we would expect that cross-modal information is stored as a unified representation in working memory. We specifically aim to focus on the questions regarding how multisensory information is encoded in working memory and whether we memorize the individual unisensory representations separately and integrate them at a later stage, or whether they are memorized as part of an integrated, multimodal representation instead.

Feature Binding in Working Memory

To fully understand the importance of considering working memory from a multisensory perspective, it is necessary to discuss how information is organized within working memory. An important question here is whether each feature of an object is remembered separately or not (e.g., Luck and Vogel, 1997; Klaver et al., 1999; Vogel et al., 2001, 2005; Wheeler and Treisman, 2002; Olsson and Poom, 2005; Luria et al., 2010; Diamantopoulou et al., 2011; Luria and Vogel, 2011). For example, Luck and Vogel (1997) used a change detection task to examine the capacity of working memory for visual objects. Participants were presented with an array of stimuli, which they had to remember during an interval without the stimuli being present. After this retention interval a second array was presented and participants responded by indicating whether any visual changes had occurred between the second and the first array. Varying the number of visual objects that need to be memorized allows estimating the capacity of visual working memory. Luck and Vogel (1997) found that capacity was limited to approximately four objects, regardless of the number of feature dimensions, or individual features that needed to be remembered per object. This led them to conclude that visual working memory has an object-based and not a feature-based organization. It is important to note that these findings have not been replicated (Oberauer and Eichenberger, 2013; Hardman and Cowan, 2015). At the very least this suggests that feature binding can, but does not always, occur automatically.

Interestingly, research has shown that an asymmetry exists in binding the visual and spatial features of an object. Multiple studies have shown that processing the visual features of an object automatically bind this object to its spatial location (e.g., Jiang et al., 2000; Olson and Marshuetz, 2005). However, processing an object’s spatial location does not result in the automatic binding of that object’s visual features (Jiang et al., 2000). While these findings show that binding of multiple features can occur within the visuo-spatial domain, other studies have shown that binding of features can even occur across domains.

Prabhakaran et al. (2000) showed that participants memorized verbal and spatial information in an integrated fashion. Participants in this study performed faster and more accurate on a verbal-spatial delayed-match-to-sample task when the probe was a letter-location combination that was presented together in the sample array compared to a letter-location combination that was presented separately. The findings on binding of verbal and spatial information have been replicated and extended in multiple studies (Bao et al., 2007; Campo et al., 2008, 2010; Elsley and Parmentier, 2009; Guérard et al., 2013; Meier et al., 2014). For example, Bao et al. (2007) found that switching attention between verbal and spatial features was faster when they were features from one object than when they were features from separate objects. Additionally, Guérard et al. (2013) showed that phonological similarity of verbal material can carry over to the recall of spatial locations in a combined verbal-spatial serial recall task. Participants were sequentially presented with letters in specific locations and were asked to either recall the order of spatial locations shown or the order of letters shown. They found that the harmful effect of phonological similarity on verbal recall carried over to spatial recall, but that the harmful effect of spatial complexity on spatial recall did not carry over to verbal recall. While the question remains under which exact circumstances automatic binding or integration of cross-domain information occurs, the asymmetry found in visual feature and location binding as well as verbal and spatial binding, suggest that the automatic integration of information across domains can occur.

Multisensory Working Memory Representations

Despite the evidence for integration of information from different domains, surprisingly little research has examined how multisensory information is represented in working memory. One of the first studies to use cross-modal stimuli was done by Thompson and Paivio (1994). Participants memorized three different types of items: visual, auditory, or audiovisual for a later free-recall test. Thompson and Paivio found an improvement of free recall of cross-modal audiovisual stimuli compared to modality-specific, audio or visual stimuli. This superior audiovisual performance was not simply due to the double presentation of information in audiovisual conditions (audio and visual dual presentation), because picture-picture and sound-sound dual presentation conditions did not yield a similar improvement. When pictures in the picture-picture dual presentation condition were two different exemplars of the same item a slight improvement in free recall was found but audiovisual performance still resulted in higher recall rates. Goolkasian and Foos (2005) also found that recall rates were higher for picture/spoken word and written/spoken word dual presentation conditions compared to the double visual presentation of pictures and written words. These findings suggest that the improved memory performance is due to the combination of information from different modalities and not because of the redundancy of the information itself.

In the multisensory literature, additive effects, such as for example linear increases of brain activity for multisensory stimuli (For an overview see; Calvert, 2001), are considered to be exemplary of multisensory processing. By contrast, in working memory research, similar additive effects, such as an increase in capacity for audiovisual material compared to modality-specific material, are considered evidence for the independence of the two modalities. For example, the advantage of cross-modal object recall, in the study of Thompson and Paivio (1994) was explained by Paivio (1971, 1986) “dual coding” theory. This theory states that a memory trace for a cross-modal stimulus is a combination of the independent sensory traces that were encoded, which in turn can be recalled separately when the task so requires. While information from different modalities can interact to provide certain behavioral benefits, this information is in fact independent.

Originally, the dual coding theory was developed to explain the independent, simultaneous processing of verbal and non-verbal information, but has later also been used to explain the independent, simultaneous processing of auditory and visual information. It is important to note that these forms of information can interact. Verbal information can be both visual (e.g., written words) and/or auditory (e.g., spoken words), and nonverbal information can also be visual (e.g., complex visual scenes) and/or auditory (e.g., white noise). We can make a distinction between the format of a working memory representation, i.e., the sensory modality in which the information is perceived and/or processed (e.g., auditory—visual), and the content of the representation, i.e., the actual information that is transferred (verbal–non-verbal). For example, when memorizing an array of blue squares or a picture of a cat, it might be more efficient to memorize this verbally as the verbal code “blue squares” or “red cat”. However, when the task requires one to describe the exact spatial location of each square, or point out a specific cat in an array of red cat pictures, it would be more efficient to use a visual code. We assume that information is processed in the format code that is most optimal for the current task. This implies that multiple format codes might be used for one and the same object, if that is more effective for memorizing that object.

Delogu et al. (2009) investigated how verbal and non-verbal auditory, visual, and audiovisual material is encoded in working memory. Participants were tested on immediate serial recall for sequentially presented visual, auditory, or audiovisual stimuli in either a non-verbal or verbal condition. In the non-verbal condition, stimuli were either pictures, environmental sounds, or a combination of both, and in the verbal condition, stimuli were either written words, spoken words, or a combination of both. Results showed that in the non-verbal condition serial recall for audiovisual stimuli was higher than recall for auditory or visual stimuli. In the verbal condition, recall for audiovisual material was still higher than recall for visual material, but auditory and audiovisual recall did not differ. The authors also found that preventing participants from articulating reduced memory performance in both the verbal and non-verbal conditions. This suggests that both in the verbal and in the non-verbal presentation conditions, the actual content of the representation was encoded in a verbal code. Furthermore, the verbal content seemed to play a key part in memorizing the stimuli in all conditions. This shows that the format in which information is presented is not necessarily the format in which the information is encoded. For example, when a participant is presented with an auditory stimulus of a meowing cat, it is possible that this sound calls forth a picture of a cat, or the word “cat”, which is then kept in working memory instead of the auditory features of the original meowing sound that was presented. It is a requirement that the participant recognizes the presented sound as the meowing produced by a cat in order to “recode” the sound into a visual or verbal representation. This requires semantic information from long term memory to be integrated with the working memory representation. Delogu et al. (2009) concluded that their findings are compatible with Baddeley’s (2000) working memory model where the existence of an episodic buffer integrates information from different modalities and combines this with semantic information from long term memory. Other studies have also shown the influence of semantic information from long term memory on visual working memory object representations (e.g., Olsson and Poom, 2005; Diamantopoulou et al., 2011) suggesting that information outside the pure visual domain can affect early visual object working memory. Similarly, Darling et al. (2012) found that accuracy on a digit serial recall task improved when the locations of presented digits matched the spatial configuration of a typical, numeral keypad (as found on a telephone or television remote) in a process they call visuospatial bootstrapping. They confirmed that this effect was due to the integration of the typical keypad representation from long-term memory with the working memory representation and not only to the binding between verbal and spatial information.

Thus far, the main goal of the studies discussed above was to provide insights into the dual code theory (Paivio, 1971, 1986) and/or the multiple component theory (Baddeley and Hitch, 1974; Baddeley, 2000) mainly by looking at recall performance for a wide variety of stimuli. To better understand how multisensory information interacts in working memory we can look at working memory capacity for cross-modal objects. As mentioned before, estimates of working memory capacity for features and objects have been used to infer that visual working memory representations are object based (Luck and Vogel, 1997). Likewise, by assuming that not only features within a modality but also across modalities are integrated into object representations, examining the number of cross-modal objects one can hold in memory compared to modality-specific objects could give insight into the organization of multisensory working memory. For instance, Saults and Cowan (2007) found that working-memory capacity for audiovisual material can exceed working-memory capacity for modality-specific material under certain conditions. In a series of five experiments, participants were presented with visual arrays of four to eight colored squares and auditory arrays of four spoken digits. They were instructed to memorize the visual array, the auditory array, or both. Interestingly, the performance advantage for audiovisual arrays disappeared when masks were used to block access to previously formed sensory memory traces. In this case, capacity for cross-modal stimuli was as high as the capacity of the highest modality-specific object, indicating that memory traces from an accessory sensory memory (echoic and/or iconic memory) contributed to the improvement of task performance. Since auditory and visual information did not additively contribute to memory performance when sensory memory traces were excluded, Saults and Cowan (2007) concluded that auditory and visual information share a common storage. Fougnie and Marois (2011) contested this interpretation by arguing that the formula used by Saults and Cowan (2007) to estimate the maximum number of object representations one can hold in working memory, might not adequately reflect the combined capacity of modality-specific stores. Fougnie and Marois argued that one item of auditory information generally places a larger load on memory than one item of visual information, suggesting that these modality-specific differences should be weighted accordingly in such a capacity estimate. Using an adapted formula in a series of three experiments, they found that even when using masks to exclude contributions of sensory memory traces, capacity for cross-modal items was superior to the capacity for modality-specific items. Contrary to Saults and Cowan (2007), they concluded that auditory and visual objects were stored in their own respective stores and contributed to performance without interfering.

Overall, there seems to be a performance benefit for the memorization of audiovisual stimuli compared to the memorization of modality-specific stimuli. It remains under debate, however, whether this benefit exists because these stimuli are integrated into a new amodal representation or because the independent storage of auditory and visual information contributes to performance in an additive fashion because they do not interfere. At this time the same effect is used to argue for both sides of the debate. Where some see the additive performance of audiovisual objects as proof for an interaction or even integration of information in working memory (e.g., Delogu et al., 2009), others see it as proof that sensory information is memorized in its own separate store (e.g., Fougnie and Marois, 2011).

In addition to examining performance benefits for the combination of auditory and visual processing, we can also study the disruption of processing for the combination of auditory and visual information. In traditional working memory research, interference paradigms have been used to show a double dissociation between two separate processing mechanisms. Meaning that when two processes use the same underlying system, interference will occur which impairs performance on both processes. The disruption of performance between modalities is referred to as cross-modal interference and would suggest that information from the different modalities interact at a certain level. For multisensory working memory this could mean that information from different modalities is maintained in a single, multisensory store. Evidence for cross-modal interference is still somewhat ambiguous, however. For instance, using a visual-pattern-recall and auditory-digit-recall dual task, Cocchini et al. (2002) did not find evidence for cross-modal interference on performance accuracy in working memory. The absence of such interference suggests that working memory operates in a domain-specific manner and is in accordance with the notion of parallel processing without interaction of information from different modalities. In contrast, Goolkasian and Foos (2005) showed that spoken words could interfere with the recall of pictures and written words when using long sequences of incongruent dually presented items. Likewise, Morey and Cowan (2004, 2005), did find cross-modal interference on performance accuracy when memory load was sufficiently high. They examined digit span using a verbal-visual dual task and found that participants showed interference for visual memory recall but only when the verbal load was sufficiently high (a load of 7 digits instead of 2). The interference patterns observed in audio-visual dual tasks are as of yet inconclusive on whether visual and auditory information share a limited capacity storage. Although interference paradigms could give us an answer on the question of whether information from different modalities share a limited capacity storage or not, they cannot answer whether the information from different modalities is integrated in this single storage, or maintained as independent modality-specific traces.

Thus far, research on multisensory working memory has shown that recall is better for cross-modal objects compared to modality-specific objects (Thompson and Paivio, 1994; Goolkasian and Foos, 2005; Delogu et al., 2009), working memory capacity is higher for cross-modal objects under certain circumstances (Saults and Cowan, 2007; Fougnie and Marois, 2011), and visual and auditory information can interfere with each other (Morey and Cowan, 2004, 2005; Goolkasian and Foos, 2005) but not always (Cocchini et al., 2002). Although a performance benefit for cross-modal objects is seen as evidence for integration in multisensory research, in working memory research it has traditionally been seen as evidence that modality-specific information from cross-modal objects is stored in separate stores. While we cannot definitively conclude that cross-modal objects are stored as fully integrated objects in working memory, it is apparent that cross-modal information interacts in working memory beyond what would be expected from modality-specific stores. The question is: at what stage or stages in the processing stream do these interactions occur?

Multisensory Processing, Selective Attention, and Working Memory

To answer this question we turn to research on multisensory processing and selective attention. The insights gained from this research could also inform questions about working memory for multisensory stimuli. In fact, more and more researchers have challenged the idea that working memory and attention are two separate systems (Cowan, 2001; Awh et al., 2006; Olivers, 2008; Oberauer and Hein, 2012; Kiyonaga and Egner, 2013; Klaver and Talsma, 2013). For example, Olivers (2008) reviews evidence for the notion that working memory and attention share the same capacity, content and control processes, suggesting they might be two aspects of the same process. Likewise, Kiyonaga and Egner (2013) discuss the literature that examined the effects of external attention on working memory representations, as well as, the effects of working memory representations on directing selective attention. These studies indicate that a competitive interaction between working memory and selective attention exists, implying that they share a limited resource. Kiyonaga and Egner (2013) state that attention and working memory should no longer be regarded as two separate concepts, but instead as one concept, where attention can be directed externally (selective attention) and/or internally (working memory). The idea of working memory as internal attention is in line with Cowan’s (2001) original idea of working memory where a capacity limited focus of attention can shift between different levels of processing.

Given the above mentioned observations that working memory and attention are presumably two different aspects of the same underlying process, and considering that several studies have shown close ties between attention and multisensory processing, it is necessary to understand the implications of these ties for working memory. Instances where multisensory events guide and focus attention (also referred to as bottom-up effects) suggest an early integration of multisensory information, while instances where attention is needed for multisensory integration (also referred to as top-down effects) are indicative of late integration processes. There is evidence for both types of interaction between multisensory integration and attention. Factors that determine the predominance of either early and/or late interactions between information from different modalities are for example, task-relevancy (e.g., Busse et al., 2005), learned associations (e.g., Molholm et al., 2007), and saliency (e.g., Van der Burg et al., 2008).

An example of top-down influence of attention on multisensory integration was given by Talsma and colleagues (e.g., Senkowski et al., 2005; Talsma and Woldorff, 2005; Talsma et al., 2007). Using a rapid succession of task-relevant and irrelevant stimuli, they found that attention could influence the integration of cross-modal stimuli. Similarly, Alsius and colleagues (Alsius et al., 2005, 2007) have shown that attending elsewhere diminishes participants susceptibility to the McGurk illusion (McGurk and MacDonald, 1976). Based on these findings it appears that attending to the relevant, to-be-integrated stimuli is necessary to build a robust, integrated representation (Talsma et al., 2010).

However, evidence for bottom-up modulation of attention by multisensory integration has made it clear that multisensory processing can already happen in very early stages of perception (Giard and Peronnet, 1999; Molholm et al., 2002; Van der Burg et al., 2011). For instance, Van der Burg et al. (2011) presented dynamic displays consisting of line elements that randomly changed orientation. When a target orientation change was synchronized with a short, spatially uninformative tone, visual search was strongly facilitated as compared to when the tone was absent. The interpretation given to these results was that the tone and the synchronized orientation change were bound together into one coherent event, thereby forming a cross-modal singleton that “popped out” between the non-synchronized visual distractors. EEG data showed that this multisensory benefit was apparent as early as 50 ms post-stimulus onset and that the strength of this effect predicted the magnitude of the behavioral benefit during visual search, due to the auditory signal.

The findings above imply that both top-down (task-relevance and learned associations) as well as bottom-up (salience) processes are involved in multisensory integration. To resolve this apparent contradiction between a bottom-up view of multisensory processing, where early multisensory effects seem to drive attention, and a top-down view of multi-sensory processing, where attention seems to be required to integrate cross-modal objects, Talsma et al. (2010) proposed a unified framework of attention and multisensory processing. According to this framework, early pre-attentive processes can bind multisensory inputs together, but only when competition among the individual inputs is low. Thus, the early latency processes serve to cross-feed low-level information between the individual sensory cortices involved in the integration processes. Early interactions might serve to realign auditory and visual input signals. Auditory information might give temporal information to visual cortex whereas visual information might provide spatial information to auditory processing.

This pre-attentive early integration would, according to Talsma et al. (2010), only be possible, however, if the stimuli presented in one modality do not need to compete for processing capacity with other stimuli in that same modality. If there is competition among multiple stimuli in one modality, top-down attentional control may be required to filter out any stimulus that is not task relevant, thereby prioritizing those stimuli that are task relevant. Consistent with this view, Van der Burg et al. (2012) found that the earlier mentioned automatic capture by a synchronized cross-modal event can be modulated by the size of the attentional window, meaning that when participants were less focused the effect of the cross-modal pop out was stronger than when participants were forced to focus on a small cue before the synchronized cross-modal event. In conclusion, stimulus-driven, bottom-up processes can automatically capture attention towards multisensory events. Top-down attention can in turn facilitate the integration of multisensory information which leads to a spread of attention across sensory modalities.

Based on the previously mentioned idea that external attention and internal attention (working memory) are two aspects of the same process, findings in attentional research could be applied to working memory. It has been shown that spatial attention can actively influence working memory representations by facilitating encoding (Uncapher et al., 2011) and improving the recall of memorized representations (Murray et al., 2013). These effects are found not only within a single modality, but also across modalities. For instance, an auditory cue can draw attention to a visual object and vice versa (Spence and Driver, 1997; Koelewijn et al., 2009). Similar effects for working memory have been found by Botta et al. (2011). They examined the effect of visual, auditory, and audiovisual cues on working memory for arrays of colored squares in a change detection task. The cross-modal and modality-specific cues could either capture attention towards the hemifield which contained the to-be-remembered objects, or towards the opposite hemifield which contained the to-be-ignored objects. They found that audiovisual cues had a larger influence on performance accuracy than modality-specific visual or auditory cues. Memory accuracy was increased when an audiovisual cue was presented on the same side as the target and it was decreased when the audiovisual cue was presented on the opposite side. Both the facilitation and impairment of memory performance was larger for audiovisual cues compared to visual cues. Although these data do not directly address the question of how a cross-modal object is represented in working memory as such, they do tell us that multisensory information has a bottom-up effect on the subsequent memorization of a unisensory object.

Investigation of top-down effects of working memory on attention has revealed that working memory content can affect the allocation of visual selective attention (Olivers et al., 2006). In a multisensory context, Murray et al. (2004) found that discrimination accuracy of visual objects, presented 20 s after initial presentation, improved when the initial presentation was a picture-sound combination compared to a unisensory picture. EEG data revealed that the neuronal response to a cross-modal stimulus happened as fast as 60–136 ms and predominantly influenced activation in the right lateral occipital complex. Where a semantically congruent picture-sound combination increased discrimination accuracy on a second presentation, a pure tone decreased discrimination accuracy on a second presentation (Lehmann and Murray, 2005; Thelen et al., 2012). Thelen et al. (2015) replicate these earlier findings, while also showing the same effects in the auditory modality. Single-trial multisensory memories affect later auditory recognition. If cross-modal objects were congruent (visual and auditory information match semantically) accuracy was higher compared to unisensory stimuli but became worse if objects were incongruent or meaningless. Unisensory percepts seem to trigger the multisensory representations associated with them, suggesting at least a partially integrated storage in memory. Yet, it seems a multisensory representation stored in memory is only beneficial for memory performance when sounds and pictures are semantically congruent. These studies show that an internal representation is formed in which both the visual and auditory information is encoded. Moreover, they also indicate that information presented in a task irrelevant modality interferes with the task relevant representation. But although this still does not address the question of whether unisensory information is still accessible it does show that the original unisensory representations are closely related. Similar to the findings in research on attention and multisensory integration, it seems that top-down and bottom-up processes play an important part in the integration of cross-modal information in working memory representations.

Predictive Coding and Multisensory Working Memory

One influential framework that can explain the intricacies of top-down and bottom-up interactions in multisensory memory is that of predictive coding. The predictive coding framework states that the brain produces a Bayesian estimate of the environment (Friston, 2010). According to this view, stochastic models of the environment exist in the brain, which are continuously updated on the basis of processed sensory information. Higher-order brain areas thus provide the lower areas with predictions (or in Bayesian terms “priors”) that influence the processing of ongoing sensory input. A strong mismatch between the prediction and the actual sensory input will then result in a major update of the internal model. Thus when we are in a complex environment with many stimuli competing for processing capacity, incongruence between the top-down predictions of the environment and the present incoming environmental information can determine the priority with which incoming stimuli need to be processed and integrated. The processed information changes the predictions and vice versa. Bottom-up sensory processing and top-down predictions mutually define each other continuously. In this way, the predictive coding view can explain how top-down and bottom-up processes interact in multisensory integration.

Talsma (2015) recently argued that the dynamic model of our environment provided by the aforementioned stochastic representations is essential to understanding the interaction between basic (multi)sensory processing on the one hand, and memory and attention on the other. For instance, Vetter et al. (2014) showed that actual auditory stimulation as well as imagined sounds could activate the visual cortex. Based on the predictive coding framework, these authors argued that visual cortex activation came about because either direct sensory information or a stored memory representation thereof could update the internal representation of the sound and therefore indirectly influence processing in visual cortex accordingly. This suggests that attention, memory, and multisensory processing are intrinsically intertwined. Similarly, Berger and Ehrsson (2013, 2014) showed that imagined sounds can mimic the effects of actual sounds in a number of well-known multisensory illusions, such as the bounce-pass illusion (Sekuler et al., 1997), the McGurk effect (McGurk and MacDonald, 1976), or the ventriloquist illusion (Howard and Templeton, 1966), and show independent of each other that visual cortex can be activated both by multisensory stimulation and by memory. Based on these findings, Talsma (2015) argued that despite the fact that several studies showed that auditory and visual inputs can interact at very early processing stages, the actual integration of the sensory inputs into a coherent mental representation occurs at later, higher-order processing stages.

An important consequence of applying the predictive coding framework is that our internal representation is assumed not only to be built on the basis of direct sensory input, but that it is also updated (and made consistent with) information stored in memory. Thus, attention is assumed to play an essential role in regulating how our sensory input is combined with these pre-existing representations stored in long-term memory. This is largely consistent with Cowan’s (2001) idea of the focus of attention, which is a part of activated long-term memory, as well as with Baddeley’s (2000) episodic buffer, although, Baddeley recently argued that attention in the form of the central executive was not necessary for the integration of multiple sources of information in the episodic buffer (Baddeley et al., 2011).

A further consequence of applying the predictive coding framework is that the internal representation is by definition always multisensory. Moreover, the active representation integrates all possible sources of information, including semantic information from long-term memory. Thus, even when only a unisensory stimulus is presented, associated representations will be activated as well. These can include information from other modalities, prior experience with the stimulus, or learned associations. Because the formation of this internal mental representation is an active process that influences ongoing processes in the sensory cortices, this model can explain why memory traces in one modality can be strengthened or corrupted by traces in another one. Furthermore, because the active representation sends feedback information to the low-level processes in sensory cortices it can be assumed that the original unisensory memory traces are still present albeit in a relatively fragile state.

Multisensory Working Memory Representations in Current Models

The active internal environmental model as proposed by the predictive coding framework would be akin to what we would describe as a multisensory working memory representation. This memory representation does not only consist of information coming from different modalities but also includes information from long-term memory such as semantic knowledge or learned associations. Taking the previously mentioned example of memorizing a cat picture the multisensory representation includes not only the visual features of the cat, but also long-term semantic knowledge of cats, autobiographical knowledge (previous personal experience with cats), and information from modalities not presented with the picture (the sound a cat makes or the knowledge that its fur is soft to the touch). We assume that working memory has an amodal central storage component. Whether this is the main component of working memory as suggested by Cowan (2001) or a part of a bigger system like the episodic buffer in (Baddeley’s, 2000; Baddeley et al., 2011) remains a point for further investigation.

The predictive coding framework would suggest that incoming sensory information is constantly used to update the internal environmental model, implying that incoming stimuli tend to integrate into a coherent multisensory representation. This framework can also explain why working memory is amodal in some cases and modality specific in others. For instance, Postle (2006) argued that working memory for modality-specific stimuli occurs in the sensory cortices. Recently, Yonelinas (2013) suggested that high-resolution bindings are stored in the hippocampus that can be used to support perception and working memory, specifically in memorizing (combinations of) complex features. In the latter case it is plausible that the multisensory representation will be activated, whereas in the former case it is not. Based on this, one important implication of the predictive coding approach is that differences in task and stimulus complexity can yield rather drastically different outcomes. With this in mind a recommendation for future research would be to consider effects of task and stimulus complexity on working memory activation.

Based on the above mentioned framework, we assume that sensory cortices can retain small amounts of modality-specific information (as suggested by Postle, 2006) and that this information supports a multisensory memory representation in higher order areas (e.g., the hippocampus; Yonelinas, 2013). Whether working memory for a specific task involves the higher-order areas or the sensory areas to retain information for limited time depends on the task and the information that needs to be memorized. For example, simple flashes and beeps could be retained in the sensory areas, whereas more complex information would also require the higher-order areas. In that sense the sensory cortices would retain information in a manner similar to separate slave systems (Baddeley and Hitch, 1974) or the recently suggested peripheral storage (Cowan et al., 2014).

Summary and Conclusions

In this paper we have reviewed recent developments in multisensory working memory research. Research has shown that cross-modal information interacts in working memory beyond what would be expected from the traditional modality-specific stores. Recall is better for cross-modal objects compared to modality-specific objects (Thompson and Paivio, 1994; Goolkasian and Foos, 2005; Delogu et al., 2009), working memory capacity can be higher for cross-modal objects than for unimodal objects (Saults and Cowan, 2007; Fougnie and Marois, 2011), and visual and auditory memory can interfere with each other (Morey and Cowan, 2004, 2005; Goolkasian and Foos, 2005). Furthermore, multisensory information has an effect on the subsequent memorization of a unisensory object (Botta et al., 2011) and multisensory memory representations can influence subsequent unisensory stimulus discrimination (Murray et al., 2004; Lehmann and Murray, 2005; Thelen et al., 2012, 2015). Taken together, these studies show that sensory representations in multiple modalities interact more with each other than can be explained by classical modal models.

Paivio’s (1971, 1986) dual coding theory states that although cross-modal information can interact it is in fact independent, because modality-specific information can still be retrieved in isolation. However, studies done by Thelen and colleagues (Thelen et al., 2012, 2015) show that this retrieval of modality-specific information from a cross-modal representation is more difficult than assumed, because a task irrelevant modality interferes with the task relevant representation. Moreover, higher-order representations of the external world built from memorized information have been shown to influence visual processing. Complex representations seem to be formed in working memory, consisting of the integration of several independent representations that can be sensory, and short- or long-term memory activations. Depending on task requirements either just the simple modal representation or the complex high-resolution binding of several features at once will become active. Therefore, we conclude that working memory is in essence multisensory, and that this must be taken into account to achieve a realistic understanding of how working memory processes maintain and manipulate information.

Acknowledgments

We would like to thank the editor and reviewers for their helpful comments and suggestions on earlier drafts of this paper.

References

Alsius, A., Navarra, J., Campbell, R., and Soto-Faraco, S. (2005). Audiovisual integration of speech falters under high attention demands. Curr. Biol. 15, 839–843. doi: 10.1016/j.cub.2005.03.046

Alsius, A., Navarra, J., and Soto-Faraco, S. (2007). Attention to touch weakens audiovisual speech integration. Exp. Brain Res. 183, 399–404. doi: 10.1007/s00221-007-1110-1

Atkinson, R. C., and Shiffrin, R. M. (1968). Human memory: a proposed system and its control processes. Psychol. Learn. Motiv. 2, 89–195. doi: 10.1016/s0079-7421(08)60422-3

Awh, E., Vogel, E. K., and Oh, S. H. (2006). Interactions between attention and working memory. Neuroscience 139, 201–208. doi: 10.1016/j.neuroscience.2005.08.023

Baddeley, A. (2000). The episodic buffer: a new component of working memory?. Trends Cogn. Sci. 4, 417–423. doi: 10.1016/s1364-6613(00)01538-2

Baddeley, A. D., Allen, R. J., and Hitch, G. J. (2011). Binding in visual working memory: the role of the episodic buffer. Neuropsychologia 49, 1393–1400. doi: 10.1016/j.neuropsychologia.2010.12.042

Baddeley, A. D., and Hitch, G. (1974). Working memory. Psychol. learn. Motiv. 8, 47–89. doi: 10.1016/S0079-7421(08)60452-1

Bao, M., Li, Z. H., and Zhang, D. R. (2007). Binding facilitates attention switching within working memory. J. Exp. Psychol. Learn. Mem. Cogn. 33, 959–969. doi: 10.1037/0278-7393.33.5.959

Beauchamp, M. S. (2005). See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex. Curr. Opin. Neurobiol. 15, 145–153. doi: 10.1016/j.conb.2005.03.011

Beauchamp, M. S., Lee, K. E., Argall, B. D., and Martin, A. (2004). Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41, 809–823. doi: 10.1016/s0896-6273(04)00070-4

Berger, C. C., and Ehrsson, H. H. (2013). Mental imagery changes multisensory perception. Curr. Biol. 23, 1367–1372. doi: 10.1016/j.cub.2013.06.012

Berger, C. C., and Ehrsson, H. H. (2014). The fusion of mental imagery and sensation in the temporal association cortex. J. Neurosci. 34, 13684–13692. doi: 10.1523/JNEUROSCI.0943-14.2014

Botta, F., Santangelo, V., Raffone, A., Sanabria, D., Lupiáñez, J., and Belardinelli, M. O. (2011). Multisensory integration affects visuo-spatial working memory. J. Exp. Psychol. Hum. Percept. Perform. 37, 1099–1109. doi: 10.1037/a0023513

Busse, L., Roberts, K. C., Crist, R. E., Weissman, D. H., and Woldorff, M. G. (2005). The spread of attention across modalities and space in a multisensory object. Proc. Natl. Acad. Sci. U S A 102, 18751–18756. doi: 10.1073/pnas.0507704102

Calvert, G. A. (2001). Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb. Cortex 11, 1110–1123. doi: 10.1093/cercor/11.12.1110

Calvert, G. A., Campbell, R., and Brammer, M. J. (2000). Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr. Biol. 10, 649–657. doi: 10.1016/s0960-9822(00)00513-3

Campo, P., Maestú, F., Capilla, A., Morales, M., Fernández, S., del Río, D., et al. (2008). Temporal dynamics of parietal activity during word-location binding. Neuropsychology 22, 85–99. doi: 10.1037/0894-4105.22.1.85

Campo, P., Poch, C., Parmentier, F. B., Moratti, S., Elsley, J. V., Castellanos, N. P., et al. (2010). Oscillatory activity in prefrontal and posterior regions during implicit letter-location binding. Neuroimage 49, 2807–2815. doi: 10.1016/j.neuroimage.2009.10.024

Cocchini, G., Logie, R. H., Della Sala, S., MacPherson, S. E., and Baddeley, A. D. (2002). Concurrent performance of two memory tasks: evidence for domain-specific working memory systems. Mem. Cognit. 30, 1086–1095. doi: 10.3758/bf03194326

Cowan, N. (2001). The magical number 4 in short-term memory: a reconsideration of mental storage capacity. Behav. Brain Sci. 24, 87–114; discussion 114–185. doi: 10.1017/s0140525x01003922

Cowan, N., Saults, J. S., and Blume, C. L. (2014). Central and peripheral components of working memory storage. J. Exp. Psychol. Gen. 143, 1806–1836. doi: 10.1037/a0036814

Darling, S., Allen, R. J., Havelka, J., Campbell, A., and Rattray, E. (2012). Visuospatial bootstrapping: long-term memory representations are necessary for implicit binding of verbal and visuospatial working memory. Psychon. Bull. Rev. 19, 258–263. doi: 10.3758/s13423-011-0197-3

Delogu, F., Raffone, A., and Belardinelli, M. O. (2009). Semantic encoding in working memory: is there a (multi) modality effect?. Memory 17, 655–663. doi: 10.1080/09658210902998054

D’Esposito, M., and Postle, B. R. (2015). The cognitive neuroscience of working memory. Annu. Rev. Psychol. 66, 115–142. doi: 10.1146/annurev-psych-010814-015031

Diamantopoulou, S., Poom, L., Klaver, P., and Talsma, D. (2011). Visual working memory capacity and stimulus categories: a behavioral and electrophysiological investigation. Exp. Brain Res. 209, 501–513. doi: 10.1007/s00221-011-2536-z

Dionne, J. K., Meehan, S. K., Legon, W., and Staines, W. R. (2010). Crossmodal influences in somatosensory cortex: interaction of vision and touch. Hum. Brain Mapp. 31, 14–25. doi: 10.1002/hbm.20841

Druzgal, T. J., and D’Esposito, M. (2003). Dissecting contributions of prefrontal cortex and fusiform face area to face working memory. J. Cogn. Neurosci. 15, 771–784. doi: 10.1162/089892903322370708

Elsley, J. V., and Parmentier, F. B. (2009). Is verbal-spatial binding in working memory impaired by a concurrent memory load?. Q. J. Exp. Psychol. (Hove) 62, 1696–1705. doi: 10.1080/17470210902811231

Fougnie, D., and Marois, R. (2011). What limits working memory capacity? Evidence for modality-specific sources to the simultaneous storage of visual and auditory arrays. J. Exp. Psychol. Learn. Mem. Cogn. 37, 1329–3141. doi: 10.1037/a0024834

Foxe, J. J., and Schroeder, C. E. (2005). The case for feedforward multisensory convergence during early cortical processing. Neuroreport 16, 419–423. doi: 10.1097/00001756-200504040-00001

Foxe, J. J., Wylie, G. R., Martinez, A., Schroeder, C. E., Javitt, D. C., Guilfoyle, D., et al. (2002). Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study. J. Neurophysiol. 88, 540–543. doi: 10.10.1152/jn.00694.2001

Friston, K. (2010). The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11, 127–138. doi: 10.1038/nrn2787

Fuster, J., Bodner, M., and Kroger, J. (2000). Cross-modal and cross-temporal association in neurons of frontal cortex. Nature 405, 347–351. doi: 10.1038/35012613

Ghazanfar, A. A., and Schroeder, C. E. (2006). Is neocortex essentially multisensory? Trends Cogn. Sci. 10, 278–285. doi: 10.1016/j.tics.2006.04.008

Giard, M. H., and Peronnet, F. (1999). Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J. Cogn. Neurosci. 11, 473–490. doi: 10.1162/089892999563544

Goolkasian, P., and Foos, P. W. (2005). Bimodal format effects in working memory. Am. J. Psychol. 118, 61–77.

Guérard, K., Morey, C. C., Lagacé, S., and Tremblay, S. (2013). Asymmetric binding in serial memory for verbal and spatial information. Mem. Cognit. 41, 378–391. doi: 10.3758/s13421-012-0275-4

Hardman, K. O., and Cowan, N. (2015). Remembering complex objects in visual working memory: do capacity limits restrict objects or features?. J. Exp. Psychol. Learn. Mem. Cogn. 41, 325–347. doi: 10.1037/xlm0000031

Howard, I. P., and Templeton, W. B. (1966). Human Spatial Orientation. Oxford, England: John Wiley and Sons.

Jiang, Y., Olson, I. R., and Chun, M. M. (2000). Organization of visual short-term memory. J. Exp. Psychol. Learn. Mem. Cogn. 26, 683–702. doi: 10.1037/0278-7393.26.3.683

Kiyonaga, A., and Egner, T. (2013). Working memory as internal attention: toward an integrative account of internal and external selection processes. Psychon. Bull. Rev. 20, 228–242. doi: 10.3758/s13423-012-0359-y

Klaver, P., Smid, H. G., and Heinze, H. J. (1999). Representations in human visual short-term memory: an event-related brain potential study. Neurosci. Lett. 268, 65–68. doi: 10.1016/s0304-3940(99)00380-8

Klaver, P., and Talsma, D. (2013). Behind the scenes: how visual memory load biases selective attention during processing of visual streams. Psychophysiology 50, 1133–1146. doi: 10.1111/psyp.12126

Klemen, J., and Chambers, C. D. (2012). Current perspectives and methods in studying neural mechanisms of multisensory interactions. Neurosci. Biobehav. Rev. 36, 111–133. doi: 10.1016/j.neubiorev.2011.04.015

Koelewijn, T., Bronkhorst, A., and Theeuwes, J. (2009). Auditory and visual capture during focused visual attention. J. Exp. Psychol. Hum. Percept. Perform. 35, 1303–1315. doi: 10.1037/a0013901

Lehmann, S., and Murray, M. M. (2005). The role of multisensory memories in unisensory object discrimination. Brain Res. Cogn. Brain Res. 24, 326–334. doi: 10.1016/j.cogbrainres.2005.02.005

Logie, R. H., Della Sala, S., Wynn, V., and Baddeley, A. D. (2000). Visual similarity effects in immediate verbal serial recall. Q. J. Exp. Psychol. A 53, 626–646. doi: 10.1080/027249800410463

Luck, S. J., and Vogel, E. K. (1997). The capacity of visual working memory for features and conjunctions. Nature 390, 279–281. doi: 10.1038/36846

Luria, R., Sessa, P., Gotler, A., Jolicoeur, P., and Dell’Acqua, R. (2010). Visual short-term memory capacity for simple and complex objects. J. Cogn. Neurosci. 22, 496–512. doi: 10.1162/jocn.2009.21214

Luria, R., and Vogel, E. K. (2011). Shape and color conjunction stimuli are represented as bound objects in visual working memory. Neuropsychologia 49, 1632–1639. doi: 10.1016/j.neuropsychologia.2010.11.031

Macaluso, E., and Driver, J. (2005). Multisensory spatial interactions: a window onto functional integration in the human brain. Trends Neurosci. 28, 264–271. doi: 10.1016/j.tins.2005.03.008

McGurk, H., and MacDonald, J. (1976). Hearing lips and seeing voices. Nature 264, 746–748. doi: 10.1038/264746a0

Meier, T. B., Nair, V. A., Meyerand, M. E., Birn, R. M., and Prabhakaran, V. (2014). The neural correlates of age effects on verbal-spatial binding in working memory. Behav. Brain Res. 266, 146–152. doi: 10.1016/j.bbr.2014.03.005

Molholm, S., Martinez, A., Shpaner, M., and Foxe, J. J. (2007). Object-based attention is multisensory: co-activation of an object’s representations in ignored sensory modalities. Eur. J. Neurosci. 26, 499–509. doi: 10.1111/j.1460-9568.2007.05668.x

Molholm, S., Ritter, W., Murray, M. M., Javitt, D. C., Schroeder, C. E., and Foxe, J. J. (2002). Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res. Cogn. Brain Res. 14, 115–128. doi: 10.1016/s0926-6410(02)00066-6

Morey, C. C., and Cowan, N. (2004). When visual and verbal memories compete: evidence of cross-domain limits in working memory. Psychon. Bull. Rev. 11, 296–301. doi: 10.3758/bf03196573

Morey, C. C., and Cowan, N. (2005). When do visual and verbal memories conflict? The importance of working-memory load and retrieval. J. Exp. Psychol. Learn. Mem. Cogn. 31, 703–713. doi: 10.1037/0278-7393.31.4.703

Murray, M. M., Michel, C. M., Grave de Peralta, R., Ortigue, S., Brunet, D., Gonzalez Andino, S., et al. (2004). Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. Neuroimage 21, 125–135. doi: 10.1016/j.neuroimage.2003.09.035

Murray, A. M., Nobre, A. C., Clark, I. A., Cravo, A. M., and Stokes, M. G. (2013). Attention restores discrete items to visual short-term memory. Psychol. Sci. 24, 550–556. doi: 10.1177/0956797612457782

Oberauer, K., and Eichenberger, S. (2013). Visual working memory declines when more features must be remembered for each object. Mem. Cognit. 41, 1212–1227. doi: 10.3758/s13421-013-0333-6

Oberauer, K., and Hein, L. (2012). Attention to information in working memory. Curr. Dir. Psychol. Sci. 21, 164–169. doi: 10.1177/0963721412444727

Olivers, C. N. L. (2008). Interactions between visual working memory and visual attention. Front. Biosci. 13, 1182–1191. doi: 10.2741/2754

Olivers, C. N. L., Meijer, F., and Theeuwes, J. (2006). Feature-based memory-driven attentional capture: visual working memory content affects visual attention. J. Exp. Psychol. Hum. Percept. Perform. 32, 1243–1265. doi: 10.1037/0096-1523.32.5.1243

Olson, I. R., and Marshuetz, C. (2005). Remembering “what” brings along “where” in visual working memory. Percept. Psychophys. 67, 185–194. doi: 10.3758/bf03206483

Olsson, H., and Poom, L. (2005). Visual memory needs categories. Proc. Natl. Acad. Sci. U S A 102, 8776–8780. doi: 10.1073/pnas.0500810102

Paivio, A. (1971). Imagery and Verbal Processes. New York: Holt, Rinehart and Winston.

Paivio, A. (1986). Mental Representation: A Dual Coding Approach. New York: Oxford University Press.

Postle, B. R. (2006). Working memory as an emergent property of the mind and brain. Neuroscience 139, 23–38. doi: 10.1016/j.neuroscience.2005.06.005

Postle, B. R., and D’Esposito, M. (1999). Dissociation of human caudate nucleus activity in spatial and nonspatial working memory: an event-related fMRI study. Brain Res. Cogn. Brain Res. 8, 107–115. doi: 10.1016/s0926-6410(99)00010-5

Prabhakaran, V., Narayanan, K., Zhao, Z., and Gabrieli, J. D. E. (2000). Integration of diverse information in working memory within the frontal lobe. Nat. Neurosci. 3, 85–90. doi: 10.10.1038/71156

Ranganath, C., DeGutis, J., and D’Esposito, M. (2004). Category-specific modulation of inferior temporal activity during working memory encoding and maintenance. Brain Res. Cogn. Brain Res. 20, 37–45. doi: 10.1016/j.cogbrainres.2003.11.017

Saults, J. S., and Cowan, N. (2007). A central capacity limit to the simultaneous storage of visual and auditory arrays in working memory. J. Exp. Psychol. Gen. 136, 663–684. doi: 10.1037/0096-3445.136.4.663

Schneider, W., and Detweiler, M. (1988). A connectionist/control architecture for working memory. Psychol. Learn. Motiv. 21, 53–119. doi: 10.1016/s0079-7421(08)60026-2

Sekuler, R., Sekuler, A. B., and Lau, R. (1997). Sound alters visual motion perception. Nature 385:308. doi: 10.1038/385308a0

Senkowski, D., Talsma, D., Herrmann, C. S., and Woldorff, M. G. (2005). Multisensory processing and oscillatory gamma responses: effects of spatial selective attention. Exp. Brain Res. 166, 411–426. doi: 10.1007/s00221-005-2381-z

Spence, C., and Driver, J. (1997). Audiovisual links in exogenous covert spatial orienting. Percept. Psychophys. 59, 1–22. doi: 10.3758/bf03206843

Stein, B. E., Burr, D., Constantinidis, C., Laurienti, P. J., Alex Meredith, M., Perrault, T. J., et al. (2010). Semantic confusion regarding the development of multisensory integration: a practical solution. Eur. J. Neurosci. 31, 1713–1720. doi: 10.1111/j.1460-9568.2010.07206.x

Stein, B. E., and Stanford, T. R. (2008). Multisensory integration: current issues from the perspective of the single neuron. Nat. Rev. Neurosci. 9, 255–266. doi: 10.1038/nrn2331

Talsma, D. (2015). Predictive coding and multisensory integration: an attentional account of the multisensory mind. Front. Integr. Neurosci. 9:19. doi: 10.10.3389/fnint.2015.00019

Talsma, D., Doty, T. J., and Woldorff, M. G. (2007). Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? Cereb. Cortex 17, 679–690. doi: 10.1093/cercor/bhk016

Talsma, D., Senkowski, D., Soto-Faraco, S., and Woldorff, M. G. (2010). The multifaceted interplay between attention and multisensory integration. Trends Cogn. Sci. 14, 400–410. doi: 10.1016/j.tics.2010.06.008

Talsma, D., and Woldorff, M. G. (2005). Selective attention and multisensory integration: multiple phases of effects on the evoked brain activity. J. Cogn. Neurosci. 17, 1098–1114. doi: 10.1162/0898929054475172

Thelen, A., Cappe, C., and Murray, M. (2012). Electrical neuroimaging of memory discrimination based on single-trial multisensory learning. Neuroimage 62, 1478–1488. doi: 10.1016/j.neuroimage.2012.05.027

Thelen, A., Talsma, D., and Murray, M. M. (2015). Single-trial multisensory memories affect later auditory and visual object discrimination. Cognition 138, 148–160. doi: 10.10.1016/j.cognition.2015.02.003

Thompson, V., and Paivio, A. (1994). Memory for pictures and sounds: independence of auditory and visual codes. Can. J. Exp. Psychol. 48, 380–398. doi: 10.1037/1196-1961.48.3.380

Uncapher, M. R., Hutchinson, J. B., and Wagner, A. D. (2011). Dissociable effects of top-down and bottom-up attention during episodic encoding. J. Neurosci. 31, 12613–12628. doi: 10.1523/jneurosci.0152-11.2011

Van der Burg, E., Olivers, C. N. L., Bronkhorst, A. W., and Theeuwes, J. (2008). Pip and pop: nonspatial auditory signals improve spatial visual search. J. Exp. Psychol. Hum. Percept. Perform. 34, 1053–1065. doi: 10.1037/0096-1523.34.5.1053

Van der Burg, E., Olivers, C. N. L., and Theeuwes, J. (2012). The attentional window modulates capture by audiovisual events. PLoS One 7:e39137. doi: 10.1371/journal.pone.0039137

Van der Burg, E., Talsma, D., Olivers, C. N. L., Hickey, C., and Theeuwes, J. (2011). Early multisensory interactions affect the competition among multiple visual objects. Neuroimage 55, 1208–1218. doi: 10.1016/j.neuroimage.2010.12.068

Vetter, P., Smith, F. W., and Muckli, L. (2014). Decoding sound and imagery content in early visual cortex. Curr. Biol. 24, 1256–1262. doi: 10.1016/j.cub.2014.04.020

Vogel, E. K., and Machizawa, M. G. (2004). Neural activity predicts individual differences in visual working memory capacity. Nature 428, 748–751. doi: 10.1038/nature02447

Vogel, E. K., McCollough, A. W., and Machizawa, M. G. (2005). Neural measures reveal individual differences in controlling access to working memory. Nature 438, 500–503. doi: 10.1038/nature04171

Vogel, E. K., Woodman, G. F., and Luck, S. J. (2001). Storage of features, conjunctions and objects in visual working memory. J. Exp. Psychol. Hum. Percept. Perform. 27, 92–114. doi: 10.1037//0096-1523.27.1.92

Wallace, M. T., Meredith, M. A., and Stein, B. E. (1993). Converging influences from visual, auditory and somatosensory cortices onto output neurons of the superior colliculus. J. Neurophysiol. 69, 1797–1809.

Wheeler, M. E., and Treisman, A. M. (2002). Binding in short-term visual memory. J. Exp. Psychol. Gen. 131, 48–64. doi: 10.1037/0096-3445.131.1.48

Wright, T. M., Pelphrey, K. A., Allison, T., McKeown, M. J., and McCarthy, G. (2003). Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cereb. Cortex 13, 1034–1043. doi: 10.1093/cercor/13.10.1034

Yonelinas, A. P. (2013). The hippocampus supports high-resolution binding in the service of perception, working memory and long-term memory. Behav. Brain Res. 254, 34–44. doi: 10.1016/j.bbr.2013.05.030

Conflict of Interest Statement: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Received: 03 December 2014; Accepted: 25 March 2015; Published online: 21 April 2015.

Citation: Quak M, London RE and Talsma D (2015) A multisensory perspective of working memory. Front. Hum. Neurosci. 9:197. doi: 10.3389/fnhum.2015.00197

Copyright © 2015 Quak, London and Talsma. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

ORIGINAL RESEARCH ARTICLE
published: 09 December 2014
doi: 10.3389/fnhum.2014.00976

Children with low working memory and children with ADHD: same or different?

Joni Holmes[1][*], Kerry A. Hilton[2], Maurice Place[3], Tracy P. Alloway[4], Julian G. Elliott[5] and Susan E. Gathercole[1]

1Medical Research Council Cognition and Brain Sciences Unit, Cambridge, UK

2Tees, Esk and Wear Valleys NHS Trust, Durham, UK

3School of Health and Life Sciences, Northumbria University, Newcastle upon Tyne, UK

4Department of Psychology, University of North Florida, Jacksonville, FL, USA

5School of Education, Durham University, Durham, UK

* Correspondence: Joni Holmes, Medical Research Council Cognition and Brain Sciences Unit, 15 Chaucer Road, Cambridge, CB2 7EF, UK e-mail: joni.holmes@mrc-cbu.cam.ac.uk

Edited by:
Steve Majerus, Université de Liège, Belgium

Reviewed by:
Gaia Scerif, University of Oxford, UK
Wendy Heller, University of Illinois, USA

The purpose of this study was to compare working memory (WM), executive function, academic ability, and problem classroom behaviors in children aged 8–11 years who were either identified via routine screening as having low WM, or had been diagnosed with ADHD. Standardized assessments of WM, executive function and reading and mathematics were administered to 83 children with ADHD, 50 children with low WM and 50 typically developing children. Teachers rated problem behaviors on checklists measuring attention, hyperactivity/impulsivity, oppositional behavior, and difficulties associated with executive function in the classroom. The ADHD and low WM groups had highly similar WM and executive function profiles, but were distinguished in two key respects: children with ADHD had higher levels of rated and observed impulsive behavior, and children with low WM had slower response times. Possible mechanisms for these common and distinct deficits are discussed.

Keywords: ADHD, working memory, executive function, developmental disorders, intervention

INTRODUCTION

Deficits in working memory (WM) are common in childhood. They are characteristic of children with specific learning difficulties in reading, mathematics, and language (e.g., Swanson and Ashbaker, 2000; Archibald and Gathercole, 2007; Szucs et al., 2013; Pimperton and Nation, 2014) and also of those with attention deficit hyperactivity disorder (ADHD) (e.g., Martinussen et al., 2005). Poor WM during development has begun to be investigated in its own right rather than as a secondary symptom of another disorder, and has been found to be closely associated both with low academic achievement (Gathercole and Alloway, 2008; Alloway et al., 2009) and with some of the attention problems typical of children with ADHD (Gathercole et al., 2008a). This study provides the first direct comparison of cognitive skills, executive functions, learning and behavior between children with low WM and those of the same age diagnosed with ADHD. The outcomes have direct implications both for the diagnosis and treatment of the broad range of cognitive and behavioral problems found in these two groups and for the interplay between inattention and poor WM in childhood.

WM is comprised of distinct but interacting cognitive and neural systems that coordinate higher-level attentional control and the temporary storage of information, providing vital ongoing support for complex cognitive activities (Baddeley, 2000; Unsworth and Engle, 2007; Cowan, 2010). There exist a variety of conceptualizations of the nature, structure, and function of WM (see Conway et al., 2007, for review). One important distinction between models is whether WM is conceived as a distinct multi-store workspace that includes an attentional component (e.g., Baddeley and Hitch, 1974; Baddeley, 2000) or is embedded within a broader limited-capacity system of controlled attention (e.g., Kane et al., 2001). In general, though, both frameworks provide adequate accounts of large bodies of empirical evidence (e.g., Cowan, 1995; Nee et al., 2008) and there is broad consensus that the storage-only capacities of short-term memory (STM) and the capacity-limited attentional control functions of WM can be distinguished (e.g., Shah and Miyake, 1999; Cowan, 2008).

Baddeley and Hitch's (1974; Baddeley, 2000) enduring multi-component model has provided a theoretical framework and set of methodologies that have been widely used for exploring WM in many cognitive developmental disorders including ADHD. In this, a domain-general limited-capacity central executive system provides executive control of attention. There are many parallels between this subsystem and the controlled attention view of WM (Kane et al., 2001). The central executive is part of a broader network of executive functions that includes inhibition, planning, and set switching and which relies on the same frontal brain networks to support flexible goal-directed behavior (Duncan and Owen, 2000; Miyake et al., 2001; St Clair-Thompson and Gathercole, 2006). Verbal and visuo-spatial short-term memory (STM) stores, and an integrative multi-modal episodic buffer, support the central executive (Baddeley, 2000). Assessments of these different components of WM are distinguished by whether or not they impose significant processing demands. Whereas STM tasks require the storage of information, tasks tapping the executive component (often termed complex span or WM tasks) involve significant processing in addition to storage (e.g., Daneman and Carpenter, 1980; Alloway et al., 2006). These components have been suggested to contribute to many everyday cognitive activities including following instructions (Yang et al., 2014), mental arithmetic (De Stefano and LeFevre, 2004) and the comprehension of language (Carretti et al., 2009).

Children selected on the basis of low scores on measures of WM that tax both the central executive and STM stores, such as backward digit span or listening span, typically perform relatively poorly on school-based evaluations of curriculum learning (e.g., Swanson and Sachse-Lee, 2001; Gathercole et al., 2003, 2004; Alloway and Alloway, 2010; Archibald et al., 2011). The majority have impairments in both reading and maths and in the classroom they frequently fail in activities that involve following instructions, storing information whilst engaged in other cognitively demanding activities, and place-keeping in complex tasks (Gathercole et al., 2006; Gathercole and Alloway, 2008). Laboratory tasks designed to simulate the high WM demands of classroom activities under more controlled conditions confirm these deficits (Engle et al., 1991; Gathercole et al., 2008b).

Children with poor WM are also reported by teachers to be inattentive and have short attention spans (Gathercole et al., 2008a; Alloway et al., 2009; Archibald et al., 2011). Similarly, adults with low WM report high levels mind-wandering under conditions of high cognitive load (Kane et al., 2007; McVay and Kane, 2009), and associations are found between poor WM and inattentive behavior in both typically developing children (Aronen et al., 2005; Lui and Tannock, 2007; Thorell, 2007) and those with poor comprehension skills (Pimperton and Nation, 2014). Children with low WM also exhibit problems in other areas of executive function. They are rated by teachers as being relatively poor in areas relating to WM, the ability to monitor work, the inhibition of impulsive responses, and in planning and organization (Gathercole et al., 2008a). On direct assessments of action planning and visual selective attention, they have also been reported to be impaired (St Clair-Thompson, 2011).

There is a high degree of overlap between this profile and the characteristics of children with combined-type ADHD, a disorder characterized both by abnormally high levels of both inattentive and hyperactive/impulsive behaviors (DSM 5 American Psychiatric Association, 2013). They too have impaired learning in reading and mathematics (Loe and Feldman, 2007), accompanied by WM difficulties (Martinussen et al., 2005; Willcutt et al., 2005a; Kofler et al., 2010) that have been linked with inattention (Willcutt et al., 2005a; Martinussen and Tannock, 2006).

The executive function problems in ADHD include response inhibition (e.g., Bledsoe et al., 2010), attentional switching (e.g., Oades and Christiansen, 2008), planning (e.g., Solanto et al., 2007), and sustained attention (e.g., Rubia et al., 2009). In the most common form of ADHD, the combined subtype, children also have excessively high levels of motor activity (hyperactivity) and impulsive behavior (Barkley, 1997; Halperin et al., 2008; Rapport et al., 2009). There is as yet no consensus regarding the origins of this complex profile of deficits. Some have argued that WM difficulties underlie other executive functions deficits in ADHD such as response inhibition (Rapport et al., 2008; Alderson et al., 2010). Others have suggested that executive function deficits (including WM) and problems of impulse control or aversion to reward delay represent impairments of two functionally distinct neurodevelopmental systems: “cool” cognitive—based executive functions that include inhibitory control and WM, and “hot” affective processes associated with aversion to delay that manifest as impulsive behavior (Tripp and Alsop, 2001; Castellanos et al., 2006).

Although the cognitive similarities between children with low WM and those with ADHD are striking, one marked difference is evident. Children with low WM do not exhibit excessive levels of motor activity and problems in impulse control that are core characteristics of ADHD (Gathercole et al., 2008a; Alloway et al., 2009). One possibility is that the two groups share a common deficit in the cool executive function system that is associated with attentional difficulties but that only those with ADHD have impairments in the hot executive system linked with hyperactivity and impulsivity. This hypothesis was investigated in the present study which, to our knowledge, is the first to compare directly the cognitive and behavioral characteristics of children who have poor WM but no ADHD diagnosis with those with ADHD. It was predicted that both groups would be impaired on direct measures of cool executive functions such as WM, planning and cognitive inhibitory control, and that they would be rated as being both inattentive and having elevated levels of other problem behaviors relating to these elements of high-level cognitive control. The ADHD group were expected to be differentiated by further problems in impulse control, and both hyperactive and impulsive behavior; the same difficulties were not predicted for the low WM group. Both direct assessments and teacher behavior ratings were obtained of a range of executive functions including WM and attentional control. Measures of IQ and learning were also included.

METHODS

PARTICIPANTS

Three groups of children participated in the study. One group consisted of 83 children (71 boys) aged 8–11 years, with a clinical diagnosis of combined-type ADHD recruited through pediatric psychiatrists based in the North-East of England. All children had a clinical diagnosis of ADHD that included a psychosocial assessment, clinical and parent observer reports and a clinical assessment of the child's mental state. Inclusion criteria for the present study were i) a DSM 5 diagnosis of combined-type ADHD for 6 months or longer ii) aged between 8 and 11 years iii) no co-morbid Autistic Spectrum Disorders. The majority of the group was receiving fast-release stimulant medication for the condition: methylphenidate (n = 64), dexamphetamine (n = 2), dexedrine (n = 2), and imipramine (n = 1). Fifteen children were not taking medication.

Two further groups were recruited from a sample of 780 children aged 8–11 years attending 10 state primary schools in the same region who were screened on two tests of verbal WM: Listening Recall and Backward Digit Recall from the Automated Working Memory Assessment (AWMA; Alloway, 2007). Fifty children (30 boys) with standard scores below 86 on both tests were assigned to a low WM group, and a further 50 children (27 boys) with standard scores above 90 on both tests formed an age-typical WM group. The ages of the children in the two groups were matched to within 30 days of 50 children in the ADHD group (mean ages: ADHD, M = 9 years, 9 months, SD = 12.64, comparison, M = 9 years 10 months, SD = 11.98, low WM, M = 9 years, 9 months, SD = 12.11). None of the children had a diagnosis of ADHD. Ethical approval was obtained through both the local National Health Service and Durham University's ethics boards. Consent was obtained from parents/guardians and children, with appropriate opportunities for withdrawal.

PROCEDURE

Testing took place on between two and five individual testing sessions according to the individual child, with a total testing time of approximately 4 h. Regular breaks were introduced as required to reduce fatigue and optimize compliance. All assessments were conducted in a quiet area of the child's school. Children with ADHD receiving drug treatment ceased ingestion at least 24 h prior to testing. As all prescribed drugs were fast release, their physiological effects were eliminated at the time of test. Both children and their teachers were asked to verify that no medication had been taken prior to testing. In line with the administration guidelines of the executive function test battery, some children in the ADHD and low WM groups did not complete the tests either due to poor performance on practice trials and/or tasks measuring the baseline component processes necessary for a higher-level task (between two and eight out of 83 in the ADHD group and between one and six in the low WM group, depending on the test). All children in the comparison group completed every task. Teacher behavior ratings assessing ADHD symptoms were returned for 71%, 54% and 46% of the ADHD, low WM and comparison groups, respectively. Ratings of executive function problem behaviors were returned for 55% of the ADHD sample, 50% of the low WM group and 38% of the comparison group.

MEASURES

Working memory

The AWMA (Alloway, 2007) provided multiple tests of verbal STM (Digit Recall, Word Recall, Non-word Recall), visuo-spatial STM (Dot Matrix, Block Recall, Mazes Memory), verbal WM (Backward Digit Recall, Listening Recall, Counting Recall), and visuo-spatial WM (Mr X, Spatial Span, Odd One Out). All tests yield standard scores.

Executive functions

Switching. The Number-Letter Sequencing test of the Delis-Kaplan Executive Function System (D-KEFS, Delis et al., 2001) assesses set-shifting/switching. The higher-level executive condition, Number-Letter Sequencing, requires children to connect letters and numbers in a progressive increasing alternating sequence (A-1-B-2-C-3, etc.). Other baseline conditions within this test, Visual Scanning, Motor Speed, Number Sequencing, and Letter Sequencing, measure the basic processes involved in Number-Letter Sequencing. Completion times are calculated for each condition and converted to scaled scores. Errors are represented as cumulative percentiles for baseline conditions and as a scaled score for the switching measure. A scaled contrast score represents differences in performance between the baseline and switching measure.

Inhibition and inhibition with switching. The Color Word Interference and Color Word Intereference with Switch tasks (D-KEFS, Delis et al., 2001) measure inhibitory control. The Color Word Interference condition involves a standard Stroop task in which the child inhibits the over-learned verbal response of naming a color word, and instead names the ink color. The Color Word with Switch condition involves two executive functions: inhibition and switching. The child is instructed to name the color of the ink for all words except those displayed in a box; on these trials, the task is to name the color word rather than the ink color. Further baseline conditions, Color Word Reading and Color Naming, assessed relevant processing abilities. Completion times are converted to scaled scores for all tasks. Errors are scored as cumulative percentiles for baseline tasks and scaled scores for the executive tasks.

Sorting. The D-KEFS Sorting Test (Delis et al., 2001) measures problem-solving and conceptual learning. It requires children to sort, or describe the sorting categories of, six cards according to different dimensions—color, shape or pattern on the cards, characteristics (upper or lower case, number of letters/syllables) or semantic information about words printed on the cards. In one condition, Free Sort, the child sorted the cards into as many categories as possible. Both the number of correct sorts and level of the description of each sort were scored. In a second condition, Recognition, the examiner sorted the cards into different categories and asked the child to describe what principles had been used to sort the cards. Both the number of correct sorts and the descriptions of the sorts were converted to scaled scores.

Planning. Planning and the ability to inhibit an impulse response were measured using the Tower Test (D-KEFS, Delis et al., 2001), in which the child moves 5 disks of different sizes arranged on three pegs from a start position to an end state one disk at a time without placing any disk on a smaller disk. Total achievement and time per move scores are converted to scaled scores. Total rule violations are scored as cumulative percentiles.

Sustained attention. The K-test of the Continuous Performance Test (CPT, Conners and Multi-Health Staff, 2004) assesses sustained attention. In this task, a series of 480 letters appear on the computer screen at a rate of 1 per second. The child's task is to press the space bar only when a K is displayed, which occurs on 140 of the trials at random intervals. The following measures are obtained: average response delay in ms, the numbers of omissions (possible range 0–140) and commissions (possible range 0–480), and total accuracy (proportion of trials correct).

Response suppression. Motor response inhibition was measured using the Walk/Don't Walk subtest from the Test of Everyday Attention for Children (TEA-Ch, Manly et al., 1999). The child is given a sheet showing paths made up of footprints and has to dot the next footprint on the path with a marker pen when they hear a frequently occurring “go” sound. The child is instructed not to dot the next footprint when an occasional “no go” sound is played, thereby inhibiting the prepotent go response. Correct responses are converted to a scaled scores.

Reading and mathematics. The basic reading, spelling and reading comprehension subtests of the Wechsler Objective Reading Dimensions (Wechsler, 1993), and both subtests of the Wechsler Objective Number Dimensions (Wechsler, 1996), mathematical reasoning and number operations, were administered. In each case, standard scores were calculated.

IQ. The four subtests of the Wechsler Abbreviated Scales of Intelligence (WASI, Wechsler, 1999) were administered: Block Design, Matrix Reasoning, Similarities and Vocabulary. Verbal and Performance IQ standard scores were calculated from subtest scores.

Behavior

Teacher rating scale. Classroom teachers completed the Conners Teacher Rating Scale Revised Short-Form (CTRS-R, Conners, 1997) for each child. Teachers rate 28 statements as not true at all (0), just a little true (1), pretty much true (2) or very much true (3). These statements comprise four subscales, which provide an index of oppositional behavior, cognitive problems and inattention, hyperactivity and ADHD symptoms. T-scores were calculated.

Behavior rating inventory of executive function. The Behavior Rating Inventory of Executive Function (BRIEF, Gioia et al., 2000), consisted of 86 statements, which teachers rated as occurring never (1), sometimes (2), or often (3). These statements formed eight subscales designed to assess executive functioning in the school environment as follows: Inhibit- the ability to control impulses; Shift—the ability to move freely from one situation, or aspect of a problem, to another; Emotional Control—the ability to modulate emotional responses; WM—the ability to hold in mind information for the completion of an activity; plan/organize—the ability to set goals, develop appropriate steps ahead of time and anticipate future events; Initiate—the ability to begin a task and work independently; Organization of Materials—the ability to maintain parts of the environment in an orderly manner; Monitor—the ability to assess performance, check work and keep track of effort. Three composite scores were derived: Behavioral Regulation Index, Metacognitive Index and Global Executive Score. T-scores were calculated for each scale and composite index.

RESULTS

Descriptive statistics for the cognitive measures are displayed by group in Table 1 and behavior ratings are shown in Table 2. Separate multivariate analyses of variance (MANOVA) were conducted for those tests that generated multiple dependent variables on a comparable scale: WM, IQ, reading, mathematics, and teacher behavior ratings. Univariate F-tests compared performance between groups on individual measures. Where there were significant group differences, pairwise comparisons were conducted for each of the three pairwise group combinations. Bonferroni corrections were applied to correct for multiple testing, yielding significance thresholds of p < 0.0125 for WM composite scores, sustained attention, and Conners behavior ratings, p < 0.004 for individual WM subtests, and BRIEF behavior scores, p < 0.005 for indices derived from the switching task, p < 0.005 for the inhibition and combined inhibition with switching task measures; p < 0.016 for sorting, planning and IQ, and p < 0.01 for reading and mathematics.

Table 1. Cognitive measures as a function of group.

Table 2. Teacher behavior ratings, by group.

COGNITIVE MEASURES

Working memory

A MANOVA revealed a significant group effect for composite WM scores, Hotelling's T^2(8, 352) = 16.567, p < 0.001. Univariate F-tests revealed significant group effects for all of the individual STM and WM subtests and the four derived composite scores. The ADHD and low WM groups performed significantly more poorly than the comparison group on all four WM component scores, and all 12 WM subtests. Performance was significantly lower for the low WM group than the ADHD group on two of three verbal WM tasks, Listening Recall and Backward Digit Recall, and for the verbal WM composite score. This is likely to reflect a selection artifact for the low WM group, as they were identified on the basis of low scores (<86) on these measures, and the two groups did not differ on the verbal WM task that was not used at screening, Counting Recall, when a Bonferroni correction was applied. No other significant differences were found between the low WM and ADHD groups.

Executive functions

Switching. Univariate analyses established significant group differences in two of the baseline tasks; Number Sequencing completion times and Letter Sequencing errors. There were no significant group differences in the other baseline conditions, Motor Speed and Visual Scanning. There were significant group differences in the frequency of errors on the higher-level switching task, Number-Letter Sequencing. However, the group difference in contrast scores, which reflect differences in performance between the baseline and higher-level conditions, was not significant.

Pairwise comparisons revealed that the low WM and ADHD groups made significantly more errors on the switching task than the comparison group. The ADHD group were also significantly less accurate than the comparison group in the baseline Letter Sequencing condition. Error rates did not differ significantly between the low WM and ADHD children on either the baseline or switching tasks. Group differences in completion time for the baseline Number Sequencing task were driven by significantly slower performance by the low WM group than either the ADHD or comparison groups.

Inhibition control and inhibition with switching. Significant group differences were established for all subtests of the Color-Word Interference and Color-Word Interference with Switch tasks, including the baseline and higher-level inhibition and inhibition with switch tasks. There were no significant group differences in contrast scores.

Subsequent pairwise comparisons revealed the ADHD group were both significantly slower and more errorful than the comparison group in the baseline Color Naming and Word Reading conditions and also on the higher-level inhibition with switch task, Color Word Interference with Switch. They also committed significantly more errors on the inhibition task than the comparison group, but completion times were not significantly different. Low WM children performed the baseline Color Naming and Word Reading tasks and the inhibition and inhibition with switch tasks significantly more slowly than the comparison group. They also made a significantly higher number of errors on both the higher-level executive tasks. There were no significant group differences between the ADHD and low WM groups on either of the baseline or higher-level subtests of Color Word Interference or Color Word Interference with Switch tasks.

Sorting. Significant group differences were observed for both the number and description of Free Sorts on the Sorting task, which measures problem-solving and conceptual learning skills. There were no significant group differences in the Recognition Sort condition. Where there were significant group differences, the comparison group performed at a significantly higher level than both the ADHD and low WM groups. The ADHD and low WM groups did not differ significantly on any measure.

Planning. There were no significant group differences in total achievement scores on the Tower Test. However, the ADHD group violated task rules significantly more often than both other groups, and also performed the task significantly faster than the comparison group. Time taken to complete the task between children with ADHD and those with low WM did not differ significantly. The low WM group made significantly more rule violations than children in the comparison group.

Sustained attention. Average response times for each trial did not differ between groups. There was, however, a significant group effect for accuracy that was accompanied by significant group differences for both error types. Overall, the ADHD and low WM children were significantly less accurate than the comparison group and both made a significantly greater number of omissions. The ADHD group additionally made significantly more commission errors than either of the other groups.

Response suppression. Significant group differences on the Walk/Don't Walk task reflected significantly poorer performance by both the ADHD and low WM groups relative to the comparison group. No significant differences were observed between those with ADHD and those with low WM.

Reading and mathematics. There was an overall significant group effect for academic ability, Hotelling's T2(8, 352) = 7.227, p < 0.001. Performance was significantly higher for the comparison group than the other two groups across measures of mathematical reasoning, written number calculations, spelling, single word reading and reading comprehension. Reading and mathematics scores did not differ significantly between the low WM and ADHD groups.

IQ. A MANOVA revealed a significant group effect for IQ, Hotelling's T^2(6, 354) = 8.315, p < 0.001. Subsequent univariate F-tests established this difference was significant for both Verbal and Performance IQ, and for the derived Full Scale IQ score. Performance IQ scores were significantly higher for the comparison group than both other groups, and for the ADHD than the low WM group.

As non-verbal reasoning is highly associated with processing speed (Fry and Hale, 1996), a series of ANCOVAs were conducted to establish whether group differences between the low WM sample and the other groups were mediated by variation in Performance IQ scores. Consider first the differences between the low WM and comparison groups. Group differences were abolished in planning rule violations (p = 0.161), both the frequency of omissions (p = 0.255) and overall accuracy levels in sustained attention (p = 0.169), two of the visuo-spatial STM tasks, Dot Matrix (p = 0.006) and Mazes Memory (p = 0.005), time taken to complete the baseline Number Sequencing test of the switching task (p = 0.006) and both the higher-level inhibition (p = 0.017) and inhibition switching tasks (p = 0.015).

Group differences between the ADHD and low WM groups were largely unaffected by controlling for Performance IQ. The exception was Backward Digit Recall, on which the group difference was no longer significant (p = 0.009).

BEHAVIOR RATINGS

Teacher rating scale

There was a significant effect of group on all four subscales of Conners behavior ratings, Hotelling's T^2(8, 204) = 8.552, p < 0.001. Teacher ratings of oppositional, inattentive and hyperactive behaviors were significantly elevated in the ADHD group relative to the comparison group. Both the ADHD and low WM groups received significantly elevated scores on the cognitive problems/inattention subscale and composite ADHD index. The ADHD group were rated as significantly more oppositional and hyperactive than the low WM group.

Behavior rating of executive function

Teacher ratings of behavioral difficulties related to executive function differed significantly by group, Hotelling's T^2(22, 152) = 2.978, p < 0.001. Significantly elevated symptoms were reported for each individual subscale of the BRIEF for the ADHD group relative to the comparison group. The low WM group received significantly higher ratings of problem behaviors on the Initiate, WM and Monitor subscales and on the composite Metacognition Index relative to the comparison group. There were no significant differences in teacher ratings between the ADHD and low WM children on any of the individual subscales or on the composite index scores of the BRIEF.

FACTOR ANALYSIS

To investigate further the differences between the ADHD and low WM groups, a principal components analysis was conducted on the WM and executive function measures for all children (N = 183). Varimax rotation was used to force differentiation between factors and amplify group differences. To satisfy the recommended 10:1 case to variable ratio (Nunnally, 1978), a reduced set of variables was entered into the analysis. The measures were selected to provide speed and accuracy scores for each of the higher-level tasks: time and error scores for switching, inhibition, and inhibition switching; number of sorts for problem-solving; time per move and frequency of rule violations for planning; total score for response suppression; frequency of each of omission and commission errors and average response time per trial for sustained attention; a mean WM score derived from the four composite scores. This single score was selected in order to avoid entering multiple highly-correlated measures.

Three factors emerged with eigenvalues in excess of 1.00, explaining 29.396, 12.855 and 10.268% of variance, respectively. Factor loadings greater than 0.30 on the rotated factor matrix are shown in bold in Table 3. A broad range of executive measures loaded highly on Factor 1: WM, switching errors, inhibition and inhibition switching, problem-solving, both planning measures, response suppression, and omission in sustained attention. The measures loading most highly on Factor 2 were response times on the switching, inhibition and inhibition switching tasks, and to a lesser extent WM. This second factor is therefore predominantly associated with speed of processing. Factor 3 is linked with impulsivity in sustained attention, with high loadings of both the frequency of commission errors and response speed on the sustained attention task. A convergent three-factor solution was derived when oblique rotation was used.

Table 3. Principal components analysis.

Factor scores are shown by group in Table 4. Univariate ANOVAs revealed significant group differences on all three factors: Factor 1, F(2, 165) = 32.413, p < 0.01; Factor 2, F(2, 165) = 7.548, p = 0.01; Factor 3, F(2, 165) = 8.492, p < 0.01. Post-hoc exploration of the group differences established scores were significantly higher for the comparison group than both the ADHD and low WM groups on Factors 1 and 2. The low WM and ADHD groups did not differ on Factor 1 (executive functions). However, the low WM group had significantly lower scores than the ADHD group on Factor 2 (speed of processing). In contrast, the ADHD group scored more poorly than the low WM group on Factor 3 (impulsivity in sustained attention).

Table 4. Factor scores displayed by group.

Due to the disproportionately high number of boys in the ADHD group compared to both the low WM and comparison groups, a series of 3 × 2 ANOVAs with group (ADHD, low WM and average WM) and gender (boys and girls) were conducted to test for gender effects. There were no significant group x gender interactions for any of the cognitive measures, teacher rating scores or factor scores (all ps > 0.05).

DISCUSSION

This study provides the first comprehensive comparison of WM, executive function, academic ability and classroom behavior in children with low WM and those with ADHD. Both groups were characterized by overlapping patterns of WM and executive function deficits compared with both a typically developing comparison group drawn from the same population as the low WM children and population-based test standardizations. There were two important differences between the groups. First, the low WM children were slower to respond on several tasks. Second, the ADHD group were more hyperactive and exhibited more difficulties in controlling impulsivity in sustained attention.

First we consider the common characteristics of the two groups. They were judged as having equivalently high levels of inattentive behavior. For the ADHD group this is expected, as inattention is one of the defining characteristics of the combined subtype. It also replicates previous reports of inattentive behavior in low WM groups (Kane et al., 2007; Gathercole et al., 2008a; McVay and Kane, 2009), although the parity in the degree of rated inattentiveness across the two groups is novel and worthy of note. Their WM characteristics were also highly similar, in terms of both the profile and magnitude of impairments. Compared both with the typically-developing group and age norms, children with low WM and those with ADHD had substantial deficits in tests of visuo-spatial STM, verbal WM and visuo-spatial WM. Their verbal STM scores fell within the age-typical range. The low WM group scored more poorly than the ADHD group on the two verbal WM tasks used to identify them at screening. However, they did not differ on a third measure that was administered after screening. This suggests that the group differences on individual measures reflected sampling artifacts. Although the low WM group were selected on the basis of poor performance on two verbal tests of WM, there was no evidence for domain-specific impairments. Their composite verbal WM scores (75.26) were equivalent to their visuo-spatial WM scores (78.12) and their IQ profile was flat across both verbal and non-verbal assessments (standard scores 86.42 and 85.20 respectively).

The general pattern of findings is consistent with a common impairment in both groups in the domain-general executive control aspect of WM (Bayliss et al., 2003; Kane et al., 2004; Alloway et al., 2006), but not in verbal STM. The low performance on visuo-spatial STM tests could reflect a particular deficit in the visuo-spatial sketchpad. Alternatively, on the basis of close links already reported between these measures and the central executive aspect of Baddeley's (2000) WM model (Miyake et al., 2001; Alloway et al., 2006; Burin et al., 2007), a more parsimonious interpretation could be a single underlying impairment in the central executive for both children with low WM and those with ADHD. Similar claims of central executive problems have already been made for both groups (e.g., Martinussen and Tannock, 2006; Gathercole et al., 2008a; Kofler et al., 2010; Kasper et al., 2012). The new finding here is the high correspondence between both the profile and severity of the WM impairments in this first direct comparison of the two samples.

Understanding the correspondences between deficits in the executive control aspect of WM and elevated ratings of inattentive behavior in these two groups raises the possibility that WM problems may be the cause of overt everyday problems in attentional focus. Consistent with Engle's model of WM (Kane et al., 2001), previous research supports a close relationship between controlled cognitive attention and WM. For example, adults with low WM spans are known to be poor at resisting distracting information in experimental tasks (Rosen and Engle, 1997; Kane and Engle, 2000; Conway et al., 2001). The current data extend these findings to suggest a link between poor WM and overt inattentive and distractible behaviors. This association has been previously observed in adults with low WM in cognitively-demanding everyday activities (Kane et al., 2007) and also in a community sample of unselected children where WM performance was correlated with levels of inattentive behavior (Lui and Tannock, 2007). The novel finding here is that these associations are present at the group level in children with poor WM and in children with ADHD. Low WM capacity may give rise to short attention spans and distractible behavior due to a failure to maintain in WM task goals, and also the intermediate products of the ongoing mental activity in order to achieve the goal, which causes attentional focus to shift away from the task in hand, either to other salient events in the environment or to internally-generated thoughts.

Impairments in the low WM and ADHD groups that were comparable in magnitude extended to other executive functions, too. High rates of problem behaviors across a wide range of executive function behaviors were reported by teachers for both the low WM and ADHD groups. They also performed poorly on direct measures of switching, inhibition, sorting, planning, sustained attention, and response suppression. These data extend previous findings in which problem behaviors relating to executive function have been reported in children with low WM (Gathercole et al., 2008a) and those with ADHD (e.g., Toplack et al., 2009). In a smaller sample selected similarly on the basis of low WM, St Clair-Thompson (2011) reported difficulties in direct measures of visual attention and planning, although not in switching, inhibition and response suppression in low WM children. With substantially greater statistical power, the present study establishes low WM children have a constellation of difficulties that extend considerably beyond WM to all assessed aspects of executive function.

The current findings are entirely consistent with patterns of pervasive executive function impairments and teacher reports of executive function problems in ADHD (e.g., Willcutt et al., 2005a; Toplack et al., 2009; Schoemaker et al., 2014). According to some theories of ADHD, executive function impairments underlie behavioral disturbances and are central to the disorder (Barkley, 1997; Zelazo and Muller, 2002). Consistent with this, results from a meta-analysis of structural neuroimaging studies indicate that ADHD is associated with neuroanatomical abnormalities in areas of the brain related to executive function (Valera et al., 2007).

There are a number of possible reasons for why the common cognitive deficits of the groups might be so pervasive. First, difficulties may be present in all executive functions, including WM, because they rely on the same frontal brain networks (Duncan and Owen, 2000). Any impairment to this network would be expected to disrupt multiple executive functions. An alternative possibility is that WM deficits may play a causal role in other executive tasks and related behaviors. For example, low performance levels in both the low WM and ADHD groups on measures of inhibition or set switching may have resulted from the loss of crucial task relevant information or goals from WM (e.g., Kane et al., 2007; McVay and Kane, 2009). Finally, and in line with behavioral inhibition theories of ADHD, difficulties in inhibitory control may adversely affect WM and other executive functions due to problems inhibiting task irrelevant information and regulating goal-directed behavior (Barkley, 1997).

However, a degree of caution is necessary in interpreting these findings simply in terms of widespread problems in the broad executive control system. This is because in some cases, children with ADHD and low WM had deficits on component (non-executive) measures that were as great as those observed on measures requiring higher-order executive control. For example, both groups had problems in simple sequencing, word reading and color naming. Thus, difficulties in the inhibition and switching conditions requiring executive control may have been mediated by problems in basic processing, consistent with previous findings that when differences in basic processing skills are controlled, apparent executive impairments disappear both in ADHD and in children with reading difficulties (Rhodes et al., 2005; van Mourik et al., 2005; Willcutt et al., 2005b; Marzocchi et al., 2008). The core deficits may therefore lie in more basic cognitive processes.

This study directly compared the academic achievements of children with low WM and those with ADHD. Poor scholastic performance has been previously reported for each group separately (Loe and Feldman, 2007; Gathercole and Alloway, 2008). These findings were replicated, and it was additionally found that underachievement was equivalent in the two groups even when group differences in IQ were taken into account. This has important educational implications as enhanced learning support was provided in school to all children in the ADHD sample but not to the low WM group. Warner-Rogers and colleagues suggest that children who find it hard to pay attention, but who are not disruptive in the classroom are at risk for being neglected in educational settings (Warner-Rogers et al., 2000). The current observations of the low WM group underscore this point and demonstrate the need to identify and support learning in children with poor WM.

Critically, though, the low WM and ADHD groups were not indistinguishable in all respects. There were two key differences. First, children with low WM performed more slowly on several processing and executive functions tasks than both the comparison group and those with ADHD. This evidence for processing speed impairments in low WM is a novel and unexpected finding. It does not appear to be part of a broader problem in fluid intelligence (often associated with processing speed, Salthouse, 1996; Jensen, 1998), as controlling statistically for performance IQ had little impact on observed group differences. These difficulties may correspond to sluggish cognitive tempo (SCT), a set of symptoms strongly associated with the predominantly inattentive form of ADHD that includes high levels of daydreaming, slow response times, poor mental alertness and hypoactivity (e.g., Barkley, 1990; McBurnett et al., 2001). Although not included in the DSM criteria for ADHD, SCT symptoms have been advanced as a marker for a subgroup of individuals with the inattentive form of ADHD who have a distinct primarily inattentive disorder (Carlson and Mann, 2002; Hinshaw et al., 2002; Hartman et al., 2004; Huang-Pollock et al., 2005). It is therefore possible that children with low WM may correspond to those with the predominantly inattentive subtype of ADHD, a diagnosis that is not commonly applied in child psychiatric services in the UK.

Secondly, ADHD children made more errors than the low WM group on some of the tasks that required the inhibition of impulsive motor responses. In particular, they violated rules more frequently during a planning task, and made more commission errors (responding impulsively to non-target stimuli) on the Continuous Performance Test of sustained attention. This is one of the most widely reported deficits in ADHD (e.g., Epstein et al., 2003; Willcutt et al., 2005a; Rubia et al., 2007, 2009), and the present data indicates that it has a high degree of specificity to ADHD. Elevated levels of impulsivity in ADHD children were also reflected in the teacher ratings of high levels of hyperactive, impulsive and oppositional behaviors. Although there were more boys in the ADHD sample these group differences, and those in the cognitive assessments, were not mediated by gender. Gender ratios in ADHD are dependent on whether the sample is drawn from clinical (referred by psychiatrists) or community-based settings (APA, 2002), with a more equal balance of boys and girls common among samples selected via routine population-based screening (Gaub and Carlson, 1997; Arica and Conners, 1998). In the current study, the ADHD sample was clinically referred whereas the low WM group were selected via routine classroom screening. These different routes to selection may explain why there were a disproportionately high number of boys in the ADHD group.

More generally, the observed pattern of shared general executive function disturbance combined with group-specific impairments in impulsive behavior in sustained attention (the ADHD group were more impulsive) and speed (the low WM children were slower) fits well with the proposal that ADHD arises from parallel disruptions in distinct cool (cognitively-based) and hot (affective, delay aversive) neurodevelopmental systems (Castellanos et al., 2006). The cool impairments in cognitive aspects of executive function (or possibly, more basic cognitive processes) and WM that are linked also with inattention (Castellanos et al., 2006) are present both in the children with ADHD and low WM, whilst the hot difficulties in controlling aversion to delay in attention-demanding tasks appears to be restricted to individuals with ADHD. The additional difficulty in slow response times observed only in the low WM group does not fit within this ADHD framework, but may be symptomatic of a subgroup of children with the predominantly inattentive form of ADHD who are characterized by SCT (Hartman et al., 2004).

In summary, this study is the first to demonstrate that children with low WM and those with ADHD have largely equivalent problems across a wide range of measures of basic and higher-level cognitive functioning, and in particular in behaviors associated with executive functions. The groups are also indistinguishable in terms of their poor learning progress in mathematics and reading. There were important differences, too. The low WM children were slower to respond than the ADHD group across tests, and the ADHD children were more hyperactive and impulsive in their behavior and in some aspects of controlling responses when required to sustain attention. These distinctions may have considerable practical value for practitioners working with developmental populations. First, despite the striking differences in classroom management challenges raised by children with ADHD (high) and those with low WM (low), their needs for educational support as indexed by low levels of attainment are equivalent and may warrant similar levels of resourcing (e.g., Warner-Rogers et al., 2000). Second, the two groups may also respond similarly to interventions that address their shared problems in WM and executive functions, as we have found with WM training (Holmes et al., 2009, 2010; Dunning et al., 2013). It may also be expected that new methods such as strategy training already shown to enhance memory function in children with low WM (St Clair-Thompson et al., 2010) will be similarly beneficial in ADHD. However, the special behavioral difficulties that distinguish children with ADHD are likely to require different kinds of intervention that target psychosocial rather than cognitive skills, such as behavioral modification (e.g., Fabiano et al., 2009). For children with ADHD, a synergistic approach combining interventions targeted at both cognitive and behavioral problems might be optimal (e.g., Rapport et al., 2013). We suggest that adopting a multi-dimensional approach to profiling individual children with WM and related executive function problems—for example, by distinguishing executive from impulsive problems—may provide a valuable means of identifying those interventions, which either singly or in combination, are likely to be effective for individual children.

REFERENCES

Alderson, R. M., Rapport, M. D., Hudec, K. L., Sarver, D. E., and Kofler, M. J. (2010). Competing core processes in attention-deficit/hyperactivity disorder (ADHD): do working memory deficiencies underlie behavioral inhibition deficits? J. Abnorm. Child Psychol. 38, 497–507. doi: 10.1007/s10802-010-9387-0

Alloway, T. P. (2007). Automated Working Memory Assessment. London, UK: Pearson Assessment.

Alloway, T. P., and Alloway, R. G. (2010). Investigating the predictive roles of working memory and IQ in academic attainment. J. Exp. Child Psychol. 106, 20–29. doi: 10.1016/j.jecp.2009.11.003

Alloway, T. P., Gathercole, S. E., Kirkwood, H., and Elliott, J. (2009). The cognitive and behavioral characteristics of children with low working memory. Child Dev. 80, 606–621. doi: 10.1111/j.1467-8624.2009.01282.x

Alloway, T. P., Gathercole, S. E., and Pickering, S. J. (2006). Verbal and visuo-spatial short-term and working memory in children: are they separable? Child Dev. 77, 1698–1716. doi: 10.1111/j.1467-8624.2006.00968.x

American Psychiatric Association. (2002). DSM-IV. Arlington, VA: American Psychiatric Association.

American Psychiatric Association. (2013). DSM 5. Arlington, VA: American Psychiatric Association.

Archibald, L. M., and Gathercole, S. E. (2007). Nonword repetition in specific language impairment: more than a phonological short-term memory deficit. Psychon. Bull. Rev. 14, 919–924. doi: 10.3758/BF03194122

Archibald, L. M., Joanisse, M., and Edmunds, A. (2011). Specific language or working memory impairments: a small scale observational study. Child Lang. Teach. Ther. 27, 294–312. doi: 10.1177/0265659010396779

Arica, E., and Conners, C. K. (1998). Gender differences in ADHD? J. Dev. Behav. Pediatr. 19, 77–83. doi: 10.1097/00004703-199804000-00003

Aronen, E. T., Vuontela, V., Steenari, M. R., Salmi, J., and Carlson, S. (2005). Working memory, psychiatric symptoms, and academic performance at school. Neurobiol. Learn. Mem. 83, 33–42. doi: 10.1016/j.nlm.2004.06.010

Baddeley, A. (2000). The episodic buffer: a new component of working memory? Trends Cogn. Sci. 4, 417–423. doi: 10.1016/S1364-6613(00)01538-2

Baddeley, A. D., and Hitch, G. (1974). Working memory. Psychol. Learn. Motiv. 8, 47–89. doi: 10.1016/S0079-7421(08)60452-1

Barkley, R. A. (1990). Attention Deficit Hyperactivity Disorder. A Handbook for Diagnosis and Treatment. New York, NY: Guildford.

Barkley, R. A. (1997). ADHD and the Nature of Self-Control. New York, NY: Guildford.

Bayliss, D. M., Jarrold, C., Gunn, D. M., and Baddeley, A. D. (2003). The complexities of complex span: explaining individual differences in working memory in children and adults. J. Exp. Psychol. Gen. 132, 71. doi: 10.1037/0096-3445.132.1.71

Bledsoe, J. C., Semrud-Clikeman, M., and Pliszka, S. R. (2010). Response inhibition and academic abilities in typically developing children with attention-deficit-hyperactivity disorder-combined subtype. Arch. Clin. Neuropsychol. 25, 671–679. doi: 10.1093/arclin/acq048

Burin, D. I., Irrazabal, N., and Quinn, J. G. (2007). Maintenance in visuo-spatial working memory. Psychologia 50, 90–101. doi: 10.2117/psysoc.2007.90

Carlson, C. L., and Mann, M. (2002). Sluggish cognitive tempo predicts a different pattern of impairment in the attention deficit hyperactivity disorder, predominantly inattentive type. J. Clin. Child Adolesc. Psychol. 31, 123–129. doi: 10.1207/S15374424JCCP3101_14

Carretti, B., Borella, E., Cornoldi, C., and De Beni, R. (2009). Role of working memory in explaining the performance of individuals with specific reading comprehension difficulties: a meta-analysis. Learn. Individ. Differ. 19, 246–251. doi: 10.1016/j.lindif.2008.10.002

Castellanos, F. X., Sonuga-Barke, E. J. S., Milham, M. P., and Tannock, R. (2006). Characterizing cognition in ADHD: beyond executive dysfunction. Trends Cogn. Sci. 10, 117–123. doi: 10.1016/j.tics.2006.01.011

Conners, C. K., and Multi-Health Staff. (2004). Continuous Performance Test. New York, NY: Multi-Health Systems Inc.

Conners, K. (1997). Conners' Teacher Rating Scale Revised Short-Form. New York, NY: Multi-Health Systems Inc.

Conway, A. R., Cowan, N., and Bunting, M. F. (2001). The cocktail party phenomenon revisited: the importance of working memory capacity. Psychon. Bull. Rev. 8, 331–335. doi: 10.3758/BF03196169

Conway, A. R., Jarrold, C. E., Kane, M. J., Miyake, A., and Towse, J. N. (2007). Variation in Working Memory. Oxford: Oxford University Press.

Cowan, N. (1995). Attention and Memory. An Integrated Framework. Oxford: Oxford University Press.

Cowan, N. (2008). What are the differences between long-term, short-term, and working memory? Prog. Brain Res. 169, 323–338. doi: 10.1016/S0079-6123(07)00020-9

Cowan, N. (2010). The magical mystery four how is working memory capacity limited, and why? Curr. Dir. Psychol. Sci. 19, 51–57. doi: 10.1177/0963721409359277

Daneman, M., and Carpenter, P. A. (1980). Individual differences in working memory and reading. J. Verbal Learn. Verbal Behav. 19, 450–466. doi: 10.1016/S0022-5371(80)90312-6

Delis, D. C., Kaplan, E., and Kramer, J. H. (2001). Delis-Kaplan Executive Function System (D-KEFS). San Antonio, TX: Pearson Education.

De Stefano, D., and LeFevre, J. A. (2004). The role of working memory in mental arithmetic. Eur. J. Cogn. Psychol. 16, 353–386. doi: 10.1080/09541440244000328

Duncan, J., and Owen, A. M. (2000). Common regions of the human frontal lobe recruited by diverse cognitive demands. Trends Neurosci. 23, 475–483. doi: 10.1016/S0166-2236(00)01633-7

Dunning, D. L., Holmes, J., and Gathercole, S. E. (2013). Does working memory training lead to generalized improvements in children with low working memory? A randomized controlled trial. Dev. Sci. 16, 915–925. doi: 10.1111/desc.12068

Engle, R. W., Carullo, J. J., and Collins, K. W. (1991). Individual differences in working memory for comprehension and following directions. J. Educ. Res. 253–262.

Epstein, J. N., Erkanli, A., Conners, C. K., Klaric, J., Costello, J. E., and Angold, A. (2003). Relations between continuous performance test performance measures and ADHD behaviors. J. Abnorm. Child Psychol. 3, 543–554. doi: 10.1023/A:1025405216339

Fabiano, G. A., Pelham, W. E. Jr., Coles, E. K., Gnagy, E. M., Chronis-Tuscano, A., and O'Connor, B. C. (2009). A meta-analysis of behavioral treatments for attention-deficit/hyperactivity disorder. Clin. Psychol. Rev. 29, 129–140. doi: 10.1016/j.cpr.2008.11.001

Fry, A. F., and Hale, S. (1996). Processing speed, working memory, and fluid intelligence: evidence for a developmental cascade. Psychol. Sci. 7, 237–241. doi: 10.1111/j.1467-9280.1996.tb00366.x

Gathercole, S., and Alloway, T. P. (2008). Working Memory and Learning: A Practical Guide for Teachers. London, UK: Sage.

Gathercole, S. E., Alloway, T. P., Kirkwood, H. J., Elliott, J. G., Holmes, J., and Hilton, K. A. (2008a). Attentional and executive function behaviors in children with poor working memory. Learn. Individ. Differ. 18, 214–223. doi: 10.1016/j.lindif.2007.10.003

Gathercole, S. E., Brown, L., and Pickering, S. J. (2003). Working memory assessments at school entry as longitudinal predictors of National Curriculum attainment levels. Educ. Child Psychol. 20, 109–122.

Gathercole, S. E., Durling, E., Evans, M., Jeffcock, S., and Stone, S. (2008b). Working memory abilities and children's performance in laboratory analogues of classroom activities. Appl. Cogn. Psychol. 22, 1019–1037. doi: 10.1002/acp.1407

Gathercole, S. E., Lamont, E., and Alloway, T. P. (2006). “Working memory in the classroom,” in Working Memory and Education, ed S. Pickering (London, UK: Elsevier Press), 219–240.

Gathercole, S. E., Pickering, S. J., Knight, C., and Stegmann, Z. (2004). Working memory skills and educational attainment: evidence from national curriculum assessments at 7 and 14 years of age. Appl. Cogn. Psychol. 18, 1–16. doi: 10.1002/acp.934

Gaub, M., and Carlson, C. L. (1997). Gender differences in ADHD: a meta-analysis and critical review. J. Am. Acad. Child Adolesc. Psychiatry 36, 1036–1045. doi: 10.1097/00004583-199708000-00011

Gioia, G. A., Isquith, P. K., Guy, S. C., and Kenworthy, L. (2000). Behavior Rating Inventory of Executive Function. Florida, FL: Psychological Assessment Resources, Inc.

Halperin, J. M., Trampush, J. W., Miller, C. J., Marks, D. J., and Newcorn, J. H. (2008). Neuropsychological outcome in adolescents/young adults with childhood ADHD: profiles of persisters, remitters and controls. J. Child Psychol. Psychiatry 49, 958–966. doi: 10.1111/j.1469-7610.2008.01926.x

Hartman, C. A., Willcutt, E. G., Rhee, S. H., and Pennington, B. F. (2004). The relation between sluggish cognitive tempo and DSM-IV ADHD. J. Abnorm. Child Psychol. 32, 491–503. doi: 10.1023/B:JACP.0000037779.85211.29

Hinshaw, S. P., Carte, E. T., Sami, N., Treuting, J. J., and Zupan, B. A. (2002). Preadolescent girls with attention-deficit/hyperactivity disorder: II. Neuropsychological performance in relation to subtypes and individual classification. J. Consul. Clin. Psychol. 70, 1099–1111. doi: 10.1037/0022-006X.70.5.1099

Holmes, J., Gathercole, S. E., and Dunning, D. L. (2009). Adaptive training leads to sustained enhancement of poor working memory in children. Dev. Sci. 12, F9–F15. doi: 10.1111/j.1467-7687.2009.00848.x

Holmes, J., Gathercole, S. E., Place, M., Dunning, D. L., Hilton, K. A., and Elliott, J. G. (2010). Working memory deficits can be overcome: impacts of training and medication on working memory in children with ADHD. Appl. Cogn. Psychol. 24, 827–836. doi: 10.1002/acp.1589

Huang-Pollock, C. L., Nigg, J. T., and Carr, T. H. (2005). Deficient attention is hard to find: applying the perceptual load model of selective attention to attention deficit hyperactivity disorder subtypes. J. Child Psychol. Psychiatry 46, 1211–1218. doi: 10.1111/j.1469-7610.2005.00410.x

Jensen, A. R. (1998). The G Factor: The Science of Mental Ability. Westport, CT: Praeger.

Kane, M. J., Bleckley, M. K., Conway, A. R., and Engle, R. W. (2001). A controlled-attention view of working-memory capacity. J. Exp. Psychol. Gen. 130, 169–183. doi: 10.1037/0096-3445.130.2.169

Kane, M. J., Brown, L. H., McVay, J. C., Sylvia, P. J., Myin-Germeys, I., and Kwapil, T. R. (2007). For whom the mind wanders, and when- An experience-sampling study of working memory and executive control in daily life. Psychol. Sci. 18, 614–621. doi: 10.1111/j.1467-9280.2007.01948.x

Kane, M. J., and Engle, R. W. (2000). Working-memory capacity, proactive interference, and divided attention: limits on long-term memory retrieval. J. Exp. Psychol. Learn. Mem. Cogn. 26, 336. doi: 10.1037/0278-7393.26.2.336

Kane, M. J., Hambrick, D. Z., Tuholski, S. W., Wilhelm, O., Payne, T. W., and Engle, R. W. (2004). The generality of working memory capacity: a latent-variable approach to verbal and visuospatial memory span and reasoning. J. Exp. Psychol. Gen. 133, 189. doi: 10.1037/0096-3445.133.2.189

Kasper, L. J., Alderson, R. M., and Hudec, K. L. (2012). Moderators of working memory deficits in children with attention-deficit/hyperactivity disorder (ADHD): a meta-analytic review. Clin. Psychol. Rev. 32, 605–617. doi: 10.1016/j.cpr.2012.07.001

Kofler, M. J., Rapport, M. D., Bolden, J., Sarver, D. E., and Raiker, J. S. (2010). ADHD and working memory: the impact of central executive deficits and exceeding storage/rehearsal capacity on observed inattentive behavior. J. Abnorm. Child Psychol. 38, 149–161. doi: 10.1007/s10802-009-9357-6

Loe, I. M., and Feldman, H. M. (2007). Academic and educational outcomes of children with ADHD. J. Pediatr. Psychol. 32, 643–654. doi: 10.1093/jpepsy/jsl054

Lui, M., and Tannock, R. (2007). Working memory and inattentive behavior in a community sample of children. Behav. Brain Funct. 3:12. doi: 10.1186/1744-9081-3-12

Manly, T., Robertson, I. H., Anderson, V., and Nimmo-Smith, A. (1999). Test of Everyday Attention for Children. Cambridge: Thames Valley Test Company.

Martinussen, R., Hayden, J., Hogg-Johnson, S., and Tannock, R. (2005). A meta-analysis of working memory impairments in children with attention-deficit/hyperactivity disorder. J. Am. Acad. Child Adolesc. Psychiatry 44, 377–384. doi: 10.1097/01.chi.0000153228.72591.73

Martinussen, R., and Tannock, R. (2006). Working memory impairments in children with attention-deficit hyperactivity disorder with and without comorbid language learning disorders. J. Clin. Exp. Neuropsychol. 28, 1073–1094. doi: 10.1080/13803390500205700

Marzocchi, G. M., Oosterlaan, J., Zuddas, A., Cavolina, P., Geurts, H., Redigolo, D., et al. (2008). Contrasting deficits on executive functions between ADHD and reading disabled children. J. Child Psychol. Psychiatry 49, 543–552. doi: 10.1111/j.1469-7610.2007.01859.x

McBurnett, K., Pfiffner, L. J., and Frick, P. J. (2001). Symptom properties as a function of ADHD type: an argument for continued study of sluggish cognitive tempo. J. Abnorm. Child Psychol. 29, 207–213. doi: 10.1023/A:1010377530749

McVay, J. C., and Kane, M. J. (2009). Conducting the train of thought: working memory capacity, goal neglect, and mind wandering in an executive-control task. J. Exp. Psychol. Learn. Mem. Cogn. 35, 196. doi: 10.1037/a0014104

Miyake, A., Friedman, N. P., Rettinger, D. A., Shah, P., and Hegarty, M. (2001). How are visuospatial working memory, executive functioning, and spatial abilities related? A latent-variable analysis. J. Exp. Psychol. Gen. 130, 621. doi: 10.1037/0096-3445.130.4.621

Nee, D. E., Berman, M. G., Moore, K. S., and Jonides, J. (2008). Neuroscientific evidence about the distinction between short-and long-term memory. Curr. Dir. Psychol. Sci. 17, 102–106. doi: 10.1111/j.1467-8721.2008.00557.x

Nunnally, J. C. (1978). Psychometric Theory, 2nd Edn. New York, NY: McGraw Hill.

Oades, R. D., and Christiansen, H. (2008). Cognitive switching processes in young people with attention-deficit/hyperactivity disorder. Arch. Clin. Neuropsychol. 23, 21–32. doi: 10.1016/j.acn.2007.09.002

Pimperton, H., and Nation, K. (2014). Poor comprehenders in the classroom: teacher ratings of behavior in children with poor reading comprehension and its relationship with individual differences in working memory. J. Learn. Disabil. 47, 199–207. doi: 10.1177/0022219412454172

Rapport, M. D., Alderson, R. M., Kofler, M. J., Sarver, D. E., Bolden, J., and Sims, V. (2008). Working memory deficits in boys with attention-deficit/hyperactivity disorder (ADHD): the contribution of central executive and subsystem processes. J. Abnorm. Child Psychol. 36, 825–837. doi: 10.1007/s10802-008-9215-y

Rapport, M. D., Bolden, J., Kofler, M. J., Sarver, D. E., Raiker, J. S., and Alderson, R. M. (2009). Hyperactivity in boys with attention-deficit/hyperactivity disorder (ADHD): a ubiquitous core symptom or manifestation of working memory deficits? J. Abnorm. Child Psychol. 37, 521–534. doi: 10.1007/s10802-008-9287-8

Rapport, M. D., Orban, S. A., Kofler, M. J., and Friedman, L. M. (2013). Do programs designed to train working memory, other executive functions, and attention benefit children with ADHD? A meta-analytic review of cognitive, academic, and behavioral outcomes. Clin. Psychol. Rev. 33, 1237–1252. doi: 10.1016/j.cpr.2013.08.005

Rhodes, S. M., Coghill, D. R., and Matthews, K. (2005). Neuropsychological functioning in stimulant-naive boys with hyperkinetic disorder. Psychol. Med. 35, 1109–1120. doi: 10.1017/S0033291705004599

Rosen, V. M., and Engle, R. W. (1997). The role of working memory capacity in retrieval. J. Exp. Psychol. Gen. 126, 211. doi: 10.1037/0096-3445.126.3.211

Rubia, K., Smith, A. B., Brammer, M. J., and Taylor, E. (2007). Temporal lobe dysfunction in medication-naive boys with attention-deficit/hyperactivity disorder during attention allocation and its relation to response variability. Biol. Psychiatry 62, 999–1006. doi: 10.1016/j.biopsych.2007.02.024

Rubia, K., Smith, A., Halari, R., Matsukura, F., Mohammad, M., Taylor, E., et al. (2009). Disorder-specific dissociation of orbitofrontal dysfunction in boys with pure conduct disorder during reward and ventrolateral prefrontal dysfunction in boys with pure ADHD during sustained attention. Am. J. Psychiatry 166, 83–94. doi: 10.1176/appi.ajp.2008.08020212

Salthouse, T. A. (1996). The processing-speed theory of adult age differences in cognition. Psychol. Rev. 103, 403. doi: 10.1037/0033-295X.103.3.403

Schoemaker, K., Bunte, T., Espy, K. A., Deković, M., and Matthys, W. (2014). Executive functions in preschool children with ADHD and DBD: an 18-month longitudinal study. Dev. Neuropsychol. 39, 302–315. doi: 10.1080/87565641.2014.911875

Shah, P., and Miyake, A. (1999). Models of Working Memory. Models of Working Memory: Mechanisms of Active Maintenance and Executive Control. New York, NY: Cambridge University Press.

Solanto, M. V., Gilbert, S. N., Raj, A., Zhu, J., Pope-Boyd, S., Stepak, B., et al. (2007). Neurocognitive functioning in ADH/HD, predominantly inattention and combined subtypes. J. Abnorm. Child Psychol. 35, 729–744. doi: 10.1007/s10802-007-9123-6

St Clair-Thompson, H. L. (2011). Executive functions and working memory behaviors in children with a poor working memory. Learn. Individ. Differ. 21, 409–414. doi: 10.1016/j.lindif.2011.02.008

St Clair-Thompson, H. L., and Gathercole, S. E. (2006). Executive functions and achievements in school: shifting, updating, inhibition, and working memory. Q. J. Exp. Psychol. 59, 745–759. doi: 10.1080/17470210500162854

St Clair-Thompson, H., Stevens, R., Hunt, A., and Bolder, E. (2010). Improving children's working memory and classroom performance. Educ. Psychol. 30, 203–219. doi: 10.1080/01443410903509259

Swanson, H. L., and Ashbaker, M. H. (2000). Working memory, short-term memory, speech rate, word recognition and reading comprehension in learning disabled readers: does the executive system have a role? Intelligence 28, 1–30. doi: 10.1016/S0160-2896(99)00025-2

Swanson, H. L., and Sachse-Lee, C. (2001). Mathematical problem solving and working memory in children with learning disabilities: both executive and phonological processes are important. J. Exp. Child Psychol. 79, 294–321. doi: 10.1006/jecp.2000.2587

Szucs, D., Devine, A., Soltesz, F., Nobes, A., and Gabriel, F. (2013). Developmental dyscalculia is related to visuo-spatial memory and inhibition impairment. Cortex 49, 2674–2688. doi: 10.1016/j.cortex.2013.06.007

Thorell, L. B. (2007). Do delay aversion and executive function deficits make distinct contributions to the functional impact of ADHD symptoms? A study of early academic skill deficits. J. Child Psychol. Psychiatry 48, 1061–1070. doi: 10.1111/j.1469-7610.2007.01777.x

Toplack, M. E., Bucciarelli, S. M., Jain, U., and Tannock, R. (2009). Executive functions: performance based measures and the behavior rating inventory of executive function (BRIEF) in adolescents with attention deficit/hyperactivity disorder (ADHD). Child Neuropsychol. 15, 53–72. doi: 10.1080/09297040802070929

Tripp, G., and Alsop, B. (2001). Sensitivity to reward delay in children with attention deficit hyperactivity disorder (ADHD). J. Child Psychol. Psychiatry 42, 691–698. doi: 10.1111/1469-7610.00764

Unsworth, N., and Engle, R. W. (2007). The nature of individual differences in working memory capacity: active maintenance in primary memory and controlled search from secondary memory. Psychol. Rev. 114, 104. doi: 10.1037/0033-295X.114.1.104

Valera, E. M., Faraone, S. V., Murray, K. E., and Seidman, L. J. (2007). Meta-analysis of structural imaging findings in attention-deficit/hyperactivity disorder. Biol. Psychiatry 61, 1361–1369. doi: 10.1016/j.biopsych.2006.06.011

van Mourik, R., Oosterlaan, J., and Sergeant, J. A. (2005). The Stroop revisited: a meta-analysis of interference control in AD/HD. J. Child Psychol. Psychiatry 46, 150–165. doi: 10.1111/j.1469-7610.2004.00345.x

Warner-Rogers, J., Taylor, A., Taylor, E., and Sandberg, S. (2000). Inattentive behavior in childhood epidemiology and implications for development. J. Learn. Disabil. 33, 520–536. doi: 10.1177/002221940003300602

Wechsler, D. (1996). Wechsler Objective Number Dimensions (WOND). London: Pearson Education.

Wechsler, D. (1993). Wechsler Objective Reading Dimensions (WORD). London: Pearson Education.

Wechsler, D. (1999). Wechsler Abbreviated Scale of Intelligence (WASI). London: Pearson Education.

Willcutt, E. G., Doyle, A. E., Nigg, J. T., Faraone, S. V., and Pennington, B. F. (2005a). Validity of the executive function theory of attention-deficit/hyperactivity disorder: a meta-analytic review. Biol. Psychiatry 57, 1336–1346.

Willcutt, E. G., Pennington, B. F., Olson, R. K., Chhabildas, N., and Hulslander, J. (2005b). Neuropsychological analyses of comorbidity between reading disability and attention deficit hyperactivity disorder: in search of the common deficit. Dev. Neuropsychol. 27, 35–78. doi: 10.1207/s15326942dn2701_3

Yang, T., Gathercole, S. E., and Allen, R. J. (2014). Benefit of enactment over oral repetition of verbal instruction does not require additional working memory during encoding. Psychon. Bull. Rev. 21, 186–192. doi: 10.3758/s13423-013-0471-7

Zelazo, P. D., and Muller, U. (2002). “Executive function in typical and atypical development,” in Handbook of Childhood Cognitive Development, ed U. Goswami (Oxford: Blackwell), 445–469.

Conflict of Interest Statement: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Received: 27 July 2014; Accepted: 16 November 2014; Published online: 09 December 2014.

Citation: Holmes J, Hilton KA, Place M, Alloway TP, Elliott JG and Gathercole SE (2014) Children with low working memory and children with ADHD: same or different? Front. Hum. Neurosci. 8:976. doi: 10.3389/fnhum.2014.00976

This article was submitted to the journal Frontiers in Human Neuroscience.

Copyright © 2014 Holmes, Hilton, Place, Alloway, Elliott and Gathercole. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

ORIGINAL RESEARCH ARTICLE
published: 05 February 2015
doi: 10.3389/fnhum.2015.00037

Temporal dynamics of attentional selection in adult male carriers of the fragile X premutation allele and adult controls

Ling M. Wong1,2[*], Flora Tassone1,3, Susan M. Rivera1,4,5 and Tony J. Simon1,2

1MIND Institute, University of California Davis School of Medicine, Sacramento, CA, USA

2Department of Psychiatry and Behavioral Sciences, University of California Davis School of Medicine, Sacramento, CA, USA

3Department of Biochemistry and Molecular Medicine, University of California Davis School of Medicine, Sacramento, CA, USA

4Department of Psychology, University of California Davis, Davis, CA, USA

5Center for Mind and Brain, University of California Davis, Davis, CA, USA

* Correspondence: Ling M. Wong, MIND Institute; Department of Psychiatry and Behavioral Sciences, University of California Davis School of Medicine, 2825 50th Street, Sacramento, CA 95817, USA e-mail: lmewong@ucdavis.edu

Edited by:
Steve Majerus, Université de Liège, Belgium

Reviewed by:
Steve Majerus, Université de Liège, Belgium
Claudine Kraan, Monash University, Australia

Carriers of the fragile X premutation allele (fXPCs) have an expanded CGG trinucleotide repeat size within the FMR1 gene and are at increased risk of developing fragile x-associated tremor/ataxia syndrome (FXTAS). Previous research has shown that male fXPCs with FXTAS exhibit cognitive decline, predominantly in executive functions such as inhibitory control and working memory. Recent evidence suggests fXPCs may also exhibit impairments in processing temporal information. The attentional blink (AB) task is often used to examine the dynamics of attentional selection, but disagreements exist as to whether the AB is due to excessive or insufficient attentional control. In this study, we used a variant of the AB task and neuropsychological testing to explore the dynamics of attentional selection, relate AB performance to attentional control, and determine whether fXPCs exhibited temporal and/or attentional control impairments. Participants were adult male fXPCs, aged 18–48 years and asymptomatic for FXTAS (n = 19) and age-matched male controls (n = 20). We found that fXPCs did not differ from controls in the AB task, indicating that the temporal dynamics of attentional selection were intact. However, they were impaired in the letter-number sequencing task, a test of executive working memory. In the combined fXPC and control group, letter-number sequencing performance correlated positively with AB magnitude. This finding supports models that posit the AB is due to excess attentional control. In our two-pronged analysis approach, in control participants we replicated a previously observed effect and demonstrated that it persists under more stringent theoretical constraints, and we enhance our understanding of fXPCs by demonstrating that at least some aspects of temporal processing may be spared.

Keywords: fragile X, FMR1 gene, attentional blink, attention, temporal processing, executive function, inhibition, letter number sequencing

1. INTRODUCTION

Fragile X premutation carriers (fXPCs) are defined as individuals who have a 55–200 CGG repeat expansion in the FMR1 gene, which is located on the X chromosome. The premutation allele is so named because it can expand in subsequent generations into the full mutation allele (>200 CGG). Individuals with a full mutation often have Fragile X Syndrome (FXS), which is associated with a ~30% chance of developing autism (Rogers et al., 2001) and low levels of the FMR1 protein (FMRP). The premutation allele has an estimated prevalence of 1 in 260– 813 in males and 1 in 113–259 in females (Hagerman, 2008). Male fXPCs have an elevated risk of developing fragile X-associated tremor/ataxia syndrome (FXTAS), a neurodegenerative disorder with age-dependent penetrance (17%, 38%, 47%, and 75% [lower-bound estimates] for participants aged 50–59, 60–69, 70–79, and ≥ 80 years, respectively), characterized by intention tremor, ataxia, and parkinsonism (Jacquemont et al., 2004). The FXTAS phenotype is thought to be due to a toxic gain of function of excess FMR1 mRNA, which is associated with increasing CGG repeat length (Hagerman and Hagerman, 2004). This potential mechanism is supported by associations between CGG repeat length and age of onset of FXTAS (Tassone et al., 2007); level of motor impairment in fXPCs (Leehey et al., 2008); negative associations of CGG repeat length with brain volume, packing density of middle cerebellar peduncle, and gray matter density of the dorsomedial frontal lobes (Cohen et al., 2006; Hashimoto et al., 2011a,b); and negative associations of both CGG repeat length and FMR1 mRNA with the connectivity strength of the superior cerebellar peduncle (Wang et al., 2013).

Increased CGG repeat length is associated with executive function impairment in fXPCs without FXTAS, so it is possible that subtle cognitive impairments precede motor impairments. Firstly, males with FXTAS can exhibit cognitive decline, particularly in executive functions such as inhibitory control and working memory (WM) (Grigsby et al., 2007, 2008; Brega et al., 2008; Cornish et al., 2009, 2011). Men with FXTAS have impairments in various aspects of inhibitory control, including interference control, cognitive control, and behavioral inhibition, as assessed by the Stroop Color-Word test, Behavioral Dyscontrol Scale, and Controlled Oral Word Association Test (Grigsby et al., 2006, 2007, 2008; Brega et al., 2008). Secondly, men with longer CGG repeat length are impaired in some of these tasks (Grigsby et al., 2006, 2007), and in male fXPCs asymptomatic for FXTAS, CGG repeat length modulates the effect of age on a behavioral inhibition task (Cornish et al., 2011; Hunter et al., 2012).

In addition to inhibitory control impairments, evidence suggests that processing of temporal information may be affected in fXPCs. This is suggested by relatively high FMRP in magnocellular (M) layers of the lateral geniculate nucleus (Zangenehpour et al., 2009). The M pathway of visual processing feeds into cortical areas responsible for motion perception and visuomotor coordination, which requires use of spatial and temporal information. Meanwhile, the parvocellular (P) pathway feeds into cortical regions responsible for color perception and object recognition, which relies much less heavily on spatial and temporal information (Van Essen and Gallant, 1994). As described in a recent review (Kraan et al., 2013), several studies report impaired visuospatial and temporal (e.g., ordering or memory) function in both male and female fXPCs. For example, fXPCs were found to exhibit a specific impairment in M pathway processing (Kéeri and Benedek, 2009, 2012), and functioning in this pathway has been linked to FMRP expression in males who were not fXPCs (Kéeri and Benedek, 2011). Female and male fXPCs exhibit impairments in tasks of visuospatial function (Goodrich-Hunsaker et al., 2011a,b; Hocking et al., 2012; Wong et al., 2012). In a mouse model of the premutation allele, female CGG knock-in (KI) mice demonstrated a CGG repeat length-sensitive impairment in temporal ordering (Hunsaker et al., 2010) and impaired temporal memory for spatial locations (Borthwell et al., 2012). To date, there has only been one study of temporal memory in human fXPCs. This study found that while typical adults show increased activation in the when pathway (i.e., right temporoparietal junction: TPJ) during temporal relative to spatial WM retrieval, adult fXPCs of both sexes failed to exhibit this pattern (Kim et al., 2014).

However, to date there have been no studies of temporal attention in fXPCs of any age. Due to the sparse literature in this area, it is unclear whether temporal processing and/or temporal attention are specifically impaired in fXPCs. For our purposes in this study, we define temporal attention as the temporal dynamics by which attention is deployed, and temporal processing as the low-level computation of temporal parameters of incoming stimuli (e.g., which item appeared first, next, or last).

The attentional blink (AB) task has been used to examine the temporal dynamics of attentional selection. During rapid serial visual presentation (RSVP) of stimuli, the “attentional blink” refers to a decrement in identification accuracy for the second of two targets in the stream, which occurs when the targets occur in close temporal proximity (Raymond et al., 1992). In a variation of the classic AB paradigm, the temporal distance between two targets remained constant, but the attend instructions and presence of intervening distractors were manipulated (Di Lollo et al., 2005; Dell'Acqua et al., 2009). In that variant, an AB was observed when distractors intervened between targets, but not when targets were continually presented (contiguous target condition). A trailing target presented in very close temporal proximity to the first target is often identified just as well as the first target, a phenomenon known as “Lag 1 sparing.” Thus, this lack of an AB for additional trailing targets has been referred to as “spreading the sparing” (Olivers et al., 2007).

This AB task provides an interesting paradigm from which to explore both the temporal dynamics of attentional selection and attentional control. Timing was held constant across conditions in the task variant used by Di Lollo et al. (2005). Thus, the authors proposed that the AB was due to a “temporary loss of attentional control” (TLC model), and not due to a purely temporal limitation of how quickly items can be processed and consolidated into WM. According to their model, RSVP processing is governed by an attention filter endogenously configured to select targets and exclude distractors, and by a central processor which switches between monitoring and consolidation processes. After the first target is identified, the central processor switches from monitoring incoming stimuli to consolidating the first target into memory. No longer under control of the central processor, the attention filter can be exogenously reconfigured by incoming stimuli. If targets continue to appear, the filter is already configured to select them, so targets proceed into consolidation. However, if a distractor appears, the filter is reconfigured, such that when a trailing target appears, the filter is no longer optimally configured for target selection, so processing of the trailing target suffers. While the TLC model posits that the AB is due to insufficient attentional control, alternate models posit the opposite, that the AB is due to excessive control (a more extensive description of models accounting for the AB is included in the Discussion). In such models, the appearance of a distractor can trigger a suppressive response, such that the trailing target is suppressed instead of selected (Raymond et al., 1992; Olivers et al., 2007; Taatgen et al., 2009). Therefore, the AB task can be modeled as assessing the temporal dynamics of attentional selection as well as inhibition of distracting information.

In this study, we had several aims. First, we sought to replicate and extend the “spreading the sparing” effect, previously observed in undergraduate students, in a sample of adults aged 18–48 years. We constrained analyses to trials in which preceding targets were accurately identified, or accurately identified and reported in the correct order, and discuss how results relate to predictions of competing models. Second, we sought to more fully characterize the temporal dynamics of attentional selection. We accomplished this by examining perception of temporal order and inter-target competition for attention resources. Third, we sought to determine whether the AB is better modeled as due to excessive or insufficient control. To do this, we related AB task performance to measures of executive WM. We will discuss how results from these three aims relate to existing models of attention in control participants. Our final aim was to determine whether fXPCs were impaired relative to controls on any of these measures, and whether performance was associated with CGG or FMR1 mRNA. Understanding the core cognitive phenotype in fXPCs may facilitate early identification of individuals most at risk for developing FXTAS.

Inhibition of distracting information is a critical component of successful WM performance. Thus, this variant of the AB task, by manipulating attention demands (i.e., the type of items to be attended or ignored) and WM load (i.e., two or three items), allows for investigation into the nature of the interaction between selective attention and WM. Specifically, in Aim 1 we compare performance when three targets are presented vs. performance when there are only two targets but with an intervening distractor. We also contrast this to performance when one target is a member of the distractor stimulus set. We find evidence that when an item from the distractor set is a target and enters WM, selective attention to a subsequent target from the target set is impaired. We also examine the effect of increasingly stringent criteria on attention demands on WM performance. In Aim 2 we examine how two items in WM compete for resources. In Aim 3, we explore the relationship between attention and WM by examining whether two commonly used tests of executive WM predict the AB phenomenon. Together, these aims capitalize on the unique design of the AB task to better understand how selective attention and WM interact and relate to attentional control.

2. MATERIALS AND METHODS

2.1. PARTICIPANTS

Participants were 41 males aged 18–48 years, including 20 control participants and 21 fXPCs. FXPCs had at least one family member with FXS. All had normal, or corrected to normal, vision.

Participants were recruited through the NeuroTherapeutics Research Institute (NTRI) at the Medical Investigation of Neurodevelopmental Disorders (MIND) Institute at the University of California, Davis Medical Center, and from the community through recruitment advertisements. FXPCs were recruited from known FXS pedigrees, and controls were recruited from pedigrees or the community. Exclusion criteria were: acute medical condition such as renal, liver, cardiac, or other disease that may be associated with brain atrophy or dysfunction; current or past history of major DSM-IV Axis I psychiatric disorder; history of head trauma, toxic encephalopathy, encephalitis, or bacterial meningitis; history of alcoholism or drug problem; and use of current medications that affect cerebral blood flow (e.g., beta blockers). This study was approved by the Institutional Review Board of the University of California, Davis and conformed to institutional and federal guidelines for the protection of human participants. Written informed consent was obtained before participation from all participants.

2.2. PROCEDURE

We conducted this experiment as part of a larger study. The study visit involved administration of cognitive tests and a blood draw. All fXPCs were evaluated by a physician and determined to be asymptomatic for FXTAS following published criteria (Jacquemont et al., 2003; Bacalman et al., 2006). All control participants completed the Tremor Disability Rating Scale (Jacquemont et al., 2004). Of 31 common actions, one control participant reported difficulty or disability on two actions (“using eyedrops” and “threading a needle”). This participant's performance was not extreme, so he was included in all analyses as a control participant.

2.2.1. Molecular assays

Molecular data were FMR1 CGG repeat length and mRNA expression level. Genomic DNA was isolated from peripheral blood leukocytes using standard methods (Puregene Kit; Gentra Inc., Valencia, CA, USA). Repeat length was determined using Southern blot analysis and PCR amplification of genomic DNA as described previously (Tassone et al., 2008). All quantifications of FMR1 mRNA were performed using a 7900 Sequence detector (PE Biosystems).

2.2.2. Psychological assessment

Full scale IQ (FSIQ) was measured using either the Wechsler Adult Intelligence Scale, third edition (WAIS-III) (Wechsler, 1997) or the Wechsler Abbreviated Scale of Intelligence (WASI) (Wechsler, 1999). Due to time constraints during testing, FSIQ data were not available from all participants.

2.2.2.1. Working memory measures. Two sub-scales from the FSIQ test were identified. Digit span backward total score is a measure of the ability to hold and process information in WM. In this task, participants hear a sequence of digits and must repeat them in the reverse order. Letter-number sequencing requires holding as well as manipulating information in WM. In this task, participants hear a mixed sequence of digits and letters, and must report the digits in ascending order, and the letters in alphabetical order. Thus, both tasks require storage and executive WM, while letter-number sequencing additionally requires the ability to switch attention between task sets. These measures were used to examine the effect of executive WM on task performance.

2.2.2.2. ADHD assessment. ADHD status was measured using the 66-item Conners' Adult ADHD Rating Scale (CAARS) (Conners et al., 1999). Participants completed a self-report, and an observer-report was completed by a spouse, partner, family member, or close friend. Scores were adjusted according to established age and sex norms. Sub-scores measured DSM-IV inattentive, hyperactive-impulsive, and total ADHD symptoms. Due to time constraints during testing and inability to collect observer reports during testing, ADHD data were not available from all participants.

2.2.3. Temporal attention task

Stimuli were presented via E-Prime 2.0.8.90 (http://www.pstnet.com) on a Tobii T120 eye tracking system (http://www.tobii.com). Participants were seated 60 cm from the eye-tracking monitor in a chin rest to maintain head position. Participants performed practice trials and were observed during task performance to ensure appropriate task performance.

This task replicated parameters used previously (Di Lollo et al., 2005). Participants pressed a button to initiate each trial. A fixation cross appeared in the center of the screen for 500 ms, and was followed by a RSVP stream. The stream consisted of 5–10 digits followed by a 3-item target sequence, and ended with one digit which served as a perceptual mask. Length of the initial stream of digits (5–10 items) was pseudo-randomized so that every six trials, all possible trial lengths were presented. The target sequence consisted of three letters or a letter-number-letter sequence, depending on condition, which will be described shortly. All stimuli appeared in Century Gothic font and subtended approximately 1.45 ° in width and 2.8 ° in height. Stimuli were black characters on a gray background presented for 30 ms, with a blank inter-stimulus interval of 70 ms. Digits ranged from 0–9, and were presented in randomized order, with the constraint that no digit was identical to the previously presented digit. Letters comprised the English alphabet, excepting I, O, Q, or Z. Letters were not repeated within a trial, and were pseudo-randomized so that no sequences formed words or common abbreviations. Participants were informed of the excluded letters and that digits would range from 0–9.

In each of three conditions (Figure 1), participants attended to 2–3 items (positions T1, T2, and T3) within the target sequence. In the Uniform condition, participants attended to three letters. They were informed they would view a stream of digits with three letters in the stream, and they were instructed to attend all three letters and to type them into the keyboard in any order when prompted by a question mark at the end of the trial. In the Varied–2 condition, participants attended to two letters but not to digits. Instructions were similar to those in the Uniform condition except participants were informed there would be two letters and they should attend both letters. In the Varied–3 condition, participants attended to two letters as well as the digit that appeared temporally between them. Instructions were similar to those in the Uniform condition except participants were informed there would be a letter-number-letter sequence embedded in the stream, and they were to identify both letters and the number that appeared in between the letters, but not any of the other numbers. The Varied–2 and Varied–3 conditions differed only in instruction; in both conditions, a digit was presented temporally between two letters. In other words, the same pattern of stimuli were presented in both conditions, but all digits were distractors to be ignored in the Varied–2 condition; meanwhile, in the Varied–3 condition the digit appearing between two letter targets was to be attended, and all remaining digits were distractors to be ignored. Each of the three conditions was presented as a block, with specific instructions preceding each block. Participants completed 10 practice trials and 60 experimental trials of each block. Trial stimuli were identical for all participants, although the order was randomized across participants. Block order was counterbalanced across participants (Uniform/Varied–2/Varied–3 and Varied–3/Varied–2/Uniform).

FIGURE 1. Attentional Blink (AB) task. In each blocked condition, participants were instructed to attend to certain types of items. Note that the Varied conditions differ only in the attend instruction. In the Varied–3 condition, the correct number is the number that occurs temporally between the two letters. Target position 2 is marked with an asterisk because in the Varied-2 condition, there are only two targets.

Thus, letters were always targets, and digits were only targets in the Varied-3 condition and if the digit occurred after the first letter and before the second letter. In the Uniform condition, the three letters were always presented in direct succession, while in the Varied-2 and Varied-3 condition, the first letter was always immediately followed by a digit, which was always immediately followed by the second letter. In these two conditions, only a single digit appeared between letters. In all conditions, the first target letter appeared in position 6–11 in the RSVP stream, and the second letter (in the Varied–2 and Varied–3 conditions) or last letter (in the Uniform condition) appeared in position 8–13 in the stream.

As described by Di Lollo et al. (2005), the dependent measure was target identification accuracy for each target in each condition. A decrement in T3 accuracy relative to T1 accuracy is defined as the AB. According to the TLC model, an AB in the Varied–2 condition would indicate that when a distractor appears between two successive targets, the distractor disrupts the selection filter. Thus, when the trailing target appears, the filter is not optimally configured to process that target. This non-optimal configuration results in poor identification accuracy for the trailing target. In the Uniform condition, no distractors appear between successive targets, so the selection filter remains optimally configured, and the trailing target does not suffer a decrement in identification accuracy.

The Varied–3 condition was included to address the concern that poor T3 performance in the Varied–2 condition might be due to the need to suppress the intervening distractor. Suppression processes might be interfering with target identification processes. Thus, in the Varied–3 condition, like in the Uniform condition, participants must report all three presented items (i.e., no suppression). Similar T1 and T3 performance between Varied–2 and Varied–3 conditions indicates that even when participants are told to ignore all digits, they perform as if they are attending to the intervening digit.

2.3. STATISTICAL ANALYSES

In Di Lollo et al. (2005), target identification accuracy was calculated regardless of accuracy for other targets within the trial (“non-conditional accuracy”). However, as Dell'Acqua et al. (2009) note, for proper interpretation of T2 accuracy, T1 responses must be accurate (“conditional accuracy”). This is an important requirement to interpret whether or not T2 accuracy decrements result from ongoing T1 processing. To replicate previous findings as well as extend their interpretation, we calculated both non-conditional and conditional accuracy.

Specifically, for Aim 1, three analyses of variance (ANOVAs) were run. In the first ANOVA, the dependent variable was target identification accuracy. For the second ANOVA, the data were divided such that T2 and T3 accuracy were computed only from trials in which the preceding target(s) were correctly identified (T1 and T2 in the Uniform and Varied–3 conditions; T1 in the Varied–2 condition). Thus, in the second ANOVA, the dependent variable was target identification accuracy with conditional accuracy. For the third ANOVA, the data were further divided such that T2 and T3 accuracy were computed only from trials in which the preceding target(s) were correctly identified and reported in the correct ordinal position. Thus, in the third ANOVA, the dependent variable was target identification accuracy with conditional accuracy and order. In each of the three ANOVAs, we examined main effects of Group, Condition, and Target; two-way interactions of Group X Condition, Group X Target, and Condition X Target; and the three-way interaction of Group X Condition X Target. The AB was defined as T1 accuracy minus T3 accuracy in a particular condition. The “T1 enhancement effect” was defined as higher T1 accuracy in the Varied–2 or Varied–3 condition than Uniform condition.

For Aim 2, we included only trials in which T1 and T3 were accurately identified, and if T2 was to be attended (i.e., Uniform and Varied–3 trials), if it was correctly identified. Thus, we included only trials in which all targets were correctly identified. The dependent variable was percentage of trials in which T1 and T3 were reported in the correct relative order (i.e., T1 reported before T3, even if T2 was also reported). We ran an ANOVA of Group X Condition on order accuracy. Ordinal position of T2 report was ignored to avoid potentially confounding effects on perception of temporal order. Specifically, because letters occurred only in the T1 and T3 positions, while both letters and numbers occurred at T2, a mismatch in stimulus set between T2 and flanking targets (i.e., in a letter-number-letter sequence), could lead to arbitrary differences in response order (e.g., letter-letter-number or number-letter-letter).

To examine the effect of inter-target competition, we tested whether (A) T1 accuracy predicted T3 accuracy, and (B) T3 accuracy predicted T1 accuracy. The Uniform and Varied–3 conditions have a memory load of three items, as opposed to two in the Varied–2 condition, so we excluded the Varied–2 condition from these analyses. We ran an ANOVA of (A) Group X Condition X T1 accuracy on T3 accuracy, and of (B) Group X Condition X T3 accuracy on T1 accuracy.

For Aim 3, the magnitude of the AB in the Varied–3 condition with conditional accuracy was calculated for each participant. We chose the Varied–3 condition, as opposed to the Varied–2 condition, because the instruction was to attend three contiguous items. Thus, while an AB should be observed in both the Varied–2 and Varied–3 conditions, the continuous attention required in the Varied–3 condition allows for ease of interpretation. To examine whether the AB relates to excessive or insufficient attentional control and whether this relationship differs between groups, we used linear regression to assess the effect of group on AB magnitude, the relationship between AB magnitude and neuropsychological measures of executive WM (i.e., digit span backward total score or letter-number sequencing score), and whether the relationship between neurospsychological measures and AB magnitude differed between groups. Specifically, we tested models with AB magnitude as the outcome variable and Group X digit span backward score as predictors or Group X letter-number sequencing score as predictors. To examine whether our results were due to differences in overall attention function, we examined the effect of ADHD status on task performance. Specifically, we used linear regression to examine the effect of group and either self- or observer-report of ADHD symptoms on AB magnitude.

FXTAS exhibits age-dependent penetrance and fXPCs are at elevated risk for developing FXTAS, so we reasoned that even though our sample was asymptomatic for FXTAS, we should test for effects of age. Accordingly, for each analysis we also ran models with an additional age covariate. For all analyses, the addition of age did not change the pattern of results. Therefore, we report results from analyses without age as a predictor. For all analyses, a p < 0.05 was considered statistically significant.

3. RESULTS

3.1. STUDY SAMPLE

A total of 20 control participants and 21 fXPCs performed the tasks (Table 1). The mean age (± SD) was 29.95 (6.48) years for controls and 32.17 (7.74) for fXPCs, which did not differ significantly (t = −1.00, p = 0.32). The mean CGG repeat length was 29.40 (5.63) (range: 20–44) for controls and 97.33 (24.62) (range: 55–146) for fXPCs, which differed significantly (t = −12.20, p < 0.001). One participant expressed two variants of the premutation allele (120 and 156). His performance was not extreme in any task, so he was included in all analyses. To assess the effect of CGG repeat length on performance, separate correlations were tested using the mean (138) or higher (156) CGG value. The mean FMR1 mRNA value was 1.41 (0.23) (range: 1.10–1.76) for controls and 3.05 (1.37) (range: 1.85–7.81) for fXPCs, which differed significantly (t = −5.09, p < 0.001). FSIQ data were missing from 4 control participants. Mean FSIQ was 119.40 (14.20) for controls, and 115.37 (13.66) for fXPCs, and did not differ between groups (t = 0.84, p = 0.41). ADHD self-report data were available from 19 controls and 18 fXPCs (i.e., missing from 1 control and 3 fXPCs), and observer-report data were available from 18 controls and 14 fXPCs (i.e., missing from 2 controls and 7 fXPCs).

Table 1. Participant descriptive statistics and FMR1 measures.

3.2. OVERALL TASK PERFORMANCE

All participants performed at > 50% accuracy for T1 identification in the Uniform condition (T1 Uniform accuracy). Thus, participants performed well above chance and were judged able to perform the task. To identify any outlier participants, we defined outliers as having T1 Uniform accuracy greater than 3 times the interquartile range (IQR) or less than 3 times the IQR within each group. No outlier participants were identified, so all participants were included in the analyses.

3.3. REPLICATING AND EXTENDING THE “SPREADING THE SPARING” EFFECT

In Aim 1, we sought to replicate and extend the “spreading the sparing” effect, previously observed in undergraduate students, in a sample of adults aged 18–48 years (Di Lollo et al., 2005). To address this, we first replicated the method of analysis, and then constrained the data by requiring that preceding targets must be accurately identified (“conditional accuracy”), and then also requiring that targets must be reported in the correct order (“conditional accuracy + order”).

3.3.1. Non-conditional accuracy

Figure 2A shows target identification accuracy for each target in each condition. Table 2 shows that the main effects of Condition and Target, as well as the Condition X Target interaction, were significant (all ps < 0.001). The main effect of Group was not significant (F(1, 312) = 0.11, p = 0.74). The remaining interaction terms were not significant (all ps > 0.29). Controls demonstrated an AB in both Varied conditions (both ps < 0.001) but not in the Uniform condition (t = −1.40, p = 0.17). Likewise, fXPCs exhibited an AB in both Varied conditions (both ps < 0.001) but not in the Uniform condition (t = 0.50, p = 0.62).

FIGURE 2. Target identification accuracy. (A) Non-conditional accuracy was computed as accuracy for each target regardless of accuracy for other items in the trial. (B) Conditional accuracy was computed for each target in which preceding target(s) were correctly identified. (C) Conditional accuracy was computed for each target in which preceding target(s) were correctly identified, and reported in the correct ordinal position. To prevent overlap, markers for the Varied–2 condition are horizontally offset. Target position 2 is marked with an asterisk because in the Varied-2 condition there are only two targets, one in position 1 and one in position 3. The AB is defined as lower accuracy for T3 than for T1 in a given condition. In (A) and (B) for both groups, and in (C) for controls, the AB was observed for the Varied–2 and Varied–3 conditions (all ps < 0.001), but not for the Uniform condition (ps < 0.17, 0.89, and 0.07, respectively). In (C) for fXPCs, the AB was observed in fXPCs for all three conditions (all ps < 0.001). Error bars represent standard error of the mean.

Table 2. ANOVA of Group X Condition X Target on accuracy.

Di Lollo et al. (2005) observed in undergraduate students that T1 accuracy was higher in the Varied than Uniform condition (“T1 enhancement effect”). We examined this effect in controls, and observed this pattern in the Varied–2 (t = −2.85, p = 0.007) but not Varied–3 (t = −0.88, p = 0.38) relative to Uniform condition. Likewise, fXPCs demonstrated this effect in Varied–2 (t = −2.10, p = 0.04) but not Varied–3 (t = −1.77, p = 0.08) relative to Uniform condition.

3.3.2. Conditional accuracy

Figure 2B shows target identification accuracy when preceding target(s) in that trial were accurately identified. Table 2 shows that the main effects of Condition and Target, as well as the Condition X Target interaction were significant (all ps < 0.001). The main effect of Group was not significant (F(1, 312) = 0.35, p = 0.56). The remaining interaction terms were not significant (all ps > 0.20), although the Group X Condition X Target interaction trended toward significance (F(3, 312) = 3.66, p = 0.057). Controls demonstrated an AB in both Varied conditions (both ps < 0.001) but not in the Uniform condition (t = −0.15, p = 0.89). Likewise, fXPCs exhibited an AB in both Varied conditions (both ps < 0.001) but not in the Uniform condition (t = 1.81, p = 0.08).

3.3.3. Conditional accuracy + order

Figure 2C shows target identification accuracy when preceding target(s) in that trial were accurately identified and reported in the correct ordinal position. Table 2 shows that the main effects of Condition and Target, as well as the Group X Target and Condition X Target interaction were significant (all ps < 0.008). The main effect of Group was not significant (F(1, 306) = 0.76, p = 0.38). The remaining interaction terms were not significant (all ps > 0.16). Controls demonstrated an AB in both Varied conditions (both ps < 0.002) but not in the Uniform condition (t = 1.91, p = 0.07). FXPCs exhibited an AB in all three conditions (all ps < 0.001).

3.3.4. Summary of conditional analyses

In sum, requiring that preceding targets within a trial must have been correctly identified (“conditional accuracy”) or that targets must also be identified in the correct order (“conditional accuracy + order”) yields largely the same pattern of results as when these criteria were not applied. This suggests that if Di Lollo et al. (2005) had applied criteria necessary to interpret their data against the predictions of current models, their findings would have remained the same.

3.3.5. Comparison of conditional and non-conditional analyses

To make our results more comparable to predictions by Dell'Acqua et al. (2009), Figure 3 displays conditional and non-conditional accuracy for the Uniform condition on the same axes. This is analogous to the third panel in Figure 3 of their manuscript. As expected, when T3 accuracy is computed from conditional accuracy + order, it is lower than when computed from conditional accuracy alone, and much lower than when computed from non-conditional accuracy. We tested whether the analysis contingency (non-conditional, conditional accuracy, or conditional accuracy and order) interacted with Group in a 3 × 2 ANOVA. We found that fXPCs were less accurate than controls (F(1, 117) = 7.72, p = 0.006, η^2p = 0.06), and accuracy decreased with increasing analysis constraints (F(2, 117) = 10.62, p < 0.001, η^2p = 0.15), while the Group X Contingency interaction was not significant (F(2, 117) = 0.17, p = 0.85, η^2p = 0.00). The difference between conditional accuracy and conditional accuracy + order was significant for fXPCs (t = 2.16, p = 0.04) but not for controls (t = 1.82, p = 0.08).

FIGURE 3. Conditional and non-conditional accuracy. Non-conditional accuracy (Figure 2A), conditional accuracy (Figure 2B), and conditional accuracy + order (Figure 2C) from the Uniform condition are re-plotted on the same axes. As expected, T3 accuracy decreases with increasing constraints on analysis. FXPCs were less accurate than controls (F(1, 117) = 7.72, p = 0.006), and accuracy decreased with increasing analysis constraints (F(2, 117) = 10.62, p < 0.001), while the Group X Contingency interaction was not significant (F(2, 117) = 0.17, p = 0.85). The difference between conditional accuracy and conditional accuracy + order was significant for fXPCs (t = 2.16, p = 0.04) but not for controls (t = 1.82, p = 0.08). To prevent overlap, markers are horizontally offset. Error bars represent standard error of the mean.

3.4. TEMPORAL ORDERING AND MEMORY

In Aim 2, we sought to more fully characterize the temporal dynamics of attentional selection. We addressed this by examining participant's perception of temporal order and level of inter-target competition.

3.4.1. Perception of temporal order

If a participant correctly identified the targets and reported them in the correct order (e.g., reporting “T” before “C” in the example in Figure 1), we can be more confident that the participant perceived “T” as appearing earlier in time than “C.” Thus, from the trials where the targets were correctly identified, we calculated the percentage of trials in which the targets were reported in the correct order (“order accuracy”).

Figure 4 shows order accuracy for each condition, and Table 3 shows the ANOVA results. There was a main effect of Condition (F(2, 115) = 5.83, p = 0.004) and Group X Condition interaction (F(2, 92) = 5.07, p = 0.008), while the main effect of Group was not significant (F(1, 115) = 1.76, p = 0.19). In controls, order accuracy was higher in the Varied conditions than Uniform condition (both ps < 0.001), while the Varied conditions did not differ (t = 0.13, p = 0.89). In fXPCs, order accuracy was higher in the Varied–3 condition than Uniform condition (t = −2.54, p = 0.02), but did not differ between the other conditions (both ps > 0.20). However, the groups did not differ within any condition (all ps > 0.12).

FIGURE 4. Perception of temporal order. For trials in which all targets were correctly identified, the order in which targets were reported was assessed for accuracy. Increased accuracy for order information represents improved perception of temporal order. In controls, order accuracy was higher in the Varied conditions than Uniform condition (both ps < 0.001), while the Varied conditions did not differ (t = 0.13, p = 0.89). In fXPCs, order accuracy was higher in the Varied-3 condition than Uniform condition (t = −2.54, p = 0.02), but did not differ between the other conditions (both ps > 0.20). However, the groups did not differ in any condition (all ps > 0.12). Error bars represent standard error of the mean. *p < 0.05.

Table 3. ANOVA of Group X Condition on order accuracy.

3.4.2. Inter-target competition

3.4.2.1. T3 accuracy predicted by T1 accuracy. Table 4 shows the ANOVA results. The main effects of Condition and T1 accuracy were both significant (both ps < 0.001), and Group X Condition interaction was significant (F(2, 231) = 4.12, p = 0.02). In controls, T3 accuracy was higher when T1 was incorrect relative to correct, in both the Uniform (91.2% vs. 76.5%; t = 3.33, p = 0.002) and Varied–3 conditions (64.2% vs. 42.7%; t = 2.89, p = 0.006). Likewise, in fXPCs, T3 accuracy was higher when T1 was incorrect relative than correct, in both the Uniform (88.0% vs. 67.1%; t = 3.84, p < 0.001) and Varied–3 conditions (74.9% vs. 52.4%; t = 3.93, p < 0.001).

Table 4. ANOVA of Group X Condition X paired target accuracy on current target accuracy.

Although the Group X Condition interaction was significant (Figure 2A), the groups did not differ at any particular target or condition. For example, although T3 accuracy appears to be lower in fXPCs than controls in the Uniform condition, this difference is not significant (t = 1.49, p = 0.14). The interaction may be significant due to minor group differences across the conditions, such that fXPCs have slightly lower T3 accuracy than controls in the Uniform condition, but slightly higher T3 accuracy in the Varied–3 condition. No other main effects or interaction terms were significant.

3.4.2.2. T1 accuracy predicted by T3 accuracy. The main effect of T3 accuracy was significant (F(1, 231) = 57.22, p < 0.001). In controls, T1 accuracy was higher when T3 was incorrect relative to correct, in both the Uniform (87.0% vs. 71.9%; t = 3.23, p = 0.003) and Varied–3 conditions (86.3% vs. 64.1%; t = 2.95, p = 0.007). Likewise, in fXPCs, T3 accuracy was higher when T1 was incorrect relative to correct, in both the Uniform (89.8% vs. 70.9%; t = 4.81, p < 0.001) and Varied–3 conditions (89.0% vs. 77.3%; t = 2.56, p = 0.01).

3.5. RELATION TO WORKING MEMORY TESTS

The mean digit span backward score did not differ between groups (controls = 8.4, fXPCs = 7.8; t = 0.69, p = 0.49), but fXPCs had significantly lower letter-number sequencing scores than controls (controls = 14.2, fXPCs = 11.9; t = 2.46, p = 0.02).

In Aim 3, we sought to determine whether the AB is better modeled as due to excessive or insufficient control. To address this, we tested whether executive WM (i.e., digit span backward or letter-number sequencing) predicted AB task performance, and whether this relationship differed between groups. We reasoned that if greater letter-number sequencing score (i.e., better executive WM) was associated with larger AB magnitude, this would provide evidence supporting the assertion that the AB is better modeled by excessive control. Meanwhile, the digit span backward task also requires updating and manipulation of information, but less so than in the letter-number sequencing task. Therefore, we did not expect that digit span backward score would be associated with AB magnitude. In Aim 4, we assessed whether the relationship between neuropsychological performance and AB magnitude differed between fXPCs and controls. Thus, we tested the interaction between Group and neuropsychological performance as predictors of AB magnitude.

Table 5 shows the linear regression results for both tests. First, we found that the main effects of Group (F(1, 32) = 0.56, p = 0.46) and digit span backward score (F(1, 32) = 3.45, p = 0.07), and the interaction of Group X score were all not significant (F(1, 32) = 0.10, p = 0.75). Second, we found that the main effect of letter-number sequencing was significant, such that a greater score was associated with larger AB magnitude (r = 0.44, p = 0.007). The main effect of Group (F(1, 32) = 0.64, p = 0.43) and the interaction of Group X score (F(1, 32) = 0.82, p = 0.37) was not significant. In sum, we observed an association between AB magnitude and executive WM (i.e., letter-number sequencing) in the combined control and fXPC group. Letter-number sequencing score and AB magnitude were positively associated, supporting the interpretation that the AB is better modeled by excessive control than by insufficient control. FXPCs did not differ from controls in this regard. We did not observe an association between AB magnitude and digit span backward. We suggest that this could be due to decreased variability in digit span backwards scores relative to letter-number sequencing scores, which would decrease the likelihood that an association could be detected.

Table 5. Linear regression of Group X neuropsychological test score.

3.6. RELATION TO ADHD SYMPTOMS

None of the ADHD sub-scale scores differed between groups on either the self- or observer-report (all ps > 0.12). No participants met ADHD criteria on both the self- and observer-report, though one control and three fXPCs met ADHD criteria on the Total Symptoms sub-scale of the observer and self-report, respectively. We reasoned that because ADHD symptoms and diagnosis prevalence tends to be increased in fXPCs, inattentive symptoms might impact performance on the AB task, and that this effect might be more pronounced in fXPCs. We used linear regression to examine the effect of Group X symptoms (either self-report or observer report) on AB magnitude. Neither main effects nor interactions were significant (all ps > 0.30).

3.7. ASSOCIATION WITH AGE AND MOLECULAR VARIABLES

We tested the correlation between AB magnitude and molecular genetic measures in fXPCs. AB was not correlated with CGG repeat length (r = 0.08, p = 0.75) or with FMR1 mRNA level (r = −0.19, p = 0.45). When we used the higher CGG repeat value for the participant who expressed two CGG values, the correlation with CGG repeat length remained not significant (p = 0.60).

4. DISCUSSION

This study is the first to assess temporal attention in fXPCs using the AB task. Stimuli were presented foveally, so manipulation of attention instructions and distractor presence assessed the ability to ignore temporally intervening distractors. In this variant of the AB task, by manipulating attention demands (i.e., the type of items to be attended or ignored) and WM load (i.e., two or three items), we are able to explore the interaction between attention and WM in control participants and fXPCs. Our main finding is that fXPCs do not differ from controls in AB magnitude, indicating that the temporal dynamics of attentional selection are intact in fXPCs.

In Aim 1, we replicated the “spreading the sparing” effect, previously observed in undergraduate students, in a sample of adults aged 18–48. We also extended those findings by demonstrating that when applying progressively more stringent performance criteria, overall effects were maintained. This suggests that the results initially reported by Di Lollo et al. (2005) are robust. The additional performance criteria implemented in this study also aid interpretation of the original findings in light of models accounting for the AB, as will be described shortly. In Aim 2, we found that control participants were more likely to report targets in the correct order when there were only two letter targets (i.e., Varied–2 and Varied–3 conditions) than when there were three letter targets (i.e., Uniform condition). This suggests that with the switch from two to three targets, temporal information about the order in which each target appears is often lost. The likelihood of reporting targets in the correct order was higher in the Varied–3 condition than Uniform condition, suggesting that attention to an additional, intervening target (T2) is less detrimental when it belongs to a different stimulus set (i.e., T2 was a digit in the Varied–3 condition while T2 was a letter in the Uniform condition). In Aim 3, we found that in the combined control and fXPC group, increased letter-number sequencing score was associated with increased AB magnitude. Because letter-number sequencing requires executive WM, our results support the interpretation that the AB is better modeled by excessive control than by insufficient control. In Aim 4, we assessed whether fXPCs were impaired relative to controls in any of these measures.

We will first discuss implications of this study for understanding fXPCs. Second, we will briefly summarize competing models of attentional selection which make predictions about performance in this task. This will provide the background and theoretical framework required to understand the implications of this study. Third, we will discuss how our results replicate and extend previous findings. Fourth, we will discuss implications for models of the temporal dynamics of attentional selection. Finally, we will discuss implications for models of conceptualizing the AB as due to excessive or insufficient attentional control.

4.1. IMPLICATIONS FOR UNDERSTANDING NEUROCOGNITIVE FUNCTION IN FXPCS

In the combined control and fXPC group, letter-number sequencing performance was positively associated with AB magnitude. Both tasks require attending to letters and numbers, but the tasks differ in several ways. For example, they operate along different timescales of presentation. Attentional control is required to filter distracting information and can thus aid performance in the letter-number sequencing task. However, attentional control takes time to implement, so it can impair performance on tasks which require operation along very short timescales. This is what we observed in this study. When we examined groups separately, we found that fXPCs exhibited worse letter-number sequencing performance than controls, but did not differ in AB magnitude. This finding of a group difference in attention switching but not AB suggests that the temporal dynamics of attentional selection are intact in fXPCs. It also suggests that differential dynamics of attentional selection (i.e., in directing attention to targets or shifting attention from distractors) do not explain the WM impairments in fXPCs which have been reported previously (Brega et al., 2008; Grigsby et al., 2008; Cornish et al., 2009, 2011). Thus, impairments in WM might be due to other factors such as inability to filter distracting information, maintain information in memory, or manipulate information in memory.

Age did not affect performance in this task. We conclude this because when we included age as a covariate, the effect of age was not significant, and the pattern of results did not differ. This null result is interesting because FXTAS exhibits age-dependent penetrance (Jacquemont et al., 2004). There are several possible explanations for this null result. First, it might be the case that the effects of age-dependent penetrance are not observable until ages more advanced than in our sample. Our sample included adults aged 18–48, while prior studies in which an age effect was observed included adults aged 18–69 (Cornish et al., 2009) or only adults older than 50 (Jacquemont et al., 2004). Second, different symptoms might exhibit differential trajectories or levels of age dependence. For example, motor symptoms might manifest earlier in life and exhibit greater dependence on age while cognitive symptoms might manifest later in life and exhibit less dependence on age. Third, the effect of age might be less pronounced in fXPCs asymptomatic for FXTAS than in those with FXTAS.

To better understand the implications of differences in patterns of performance between fXPCs and controls, it would be helpful to understand the mechanisms that produce the AB. Therefore, in the next section we provide a brief summary of the theoretical models that have been proposed to describe and explain the AB and which have helped shape our understanding of attention and memory.

4.2. MODELS ACCOUNTING FOR THE AB

The AB refers to an impairment in perceiving the second of two targets presented closely in time in a RSVP stream (Raymond et al., 1992). With increasing time and number of intervening items, this effect diminishes. Thus, because the blink occurs only when two targets occur close together in time, the blink is thought to be due to difficulty engaging attention twice in a short time period (Nieuwenstein et al., 2009). In the current AB task, the AB was defined as decreased accuracy for T3 relative to T1.

The relevant models can be categorized as predicting that the AB is due to T1 processing or distractor processing (see Dux and Marois, 2009 for a review). Generally, the former models can be described as resource-depletion models, and the latter models can be described as distractor-based models. We will discuss examples of both types of models.

4.2.1. Resource-depletion (RD) models

RD models stipulate that while T1 is being consolidated, T2 cannot be encoded and is susceptible to interference from trailing distractor stimuli (Ward et al., 1996). For example, according to interference theory, both targets enter limited-capacity WM and interfere with one another during retrieval (Shapiro and Raymond, 1994). According to bottleneck models, targets are rapidly identified during Stage 1 processing, but must then be consolidated into WM during capacity-limited Stage 2 processing (e.g., Chun and Potter, 1995). Similar to bottleneck models, the episodic simultaneous type/serial token (eSTST) model predicts that targets are identified during Stage 1, but must have identity and episodic information, such as the relative temporal position of the item in the stream, bound to a token during Stage 2 processing (Wyble et al., 2009). Unlike in the authors original STST model, the binding of items to separate tokens in the eSTST model allows for preservation of order information. These RD models predict that in this task, an AB should be observed in the Uniform as well as Varied conditions, because the same limited-capacity resources are devoted to T1 processing regardless of the presence of or attention to intervening distractors.

Distinct from other T1-based models, the boost and bounce model posits that WM capacity limitations play no role in the AB (Olivers and Meeter, 2008). Instead, target identification triggers an attentional boost, which is then followed by a bounce to prevent distractors from entering WM. Contiguous targets elicit a recurring boost, but after an intervening distractor appears, the bounce negatively affects identification of the trailing target.

4.2.2. Distractor-based models

According to distractor-based models, the AB is produced by processes following the appearance of the distractor after T1. One such model is the TLC model, which we have already described (Di Lollo et al., 2005). As noted by Olivers et al. (2007), the TLC model thus assumes that resources are limited not at the low level of processing individual targets, but rather at a higher, executive level such that only one task can be prioritized at a time (i.e., monitoring or consolidation). This notion of limited executive level resources predicts that simultaneously performing two tasks which both require executive level resources should result in a decrement in performance, either in the non-prioritized task or both tasks. Indeed, the AB is reduced by manipulations that divide attention, such as the presentation of distracting visual motion and flicker (Arend et al., 2006), or the instruction to “pay a little less attention” (Olivers and Nieuwenhuis, 2006). These findings suggest that the AB is due to excessive attentional control. A related computational model posits that control processes suppress target detection while T1 is being consolidated, and would thus predict an AB in both the Uniform and Varied conditions (Taatgen et al., 2009).

4.2.3. “Spreading the sparing”

One feature of AB task performance that is explicitly predicted by some models of the AB is the “spreading the sparing” effect, in which a trailing target presented in very close temporal proximity to the first target is often identified just as well as the first target (Olivers et al., 2007). Understanding the conditions under which sparing can be spread to additional targets and identifying capacity limits on the number of targets which can receive the spread would help inform and update these models.

4.3. REPLICATING AND EXTENDING THE “SPREADING THE SPARING” EFFECT

In this study, we replicated in a sample of adults the “spreading the sparing” effect observed in an undergraduate sample (Di Lollo et al., 2005; Dell'Acqua et al., 2009). This sparing effect was observed as an AB in the Varied–2 and Varied–3 conditions, but not in the Uniform condition. When evaluating accuracy to trailing targets in an AB paradigm, it is critical to consider accuracy only from trials in which T1 performance was accurate (e.g., T3|T1+T2), in other words applying the within-trial contingency (WTC) principle (Dell'Acqua et al., 2009). We applied this principle in our “conditional accuracy” analysis and observed an overall similar pattern of results. While we expected conditional T3 accuracy to be lower than non-conditional T3 accuracy, a lack of marked difference suggests that T3 and T1 are often co-reported.

We then performed a more stringent “conditional accuracy + order” analysis. We reasoned that this additional constraint would allow us to visualize performance under conditions that best fit the assumptions of theoretical models. Specifically, these constraints ensure that T1 consolidation occurred successfully, and can therefore impact T3 perception or consolidation, and that episodic information about the temporal order in which the items appeared remained intact. Notably, neither RD models nor distractor-based models generate explicit predictions regarding the additional constraint of correct reporting of target order. Instead, this additional constraint reveals behavioral performance when 1) preceding targets were successfully encoded and 2) episodic information was correctly associated with each preceding target, such that relative temporal position of each target was maintained. Thus, this analysis imposes the most stringent criteria defining “correct task performance.” We observed an interactive Group X Condition effect on accuracy, such that fXPCs exhibited an AB in the Uniform as well as Varied conditions. In contrast, in the less stringent analyses fXPCs performed similarly to controls, exhibiting an AB in the Varied conditions but not the Uniform condition. Thus, this more stringent analysis revealed that fXPCs are subtly impaired relative to controls in terms of the ability to retain episodic information about temporal order.

4.4. TEMPORAL ORDERING AND MEMORY

We examined the extent to which perception of temporal order is preserved in an AB task. This has been discussed previously to explain Lag 1 sparing (Hommel and Akyürek, 2005) and to identify whether the AB impairs perception of temporal order (Spalek et al., 2012). In the latter study, presence and absence of preceding and intervening distractors was systematically manipulated and found to have similar effects on the AB. Notably, the authors observed that distractors impaired perception of temporal order even while target identification accuracy remained unimpaired. According to their criteria, T2 and T3 could be reported in any order as long as (1) they were reported correctly and (2) T1 was reported correctly in the first ordinal position.

In the present study, to analyze perception of temporal order we required that T1 and T3 be: (1) correctly identified; (2) reported in the correct relative order; and (3) in the Uniform and Varied–3 conditions, that T2 was correctly identified. As described in the Materials and Methods, these constraints removed the potentially confounding effect of T2 stimulus type (i.e., either letter or number) on perception of temporal order. We found that in controls, order accuracy was higher in the Varied conditions than Uniform conditions. This was to be expected, because the Uniform condition, unlike the other conditions, contained an additional target from the same stimulus set as the other targets (i.e., letters), and episodic information for this target could be more easily lost or incorrectly attributed to another target. Additionally, we found that in fXPCs, order accuracy was higher in the Varied–3 condition than Uniform condition, but did not differ between the other conditions. The finding of higher order accuracy in the Varied–2 relative to Uniform condition in controls, but not fXPCs, suggests that fXPCs were attending to and encoding episodic information for the number distractor despite instructions to ignore all numbers.

To further examine the dynamics of attentional selection, we examined the effect of T1 accuracy on T3 accuracy, and the effect of T3 accuracy on T1 accuracy. In both instances, we found that accuracy for one target was higher when identification of the other target was incorrect. This supports models in which targets interfere with each other, regardless of which entered WM first (Shapiro and Raymond, 1994). However, it is less clear how other T1-based models would account for the effect of T3 accuracy on T1 accuracy.

No group differences in accuracy were observed, so we found little evidence for impaired temporal attention in adult male fXPCs asymptomatic for FXTAS relative to controls. Previous research reported that M pathway function was impaired in adult fXPCs (Kéeri and Benedek, 2009, 2012), and because the M pathway processes temporal information, this dysfunction might manifest as a disrupted AB. However, disrupting M pathway function in controls does not affect the AB (Nieuwenhuis et al., 2008). This supports models that describe the AB as the result of control processes, and suggests that the AB is more dependent on cortical than subcortical processing. Instead of a temporal processing impairment, we found that fXPCs exhibit relatively intact perception of temporal order, and similar patterns of inter-target competition in WM.

Kim et al. (2014) observed differences in neural processing of temporal WM in adult fXPCs, which is not necessarily inconsistent with our finding. For example, differential recruitment of brain regions in fXPCs relative to controls might nonetheless exhibit similar temporal dynamics, or result in similar response profiles. Alternatively, WM retrieval and attentional selection are related but distinct processes, such that one could be altered in fXPCs while the other is not. Exploring the relationship between brain activation and behavior, and clarifying which cognitive processes are impacted by FMR1 mutations, is a potentially fruitful area for further research.

Overall performance was high, and effects were not modulated by ADHD symptoms, so it is unlikely that results are due to overall inattentiveness. This lack of relationship with attentiveness is consistent with previous findings that although alerting enhances target identification, alerting does not affect AB magnitude (Spalek and Lollo, 2010).

4.5. EXCESSIVE VS. INSUFFICIENT CONTROL

To discern between different models of attentional selection that attribute the AB to either excessive or insufficient attentional control, we examined associations between the AB and traditional neurospsychological measures of executive WM. We reasoned that if the AB were due to excessive control, AB magnitude should correlate positively with letter-number sequencing score, but that if it were due to insufficient control, it should correlate negatively with letter-number sequencing score. Consistent with reported literature (Cornish et al., 2009, but see also Allen et al., 2011), we found that fXPCs exhibited lower letter-number sequencing scores than controls. We also found that both groups exhibited a positive association between letter-number sequencing score and AB magnitude, supporting models of the AB as the result of excessive control (Raymond et al., 1992; Olivers et al., 2007; Taatgen et al., 2009). These results are in line with findings that individuals who do not exhibit an AB effect (non-blinkers) are more efficient in ignoring distractors than blinkers (Martens and Valchev, 2009), and that blink magnitude increases when targets and distractors are more similar and therefore require more top-down control to distinguish (Chun and Potter, 1995).

Given lower letter-number sequencing scores in fXPCs, and a positive association between score and AB magnitude, we would predict that fXPCs also exhibit decreased AB magnitude (i.e., greater T3 accuracy), which was not the case. A potential confounding factor is that both the AB and letter-number sequencing tasks require a switch between attending to letters and numbers. It is possible that a cost associated with switching from T2 (number) to T3 (letter) identification resulted in decreased T3 accuracy, producing a larger rather than a smaller blink. Future studies using targets from a single stimulus set in a more traditional AB paradigm, or studies of executive WM that do not require task-switching, are needed to explore this possibility.

Finally, we should specify that our conclusion that the AB is better modeled by excessive rather than insufficient control is based on the finding of a correlation between AB magnitude and performance on the letter-number sequencing task, which requires executive WM. The idea that these two tasks might be related is not new; the relationship between AB duration and letter-number sequencing performance has been assessed previously (Gillard-Crewther et al., 2007). Executive function is required to switch between letters and numbers in the letter-number sequencing task, which puts higher load on the central executive of WM, and is similarly required to switch between letters and numbers in the AB task. However, because successful letter-number sequencing performance involves memory for increasingly longer lists of items, while this AB task requires memory for only 2-3 items, the tasks are not completely analogous. Future studies utilizing direct measures of attentional control in conjunction with the AB task will help clarify this emerging evidence for the AB as a task of excessive attentional control.

4.6. LIMITATIONS

One limitation of our study is our choice of AB task. MacLean and Arnell (2012) argue that the AB cannot be properly assessed without sampling at least two trailing target lags: one shorter lag within the window of a typical AB, and one longer lag at which point the AB has typically resolved. Otherwise, any group effects could be due to differences in T2 accuracy alone. Our task design, which held trailing target lag constant, allowed us to examine the effect of attend instructions and intervening distractors. Instead of using accuracy from a long inter-target lag condition as a control condition, we used accuracy from the Uniform condition. Although this design best suits the goals of this study, differences between our task design and traditional AB task designs limit the extent to which our findings can directly extend existing literature.

Another limitation of this study is that participants were allowed to identify targets in any order. This replicates how the task was administered in previous studies, so that identification accuracy was the primary outcome measure. However, because participants were not explicitly required to encode order information, different response strategies may have caused variations in the order of responses and affected our analyses of temporal perception. We speculate that a requirement to encode order information would make the task more challenging and potentially more sensitive to group differences. Thus, future studies manipulating attention to temporal information are needed for a more complete understanding of the temporal dynamics of attention in fXPCs.

A third limitation of this study is that in the conditional accuracy and conditional accuracy + order analyses, by becoming more stringent with the selection criteria, fewer and fewer trials for T3 were eligible for analysis. Thus, one must consider the difference in the number of included trials when comparing the results of these analyses. Similarly, the number of included subjects could differ (e.g., if one participant consistently reported targets in the incorrect order).

A fourth limitation of this study is that we did not include female fXPCs. We reasoned that male fXPCs, because they lack a second, unaffected FMR1 allele and are at greater risk of developing FXTAS, would be more likely than female fXPCs to exhibit group differences in cognitive performance. However, reports of impairments in female fXPCs (Goodrich-Hunsaker et al., 2011a,b) but not male fXPCs in the same task (Wong et al., 2012), or of enhanced psychomotor speed in female fXPCs but not male fXPCs (Goodrich-Hunsaker et al., 2011c; Wong et al., 2012) challenge this potentially overly simplistic view.

Finally, although we examined processes that may be affected in FXTAS, the fXPCs we studied were asymptomatic for FXTAS. To identify whether cognitive impairments precede or characterize FXTAS, longitudinal studies are needed. We observed no differences in temporal attention between fXPCs and controls, but this could be because participants in our sample will not go on to develop FXTAS, temporal attention is truly unaffected in fXPCs, our sample size was insufficient to detect a small effect, or the AB task was insensitive to the specifically affected processes.

4.7. CONCLUSION

This study was the first to examine the dynamics of temporal attention in fXPCs. We found no differences between adult male fXPCs asymptomatic for FXTAS and controls, suggesting that attentional selection processes were intact. Meanwhile, fXPCs exhibited impaired attentional control, observed as impaired performance in the letter-number sequencing task. Understanding what cognitive processes are intact or impaired in fXPCs may facilitate early identification of individuals most at risk for developing FXTAS. Furthermore, we replicated and extended findings in controls using an AB paradigm. Results from this study support models of attentional selection that posit that the AB is due to excessive, and not insufficient, attentional control. However, future studies are needed to develop a more complete understanding of the dynamics of temporal attention and the effect of control processes on those dynamics.

AUTHOR CONTRIBUTIONS

Ling M. Wong played a primary role in experiment design, data analysis, data interpretation, and manuscript preparation. FT performed the molecular genetics analysis. Susan M. Rivera participated in the design of the study, and supported data interpretation and manuscript preparation. Tony J. Simon participated in the design of the study, and supported data interpretation and manuscript preparation. All authors read and approved the final manuscript.

FUNDING

This work was supported by National Institute of Health (NIH) grants: NIA RL1 AG032119, NINDS RL1 NS062412, NIDA TL1 DA024854, and NIMH MH078041. This work was also made possible by a Roadmap Initiative grant (UL1 DE019583) from the National Institute of Dental and Craniofacial Research (NIDCR) in support of the NeuroTherapeutics Research Institute (NTRI) consortium. The NIH had no further role in study design; in the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the study for publication.

ACKNOWLEDGMENT

We thank the participants who made this work possible. We also thank the reviewers for their insightful comments, which improved the clarity and scope of this manuscript.

ABBREVIATIONS

AB, attentional blink; CAARS, Conners' Adult ADHD Rating Scale; FMRP, fragile X mental retardation protein; fXPC, fragile x premutation carrier; FSIQ, full scale IQ; FXS, fragile X syndrome; FXTAS, fragile x-associated tremor/ataxia syndrome; LNS, letter-number sequencing; TLC, temporary loss of control; RD, resource-depletion.

REFERENCES

Allen, E. G., Hunter, J. E., Rusin, M., Juncos, J., Novak, G., Hamilton, D., et al. (2011). Neuropsychological findings from older premutation carrier males and their noncarrier siblings from families with fragile X syndrome. Neuropsychology 25, 404–411. doi: 10.1037/a0021879

Arend, I., Johnston, S., and Shapiro, K. (2006). Task-irrelevant visual motion and flicker attenuate the attentional blink. Psychon. Bull. Rev. 13, 600–607. doi: 10.3758/BF03193969

Bacalman, S., Farzin, F., Bourgeois, J., Cogswell, J., Goodlin-Jones, B., Gane, L. W., et al. (2006). Psychiatric phenotype of the fragile X-associated tremor/ataxia syndrome (FXTAS) in males: newly described fronto-subcortical dementia. J. Clin. Psych. 67, 87–94. doi: 10.4088/JCP.v67n0112

Borthwell, R. M., Hunsaker, M. R., Willemsen, R., and Berman, R. F. (2012). Spatiotemporal processing deficits in female CGG KI mice modeling the fragile X premutation. Behav. Brain Res. 233, 29–34. doi: 10.1016/j.bbr.2012.04.029

Brega, A. G., Goodrich, G., Bennett, R. E., Hessl, D., Engle, K., Leehey, M. A., et al. (2008). The primary cognitive deficit among males with fragile X-associated tremor/ataxia syndrome (FXTAS) is a dysexecutive syndrome. J. Clin. Exp. Neuropsychol. 30, 853–869. doi: 10.1080/13803390701819044

Chun, M. M., and Potter, M. C. (1995). A two-stage model for multiple target detection in rapid serial visual presentation. J. Exp. Psychol. Hum. Percept. Perform. 21:109. doi: 10.1037/0096-1523.21.1.109

Cohen, S., Masyn, K., Adams, J., Hessl, D., Rivera, S., Tassone, F., et al. (2006). Molecular and imaging correlates of the fragile X–associated tremor/ataxia syndrome. Neurology 67, 1426–1431. doi: 10.1212/01.wnl.0000239837.57475.3a

Conners, C. K., Erhardt, D., and Sparrow, E. P. (1999). Conners Adult ADHD Rating Scales (CAARS) technical manual. North Tonawanda, NY: Multi-Health Systems Inc.

Cornish, K., Kogan, C., Li, L., Turk, J., Jacquemont, S., and Hagerman, R. J. (2009). Lifespan changes in working memory in fragile X premutation males. Brain Cogn. 69, 551–558. doi: 10.1016/j.bandc.2008.11.006

Cornish, K., Hocking, D., and Moss, S. (2011). Selective executive markers of at-risk profiles associated with the fragile X premutation. Neurology 77, 618–622. doi: 10.1212/WNL.0b013e3182299e59

Dell'Acqua, R., Jolicøeur, P., Luria, R., and Pluchino, P. (2009). Reevaluating encoding-capacity limitations as a cause of the attentional blink. J. Exp. Psychol. Hum. Percept. Perform. 35, 338–351. doi: 10.1037/a0013555

Di Lollo, V., Kawahara, J.-I., Shahab Ghorashi, S. M., and Enns, J. T. (2005). The attentional blink: resource depletion or temporary loss of control? Psychol. Res. 69, 191–200. doi: 10.1007/s00426-004-0173-x

Dux, P. E., and Marois, R. (2009). The attentional blink: a review of data and theory. Attent. Percept. Psychophys. 71, 1683–1700. doi: 10.3758/APP.71.8.1683

Gillard-Crewther, S., Lawson, M. L., Bello, K., and Crewther, D. P. (2007). The visual attentional blink reflects constraints on temporal visual processing, not just a lapse of visual memory. Clin. Exp. Optom. 90, 282–289. doi: 10.1111/j.1444-0938.2007.00160.x

Goodrich-Hunsaker, N. J., Wong, L. M., McLennan, Y., Srivastava, S., Tassone, F., Harvey, D., et al. (2011a). Young adult female fragile X premutation carriers show age- and genetically-modulated cognitive impairments. Brain Cogn. 75, 255–260. doi: 10.1016/j.bandc.2011.01.001

Goodrich-Hunsaker, N. J., Wong, L. M., McLennan, Y., Tassone, F., Harvey, D., Rivera, S. M., et al. (2011b). Adult female fragile X premutation carriers exhibit age- and CGG repeat length-related impairments on an attentionally based enumeration task. Front. Hum. Neurosci. 5:63. doi: 10.3389/fnhum.2011.00063

Goodrich-Hunsaker, N. J., Wong, L. M., McLennan, Y., Tassone, F., Harvey, D., Rivera, S. M., et al. (2011c). Enhanced manual and oral motor reaction time in young adult female fragile X premutation carriers. J. Int. Neuropsychol. Soc. 17, 746. doi: 10.1017/S1355617711000634

Grigsby, J., Brega, A. G., Jacquemont, S., Loesch, D. Z., Leehey, M. A., Goodrich, G. K., et al. (2006). Impairment in the cognitive functioning of men with fragile X-associated tremor/ataxia syndrome (FXTAS). J. Neurol. Sci. 248, 227–233. doi: 10.1016/j.jns.2006.05.016

Grigsby, J., Brega, A. G., Leehey, M. A., Goodrich, G. K., Jacquemont, S., Loesch, D. Z., et al. (2007). Impairment of executive cognitive functioning in males with fragile X-associated tremor/ataxia syndrome. Mov. Disord. 22, 645–650. doi: 10.1002/mds.21359

Grigsby, J., Brega, A. G., Engle, K., Leehey, M. A., Hagerman, R. J., Tassone, F., et al. (2008). Cognitive profile of fragile X premutation carriers with and without fragile X-associated tremor/ataxia syndrome. Neuropsychology 22, 48–60. doi: 10.1037/0894-4105.22.1.48

Hagerman, P. J., and Hagerman, R. J. (2004). The fragile-X premutation: a maturing perspective. Am. J. Hum. Genet. 74, 805–816. doi: 10.1086/386296

Hagerman, P. J. (2008). The fragile X prevalence paradox. J. Med. Genet. 45, 498–499. doi: 10.1136/jmg.2008.059055

Hashimoto, R., Javan, A. K., Tassone, F., Hagerman, R. J., and Rivera, S. M. (2011a). A voxel-based morphometry study of grey matter loss in fragile X-associated tremor/ataxia syndrome. Brain 134, 863–878. doi: 10.1093/brain/awq368

Hashimoto, R.-I., Srivastava, S., Tassone, F., Hagerman, R. J., and Rivera, S. M. (2011b). Diffusion tensor imaging in male premutation carriers of the fragile X mental retardation gene. Mov. Disord. 26, 1329–1336. doi: 10.1002/mds.23646

Hocking, D., Kogan, C. S., and Cornish, K. M. (2012). Selective spatial processing deficits in an at-risk subgroup of the fragile X premutation. Brain Cogn. 79, 39–44. doi: 10.1016/j.bandc.2012.02.005

Hommel, B., and Akyürek, E. G. (2005). Lag-1 sparing in the attentional blink: benefits and costs of integrating two events into a single episode. Q. J. Exp. Psychol. A 58, 1415–1433. doi: 10.1080/02724980443000647

Hunsaker, M. R., Goodrich-Hunsaker, N. J., Willemsen, R., and Berman, R. F. (2010). Temporal ordering deficits in female CGG KI mice heterozygous for the fragile X premutation. Behav. Brain Res. 213, 263–268. doi: 10.1016/j.bbr.2010.05.010

Hunter, J. E., Sherman, S., Grigsby, J., Kogan, C., and Cornish, K. (2012). Capturing the fragile X premutation phenotypes: a collaborative effort across multiple cohorts. Neuropsychology 36, 156–164. doi: 10.1037/a0026799

Jacquemont, S., Hagerman, R. J., Leehey, M., Grigsby, J., Zhang, L., Brunberg, J. A., et al. (2003). Fragile X premutation tremor/ataxia syndrome: molecular, clinical, and neuroimaging correlates. Am. J. Hum. Genet. 72, 869–878. doi: 10.1086/374321

Jacquemont, S., Hagerman, R. J., Leehey, M. A., Hall, D. A., Levine, R. A., Brunberg, J. A., et al. (2004). Penetrance of the fragile X-associated tremor/ataxia syndrome in a premutation carrier population. J. Am. Med. Assoc. 291, 460–469. doi: 10.1001/jama.291.4.460

Kéeri, S., and Benedek, G. (2009). Visual pathway deficit in female fragile X premutation carriers: a potential endophenotype. Brain Cogn. 69, 291–295. doi: 10.1016/j.bandc.2008.08.002

Kéri, S., and Benedek, G. (2011). Fragile X protein expression is linked to visual functions in healthy male volunteers. Neuroscience 192, 345–350. doi: 10.1016/j.neuroscience.2011.06.074

Kéri, S., and Benedek, G. (2012). Why is vision impaired in fragile X premutation carriers? The role of fragile X mental retardation protein and potential FMR1 mRNA toxicity. Neuroscience 206, 183–189. doi: 10.1016/j.neuroscience.2012.01.005

Kim, S.-Y., Tassone, F., Simon, T. J., and Rivera, S. M. (2014). Altered neural activity in the when pathway during temporal processing in fragile X permutation carriers. Behav. Brain Res. 261, 240–248. doi: 10.1016/j.bbr.2013.12.044

Kraan, C. M., Hocking, D. R., Bradshaw, J. L., Fielding, J., Cohen, J., Georgiou-Karistianis, N., et al. (2013). Neurobehavioral evidence for the involvement of the FMR1 gene in female carriers of fragile X syndrome. Neurosci. Biobehav. Rev. 37, 522–547. doi: 10.1016/j.neubiorev.2013.01.010

Leehey, M. A., Berry-Kravis, E., Goetz, C. G., Zhang, L., Hall, D. A., Li, L., et al. (2008). FMR1 CGG repeat length predicts motor dysfunction in premutation carriers. Neurology 70(16 Pt 2), 1397–1402. doi: 10.1212/01.wnl.0000281692.98200.f5

MacLean, M. H., and Arnell, K. M. (2012). A conceptual and methodological framework for measuring and modulating the attentional blink. Attent. Percept. Psychophys. 74, 1080–1097. doi: 10.3758/s13414-012-0338-4

Martens, S., and Valchev, N. (2009). Individual differences in the attentional blink. Exp. Psychol. 56, 18–26. doi: 10.1027/1618-3169.56.1.18

Nieuwenhuis, S., Jepma, M., La Fors, S., and Olivers, C. N. L. (2008). The role of the magnocellular and parvocellular pathways in the attentional blink. Brain Cogn. 68, 42–48. doi: 10.1016/j.bandc.2008.02.119

Nieuwenstein, M. R., Potter, M. C., and Theeuwes, J. (2009). Unmasking the attentional blink. J. Exp. Psychol. Hum. Percept. Perform. 35, 159–169. doi: 10.1037/0096-1523.35.1.159

Olivers, C. N. L., and Meeter, M. (2008). A boost and bounce theory of temporal attention. Psychol. Rev. 115, 836–863. doi: 10.1037/a0013395

Olivers, C. N. L., and Nieuwenhuis, S. (2006). The beneficial effects of additional task load, positive affect, and instruction on the attentional blink. J. Exp. Psychol. Hum. Percept. Perform. 32, 364–379. doi: 10.1037/0096-1523.32.2.364

Olivers, C. N. L., Stigchel, S., and Hulleman, J. (2007). Spreading the sparing: against a limited-capacity account of the attentional blink. Psychol. Res. 71, 126–139. doi: 10.1007/s00426-005-0029-z

Raymond, J. E., Shapiro, K. L., and Arnell, K. M. (1992). Temporary suppression of visual processing in an RSVP task: an attentional blink? J. Exp. Psychol. Hum. Percept. Perform. 18, 849–860. doi: 10.1037/0096-1523.18.3.849

Rogers, S. J., Wehner, E. A., and Hagerman, R. (2001). The behavioral phenotype in fragile X: symptoms of autism in very young children with fragile X syndrome, idiopathic autism, and other developmental disorders. J. Dev. Behav. Pediatr. 22, 409–417. doi: 10.1097/00004703-200112000-00008

Shapiro, K. L., and Raymond, J. E. (1994). “Temporal allocation of visual attention: inhibition or interference?” in Inhibitory Mechanisms in Attention, Memory and Language, eds D. Dagenbach (Boston, MA: Academic Press), 151–188.

Spalek, T. M., and Lollo, V. (2010). Alerting enhances target identification but does not affect the magnitude of the attentional blink. Attent. Percept. Psychophys. 73, 405–419. doi: 10.3758/s13414-010-0044-z

Spalek, T. M., Lagroix, H. E. P., Yanko, M. R., and Di Lollo, V. (2012). Perception of temporal order is impaired during the time course of the attentional blink. J. Exp. Psychol. Hum. Percept. Perform. 38, 402–413. doi: 10.1037/a0025050

Taatgen, N. A., Juvina, I., Schipper, M., Borst, J. P., and Martens, S. (2009). Too much control can hurt: a threaded cognition model of the attentional blink. Cogn. Psychol. 59, 1–29. doi: 10.1016/j.cogpsych.2008.12.002

Tassone, F., Adams, J., Berry-Kravis, E. M., Cohen, S. S., Brusco, A., Leehey, M. A., et al. (2007). CGG repeat length correlates with age of onset of motor signs of the fragile X-associated tremor/ataxia syndrome (FXTAS). Am. J. Med. Genet. A 144B, 566–569. doi: 10.1002/ajmg.b.30482

Tassone, F., Pan, R., Amiri, K., Taylor, A. K., and Hagerman, P. J. (2008). A rapid polymerase chain reaction-based screening method for identification of all expanded alleles of the Fragile X (FMR1) gene in newborn and high-risk populations. J. Mol. Diagn. 10, 43–49. doi: 10.2353/jmoldx.2008.070073

Van Essen, D. C., and Gallant, J. L. (1994). Neural mechanisms of form and motion processing in the primate visual system. Neuron 13, 1–10. doi: 10.1016/0896-6273(94)90455-3

Wang, J. Y., Hessl, D., Schneider, A., Tassone, F., Hagerman, R. J., and Rivera, S. M. (2013). Fragile X-associated tremor/ataxia syndrome: influence of the FMR1 gene on motor fiber tracts in males with normal and premutation alleles. JAMA Neurol. 70:1022. doi: 10.1001/jamaneurol.2013.2934

Ward, R., Duncan, J., and Shapiro, K. (1996). The slow time-course of visual attention. Cogn. Psychol. 30, 79–109. doi: 10.1006/cogp.1996.0003

Wechsler, D. (1997). WAIS-III: Wechsler Adult Intelligence Scale. Technical report.

Wechsler, D. (1999). Wechsler Abbreviated Scale of Intelligence. Technical report.

Wong, L. M., Goodrich-Hunsaker, N. J., McLennan, Y., Tassone, F., Harvey, D., Rivera, S. M., et al. (2012). Young adult male carriers of the fragile X premutation exhibit genetically modulated impairments in visuospatial tasks controlled for psychomotor speed. J. Neurodev. Disord. 4:26. doi: 10.1186/1866-1955-4-26

Wyble, B., Bowman, H., and Nieuwenstein, M. (2009). The attentional blink provides episodic distinctiveness: sparing at a cost. J. Exp. Psychol. Hum. Percept. Perform. 35, 787–807. doi: 10.1037/a0013902

Zangenehpour, S., Cornish, K. M., and Chaudhuri, A. (2009). Whole-brain expression analysis of FMRP in adult monkey and its relationship to cognitive deficits in fragile X syndrome. Brain Res. 1264, 76–84. doi: 10.1016/j.brainres.2009.01.059

Conflict of Interest Statement: The Review Editor Dr. Claudine Kraan declares that, despite having collaborated with one of the authors, the review process was handled objectively and no conflict of interest exists. The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Received: 21 August 2014; Accepted: 14 January 2015; Published online: 05 February 2015.

Citation: Wong LM, Tassone F, Rivera SM and Simon TJ (2015) Temporal dynamics of attentional selection in adult male carriers of the fragile X premutation allele and adult controls. Front. Hum. Neurosci. 9:37. doi: 10.3389/fnhum.2015.00037

This article was submitted to the journal Frontiers in Human Neuroscience.

Copyright © 2015 Wong, Tassone, Rivera and Simon. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

ORIGINAL RESEARCH ARTICLE
published: 16 December 2014
doi: 10.3389/fnhum.2014.01011

How do selective attentional processes contribute to maintenance and recall in children’s working memory capacity?

Hannah E. Roome[1][*], John N. Towse[1] and Chris Jarrold[2]

1Department of Psychology, Lancaster University, Lancaster, UK

2School of Experimental Psychology, University of Bristol, Bristol, UK

*Correspondence:

Hannah E. Roome, Department of Psychology, Lancaster University, Lancaster, LA1 4YW, UK
e-mail: h.roome@lancaster.ac.uk

Edited by:
Elger Abrahamse, Ghent University, Belgium

Reviewed by:
Michael J. Kane, University of North Carolina at Greensboro, USA
Nelson Cowan, University of Missouri, USA

The development of working memory capacity is considered from the perspective of the active maintenance of items in primary memory (PM) and a cue-dependent search component, secondary memory (SM). Using free recall, plus a more novel serial interleaved items task, age-related increases in PM estimates were evident in both paradigms. In addition to this, age-related improvements in attentional selectivity were observed, indexed by the recall of target and non-target information respectively. To further characterize PM, presentation modality was varied in the serial interleaved items task (auditory, visual and dual presentation). Developmental differences were found in the effectiveness of presentation formats. Older children’s recall was enhanced by the combination of labeled visual items and enduring auditory information, whilst the same format was detrimental to younger children’s recall of target information. The present results show how estimates of PM and SM in children relate to the development of working memory capacity, but measurement of these constructs in children is not straightforward. Data also points to age-related changes in selective attention, which in turn contributes to children’s ability to process and maintain information in working memory.

Keywords: selective attention, working memory capacity, primary memory, secondary memory, dual-component model, presentation modality

INTRODUCTION

Working memory is frequently described as a memory system responsible for the active maintenance of task-relevant information, alongside other concurrent processing (Baddeley and Hitch, 1974). The enormous interest in, and popularity of, working memory capacity tasks such as reading span (Daneman and Carpenter, 1980) and counting span (Case et al., 1982) partly reflects the way in which these paradigms operationalize this conceptual description. At its core, working memory capacity involves the management of to-be-remembered items on the one hand and concurrent representations on the other. This has generated interest in for example inhibitory processes for the gating of information into the system, the ‘housekeeping’ of information already there (Kane and Engle, 2002; Hasher et al., 2007) and the impact of representational overlap between memory items and processing items (Saito and Miyake, 2004). It is therefore important to understand the attentional mechanisms involved in which items are actively maintained and which are not. In this paper, one key objective is to describe the contribution of attentional selectivity to the development of working memory processes.

The dual-component model (Unsworth and Engle, 2007) provides an influential analysis of working memory capacity in adults, which also draws on ideas relevant to attentional mechanisms. According to this perspective, working memory capacity comprises two memory systems: primary memory (PM) and secondary memory (SM). PM is a flexible memory system that actively maintains a fixed number of memory representations (Waugh and Norman, 1965; Unsworth and Engle, 2007), whilst SM is driven by cue-dependent search processes to recall target-relevant information (see Towse et al., 2008, for one perspective on search mechanisms underpinning working memory recall). Unsworth and Engle (2007) argued that the division of responsibility between these two systems explains individual differences in working memory capacity and how such differences are linked to wider cognition. To explore this dichotomy, Unsworth and colleagues (Unsworth and Engle, 2007; Unsworth et al., 2010, 2011) initially used free recall to obtain estimates of PM and SM. Based upon the serial positions of recalled items, the superior recall of recency items reflected the unloading of items from PM, whilst recall of the primacy or asymptote sections reflected cue-dependent search processes required for retrieval from SM. Yet, to prove that the same mechanisms required in free recall are also required in complex span tasks (a traditional measure of working memory capacity), free recall measures, were shown to load as highly as complex span tasks onto working memory capacity (Engle et al., 1999). From this, the dual-component model (Unsworth and Engle, 2006, 2007; Unsworth et al., 2010, 2011) have delivered an intriguing perspective to working memory, demonstrating how PM and SM use provide unique contributions to working memory capacity.

The use of free recall to obtain estimates of PM and SM has been pivotal to the implementation of the framework to explain adult working memory capacity. However, little is known about the developmental performance on free recall paradigms and the emergence of PM and SM capacities through childhood. Accordingly, a second objective is to establish whether the acknowledged change in memory in childhood (Case et al., 1982) is accompanied by developmental increases in PM capacity, SM capacity, or both. There are few published studies directly relevant to the development of PM and SM. One exception is De Alwis et al. (2009), who argued for age-related increases in SM, but not PM. The idea that recency performance does not change with age echoes earlier work by Cole et al. (1971) and Thurm and Glanzer (1971). Yet recent work by Jarrold et al. (in press) has led to an alternative conclusion that developmental increases in PM are observable, after controlling for individual’s order of report. The current work aims to provide further evidence as to whether free recall supports the idea that PM develops with age. Further, presentation rate and list length were varied, thought to affect primacy effects (Murdock, 1962) to explore whether such manipulations affect the relative contributions of PM and SM to output.

Currently, free recall carries a heavy burden in deriving measures of PM and SM, especially as the adult-based algorithm of Tulving and Colotla (1970) for separating these systems is of questionable validity among children. Their method assumes that the lag length between the presentation and recall of items [labeled an intratrial retention interval (ITRI)] specifies which system is used. Items with an ITRI of seven or below are believed to be recalled from PM, whilst items with an ITRI above seven are deemed to be recalled from SM. However, it is unclear whether this cutoff reasonably applies to children, who recall fewer items overall than adults (Jarrold et al., in press). Tulving and Colotla’s (1970) approach also assumes participants consistently begin their response from recency sections of a list. Variation in the recall order affects the recall lags in ways that may not always map straightforwardly onto the proposed partitioning of memory. Therefore, the distribution of ITRI values generated by children are described to help clarify the extent to which they can accurately recall items despite long lags, and quantifying the prevalence of various lag distances in successful recall. This article also reports what children are able to recall, and where children begin their recall. All these measures offer clues as to whether PM and SM, as derived from Tulving and Colotla (1970), represent distinct and coherent constructs.

In order to help clarify the developmental trajectory of PM, an independent paradigm was administered to provide complimentary measures of this construct. The convergent measure of PM, called the serial interleaved items task, is based on a dichotic listening paradigm (Bryden, 1971) recently revived and adapted by Hall et al. (submitted). Bryden (1971) found that adults recall of ‘attended information’ was impaired when delayed through the requirement to report ‘unattended’ items first. This is consistent with the characterization of PM functioning among adults (e.g., Broadbent, 1958; Parkinson, 1974; Martin, 1978). Therefore, a developmentally appropriate implementation of the dichotic task was deployed, largely following Hall et al. (submitted). Instead of using simultaneous presentation of two auditory streams, the two presentation sources were alternated. The two types of stimuli were labeled as focal and non-focal to indicate whether items were designated targets. These terms were preferred over the original labels of ‘attended’ and ‘unattended’ as they refer to experimenter-assigned priorities, but are neutral with respect to attentional control processes. Hall et al. (submitted) argued that performance provides several ‘signatures’ indicative of PM, and reported a developmental increase in PM capacity. Therefore, the current experiment provided the opportunity to replicate and extend this finding that focal recall (an estimate of PM) increases with age.

According to the dual-component model, incoming memory items are actively maintained in PM, however, information that should be ignored and may act as a distraction might potentially displace such memories. To try and capture this account of PM processes, for 80% of trials children were asked to recall focal items in serial order, whilst for the remaining 20% children were instructed to recall non-focal information. By implementing different priorities for the two streams, one can explore the robustness of both the focal items and the non-focal items. PM is thought to retain focal information, but when children are asked to recall lower priority information the involvement of SM may be required. This will be assessed within the inter-relations between free recall measures of PM and SM and focal and non-focal recall.

The relative success at recalling focal and non-focal targets can shed light on the relationship between item management, PM capacity and working memory capacity. Dichotic listening tasks have previously been linked to working memory capacity and attention (e.g., Conway et al., 2001). In order to be successful at such tasks participants have to direct cognition both to form robust memory representations of the focal or to-be-remembered items, and avoid confusing these with non-focal or irrelevant information. Older children are found to be better at selecting just focal items, whilst younger children are less efficient at preventing non-focal intrusions appearing in output (Doyle, 1973; Sexton and Geffen, 1979). This is potentially explained by age-related increases in the ability to focus attention on task–relevant cues, making recall less affected by distracting stimuli (Hagen, 1967). The implementation of an 80–20% split between the recall of focal and non-focal information requires children to filter necessary information in order to be successful at the task, minimizing the number of irrelevant items in working memory (Cowan et al., 2010). Under such conditions, older children should be able to focus attention better on the task at hand and be less affected by distracting stimuli. Overall, using the estimates of PM, SM and working memory capacity, this experiment will assess whether selective attention is relevant to these constructs with respect to the inter-correlations between them.

The interlink between working memory capacity and selective attention in children has been investigated by Cowan et al. (2010, 2011) within the visual domain. As part of assessing visual working memory capacity, the authors used simultaneous (Cowan et al., 2010) and interleaved presentations (Cowan et al., 2011) of attended and unattended stimuli in a visual array task. Cowan et al. (2010, 2011) reported that younger children retained fewer items in working memory, implying that a developmental increase in visual working memory capacity is central to performance. However, age-related differences in the allocation of attention between attended and unattended stimuli was only apparent when the memory load was large relative to working memory capacity. Thus, the developmental changes observed were attributed to an individuals working memory capacity as opposed to their ability to allocate attention effectively. In the study described here, consistent with Cowan et al. (2011) the serial interleaved items task will show an age-related increase in PM capacity, positively linked to increases in working memory capacity. Further, the experiment lends itself to explore how generalizable Cowan’s findings are in a different context, exploring age-related differences in the proportion of focal and non-focal recall, indicative of an effective use of selective attention.

The presentation modality of stimuli on the serial interleaved items task was manipulated, using an auditory, visual, or combined auditory and visual format. This allowed the assessment of whether PM capacity and attentional selectivity is modulated by the nature of representational codes (see Penney, 1989 for a review). In serial recall, auditory presentation produces a stronger recency advantage than visual presentation (Cowan et al., 2002). Further, auditory information has been argued to have privileged or obligatory access to some cognitive systems (Salamé and Baddeley, 1982; McLeod and Posner, 1984). However, within the context of dichotic listening paradigms, Martin (1978) reported similar forgetting of attended visual and auditory items after a delay in adults. Therefore, the current experiment provides the opportunity to explore how different presentation modalities affect the attentional capture and active maintenance of target information in the age range chosen.

In summary, the current experiment compares serial interleaved items and free recall as tasks that draw on PM. Firstly, the proportion of recalled focal and nonfocal items in the interleaved items task provides indpendement indices of PM use, and secondly the ability to selectively attend to target items. Two age groups were chosen: 5–6 year olds and 7–8 year olds, thereby describing early, primary school development, and permitting assessment of whether recall priorities change as selective attention processes mature over this period. Overall, this article considers how an estimate of working memory capacity is composed from a suite of cognitive systems and capacities providing not only theoretical relevance but also practical implications in educational practice.

MATERIALS AND METHODS

PARTICIPANTS

Eighty children were recruited from three primary schools in the North-West of England, having obtained parental consent. Children were classified by class into younger (5- to 6-year-olds; N = 40, M = 6.02 years and months, range: 5.07–6.11, 25 female), and older (7- to 8-year-olds; N = 40, M = 8.00 years and months, range: 7.02–9.00, 21 female) groups. The sample size was based on previous relevant studies, reporting between 36 and 136 participants. All participants completed all experimental measures within the task, with no exclusions of data. We therefore comply with the recommendations of Simmons et al. (2012), in that “We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study.”

MATERIALS

A stimulus pool of 380 words was extracted from the MRC linguistic database (Wilson, 1988). The corpus comprised 236 words used in the free recall experiment; 97 in the interleaved items task and 47 for the listening span task. All stimuli were monosyllabic, concrete nouns, with age of acquisition ratings below 6.2 years. Colored pictures were also presented for the free recall and interleaved items tasks. These were the same visual stimuli used by Hall et al. (submitted) and supplemented with additional items.

PROCEDURE

Participants took part in the serial interleaved items trials, free recall and listening span tasks in one session lasting approximately 40 min. The order of tasks within the session were counterbalanced across participants. The interleaved items and free recall tasks were programmed using Livecode 5.5 and the listening span task using Psyscript, version 2.3.0 (Slavin, 2003–2014). All experimental events were delivered on a 15-inch screen MacBook laptop, in a quiet, classroom setting.

Serial interleaved items task

Children were assigned to one of three presentation conditions: visual (colored illustration only, younger: N = 13; older: N = 13), auditory (spoken words, younger: N = 14; older: N = 13) and dual presentation (both presentation forms occurred simultaneously, younger 1: N = 13; older: N = 14). This task involved two cartoon characters, Spongebob and Patrick, distinguished by two male voices and colored illustrations. Participants were instructed to try and remember Spongebob’s items (focal items) and ignore Patrick’s items (non-focal items), ensuring that the recall of focal information was the focus.

Presentation consistently began with a focal item on the left hand side of the screen with the cartoon Spongebob and then alternated with the non-focal stimuli and cartoon character Patrick on the right hand side. The task included 20 trials with list lengths ranging from three to six items in total. For example, list length three included the alternation of two focal items and one non-focal item. The longest list of six items included the interleaved pattern of three focal and three non-focal items. Stimuli appeared for 1,000 ms with a 250 ms interstimulus interval. The list lengths used were pseudo-randomized and children were not aware of which list length would be presented. After stimulus presentation, 80% of lists were followed with a highlighted red speech bubble appearing above Spongebob on the left hand side of the screen, indicating the recall of focal items. For the remaining 20% of trials the red speech bubble appeared above Patrick on the right hand side of the screen, indicating the recall of non-focal items. The position of the red speech bubble was distributed randomly and therefore participants were unaware of where it was going to appear on each trial. Participants were instructed to use serial recall, thus recalling the focal items in the order in which they were presented.

Free recall

List length (8- and 10-items) and presentation rate (1- and 2-s) were manipulated in a blocked format; four blocks comprising six trials. All list items were presented auditorily alongside a colored illustration. Once a list was finished, participants were instructed to recall all the items they could remember in any order.

Listening span

The listening span task was adapted from procedures described in Threadgold (2012). Participants listened to sentences whilst trying to remember a set of unrelated words. List length increased sequentially from two to five items, with three trials at each list length, generating 12 trials. There were 42 sentences available, half of which were “silly” (i.e., semantically inappropriate); the other half were not (based on early acquired semantic information, for example “A book is a musical instrument.” in contrast to “I can see with my eyes ”). If children thought the sentence was silly they pressed “Y” on the keyboard; otherwise “N.” Immediately following this response, the unrelated word was presented in a different voice to that of the preceding sentence. At recall, participants were instructed to recall words in serial order.

RESULTS

SERIAL INTERLEAVED ITEMS TASK

Three different measures were used to ascertain age- and presentation modality differences within this task. The proportion of recalled focal targets was used as a measure of PM, but also the ability to selectively attend to the target information. This follows Cowan et al. (2011) who divided the capacity of items held in memory into different proportions according to the allocation of attention. The same analysis was also carried out for the trials that participants were instructed to recall non-focal information. Finally, children’s total recall (i.e., the sum of focal and non-focal information), labeled as k, was defined as an estimate of the total number of items loaded into working memory per trial (Cowan et al., 2011). Each of these in turn should provide evidence of PM capacity and selective attentional differences in PM capacity and working memory.

Focal recall

Analyzing overall proportion of correct focal recall, a 2(age: younger vs. older) × 3(presentation modality: visual vs. auditory vs. dual) ANOVA showed a significant effect of age, F(1,79) = 7.561, p = 0.007, = 0.093 and presentation modality, F(2,79) = 10.199, p = 0.001, = 0.216. Older children recalled a higher proportion of focal items (M = 0.819; SE = 0.023) than younger children (M = 0.729; SE = 0.023), whilst individuals in the visual condition recalled a higher proportion (M = 0.875; SE = 0.029) than both the auditory (M = 0.697; SE = 0.028; p = 0.001) and dual conditions (M = 0.750; SE = 0.028, p = 0.011). The interaction between the two variables, F(2,79) = 3.641, p = 0.031, = 0.090 arises because age differences were evident only in the dual condition, F(1,26) = 7.735, p = 0.010, = 0.236 but not the visual: F(1,25) = 0.160, p = 0.693, = 0.007 or auditory: F(1,26) = 3.361, p = 0.079, = 0.119 (Figure 1). For additional analyses including list length as a variable please see supplementary materials.

FIGURE 1. Proportion of correct focal recall as a function of presentation modality and age. Error bars represent one SE of the mean.

Non-focal recall

On those occasions when children were probed for non-focal targets, younger children recalled more than older children (younger: M = 0.466; SE = 0.031; older: M = 0.328; SE = 0.032), F(1,79) = 13.757, p = 0.002, = 0.157. The dual condition afforded greater non-focal recall, (M = 0.531; SE = 0.032), than either the visual, (M = 0.274; SE = 0.033), or auditory conditions, (M = 0.380; SE = 0.032), F(2,79) = 15.644, p = 0.001, = 0.297. The interaction was marginally significant, F(2,79) = 3.016, p = 0.055, = 0.075, whereby the recall of visual non-focal targets was least accurate and did not differ between age groups, F(1,25) = 0.019, p = 0.891, = 0.001, whilst in both auditory and dual conditions, younger children recalled more items than older children, auditory: F(1,26) = 18.545, p = 0.001, = 0.426; dual: F(1,26) = 6.594, p = 0.017, = 0.209.

The use of k as a measure of working memory

The mean number of items in working memory were also analyzed as a function of age and presentation modality. Analysis of variance confirmed older children held more items in working memory than younger children (M = 1.660; SE = 0.060 vs. M = 1.213; SE = 0.060), F(1,79) = 28.091, p = 0.001, = 0.275. k was smallest with visual presentation (M = 1.228; SE = 0.074) compared with auditory (M = 1.567; SE = 0.073, p = 0.006) and dual conditions (M = 1.514; SE = 0.073, p = 0.015), F(2,79) = 6.146, p = 0.003, = 0.142. A breakdown of the age by presentation interaction, F(2,79) = 3.672, p = 0.030, = 0.090, showed that younger children did not show reliable modality effects, F(2,39) = 2.004, p = 0.149, = 0.098, whilst older children did, F(2,39) = 8.278, p = 0.001, = 0.309, k being significantly larger for the dual than the visual condition (p = 0.001, Figure 2).

FIGURE 2. The mean number of items loaded in working memory (k) as a function of presentation modality and age. Error bars represent one SE of the mean.

FREE RECALL

Three measures were extracted here; (1) the probability of recall, revealing the serial positions of children’s successful and unsuccessful recall; (2) the probability of first recall, to establish the starting point of children’s recall; and (3) a decomposition of the recall report into PM and SM. Each of these are considered in turn.

Probability of recall

Each list length was analyzed separately, investigating the effect of age, presentation rate and serial position (see Figure 3). As expected, both analyses showed older children recalled more items than younger children, 8-item lists: F(1,78) = 54.520, p = 0.001, = 0.411; 10-item lists: F(1,78) = 44.438, p = 0.001, = 0.363. Alongside this, there were highly significant main effects of serial position, 8-items: F(7,546) = 274.131, p = 0.001, = 0.778; 10-items: F(9,702) = 323.351, p = 0.001, = 0.806. There were clear recency effects at both list lengths, in which the final items significantly differed from each other, but also all pre-recency items (all ps = 0.001). A primacy effect was only established for the 10-item list and was evident only among older children, leading to an interaction between serial position and age, F(9,702) = 1.946, p = 0.043, = 0.024.

FIGURE 3. Probability of recall as a function of serial position, presentation rate and age for 8- and 10-item lists. Error bars represent one SE of the mean.

Children recalled more items at the faster presentation rate (1- vs. 2-s per item), for both 8- and 10-item lists, F(1,78) = 18.200, p = 0.001, = 0.189 and F(1,78) = 17.020, p = 0.001, = 0.179 respectively. With 8-item lists, the presentation rate effect was only evident in younger children, F(1,39) = 21.648, p = 0.001, = 0.357, thus the significant interaction between presentation rate and age, F(1,78) = 5.915, p = 0.017, = 0.070. In addition, we found significant interactions between serial position and presentation rate, F(7,546) = 5.848, p = 0.001, = 0.070, and a three-way interaction between serial position, presentation rate and age, F(7,546) = 5.921, p = 0.001, = 0.071, reflecting how the effect of presentation rate on the different age groups was portrayed across the eight serial positions. As shown in Figure 3, younger children produced higher recall at the 1-s rate across different serial positions [position three: t(39) = 4.000, p = 0.001, five: t(39) = 5.267, p = 0.001 and seven: t(39) = 3.902, p = 0.001], whilst this was only evident at position seven in older children, t(39) = 4.451, p = 0.001.

The 10-item lists generated a significant interaction between serial position and presentation rate, F(9,702) = 23.428, p = 0.001, = 0.231, with the faster rate producing greater primacy and recency, at positions one, two, eight, and nine [all ts(79) > 3.789, ps = 0.001]. Yet, the slower presentation rate generated higher recall between middle positions three and six, [all ts(79) > -2.552, ps < 0.05]. The interaction between presentation rate and age was not significant, F(1,78) = 0.260, p = 0.611, = 0.003, nor was the three-way interaction between serial position, presentation rate and age, F(9,702) = 1.430, p = 0.171, = 0.018.

Probability of first recall

Where did children begin their recall? For 8-item lists, the last serial position was the most likely entry point for all children’s recall, F(7,546) = 158.585, p = 0.001, = 0.670 (Figure 4). This was the case for both age groups as there was no significant age group effect, F(1,78) = 0.275 p = 0.601, = 0.004. The interaction between serial position and presentation rate, F(7,546) = 5.690, p = 0.001, = 0.068, revealed that at the faster presentation rate, children showed a raised probability of beginning their recall at positions six, t(79) = 1.966, p = 0.050 and seven, t(79) = 3.232, p = 0.002 in comparison to the slower 2-s rate. However, when making the same comparison, the probability of beginning recall with the last item was higher at the slower rate, t(79) = -2.926, p = 0.004 (Figure 4). The 10-item lists only showed a significant effect of serial position, F(9,702) = 232.789, p = 0.001, = 0.749, whereby the final position was the most likely point for participants to begin their recall, but no effect of age, F(1,78) = 1.070, p = 0.304, = 0.014, and no interaction between serial position and presentation rate, F(9,702) = 1.396, p = 0.186, = 0.018 were evident.

FIGURE 4. The probability of first recall as a function of serial position and presentation rate for 8- and 10-item lists. Error bars represent one SE of the mean.

PM and SM

Using the original method by Tulving and Colotla (1970), PM and SM estimates were extracted from the free recall method. A 2(list length: 8-items data vs. 10-items) × 2(presentation rate: 1 vs. 2 s) × 2(memory systems: PM vs. SM) × 2(age: younger vs. older) mixed factor ANOVA confirmed older children recalled more items than younger children, F(1,78) = 69.050, p = 0.001, = 0.470, plus greater levels of PM than SM were produced, F(1,78) = 2346.401, p = 0.001, = 0.968. In addition, greater recall was generated at the faster rate, F(1,78) = 17.212, p = 0.001, = 0.181, plus a recall advantage for shorter list lengths, F(1,78) = 7.614, p = 0.007, = 0.089 (see Table 1).

TABLE 1. Descriptive statistics of total recall as a function of age, presentation rate, list length and memory systems (one SE of the mean).

Significant interactions between list length and age, F(1,78) = 5.440, p = 0.022, = ,065; and list length and memory system, F(1,78) = 83.425, p = 0.001, = 0.517, was further qualified by a significant three-way interaction between the three factors, F(1,78) = 10.694, p = 0.002, = 0.121. Both age groups showed a trade-off between PM and SM use. PM use decreased as a function of list length, younger children: F(1,158) = 7.477, p = 0.007, = 0.045; older children: F(1,158) = 55.035, p = 0.001, = 0.258, whilst SM use increased, younger children: F(1,158) = 14.245, p = 0.001, = 0.083; older children: F(1,158) = 14.010, p = 0.001, = 0.081. However, the interaction between list length and age highlighted that the amounted recalled across list lengths did not differ in younger children, F(1,39) = 0.077, p = 0.783, = 0.002, whilst older children recalled more items from the shorter list length, F(1,39) = 12.025, p = 0.001, = 0.236 (Figure 5).

FIGURE 5. Mean number of items recalled as a function of list length, memory system and age. Error bars represent one SE of the mean.

The analysis reported thus far is predicated on the assumption that PM and SM are distinguishable by a lag value of ± seven items. It is doubtful that this is appropriate for children, given their pattern of free recall. At the same time, using any other (smaller) threshold for children, without convergent evidence, might be, similarly arbitrary. Therefore, the total frequency of recalled items for each ITRI was calculated to visually depict the recall profile, see Figure 6. It is clear that the majority of children based their recall between ITRIs zero and two. The lag profile serves to emphasize how much the recall is based on very short lags. However, these data are not of themselves diagnostic of memory systems, and we do not take them to imply that recall is entirely a reflection of PM.

FIGURE 6. Frequency of items recalled as a function of intratrial retention interval (ITRI) and age. Error bars represent one SE of the mean.

THE RELATION BETWEEN SELECTIVE ATTENTION, PM, SM AND WORKING MEMORY CAPACITY

Table 2 reports the bivariate and partial correlations controlling for age between measures of k, focal and non-focal recall taken from the serial interleaved items tasks, working memory capacity (derived from listening span performance) and free recall estimates of PM and SM. There are consistent, significant relationships between k, working memory capacity, PM and SM. Therefore, children who recalled more items overall in the interleaved items task showed a higher usage of PM and SM and a higher working memory capacity. Focal recall also significantly related to the measures of working memory capacity, PM and SM, consistent with the view that the ability to select target information is relevant to the listed memory measures. Non-focal recall did not hold significant correlations with any of these variables.

TABLE 2. The correlational analysis between all experimental measures.

DISCUSSION

An extensive body of research has shown that complex span measures, as indices of working memory capacity, strongly predict complex cognition among children (see Jarrold and Towse, 2006). There have also been many attempts to identify the key components of complex span performance responsible for its psychological profile, in particular the nature of active maintenance (Towse et al., 2007) and the coordination of processing and memorial demands (Jarrold and Bayliss, 2007). The current study enriches such analyses by highlighting the attentional processes that contribute to performance, and distinguishing between highly accessible information (PM) and search processes that operate upon more distributed and diverse representations (SM).

The serial interleaved items task is believed to reflect PM processes (Hall et al., submitted). Older children maintained a higher total number of items in working memory (k) as well as a higher proportion of target, focal items, indicating an age-related increase in capacity and PM. We also found that focal recall was maximized and non-focal recall was minimized following visual item presentation. We suggest this may be a labeling effect. Labeling (i.e., vocal naming) only focal items reduced the requirement to filter out irrelevant information, as non-focal items would not be verbally encoded, therefore not interfering with targets. In contrast, auditory presentation produced the lowest proportion of focal recall, a modality believed to have obligatory access to cognitive systems (Salamé and Baddeley, 1982; McLeod and Posner, 1984), even though the k measure indicated higher capacity than the visual presentation. Interestingly, the age-related increase in focal recall performance was only evident in the dual condition. The combination of labeled visual items and enduring auditory information enhanced the recall of focal items in older children, whilst younger children’s recall did not change across presentation conditions.

In addition to evidence that PM increased with age, a developing efficiency in selective attention was also observed: an increase in focal recall and decrease in non-focal recall. Differing presentation formats and the nature of representational codes seemed to affect the attentional capture, active maintenance and recall of information. Despite finding low levels of non-focal visual recall in both age groups, younger children’s non-focal recall increased for the other two conditions. This implies that the auditory format hindered the selective maintenance and recall of target information.

The current findings resonate with aspects of Cowan et al. (2011), who found no age differences in attentional allocation with visual working memory capacity. In the current experiment this was also the case, but age differences were obtained in the auditory and dual presented stimuli. Obviously there are key differences between the current experiment and Cowan’s work, in terms of the experimental designs and age groups tested. However, the findings still converge in highlighting the relevance of attention for visual items in working memory. Future work might usefully include the age ranges covered by both experiments, and explore further the modality differences found here. Such work needs to consider whether the distribution of attention across the three presentation formats is equivalent (Cowan et al., 2006), ensuring the visual condition requires equivalent levels of attention as the auditory and dual conditions of the task. This in turn will enable a detailed examination of the specific selective attentional processes involved across the different modalities.

We caution against the conclusion that the serial interleaved items task solely relies on PM. One criterion for PM is near-perfect serial output of items (Unsworth and Engle, 2006). This was only evident in the three-item list (mean correct proportion of.93). Only visual recall remained near perfect across list lengths (see Supplementary Materials). Further support for SM contributing to the task was provided by the k measure and focal recall correlating with working memory capacity, PM and SM. Together, the results suggest that children’s working memory may face interference from irrelevant, non-focal information, making it harder to recall the target items from PM and potentially forcing the use of cue-dependent search processes in SM. This particular memory system is believed to contribute to performance on complex span tasks as memory items are interleaved with the processing of other information in the environment. The current task also follows this experimental layout, and therefore it may be the case that SM contributes more in longer lists as items are displaced into SM due to the maintenance of new, incoming information in PM.

Children with a higher working memory capacity tended to have a larger k score also, and recalled a higher proportion of focal items. This suggests a common role for efficient selective attentional processes in working memory. We were intrigued by the idea of non-focal recall involving retrieval from SM, assuming such items were processed and maintained in the first place. Unfortunately we did not find clear-cut evidence for this view, insofar as non-focal recall did not correlate with any memory measures. However, future research could help verify whether PM and/or SM are involved in the recall of irrelevant information when performing such tasks. The current task only used a small number of trials to assess non-focal recall, and thus it may be the case that this affected the profile of non-focal recall and its relations to the other working memory measures.

In conjunction with the serial interleaved items task, traditional free recall measures were used to illustrate where children begin their recall and what items they were able to recall. The majority of children, regardless of age, began their recall with the final list item, contrasting with reports that show adolescents (Gibson et al., 2010) and adults (Unsworth et al., 2011) more commonly initiate recall with primacy items, This may imply a qualitative change in recall strategy at some point from primary to secondary school. In terms of the serial positions effects, reflecting what items children were able to recall, it was apparent that older children were better at recalling both primacy and recency serial positions than younger children. This is consistent with the recent conclusions of Jarrold et al. (in press), but stands in contrast to previous studies of children’s free recall that suggest age-related increases in primacy but not recency (Cole et al., 1971; Thurm and Glanzer, 1971). The age-related increases in primacy and recency effects also showed up in developmental increases in PM and SM, supporting the work of Jarrold et al. (in press), but conflicting with evidence provided by De Alwis et al. (2009) that PM does not develop with age. One may attribute this conflict in findings to the different methods used to categorize items into the different memory systems. De Alwis et al. (2009) categorized the final four list items as maintenance in PM, and the remaining 10 items retrieved from SM. However, this may be considered an oversimplification, In fact, Jarrold et al. (in press) used the categorization method described by De Alwis et al. (2009) and replicated their results. However, when participants’ order of report was included within the analysis, the age-related increases in PM recall were once again evident. Thus, the data emphasizes the need to use independent measures to derive estimates of PM and SM.

It is important to reflect on the categorization of PM and SM among children from free recall. The original method by Tulving and Colotla (1970) assumes adults levels of PM capacity using a threshold of seven items. Implementing this metric makes it simpler to compare children’s performance here with data from the adult literature. Notwithstanding, if PM capacity is smaller and develops with age, as the interleaved items data suggests, then this method will not truly capture the capacity of either system. SM will be underestimated, and PM exaggerated. Figure 6 shows that the majority of items recalled were given smaller ITRIs (the highest frequency of ITRIs between zero and two), but the frequency data alone cannot be taken to suggest where to differentiate PM and SM; but to direct to the points of continuity and discontinuity in the profile. In addition, the use of Tulving and Colotla’s (1970) method assumes that recall reports proceed in the same way for all participants (children and adults), which is not borne out by the empirical data (Unsworth et al., 2011), and needs to be considered when debating the validity of this method.

This experiment has provided evidence of the development of PM and SM and the potential application of the dual-component model to children’s memory performance. Correlational evidence linked working memory capacity to both PM and SM. Further, Unsworth and Engle’s (2006; 2007) explanation of complex span as a predominantly SM based task, is also supported by the children’s data. The correlation between working memory capacity (as measured by listening span) and SM was numerically stronger than the correlation between working memory capacity and PM. This suggests that from a young age, the way in which the two systems interact may be similar to adults, although such a conclusion is predicated on the comparability in the algorithms for extracting PM and SM, which have been questioned here (see Jarrold et al., in press).

The current study highlights an increased aged-related ability to distribute attention to target information whilst also ignoring irrelevant information. Younger children are less efficient in their ability to allocate attention, thus they are less able to exclude unnecessary information from working memory. This provides the cognitive underpinnings of the development of PM and SM and how the dual-component model can be applied as an explanation of the development of working memory capacity. The fixed number of memory representations actively maintained in PM and the use of contextual cue-dependent search processes driving SM increase with age throughout childhood. The current article argues though that detailed, independent analyses of the separate components of working memory will help to better model this key cognitive construct. The concepts of PM and SM can help in this, but so can details of processing mechanisms, including selective attention.

ACKNOWLEDGMENTS

We are thankful for the cooperation of staff and students at Platt Bridge Community School, Wigan, St Andrews Primary School, Wigan, and St Anthony’s Catholic Primary School, Preston. We would also like to thank Debbora Hall for many discussions about the research program in general, as well as advice on this article. This research was supported by a grant from the UK’s Economic and Social Research Council to Chris Jarrold and John Towse (RES-062-23-2467). We would also like to thank RCUK at Lancaster University for funding this open access article. The underlying research materials and data are part of a Ph.D thesis and can be accessed by contacting the first author.

SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fnhum.2014.01011/abstract

REFERENCES

Baddeley, A. D., and Hitch, G. J. (1974). “Working memory,” in The Psychology of Learning and Motivation: Advances in Research and Theory, Vol. 8, ed. G. A. Bower (New York: Academic Press), 47–90.

Broadbent, D. E. (1958). Perception and Communication. London: Pergamon Press. doi: 10.1037/10037-000

Bryden, M. P. (1971). Attentional strategies and short-term memory in dichotic listening. Cogn. Psychol. 2, 99–116. doi: 10.1016/0010-0285(71)90004-1

Case, R., Kurland, D. M., and Goldberg, J. (1982). Operational efficiency and the growth of short-term memory span. J. Exp. Child Psychol. 33, 386–404. doi: 10.1016/0022-0965(82)90054-6

Cole, M., Frankel, F., and Sharp, D. (1971). Development of free recall learning in children. Dev. Psychol. 4, 109–123. doi: 10.1037/h0030435

Conway, A. R. A., Cowan, N., and Bunting, M. F. (2001). The cocktail party phenomenon revisited: the important of working memory capacity. Psychon. Bull. Rev. 8, 331–335. doi: 10.3758/BF03196169

Cowan, N., AuBuchon, A. M., Gilchrist, A. L., Ricker, T. J., and Saults, J. S. (2011). Age differences in visual working memory capacity: not based on encoding limitations. Dev. Sci. 14, 1066–1074. doi: 10.1111/j.1467-7687.2011.01060.x

Cowan, N., Fristoe, N. M., Elliott, E. M., Brunner, R. P., and Saults, J. S. (2006). Scope of attention, control of attention, and intelligence in children and adults. Mem. Cogn. 34, 1754–1768. doi: 10.3758/BF03195936

Cowan, N., Morey, C. C., AuBuchon, A. M., Zwilling, C. E., and Gilchrist, A. L. (2010). Seven-year-olds allocate attention like adults unless working memory is overloaded. Dev. Sci. 13, 120–133. doi: 10.1111/j.1467-7687.2009.00864.x

Cowan, N., Saults, S., Elliott, E. M., and Moreno, M. V. (2002). Deconfounding serial recall. J. Mem. Lang. 46, 153–177. doi: 10.1006/jmla.2001.2805

Daneman, M., and Carpenter, P. A. (1980). Individual differences in working memory and reading. J. Verb. Learn. Verb. Behav. 19, 450–466. doi: 10.1016/S0022-5371(80)90312-6

De Alwis, D., Myerson, J., Hershey, T., and Hale, S. (2009). Children’s higher order cognitive abilities and the development of secondary memory. Psychon. Bull. Rev. 16, 925–930. doi: 10.3758/PBR.16.5.925

Doyle, A. (1973). Listening to distraction: a developmental study of selective attention. J. Exp. Child Psychol. 15, 100–115. doi: 10.1016/0022-0965(73)90134-3

Engle, R. W., Tuholski, S. W., Laughlin, J. E., and Conway, A. R. A. (1999). Working memory, short-term memory and general fluid intelligence: a latent-variable approach. J. Exp. Psychol. Gen. 128, 209–331. doi: 10.1037/0096-3445.128.3.309

Gibson, B. S., Gondoli, D. M., Flies, A. C., Dobrzenski, B. A., and Unsworth, N. (2010). Application of the dual-component model of working memory to ADHD. Child Neuropsychol. 16, 60–79. doi: 10.1080/09297040903146958

Hagen, J. W. (1967). The effect of distraction on selective attention. Child Dev. 38, 685–694. doi: 10.2307/1127246

Hasher, L., Lustig, C., and Zacks, R. T. (2007). “Inhibitory mechanisms and the control of attention,” in Variation in working memory, eds A. Conway, C. Jarrold, M. Kane, A. Miyake, and J. Towse (New York: Oxford University Press), 227–249.

Jarrold, C., and Bayliss, D. M. (2007). “Variation in working memory due to typical and atypical development,” in Variation in working memory, eds A. Conway, C. Jarrold, M. Kane, A. Miyake, and J. Towse (New York: Oxford University Press), 227–249.

Jarrold, C., and Towse, J. N. (2006). Individual differences in working memory. Neuroscience 139, 39–50. doi: 10.1016/j.neuroscience.2005.07.002

Jarrold, C. R., Hall, D., Harvey, C. E., Tam, H. H. Y., Towse, J. N., and Zarandi, A. L. (in press). What can we learn about immediate memory from the development of children’s free recall? Q. J. Exp. Psychol. doi: 10.1080/17470218.2014.995110

Kane, M. J., and Engle, R. W. (2002). The role of prefrontal cortex in working-memory capacity, executive attention, and general fluid intelligence: an individual-differences perspective. Psychon. Bull. Rev. 9, 637–671. doi: 10.3758/BF03196323

Martin, M. (1978). Retention of attended and unattended auditorily and visually presented material. Q. J. Exp. Psychol. 30, 187–200. doi: 10.1080/14640747808400669

McLeod, P., and Posner, M. I. (1984). “Privileged loops from percept to act,” in Attention and Performance, eds H. Bouma and D. Bowhuis (Hillsdale, NJ: Lawrence Erlbaum Associates), 55–66.

Murdock, B. B. (1962). The serial position effect of free recall. J. Exp. Psychol. 64, 482–488. doi: 10.1037/h0045106

Parkinson, S. R. (1974). Variability and control in dichotic memory. J. Exp. Psychol. 102, 67–80. doi: 10.1037/h0035701

Penney, C. G. (1989). Modality effects and the structure of short-term verbal memory. Mem. Cogn. 17, 398–422. doi: 10.3758/BF03202613

Saito, S., and Miyake, A. (2004). On the nature of forgetting and the processing-storage relationship in reading span performance. J. Mem. Lang. 50, 425–443. doi: 10.1016/j.jml.2003.12.003

Salamé, P., and Baddeley, A. D. (1982). Disruption of short-term memory by unattended speech: implications for the structure of working memory. J. Verb. Learn. Verb. Behav. 21, 150–164. doi: 10.1016/S0022-5371(82)90521-7

Sexton, M., and Geffen, G. (1979). The development of three strategies of attention in dichotic monitoring. Dev. Psychol. 15, 299–310. doi: 10.1037/0012-1649.15.3.299

Simmons, J. P., Nelson, L. D., and Simonsohn, U. (2012). A 21 word solution, Dialogue. Off. Newslett. Soc. Pers. Soc. Psychol. 26, 4–7.

2003–2014Slavin, S. (2003–2014). PsyScript Version 2.3.0 [Computer software].Lancaster University. Available at: https://open.psych.lancs.ac.uk/software/PsyScript.html [accessed Janurary 2012]

Threadgold, E. (2012). The Relationship between Processing and Memory in Working Memory Development. Ph.D. thesis, Lancaster University, Lancaster.

Thurm, A. T., and Glanzer, M. (1971). Free recall in children: long-term store vs short-term store. Psychon. Sci. 23, 175–176. doi: 10.3758/BF03336538

Towse, J. N., Cowan, N., Hitch, G., and Horton, N. (2008). The recall of information from working memory: insights from behavioural and chronometric perspectives. Exp. Psychol. 55, 371–383. doi: 10.1027/1618-3169.55.6.371

Towse, J. N., Hitch, G. J., and Horton, N. (2007). Working memory as the interface between processing and retention: a developmental perspective. Adv. Child Dev. Behav. 35, 219–251. doi: 10.1016/B978-0-12-009735-7.50011-6

Tulving, E., and Colotla, V. A. (1970). Free recall for trilingual lists. Cogn. Psychol. 1, 86–98. doi: 10.1016/0010-0285(70)90006-X

Unsworth, N., Brewer, G. A., and Spillers, G. J. (2011). Inter-and Intra individual variation in immediate free recall: an examination of serial position functions and recall initiation strategies. Memory 19, 67–82. doi: 10.1080/09658211.2010.535658

Unsworth, N., and Engle, R. W. (2006). Simple and complex memory spans and their relation to fluid abilities: evidence from list-length effects. J. Mem. Lang. 54, 68–80. doi: 10.1016/j.jml.2005.06.003

Unsworth, N., and Engle, R. W. (2007). The nature of individual differences in working memory capacity: active maintenance in primary memory and controlled search from secondary memory. Psychol. Rev. 114, 104–132. doi: 10.1037/0033-295X.114.1.104

Unsworth, N., Spillers, G. J., and Brewer, G. A. (2010). The contributions of primary and secondary memory to working memory capacity: an individual differences analysis of immediate free recall. J. Exp. Psychol. Learn. 36, 240–247. doi: 10.1037/a0017739

Waugh, N. C., and Norman, D. A. (1965). Primary memory. Psychol. Rev. 72, 89–104. doi: 10.1037/h0021797

Wilson, M. D. (1988). The MRC psycholinguistic database: machine readable dictionary, version 2. Behav. Res. Methods Instrum. Comput. 20, 6–10. doi: 10.3758/BF0320259

Conflict of Interest Statement: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Received: 31 July 2014; accepted: 28 November 2014; published online: 16 December 2014.

Citation: Roome HE, Towse JN and Jarrold C (2014) How do selective attentional processes contribute to maintenance and recall in children’s working memory capacity? Front. Hum. Neurosci. 8:1011. doi: 10.3389/fnhum.2014.01011

This article was submitted to the journal Frontiers in Human Neuroscience.

Copyright © 2014 Roome, Towse and Jarrold. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

ORIGINAL RESEARCH ARTICLE
published: 21 January 2015
doi: 10.3389/fnhum.2014.01063

Fixation duration surpasses pupil size as a measure of memory load in free viewing

Radha Nila Meghanathan*, Cees van Leeuwen and Andrey R. Nikolaev

Laboratory for Perceptual Dynamics, Faculty of Psychology and Educational Sciences, University of Leuven, Leuven, Belgium

* Correspondence: Radha Nila Meghanathan, Laboratory for Perceptual Dynamics, Faculty of Psychology and Educational Sciences, KU Leuven - University of Leuven, Tiensestraat 102, Box 3711, Leuven B-3000, Belgium e-mail: radhanila.meghanathan@ppw.kuleuven.be

Edited by:
Wim Fias, Ghent University, Belgium

Reviewed by:
Jan W. De Fockert, Goldsmiths, University of London, UK
Jan Lauwereyns, Kyushu University, Japan

Oculomotor behavior reveals, not only the acquisition of visual information at fixation, but also the accumulation of information in memory across subsequent fixations. Two candidate measures were considered as indicators of such dynamic visual memory load: fixation duration and pupil size. While recording these measures, we displayed an arrangement of 3, 4 or 5 targets among distractors. Both occurred in various orientations. Participants searched for targets and reported whether in a subsequent display one of them had changed orientation. We determined to what extent fixation duration and pupil size indicate dynamic memory load, as a function of the number of targets fixated during the search. We found that fixation duration reflects the number of targets, both when this number is within and above the limit of working memory capacity. Pupil size reflects the number of targets only when it exceeds the capacity limit. Moreover, the duration of fixations on successive targets but not on distractors increases whereas pupil size does not. The increase in fixation duration with number of targets both within and above working memory capacity suggests that in free viewing fixation duration is sensitive to actual memory load as well as to processing load, whereas pupil size is indicative of processing load only. Two alternative models relating visual attention and working memory are considered relevant to these results. We discuss the results as supportive of a model which involves a temporary buffer in the interaction of attention and working memory.

Keywords: eye movements, fixation duration, pupil size, memory load, processing load, attention, working memory, visual search

INTRODUCTION

Vision science is shifting its focus from the traditional stimulus-response paradigm to a more ecologically valid approach: the analysis of continuous visual processes in free viewing. In these conditions, visual information is acquired via eye movements and thus their measurement may provide crucial insights into the time course of processing and accumulation of visual information.

In freely viewing, the eyes are directed to different locations of interest via saccades. The eyes fixate these locations in order to identify and encode information (Henderson and Hollingworth, 1999; Irwin, 2004). Over time, information from the fixated locations is accumulated in memory across multiple fixations (Melcher, 2001; Hollingworth and Henderson, 2002; Tatler et al., 2003). In order to study accumulation of information in memory in free viewing conditions, it is important to determine which measures of eye movement are sensitive to memory load. Such measures could be useful in combination with behavioral responses, but also in combination with neurophysiological recording, for example, of electrical brain activity.

One of the most common and informative eye movement measures is fixation duration. The duration of fixation increases under processing load, i.e., it increases when processing becomes more effortful (Inamdar and Pomplun, 2003; Peterson et al., 2008; He and McCarley, 2010). In particular, fixation duration is sensitive to the amount of attention deployed to a fixated location (Just and Carpenter, 1980; Irwin, 2004; Henderson, 2007) and, crucially, memory load lengthens the average duration of fixation in free viewing conditions, as shown in visual search studies (McCarley et al., 2006).

Another potentially useful information processing indicator is pupil size. When measured at fixation, the pupillary response increases with memory accumulation and task difficulty (Beatty, 1982). Pupil size saturates at the limit of working memory capacity and decreases during use of memory strategies that reduce load (Beatty, 1982; Andreassi, 2000; Beatty and Lucero-Wagoner, 2000). Pupil size is relatively unexplored, however, as a measure in free viewing conditions. We are aware of only one study where pupil size increased with memory load during unrestricted eye movements (Porter et al., 2007). In their study, the pupil dilated more for larger search set size, larger number of search targets, and for heterogeneous as compared to homogeneous distractors.

This study aims at assessing fixation duration and pupil size, specifically, as measures of accumulating memory load in free viewing conditions. Whereas previous studies considered the effect of memory load on eye movement measures averaged across the duration of free viewing (McCarley et al., 2006), here we will focus on how these measures change dynamically as memory load increases as participants gather information during viewing of a display. To do this, we used a visual search task with multiple targets. The presence of multiple targets necessitated them to be accumulated in memory during free viewing. The visual search was followed by a change detection task in order to assess memory for the targets. The number of targets in a display was varied between 3, 4 and 5, a number chosen to lie around the working memory capacity limit of 4 (Luck and Vogel, 2013). We predict that both fixation duration and pupil size will reflect the number of targets in the search task. Moreover, these measures might also reflect the number of distractors viewed, to the extent that they enter memory.

Besides memory load, the task also involves attentional selection, control operations, and other cognitive processes. Fixation duration and pupil size may, in principle, be differently sensitive to all of these processes and their interactions. To predict the interaction of these processes and memory load during the task, we draw on two recent models.

According to Model I (Bowman and Wyble, 2007; Wyble et al., 2011, 2015), limited resources are shared between working memory and attention. This implies that during the course of a search task, as each new target is detected, working memory becomes increasingly loaded, resulting in decreasing processing rate. Therefore, in our visual search task, a difference between 3, 4 and 5 targets should be seen from early on. According to Model II (Simione et al., 2012; Raffone et al., 2014), new targets will be loaded automatically into a temporary global workspace buffer with limited capacity. When this buffer is full, control operations involving interactions between attention, working memory and global workspace are needed, in order to select and manage the content for consolidation in memory. These interactions give rise to additional processing load. Therefore, in the early stage of a search task, effects of memory load alone will be obtained. When the buffer is loaded to capacity, additional effects of attentional and processing load will appear. Hence according to Model II, a difference between 3 and 4 targets should be seen early, since they lie within memory capacity limit. But, in addition, a difference between 4 and 5 targets would arise with a late onset, because of the additional processing load that becomes necessary when the buffer is full. This prediction contrasts with Model I. Both models have in place an early mechanism of attentional filtering, which warrants that only targets are loaded into working memory. Therefore, the effect of memory load will be seen only in target processing and not on distractor processing.

To evaluate fixation duration and pupil size as memory load measures, we test the extent to which the two measures adhere to one of these alternative models. The degree of conformity with one of the models would legitimate the measure; at the same time it would support the model to which it adheres.

MATERIAL AND METHODS

PARTICIPANTS

Twenty-three participants (ages 18–29 years, mean 20.86 years, 7 male) took part in the experiment. Of these, 15 reported normal vision, 6 had their vision corrected to normal with glasses and two with contact lenses. Two participants were excluded because of noisy eye-movement data and two others for failure to meet the criterion number of trials in each condition as explained below, leaving 19 participants (6 male), whose data were used for the final analysis. All participants gave their written consent. The study was approved by the departmental Ethics Committee of the KU Leuven.

STIMULI AND APPARATUS

We used displays with target and distractor items in various orientations (Figure 1). There were 40 items in each display, 3, 4 or 5 of which were targets. In a pilot experiment using a staircase procedure, we found 40 items to be the optimal set size for 70% correct responses in our task. To avoid any luminance differences between conditions, we kept the number of items constant at 40 across all conditions, while varying the number of distractors as 40 minus the number of targets in the display.

FIGURE 1. A sample display with 40 items: 5 targets (“T”) and 35 distractors (“L”).

The target stimuli were “T”s, 0.41° × 0.41° of visual angle, and the distractor stimuli were “L”s, 0.31° × 0.41° of visual angle. Both types of items were rotated 20°, 80°, 140°, 200°, 260° or 320°. No item was in vertical or horizontal orientation and no two targets in a display had the same orientation. Each item was surrounded by a circle subtending 0.83° of visual angle. This was done in order to make it too difficult for our participants to discriminate targets without fixating on them (Peterson et al., 2001; Körner and Gilchrist, 2007).

The stimuli were presented in black (0.48 cd/m^2) on a gray background (32.84 cd/m^2). The gray background spanned the entire monitor screen and subtended 39.9° × 30.5° of visual angle. The stimuli themselves appeared within a virtual rectangle of size 32.9° × 23.12°. Distractor locations were chosen randomly within this rectangle under the constraint that the minimum distance between two items was 3.12°. Target locations were chosen randomly but with the constraint that targets appeared within a donut shaped region of inner radius 3.12° and outer radius 10.4° from the center of the display, with a probability of 0.86, 0.9 and 0.92 in the 3-, 4- and 5-target conditions, respectively. This was done in order to deter easy detection of targets close to the central fixation dot or near the border of the display by reducing the odds of targets occurring there. Two targets were always separated by a minimum of 6.24° so as to prevent two targets from being detected in a single fixation. Stimuli were presented on a 40 cm × 30 cm monitor with a refresh rate of 75 Hz and a screen resolution of 1600 × 1200 pixels. The viewing distance was 55 cm. The stimulus presentation program was written in Python 2.7.

PROCEDURE

We used a change-detection task, in which participants searched a first display, the search display, for 10 s. This display was followed by a change detection display where, in half of the trials, one of the targets (changed target) had changed orientation (different trials), and in the other half, a display identical to the search display was shown (same trials). Participants were asked to report whether a change had occurred.

Each trial comprised of a blank screen with a fixation dot, a search display, another blank screen with a fixation cross, a change detection display, and a feedback display. At the start of each trial, participants were asked to fixate on the central dot and press the space bar to start the task. Immediately after the space bar was pressed, a visual search display was presented for 10 s. Participants were not informed about the number of targets in the display, but had been told in advance that each display would contain 3–5 targets. Participants were asked to search for the targets and memorized the orientation of each target in order to perform change detection in the subsequent display. After the first display, the fixation cross was shown for a duration randomly varied according to a uniform distribution in the interval of 1–1.5 s. The change detection display was shown afterwards until participants responded or after another 10 s, whichever was earlier. Participants were asked to respond with the left arrow key to indicate “change” or the right arrow key to indicate “no change”. The response keys were counterbalanced across participants. After response, a feedback screen was displayed for 0.8 s with the targets encircled in green for correct responses and in red for incorrect ones. The feedback for a changed target was given by substituting the response circle around the target with a larger one. The new trial started immediately after the feedback screen.

Participants performed a practice block of 12 trials before the experiment. They were then asked to complete ten blocks of 27 experimental trials each for a total of 270 trials, lasting around 100 min.

EYE MOVEMENT AND PUPIL SIZE RECORDING

The desktop system of the EyeLink 1000 eye tracker (SR Research Ltd.) was used for recording eye movements and pupil size. The accuracy of the system is typically between 0.25° and 0.5° and pupil size resolution is 0.2% of diameter. The system records pupil area or diameter as integer numbers in arbitrary units using centroid or ellipse fitting for pupil detection. In our experiment, pupil area was recorded and ellipse fitting was used in order to reduce dependance of pupil size on gaze direction. Eye position and pupil size were tracked at a sampling frequency of 250 Hz.

Participant’s head was stabilized using a chin rest. At the beginning of the experiment, the eye to be tracked was determined from the quality of calibration. A 9-point calibration was done for calibration points at the center, four corners and mid-points of the edges of the stimulus display area. A tolerance of 2° was maintained for error between calibration and validation. If the error was larger, calibration was repeated. For participants with consistently poor calibration or validation in the left eye, the right eye was tracked. For 14 of the 19 participants whose data were finally analyzed, the left eye was tracked.

Before each trial, drift correction was performed during the fixation display automatically by the EyeLink system. If drift was greater than 2°, the trial did not proceed and calibration was repeated. Calibration was also done at the start of each block.

During the experiment, EEG was also recorded, but these data will be reported elsewhere.

ANALYSIS OF DATA

Eye movements

Fixation locations, fixation durations, and pupil sizes were determined and output by the EyeLink software. This software detected saccades based on an eye velocity threshold of 22°/s and an acceleration of 3800°/s^2.

Since calibration was performed only once before a block, which lasted about 9 min, and the head was fixed with a chin rest only, errors in eye location because of head movements cannot be excluded. To check location accuracy, we visually inspected scan paths for each participant both trial-by-trial and by superimposing scan paths of all trials in each block over the rectangular viewing area. To make corrections in fixation locations, all targets from all trials in a block were centered and superimposed. For fixations within 3.12° around the target, a fixation density array was generated and a Gaussian filter was applied to generate a heat map. A heat map that was not centered at the target indicated a shift in the recorded eye location. The vector from the center of mass of the heat map to the center of the target was used as a correction factor for all fixation locations in that block. 2 of 23 participants who had excessive eye movements outside the display in spite of the corrections were excluded from analysis.

During analysis of fixations, a target was considered detected when a fixation was located within 2° radius around the target. Körner and Gilchrist (2007) showed that for stimuli with bounding circles that reduced discriminability similar to the ones used in our experiment, fixation distances beyond 3° reduced item detection to chance levels. Moreover, since no difference had previously been found between a 2° criterion and a criterion computed with a nearest neighbor algorithm for assigning fixations to visual search targets (McCarley et al., 2006), we used a criterion of 2°.

Fixation duration and pupil size

Fixation duration and pupil size recorded during the search display were analyzed for trials with correct responses. We divided the 10 s duration of search interval into bins of 1 s each. For each participant, in each bin, fixation durations were averaged across fixations on both targets and distractors.

Pupil size was analyzed during fixations and additionally as a time series. In fixations, the sampling points of pupil size were first averaged in time within each fixation and then, across all fixations in the bin. Before both analyses, the pupil signal was preprocessed. Since pupil response is known to be slow while the eye tracker is a source of high-frequency noise (Klingner et al., 2008), the pupil size signal was low-pass filtered at 10 Hz to remove noise from the eye tracker. For blinks detected by the EyeLink software, 10 samples before the blink, all samples during the blink, and 20 samples after the blink were removed and replaced by spline interpolation. Fewer than 1% of samples were replaced in this way. Other spike-like artefacts probably related to partial occlusion of the pupil by eyelids were detected by custom-made software and also replaced by spline interpolation. On average, 3.26% of the data was replaced in this process. For each participant, pupil sizes series were normalized by finding the mean pupil size during the search display irrespective of conditions and subtracting this mean value from each sample point.

Statistical analysis

We used repeated-measures ANOVA with the Huynh-Feldt correction (ε) of p-values associated with two or more degrees of freedom in order to compensate for violation of sphericity. We used the Fisher’s LSD (least significant difference) test for post-hoc analyses.

RESULTS

PERFORMANCE

The average accuracy in the task was greater than 70% (Table 1).

Table 1. The percentage of correct trials for 19 participants.

The accuracy in same trials was higher than that in different trials (F(1,18) = 36.4, p < 0.001). Accuracy decreased with the number of targets, as indicated by the effect of target (F(2,36) = 30.3, p < 0.001, ε = 0.96). Accuracy was higher for 3-target condition compared to 4-target condition (p < 0.001) and 5-target condition (p < 0.001). There was no difference in accuracy between 4- and 5-target conditions.

A correct response does not mean that all the targets presented were fixated. Participant could score well above chance level by inspecting a few targets and gauge the rest from peripheral vision, or even guess the rest. Since we could not eliminate either of these possibilities, we grouped trials according to the number of targets fixated irrespective of the number presented. This way, a target condition corresponded to the number of targets that were fixated and most likely attended and memorized.

When initially using the criterion that a target fixation should last longer than 200 ms, some participants had very few trials in the 5-target condition. Therefore, we took into account the minimum number of trials needed in each condition to determine the criterion for target fixation duration. Histograms of fixation durations within 2° of the target peaked between 120 ms and 180 ms. In a range of 10–20 for number of trials and 120 ms–150 ms for fixation duration, we generated exhaustive combinations of number of trials and fixation duration. We used each combination of number of trials and target fixation duration as a criterion to find the number of participants satisfying this criterion. We found that for a criterion of 10 for number of trials and 150 ms for target fixation duration, the maximum number of participants satisfied the criterion. Data from two participants who could not meet this criterion were excluded from analysis, leaving us with data for the 19 participants that we finally analyzed.

Since we presented a minimum of 3 targets, we analyzed only those trials where at least 3 targets were fixated. In the 3-target condition all the targets presented were always fixated. For the 4- and 5-target condition, in 94.2% and 89.6% of different trials, the changed target was fixated and in 58.3% and 36.2% of the same trials, all presented targets were fixated, respectively. This result confirms that even when participants responded correctly, they did not always fixate on the necessary targets.

To evaluate the time course of target fixations in a trial, we calculated the average cumulative target fixation scores as the number of targets fixated up to each second of the search interval (Figure 2). An ANOVA with target condition and time as factors showed a significant effect of target (F(2,36) = 1605.9, p < 0.001, ε = 0.58), an effect of time (F(9,162) = 971.4, p < 0.001, ε = 0.28) and an interaction between target and time (F(18,324) = 128.6, p < 0.001, ε = 0.6).

FIGURE 2. Cumulative target fixation scores calculated for each second of search interval. The means across 19 participants are shown. Standard errors are not displayed because they were so small that they did not exceed the mean markers in size.

FIXATION DURATION VS. PUPIL SIZE

We applied an ANOVA with factors of time and target condition on fixation duration and pupil size. There was an effect of time for both fixation duration (F(9,162) = 13.99, p < 0.0001, ε = 0.4) (Figure 3A) and pupil size (F(9,162) = 7.9, p < 0.001, ε = 0.24) (Figure 3B). There was a prominent effect of target condition for fixation duration (F(2,36) = 50.4, p < 0.0001, ε = 0.71) and post-hoc tests showed a significant increase in fixation duration with number of targets (all p < 0.001). For pupil size, the effect of target condition was significant (F(2,36) = 4.3, p = 0.02, ε = 0.95) with post-hoc tests showing that pupil size was larger for the 5-target condition than the 3-target condition (p < 0.01) and the 4-target condition (p < 0.05).

FIGURE 3. Average fixation duration and pupil size. Fixation duration (A) and pupil size (arbitrary units) (B) in 1-s intervals for the entire search interval. Along the time axis the target conditions having significant difference are indicated as found from post-hoc tests. The difference between 3- and 5-target conditions is not shown because it occurred from the 2nd to the 10th second for fixation duration and from the 3rd to the 10th second for pupil size. All plots show the data averaged across 19 participants. The error bars indicate the standard errors of the means.

There was an interaction between target condition and time for fixation duration (F(18,324) = 2.3, p = 0.04, ε = 0.31). Post-hoc tests showed the emergence of a difference between 3- and 5-target conditions beginning at two seconds and staying until the end of search, a difference between 3- and 4-target conditions emerging at the 4th second and staying till the 9th second and a difference between 4- and 5-target conditions beginning at the 7th second and staying till the end of search (Figure 3A). For pupil size the interaction between target condition and time only approached significance (F(18,324) = 2, p = 0.06, ε = 0.36) though it was significant (p = 0.009) before correcting for sphericity. Post-hoc tests revealed a difference between 3- and 5-target conditions beginning from the 3rd second and staying until the end of search, no difference between 3- and 4-target conditions and a difference between 4- and 5-target conditions from the 5th second till the end of search (Figure 3B).

Thus, both fixation duration and pupil size showed sensitivity to the number of targets in the display. In contrast to pupil size, fixation duration showed a difference between the target conditions within memory capacity (3 and 4). This difference lasted for 6 s of the entire interval of visual search, suggesting sensitivity of fixation duration to memory load. In the middle of the interval, after about 5–6 s, a difference between target conditions above memory capacity (4 and 5) appeared. This effect occurred simultaneously in both measures, suggesting a late onset of additional processing load effects.

In addition to the assessment of pupil size within fixations we analyzed the evolution of pupil size over all 2500 sampling points. We applied an ANOVA with 3 targets as a factor on each of the 2500 sampling points. The false discovery rate was controlled using the procedure described by Storey (2002).

In the pupil size series, there was a steep increase in the first 1.5 s, (Figure 4A), which could be attributed to luminance change from the fixation cross to the onset of the search display. After this time, pupil size generally increased throughout the search interval. Figure 4B illustrates this increase after omitting the initial pupil response to luminance change. The point-wise ANOVA showed a main effect of target in the interval 5.5–7.5 s, which corresponds to the maximal deviation of the 5-target condition. This result indicates that an effect of target on pupil size occurs, albeit only for a limited time interval.

FIGURE 4. Pupil size series (arbitrary units). (A) Pupil size series averaged across all target conditions for entire duration of search interval. (B) Pupil size series for the 3 target conditions after omitting the initial pupil response to luminance change. In (B) the bar along the time axis indicates the interval of significant difference between target conditions as found in the point-wise ANOVA. All plots show the pupil size series averaged across 19 participants. The shaded areas indicate standard errors of the means.

TARGETS VS. DISTRACTORS

To evaluate the contribution of targets and distractors to the effect of target condition, we assessed fixation duration and pupil size separately for target and distractor fixations. We pooled the 3-, 4-, and 5-target conditions and distinguished target fixations based on order of visiting during search into 1st, 2nd, 3rd and 4th target visited. Targets visited 5th were not considered because their number was too small. To have the same amount of target and distractor data, for each target we selected the fixation on either the preceding or the following distractor alternatingly, after having established that these two do not differ systematically. Fixation duration and pupil size were averaged for each target visiting order, separately for target and distractor fixations. Only first target fixations were used for this analysis. It could not be excluded, however, that some of the preceding and following distractor fixations are re-fixations.

We performed an ANOVA with fixation type (target vs. distractor) and visiting order (1st, 2nd, 3rd, and 4th) as factors. Target fixations were much longer than distractor fixations (F(1,18) = 149.1, p < 0.0001) (Figure 5A). But there was no fixation type effect on pupil size. There was an effect of target visiting order for fixation duration (F(3,54) = 4.1, p = 0.01, ε = 1), while for pupil size this effect did not reach significance (F(3,54) = 2.3, p = 0.08) despite visible increase of pupil size between 1st and 2nd targets (Figure 5B). From the cumulative target scores we know that the first target is fixated by the first second (Figure 2), in which time, pupil size responds predominantly to luminance changes (Figure 4A). This suggests that the pupil size during the fixation on the 1st target is defined by luminance and it is unlikely that it reflects any difference between targets and distractors related to cognitive processing.

FIGURE 5. Fixation duration and pupil size for targets and distractors. Fixation duration (A) and pupil size (arbitrary units) (B) for fixations on successive targets and their preceding distractors in the order of target fixations. Only for fixation duration, a significant difference between targets and distractors as found from post-hoc tests was seen for all target visiting orders. All plots show the data averaged across 19 participants. The error bars indicate standard errors of the means.

For fixation duration there was a significant interaction between fixation type and visiting order (F(3,54) = 9.6, p < 0.0001, ε = 1). Post-hoc tests showed that for targets there was a significant increase in fixation duration with target visiting order between all target visiting order combinations except between 2nd and 3rd targets (all p < 0.01). For distractors, there was no difference between target visiting orders. Post-hoc tests also revealed highly significant differences between targets and distractors for all target visiting orders (all p < 0.001) (Figure 5A). We tested the changes across the target visiting orders using a planned comparison with linear contrast over visiting order. There was a linear trend for targets (F(1,18) = 18.6, p < 0.001) but not for distractors (p = 0.2). In sum, fixation duration increases with visiting order only for targets and not for distractors.

Pupil size showed no interaction between fixation type and target visiting order. Pupil size, therefore, reveals no difference between targets and distractors. Pupil size, being a slow signal, is hence ineffective in differentiating eye fixations on targets and distractors in free viewing.

DISCUSSION

We asked how memory load affects fixation durations and pupil size in free viewing behavior. To this end, we recorded eye movements and pupil size in a multiple-targets visual search task in which participants could freely move their eyes. The accuracy of memory for targets was tested in a subsequent change detection task where participants detected the change in orientation of one of the targets. The effect of memory load on fixation duration and pupil size was evaluated in the course of the visual search task. The number of targets (3, 4 and 5) fixated in the search display was an index of memory load. Fixation durations and pupil size were also compared for successive target and distractor visits.

Our findings suggest that the 10 s of the visual search task can be approximately divided in two sequential stages of processing: the first stage is dominated by loading targets into memory and in the second stage memory accumulation is accompanied by other cognitive processes. The following line of evidence supports this conclusion.

The first 4 targets used in the targets vs. distractors analysis were visited within 6 s (Figure 2). In this interval there was a linear increase in the duration of fixation on targets but not on distractors (Figure 5A). Later during this interval (4–6 s), fixation duration was longer for 4- than 3-target conditions, both within the memory capacity limit. In the following interval, about 7–10 s, both fixation duration and pupil size increase with time and both differentiate between 4 and 5 target conditions, i.e., above the memory capacity limit (Figure 3). Thus, the initial difference in fixation duration within the memory capacity was associated with increasing duration of target fixations, whereas the later part with the number of targets above memory capacity was additionally characterized by still longer fixation duration and larger pupil size.

FIXATION DURATION

Cumulative target fixation scores (Figure 2) show that targets are fixated sooner in conditions which have more targets, leading to a prominent difference between the three conditions at any time. This pattern is discernible in the dynamics of fixation duration. The deviation in fixation duration between 3- and 5-target conditions starts at 2 s, followed by a deviation between 3- and 4-target conditions. The early increase in fixation duration coincides with the early increase in the number of targets fixated (Figure 2). As well, the different rates of increase of fixation duration for 3-, 4- and 5- target conditions (Figure 3A) mirror the rates of increase in the number of targets fixated (Figure 2). The difference between 3- and 4-target conditions, in which the numbers of targets are still within the limit of working memory capacity, indicates sensitivity of fixation duration to memory load. The fixation durations in the 4- and 5-target conditions begin to differ only from the 7th second onwards (Figure 3A) when in the 5-target condition 4 targets have already been seen. Assuming memory is fully loaded at this time, the following prominent increase in the 5-target condition may reflect cognitive control operations involved in managing the number of items exceeding the capacity limit.

We observed a huge difference in fixation duration between targets and distractors. This is not surprising, because after detection of targets attention stays on targets for encoding, while after detection of distractors the focus of attention shifts away.

The target-distractor comparison revealed that the observed difference between the 3-, 4-, and 5-target conditions occurs because of the increasing fixation durations on successive targets but not on distractors (Figure 5A). No increase in duration for distractor fixations indicates that the effect is specific to accumulating targets into memory rather than decreasing resources or increasing cognitive effort while performing the task.

A general trend of fixation durations increasing with time is known to occur because of transition from global to local scanning strategies and is typically limited to the first 2 s of free visual exploration (Unema et al., 2005). Such an effect was seen for all of 3-, 4- and 5-target conditions (Figure 3A). The effects suggesting memory accumulation are reflected later both in the difference between 3-, 4- and 5-target conditions and in the linear trend for successively visited targets.

The increase of fixation duration with memory load is consistent with the results of previous studies of multiple target visual search (McCarley et al., 2006), single target visual search (Peterson et al., 2008; He and McCarley, 2010) and comparative search tasks (Inamdar and Pomplun, 2003).

Fixation duration, thus, reflects memory load precisely when the number of items lies within memory capacity limit. In the second half of the search interval in our task, when memory load exceeds the memory capacity, fixation duration probably indicates the effect of processing load in addition to memory load. These results show no support for Model I (Bowman and Wyble, 2007; Wyble et al., 2011), however, they are in accordance with Model II (Simione et al., 2012; Raffone et al., 2014). As predicted by Model II, fixation duration divulges the dichotomy between memory load and processing load in the course of the search interval. An early difference occurs between 3- and 4-target conditions and between 3- and 5-target conditions indicating increase in memory load within the memory capacity. This corresponds to the filling of the temporary buffer in Model II. According to Model I, a difference should have been seen between 4- and 5-target conditions as well at this stage, but no evidence bearing this was found in our results. Model II also posits that information that is filtered by attention occupies the temporary buffer for later consolidation into working memory. Such filling of the temporary buffer is supported by the increase in duration of fixations on successive targets and not on distractors. As memory load exceeds capacity limit, the difference between 4 and 5-target conditions emerges, when, according to Model II, processing load is heightened because encoding becomes more effortful.

PUPIL SIZE

For 7 s of the entire search duration, pupil size showed a difference between 3- and 5-target conditions. 4- and 5-target conditions deviated from the 5th second onwards (Figure 3B). This coincides with the onset of the difference between 4- and 5-target conditions in fixation duration (Figure 3A). The cumulative target fixation score (Figure 2) indicates that at this moment the 5-target condition exceeded memory capacity of 4. The pupil size series also showed an effect of number of targets only in this interval of maximal deviation of the 5-target condition.

Pupil size does not differ between targets and distractors (Figure 5B). Furthermore, pupil size does not change with target visiting order. Its increase during the search interval is most prominent after 5 s (Figure 3B). These findings constitute additional evidence that pupil size is unlikely to be related to memory load and probably reflects cognitive effort which is most prominent in the second half of our task.

Porter et al. (2007) also reported an increase of pupil size with time in free viewing. In this study, the pupil dilated quickly and sustained till the end of a counting task, while it dilated gradually throughout a visual search task. These different dilatory patterns were ascribed to the differences in spatial memory requirements between the tasks. In our experiment, a similar gradual increase in pupil size in time is seen (Figures 3B, 4), which might correspond to spatial memory requirements imposed by the visual search task for tagging of found targets and visited distractors (Shore and Klein, 2000; Körner and Gilchrist, 2008). Besides this, when memory load increased within the task, pupil size did not appear to have the temporal resolution necessary to observe memory load changes at the level of a single fixation.

Thus, pupil size reflects the number of targets only when it exceeds the memory capacity limit. This suggests that pupil size reflects processing load rather than memory load. In our task, the processing load might involve multiple collateral processes accompanying memory management. This additional effort required for encoding in the case of 5 targets, which is reflected in pupil size, was predicted by Model II as the late emergence of difference between 4 and 5 targets.

CONCLUSION

In sum, our findings indicate that both fixation duration and pupil size depend on the number of targets which are fixated in a search task with unrestricted eye movements. Fixation duration is selective to memory load for targets. In contrast, pupil size is too slow for isolating instances of memory accumulation such as target encoding in the free viewing search task. Pupil size most likely reflects an overall processing load which incorporates several cognitive processes. The slowness of pupil size dynamics renders it unlikely that a combination of fixation duration and pupil size may enhance our understanding of memory accumulation, compared to fixation duration alone.

The dynamics of the search task and the different target-distractor processing are in good correspondence with the predictions of Model II. This could be understood as evidence for involvement of a temporary buffer in memory accumulation of multiple targets across sequential fixations in free viewing behavior.

ACKNOWLEDGMENTS

The authors were aided by an Odysseus grant from the Flemish Organization for Science (FWO).

REFERENCES

Andreassi, J. L. (2000). “Pupillary response and behavior,” in Psychophysiology: Human Behavior and Physiological Response, (Mahwah, NJ: Lawrence Erlbaum Assoc.), 218–233.

Beatty, J. (1982). Task-evoked pupillary responses, processing load and the structure of processing resources. Psychol. Bull. 91, 276–292. doi: 10.1037//0033-2909.91.2.276

Beatty, J., and Lucero-Wagoner, B. (2000). “The pupillary system,” in Handbook of Psychophysiology, eds J. T. Cacioppo, L. G. Tassinary and G. G. Berntson 2nd Edn. (Cambridge, England: Cambridge University Press), 142–162.

Bowman, H., and Wyble, B. (2007). The simultaneous type, serial token model of temporal attention and working memory. Psychol. Rev. 114, 38–70. doi: 10.1037/0033-295X.114.1.38

He, J., and McCarley, J. S. (2010). Executive working memory load does not compromise perceptual processing during visual search: evidence from additive factors analysis. Atten. Percept. Psychophys. 72, 308–316. doi: 10.3758/app.72.2.308

Henderson, J. M. (2007). Regarding scenes. Curr. Dir. Psychol. Sci. 16, 219–222. doi: 10.1111/j.1467-8721.2007.00507.x

Henderson, J. M., and Hollingworth, A. (1999). The role of fixation position in detecting scene changes across saccades. Psychol. Sci. 10, 438–443. doi: 10.1111/1467-9280.00183

Hollingworth, A., and Henderson, J. M. (2002). Accurate visual memory for previously attended objects in natural scenes. J. Exp. Psychol. Hum. Percept. Perform. 28, 113–136. doi: 10.1037//0096-1523.28.1.113-136

Inamdar, S., and Pomplun, M. (2003). “Comparative search reveals the tradeoff between eye movements and working memory use in visual tasks,” in Proceedings of the Twenty-Fifth Annual Meeting of the Cognitive Science Society, eds R. Alterman and D. Kirsh (Boston, MA: Cognitive Science Society), 599–604.

Irwin, D. E. (2004). “Fixation location and fixation duration as indices of cognitive processing,” in The interface of Language, Vision and Action: Eye Movements and the Visual World, eds J. M. Henderson and F. Ferreira (New York, NY: Psychology Press), 105–133.

Just, M. A., and Carpenter, P. A. (1980). A theory of reading: from eye fixations to comprehension. Psychol. Rev. 87, 329–354. doi: 10.1037/0033-295x.87.4.329

Klingner, J., Kumar, R., and Hanrahan, P. (2008). “Measuring the task-evoked pupillary response with a remote eye tracker,” in Proceedings of the 2008 Symposium on Eye Tracking Research and Applications—ETRA (New York: ACM Press), 69–72.

Körner, C., and Gilchrist, I. D. (2007). Finding a new target in an old display: evidence for a memory recency effect in visual search. Psychon. Bull. Rev. 14, 846–851. doi: 10.3758/bf03194110

Körner, C., and Gilchrist, I. D. (2008). Memory processes in multiple-target visual search. Psychol. Res. 72, 99–105. doi: 10.1007/s00426-006-0075-1

Luck, S. J., and Vogel, E. K. (2013). Visual working memory capacity: from psychophysics and neurobiology to individual differences. Trends Cogn. Sci. 17, 391–400. doi: 10.1016/j.tics.2013.06.006

McCarley, J. S., Kramer, A. F., Boot, W. R., Peterson, M. S., Wang, R. F., and Irwin, D. E. (2006). Oculomotor behaviour in visual search for multiple targets. Vis. Cogn. 14, 685–703. doi: 10.1080/13506280500194147

Melcher, D. (2001). Persistence of visual memory for scenes. Nature 412:401. doi: 10.1038/35086646

Peterson, M. S., Kramer, A. F., Wang, R. F., Irwin, D. E., and McCarley, J. S. (2001). Visual search has memory. Psychol. Sci. 12, 287–292. doi: 10.1111/1467-9280.00353

Peterson, M. S., Melissa, R. B., and Wong, J. H. (2008). Were you paying attention to where you looked? The role of executive working memory in visual search. Psychon. Bull. Rev. 15, 372–377. doi: 10.3758/pbr.15.2.372

Porter, G., Troscianko, T., and Gilchrist, I. D. (2007). Effort during visual search and counting: insights from pupillometry. Q. J. Exp. Psychol. (Hove) 60, 211–229. doi: 10.1080/17470210600673818

Raffone, A., Srinivasan, N., and van Leeuwen, C. (2014). The interplay of attention and consciousness in visual search, attentional blink and working memory consolidation. Philos. Trans. R. Soc. Lond. B Biol. Sci. 369:20130215. doi: 10.1098/rstb.2013.0215

Shore, D. I., and Klein, R. M. (2000). On the manifestations of memory in visual search. Spat. Vis. 14, 59–75. doi: 10.1163/156856801741369

Simione, L., Raffone, A., Wolters, G., Salmas, P., Nakatani, C., Belardinelli, M. O., et al. (2012). ViSA: a neurodynamic model for visuo-spatial working memory, attentional blink and conscious access. Psychol. Rev. 119, 745–769. doi: 10.1037/a0029345

Storey, J. D. (2002). A direct approach to false discovery rates. J. R. Stat. Soc. Ser. B 64, 479–498. doi: 10.1111/1467-9868.00346

Tatler, B. W., Gilchrist, I. D., and Rusted, J. (2003). The time course of abstract visual representation. Perception 32, 579–592. doi: 10.1068/p3396

Unema, P. J. A., Pannasch, S., Joos, M., and Velichkovsky, B. M. (2005). Time course of information processing during scene perception: the relationship between saccade amplitude and fixation duration. Vis. Cogn. 12, 473–494. doi: 10.1080/13506280444000409

Wyble, B., Bowman, H., and Nieuwenstein, M. R. (2015). On the interplay between working memory consolidation and attentional selection in controlling conscious access: parallel processing at a cost-a comment on ‘The interplay of attention and consciousness in visual search, attentional blink and working memory consolidation’. Philos. Trans. R. Soc. Lond. B Biol. Sci. 370:20140197. doi: 10.1098/rstb.2014.0197

Wyble, B., Potter, M. C., Bowman, H., and Nieuwenstein, M. (2011). Attentional episodes in visual perception. J. Exp. Psychol. Gen. 140, 488–505. doi: 10.1037/a0023612

Conflict of Interest Statement: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Received: 27 October 2014; Accepted: 22 December 2014; Published online: 21 January 2015.

Citation: Meghanathan RN, van Leeuwen C and Nikolaev AR (2015) Fixation duration surpasses pupil size as a measure of memory load in free viewing. Front. Hum. Neurosci. 8:1063. doi: 10.3389/fnhum.2014.01063

This article was submitted to the journal Frontiers in Human Neuroscience.

Copyright © 2015 Meghanathan, van Leeuwen and Nikolaev. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

ORIGINAL RESEARCH ARTICLE
published: 29 September 2014
doi: 10.3389/fnhum.2014.00765

The influences of working memory representations on long-range regression in text reading: an eye-tracking study

Teppei Tanaka[1]*, Masashi Sugimoto1,[2], Yuki Tanida[1] and Satoru Saito[1]*

1Department of Cognitive Psychology in Education, Graduate School of Education, Kyoto University, Yoshida-Honmachi, Sakyo-ku, Kyoto, Japan

2The Japan Society for the Promotion of Science, Tokyo, Japan

* Correspondence: Teppei Tanaka and Satoru Saito, Department of Cognitive Psychology in Education, Graduate School of Education, Kyoto University, Yoshida-Honmachi, Sakyo-ku, Kyoto 606-8501, Japan e-mail: tp.neyu@gmail.com;
saito.satoru.2z@kyoto-u.ac.jp

Edited by:
Steve Majerus, Université de Liège, Belgium

Reviewed by:
Angela Heine, Hochschule Rhein-Waal, Germany
Alessandro Guida, University of Rennes 2, France

The present study investigated the relationship between verbal and visuospatial working memory (WM) capacity and long-range regression (i.e., word relocation) processes in reading. We analyzed eye movements during a “whodunit task”, in which readers were asked to answer a content question while original text was being presented. The eye movements were more efficient in relocating a target word when the target was at recency positions within the text than when it was at primacy positions. Furthermore, both verbal and visuospatial WM capacity partly predicted the efficiency of the initial long-range regression. The results indicate that WM representations have a strong influence at the first stage of long-range regression by driving the first saccade movement toward the correct target position, suggesting that there is a dynamic interaction between internal WM representations and external actions during text reading.

Keywords: working memory, reading, eye-tracking, working memory capacity, reading span test, spatial span test

Reading involves dynamic interactions between internal representations and external actions. For example, when a reader encounters Sherlock Holmes saying, “the murderer is the man…” while reading a novel, identification of the person as well as his previously described utterances, attributes, and actions are important contributors to the enjoyment of the detective story. The reader may remember “the man” and easily identify the attributes that are central to the plot. In this case, internal representations in working memory (WM) are indispensable to reading comprehension, as suggested by a seminal study by Daneman and Carpenter (1980), who showed that individual differences in WM capacity (WMC) predicted successful remembering of the referent. Even with high WMC, when the story and the relationships among characters are complex, it may be difficult to remember previous descriptions. In such cases, readers need to relocate previous sentences to obtain the lost information. This process, called long-range regression, usually involves an external action, e.g., eye movements, but must be driven by internal representations, which direct the reader’s attention to the location where the lost information was presented. The external actions may in turn contribute to the (re)construction of internal representations by enabling readers to find the lost information that is important for understanding the story.

The primary purpose of this study was to explore the nature of the dynamic interaction between internal representations and external actions, which underpins the long-range regression process and is an important part of natural reading. It has already been shown that readers can identify (or relocate) the location of a word that was read previously, after reading text with a high level of accuracy (e.g., Rothkopf, 1971; Baccino and Pynte, 1994). These processes should be supported by internal representations generated from previously read text. In these cases, two types of representations could potentially underlie long-range regression in text reading: verbal representations and spatial representations.

Fischer (1999) suggested that when readers could use the spatial location of words, the location information could support word localization processes. However, when readers could not use spatial information due to its short-lived nature, they used nonspatial representations, such as information about a word’s temporal order, for the localization of words.

Some characteristics of readers’ word localization (relocation) abilities were examined by Rawson and Miyake (2002). In their experiment, participants were first asked to read 12 pages of text using a virtual book; they later engaged in an unexpected relocation task in which they were explicitly required to identify the locations (pages and lines) of some words in the text. The results showed that participants’ relocation accuracy was predicted by their verbal abilities, which were measured by a set of language tasks that included the reading span task (RST). However, there was no significant correlation between the relocation performance and visuospatial abilities, which were measured by a set of visual and spatial tasks, including a spatial WM task. The authors echo Fischer (1999) in suggesting that this difference can be explained by the faster decay of visuospatial representations compared to verbal representations (their relocation task was administered on a scale of minutes, rather than of seconds).

One of the notable features of Rawson and Miyake (2002) experiment is that all sentence letters were replaced by Xs during the relocation task (at the test phase). This method might have increased verbal cognitive load; consequently, only verbal (but not visuospatial) abilities showed correlation with relocation scores. In addition, Fischer (1999) pointing task and Rawson and Miyake (2002) relocation task were both off-line tasks rather than those requiring natural regression during reading. Thus, readers might execute their actions by using their explicit and/or episodic knowledge (see also Inhoff et al., 2005; Weger and Inhoff, 2007).

On the other hand, numerous studies demonstrate the beneficial effect of spatial representation on localization in reading (e.g., Kennedy and Murray, 1987; Kennedy, 1992; Kennedy et al., 2003). Furthermore, one eye-tracking study (Weger and Inhoff, 2007) revealed that both verbal and spatial representations play an important role in regression in reading sentences. Specifically, Weger and Inhoff argued that the first pass regression, which was the first regression to a particular word, was affected by spatial representations, whereas cumulative regressions, which were complementary regressions after the first pass regression, were supported by verbal representations. In their within-line condition, a sentence was presented in two separate lines; a target word, which appeared in the sentence, was aurally presented when readers’ eyes reached a predetermined sentence location. The predetermined position was at the right side of the sentence, in the same line in which the target word was visually presented. Participants were asked to read the sentence and required to regress their eyes on the target when the target was aurally presented. Two factors were manipulated: target distance (distant/close to the right edge of the sentence line) and target position (first line/second line). The results of Weger and Inhoff showed that the first pass regression was more accurate when the target was close; however, there was no influence of target position on the first pass regression. On the cumulative regressions, however, readers regressed their eyes for the target word in much more selective ways when the target was presented in the first line (i.e., verbal load was assumed to be low) than when the target was presented in the second line (i.e., the load was assumed to be high). In contrast, their between-line condition revealed the contribution of spatial knowledge during long-range regressions. In this condition, the predetermined region, in which the target word was aurally presented when readers’ eyes reached this location, was on the front of the second line of the sentence, and the target candidates were in the middle or at the end of the first line of the sentence. Thus, a target in the middle position was spatially close to, but verbally distant from, the predetermined region (a target in the end position was distant spatially, but close verbally). The result showed that the first pass regression was more spatially selective and more accurate when the target was spatially close (verbally distant) than spatially distant (verbally close). Results from their two conditions suggest that both spatial and verbal representations contribute to long-range regressions; however, their effectiveness is different in the two regression stages.

These previous studies used unique and powerful methods to examine the interaction between internal representations and external actions during reading, and successfully revealed the nature of representations employed for external actions (i.e., eye-movements). These experimental reading settings, however, are slightly different from the natural reading situation. For example, our long-range regressions are initiated spontaneously; thus, we are not explicitly required to regress to any words in response to an auditory word. A set of sentences (i.e., text), rather than a single sentence, are presented when reading a page of a book. Because they used single sentences in their study, analyses of eye-movements in Weger and Inhoff (2007) study were restricted to horizontal distances. But our real-world reading inevitably requires eye-movements in both horizontal and vertical directions.

In this study, we examined the nature and characteristics of representations that underpin long-range regression in text reading using an eye-tracking analysis, in which eye-movement accuracy was calculated based on both horizontal and vertical distances. In addition, we tested the influences of individual differences in verbal and visuospatial WM abilities on such processes. Our experimental task was based on a text comprehension task similar to a pronoun reference task (Daneman and Carpenter, 1980). For each trial, participants read a text, and were then asked to identify a particular person from the reading. Each text was one paragraph long and was presented on a single display page. This made the task duration quite short, which protected text representations (both verbal and visuospatial) from complete forgetting (due to decay or interference). The text paragraph remained available on the display while participants were answering the questions. This mimics a natural reading situation, in which all sentences are available even when readers forget some information that was read earlier. This also made participants execute spontaneous regressions.

Three types of eye movement indices were employed here. One was the number of fixations (NF), which is frequently used in eye-movement analyses. Fixations were defined as gaze points at which the reader’s eye movements stopped for more than 100 ms. If the reader’s regression process is inefficient, a larger NF should be counted between the presentation of the question and the participant’s response. The second index was the distance to the first fixation point while answering the question from the last fixation point on the question (i.e., regression size, RS). This index was used by Weger and Inhoff (2007) and is assumed to measure the spatial selectiveness, which reflects the usage of internal spatial representation in reading. The third index was introduced to estimate the preciseness of long-range regression—the distance to the target word (DT) from the first fixation point (Figure 1). These distances were calculated from x- and y-coordinates of the target word and of the first fixation point. We also recorded reaction time (RT) between the presentation of the question and the participant’s response.

FIGURE 1. An example of eye movements and indices on the whodunit task in Japanese.

If readers execute their regression with spatial selectiveness induced by internal representations, a longer RS should be observed when the target is far from the question. If the readers know the location of the target word in the text precisely, the first saccade should directly move to the target from the question sentence. Consequently, DT should be shorter when the internal representations successfully guide the eye movements than when they do not. Our central assumptions here were that such internal (verbal and/or visuospatial) representations are supported by WM and that the efficiency of the long-range regression processes measured by the eye-tracking technique could be predicted by individuals’ WMC. Although it would be expected that WMC could predict the accuracy level of this task as well, the individual differences in accuracy would not be detected as a ceiling effect should be expected due to the presence of the target and the correct answer on the display while answering the questions.

METHODS

PARTICIPANTS

A total of 51 undergraduate and graduate students who were all native Japanese speakers from Kyoto University participated in the experiment. We recruited participants only those who had good eyesight (without eyeglasses) or who were contact lens users. Therefore, all of them could participate in this study without glasses. The test session was divided over 2 days. The eye-tracking reading experiment was conducted on the first day, and two WM tasks took place on the second day. Participants provided written informed consent before the experiment on the first day, and received 1500 yen after the 2-day session.

STIMULI AND PROCEDURE

Reading task (whodunit task)

Participants were required to read aloud a text presented on a PC screen. Although the reading aloud procedure did not provide a perfect natural reading situation, this setting had produced several benefits for our experiment. First was that we could easily monitor the participants’ text reading. Second was that the reading aloud setting inhibited readers’ articulatory rehearsal during reading phase. Moreover, this procedure could prevent readers’ unregulated regressions during the reading phase, securing eye-tracking immediately after the text reading. One text contained one paragraph consisting of a mean of 348 (SD = 2.43) Japanese kanji and kana letters and of 11.55 (SD = 1.01) sentences (Figure 2A). Most of the sentences held an SOV structure common in Japanese. Letters (Gothic font, font size 18 pt.) were presented in black color on a white screen. Sentence text (9 lines) was displayed with double spacing and there were triple line spaces above and beneath the text region.

FIGURE 2. The procedure of the whodunit task. Sa, Sb, and Sc are character names. A1 and A2 are target actions of Sa (B1, B2, and C1, C2 are target actions of Sb and Sc, respectively). After participants read the text aloud (A), the experimenter presented a question by pressing a key (B). After the participants answered the question, the experimenter presented the next question by pressing a key (C). The questions, the targets of which were randomly assigned to the top, middle, or bottom position, were presented three times in total.

Three characters with female Japanese names, i.e., Kaori, Megumi, and Yuuko, appeared in every text (trial), and their characteristics, activities, and/or thoughts were described in a story. In total, six descriptions were presented in each text (two for each character). Below, we refer to these descriptions as the target. Each text started with one or a few sentence(s) that described a situation of the story (i.e., the number of sentences was 1, 2, or 3) and that was/were followed by the six target descriptions. The first three target descriptions (one for each character) and the last three target descriptions (one for each character) consisted of three to six sentences, respectively. In the majority of stories (16 among 18 trials), one or two sentences was/were presented between the third and fourth target descriptions in order to describe a situation for the last three target descriptions. The final one or two sentence(s) finished the story. When participants finished reading a text aloud, the experimenter pressed a key to display a question under the text (Figure 2B). The question asked who was responsible for performing the act named in the target word (e.g., “Who ordered a cup of coffee?”); we called this a “whodunit question” and in this case, the target word is “coffee”. All target words were a noun. Participants were asked to answer the question by pressing one of three keys, each of which was assigned to one of the three female names. The same three female names appeared in all trials. This setting allowed participants to learn the correspondence between names and keys perfectly. Thus, they did not need to look down at the keyboard when they made their response. Participants were told to answer the question as quickly and as accurately as possible, and they were permitted to reread the original text before giving their responses, as the text was still present in front of them. Eye movements were recorded between the presentation of the question and each participant’s response. Three questions were asked in each trial. After participants answered the first question, the next question was presented immediately (Figure 2C). The three questions were derived from the top position (the first or second targets; presented on lines between 2 and 4), middle position (the third or fourth targets; presented on lines between 4 and 7), and bottom position (the fifth or sixth targets; presented on lines between 6 and 8) within a text, respectively. The presentation order of the three questions was randomized for each trial. Before and during two practice trials, participants received instructions about the experimental procedure and learned the associations between the three character names and their respective key positions. 18 test trials were divided into three blocks, and at the beginning of each block, eye movements were recalibrated. The order of the three blocks was randomized.

WM tasks

We employed the Japanese version of the RST (Osaka, 2002) for the measurement of verbal WMC, and the spatial span task (SST; Maehara and Saito, 2007) for the measurement of spatial WMC. We used RST stimuli from Osaka (2002), and SST stimuli from Maehara and Saito (2007). In both WM tasks, set size was from 2 to 5 and there were three trials in each set size. Thus, the maximum score on both the RST and SST was 42 points. The order of the 12 trials was randomized. The RST required participants to read aloud a series of unrelated sentences (presented one by one), in which a to-be-remembered word was underlined with a red line, and asked to recall the set of target words in the presented order (e.g., in set size three, participants were asked to read aloud three sentences and asked to recall three target words in total). The SST required participants to engage in a spatial processing task (same-different judgment on two meaningless shapes) and a spatial memory task (remembering the position of a dot in 4 × 4 matrices) alternatively. During the recall phase, participants were asked to recall the dot positions in the presented order by writing them down (e.g., in set size three, participants repeated the “spatial judgment—dot memory” cycle three times and were asked to recall the three dot positions by writing them down). Performance of RST and SST was evaluated by the total number of correctly recalled items, which were target words for RST and dots in matrices for SST, at the correct serial position (see Friedman and Miyake, 2005).

APPARATUS

A Tobii T120 eye-tracker was used for recording eye movement during the reading task. Participants were not fixed by chinrest or bitebar, as the Tobii T120 can capture and record eye movement in a natural reading state. The distance between participants and display was approximately 40 cm. The screen resolution was 1280 × 1024 pixels (17″). The reading task was controlled by Tobii Studio 1.7.2. Both RST and SST were administered on another PC (screen resolution was 1920 × 1200, 23″) and controlled by Super Lab., version 4.0.

RESULTS

Data from 17 females and 23 males (ages ranged from 18 to 29 years) were included in the following analyses. Data from other participants were not analyzed because of eye-tracking failure (6 participants), inadequate reading (e.g., skipped some words in text reading; 2 participants), very low average correct answer rates (not different from or lower than the chance level; 2 participants), and experimenter’s failure (1 participant).

The average score for RST was 26.65 (SD = 5.01), and that for SST was 28.83 (SD = 6.48). A significant correlation between RST and SST (r(40) = 0.64, p < 0.01) was found. The average total number of correct responses in the whodunit task was 52.65 (SD = 1.87) for 54 questions (three questions in each of 18 trials). Our participants frequently looked back to the text region after the presentation of the whodunit question (45.28 times in average, SD = 8.87) and most of their responses were correct in those cases (M = 44.33, SD = 9.13). Even when they did not look back (i.e., without regression; 8.73 times in average, SD = 8.87), our participants responded correctly in the most of the cases (M = 8.33, SD = 8.78). As predicted, the accuracy of the whodunit task was almost at a ceiling and the majority of the responses were accompanied by a long-range regression. In the present study, we analyzed RT and three types of eye-movement measures, which were recorded between the presentation of the question and the participant’s response only when participants answered correctly with regressions.

The RT, NF, RS, and DT were subjected to an analysis of variance (ANOVA) with the within-subjects variable of target position (three levels: top, middle, and bottom).[1] Descriptive statistics from these measures are presented in Table 1. A significant main effect of target position was found for all indices, RT: F(2,78) = 16.24, p < 0.01, MSE = 75467.50, η^2 = 0.29; NF: F(2,78) = 17.93, p < 0.01, MSE = 1.45, η^2 = 0.31; RS: F(2,78) = 22.39, p < 0.01, MSE = 2707.44, η^2 = 0.36; DT: F(2,78) = 59.74, p < 0.01, MSE = 2661.61, η^2 = 0.60. Subsidiary analyses (with Bonferroni corrections) indicated that the longest RT, the largest NF, and the longest RS and DT occurred when target words were at the top position. In contrast, the shortest RT, the least NF, and the shortest RS and DT occurred when target words were at the bottom position. There were significant differences between top and middle positions (p < 0.05 (RT), p < 0.05 (NF), p < 0.01 (DT)), top and bottom positions (all ps < 0.01), middle and bottom positions (all ps < 0.01). There were significant differences between top/middle positions and bottom position on the RS index (both ps < 0.01); however, there was no significant difference between top and middle positions on this index (p = 0.14, n.s.).

Table 1. Descriptive statistics of RT, NF, RS, and DT on each target position.

Table 2 shows correlations among the three eye-movement measures (i.e., NF, RS, and DT) at each target position, the two WM measures and RT. NF was not correlated with RST and SST scores at any position. However, DT for the middle-position targets showed significant negative correlations with RST and SST scores, although DT for other positions did not. A subsequent partial correlation analysis revealed that when SST performance was controlled, the partial correlation between DT for the middle position and RST scores dropped to a non-significant level, r(37) = −0.21, p = 0.18. After controlling for RST performance, the partial correlation between DT for the middle position and SST scores also decreased to a non-significant level, r(37) = −0.21, p = 0.19. As Table 2 shows, RT and NF at each target position were positively correlated. DT showed a significant correlation with NF and RS only at the bottom position but not at the other target positions. The mean text reading time was 55.01 s (SD = 61.71), and no significant correlations were found between RT and RST performance in any positions (r(40) = −0.14, p = 0.36 (top); r(40) = −0.25, p = 0.11 (middle); r(40) = −0.26, p = 0.09 (bottom)) and between RT and SST performance (r(40) = 0.17, p = 0.28 (top); r(40) = −0.01, p = 0.94 (middle); r(40) = −0.15, p = 0.34 (bottom)).

Table 2. Correlation between eye-movement measures (NF, RS and DT) at each target position and reaction time at each position and two working memory task scores.

DISCUSSION

The current study reconfirms spatial selectiveness during long-range regressions when reading text and offers two sets of novel findings. Analysis of RS shows that RS was smaller when the target word was at a bottom position than at a top or middle position. This result suggests that when readers execute long-range regression, their eyes regress spatially selectively rather than randomly landing (Weger and Inhoff, 2007). One of the novel findings is the target position effect observed in three measures (RT, NF, and DT) of our text-reading task; that is, the least efficient regression performance for the top-position targets and the better performance for the bottom-position targets. The other new finding is a specific correlational pattern between the eye-movement data and the WM scores.

The former finding indicates that it was more difficult for participants to regress accurately and efficiently when target words were at the top position than when they were at the middle or bottom position. This suggests that the availability of text/target representations decreases, either over time or due to interference from subsequent reading activities. Thus, the effect seems to reflect the characteristics of forgetting in WM (Towse et al., 1998; Maehara and Saito, 2007; Barrouillet and Camos, 2012; Oberauer et al., 2012). Although the current dataset cannot specify the mechanisms of forgetting in WM, it is certainly consistent with the idea that WM is involved in our text-reading task.

The results from the correlation analyses provide an additional clue to understanding the nature of WM representations/processes in the text-reading and long-range regressions. One of the crucial results in this study is the correlation between DT and WM scores. Both verbal and visuospatial WM performance predicted DT for the middle-position targets. Furthermore, the partial correlation analyses indicated that verbal and visuospatial WM shared variance in predicting DT at the middle position. Although DT showed this correlational pattern with WM scores, other eye-movement measures (NF and RS) were not correlated with WM measures. Note that as Table 2 shows, this NF measure was correlated with RT at all three target positions, but RS and DT were not, indicating that NF and RS/DT reflect different cognitive processes, although both are eye-movement measures.

NF reflects the visual search process while participants are looking for the target word after the first saccade, and increasing NF leads to the linear increase in RT. Because our text-reading task left all text information on the display even after the question was presented, the search process might be similar to that required in a typical visual search experiment (e.g., Treisman and Gelade, 1980). Although the efficiency of the visual search is sometimes affected by WMC in a very specific situation (e.g., Poole and Kane, 2009), it has been widely accepted that WM scores do not modulate the patterns of typical visual search processes (Kane et al., 2006). Therefore, the lack of correlation between NF (and RT) and WM scores is consistent with findings from the previous WM literature.

RS and DT do not reflect eye movements during visual search processes, but do reflect the accuracy of the first saccade before starting the visual search. On the first saccade for the long-range regression, participants cannot use the external text information because that saccade starts before they look back over the text. Instead, they have to rely on internal text representations (either rough gist or detailed representations) in guiding their first saccade eye movement. We assume that RS might reflect gist text information, which can provide rough information regarding the target position (e.g., top, middle, or bottom). The results of RS reconfirmed this spatial selectivity of long-range regression (Weger and Inhoff, 2007). On the other hand, DT might reflect the precise information regarding the target location, which could be the internal detailed representations (verbal, visuospatial, or both) that are supported by WM. Thus, the availability of such detailed representations may reflect individual differences in WMC, predicting the correlation between DT and WM scores. It is assumed that when readers could not use detailed representations for initial regression, they might use a rough gist. We observed no correlations between RS and WM scores. This result is consistent with the assumption that RS might reflect gist text information but not reflect individual differences in WMC. One may note that there is a significant correlation between RS and DT at the bottom position and that this correlation seems to be strange if the two indices are reflecting different aspects of internal representations. Although the most of RSs and DTs for targets at the bottom positions were very short, some of the first saccades for those targets occasionally went beyond the bottom and arrived at the middle or at the top positions erroneously. In these cases, RS and DT were both very long, leading to the strong positive correlation between the two indices.

In summary, long-range regression entails at least two steps with differential scaffolds. The first step, in which readers could not use external text information, might be guided by readers’ internal representations. In this step, when readers have detailed representations (e.g., detailed word position information) they could regress accurately near the target (reflected in DT). Even when they have only rough gist representations, however, they could regress roughly around the target (reflected in RS). The second step is underpinned by external text information, and its operation is measured by NF and RT. The idea that WMC seems to affect only the first process by holding the internal text representations can generally explain the presence, and the absence, of correlations between WMC and other measures. Here, two unresolved questions remain. First, why did we find the correlation between DT and WM scores only for the middle target position? Second, what is the nature of the internal representations (or mechanisms) that mediate correlations among DT, verbal WMC, and visuospatial WMC?

Why should we find a correlation between DT and WM scores only for the middle target position but not for the top and the bottom target positions? One approach to this issue can be provided by the notion of the capacity limitation of WM. Cowan (2001, 2010) indicated that our ability to hold bound objects in WM is limited to a certain number, i.e., four. The present text-reading task had six target actions. As we discussed later, participants would be more likely to bind the verbal and spatial information of the targets during text reading (a recent estimation of the capacity limit of verbal-spatial bound objects is said to be three; Langerock et al., 2014). If we accept that the bound representations for both verbal and spatial aspects operate as bound objects, then we can assume that only the last three or four target actions (one or two middle positions and two bottom positions) were within WM. The accessibility of these four targets was higher than that of the first two targets (i.e., those in the top position), which had already gone to long-term memory. Within WM, due to decay or representational interference, the targets in the middle positions, at the boundary of the capacity limit, may have had degraded representational quality that was different from that of the targets in the bottom positions. It is likely that the degree of representational degradation would show large inter-individual variations based on WM functioning. Therefore, it is expected that the individual differences in WM scores should have an impact on relocation processes measured by DT predominantly at the middle target positions. There might be another possible explanation of the significant correlations at the middle positions. The majority of our text stories used in this study consisted of two situations of which the boundaries located between the third and fourth target descriptions. As a shift of a mental model from one to another could potentially require additional resources of WM, it might be the case that, due to the WM load, the targets in the middle positions might have had degraded representations compared to that of the targets at the top and bottom positions. This view also predicts that the individual differences in WM scores should have an impact on relocation processes at the middle target positions. Although these two views provide speculative explanations, both expect that the individual differences in WM should exhibit its predictive power at the middle positions.

Then, what is the nature of the internal representations (or mechanisms) that mediate correlations among DT, verbal WMC, and visuospatial WMC? The correlational analyses suggest that the internal representations and the representational mechanisms that can guide the first regression may be domain-general. Some shared variances between verbal and visuospatial WM are associated with individual differences in DT. To accomplish the relocation smoothly, participants should return their eyes to a location near the target word. For this action, not only verbal information from the context, but also spatial information—such as the estimated position of the target—would be important scaffolds. In this situation, there were at least two possible ways that verbal and visuospatial WM could have had an impact on relocation processes. The first is that two WM domains may affect relocation independently. Another is that a common domain-general component of WM (i.e., executive attention; see also Kane et al., 2006) underpins long-range regression processes. The partial correlation analysis suggests that the latter assumption seems to be the correct one. A study by Cowan et al. (2006) that examined verbal-spatial associations may support this assumption. In their study, participants were presented with a series of names sequentially, one at a time, each located in one of a group of schematic houses at different locations. Participants were then required to remember the names and their locations (houses). Adults could perform this task efficiently by remembering the sequence of the names and that of the locations separately and then combining these modality-dependent memories in a recall phase. Third-grade children, however, did not employ this efficient strategy. Instead, they seemed to combine the names and the houses at an encoding phase and try to maintain these bound representations throughout the experiment. The adult participants did not use this binding strategy because it requires high attentional demands. However, the results showed that the adult participants were forced to use the binding strategy when they performed the task under articulatory suppression, which is assumed to prevent participants from using articulatory rehearsal for the retention of verbal materials (e.g., a sequence of the names).

In our reading task, both verbal and visuospatial aspects of the task-related information were important to accomplish the task. Moreover, the task required participants’ oral reading, which could potentially prevent them from employing articulatory rehearsal during performance of the whodunit task. Thus, it is possible that our participants may have tried to use the binding strategy, attempting to bind the potential target words and their locations while reading the text. Although the precise mechanisms of this binding function have not been specified, this function likely requires domain-general resources involving attentional control mechanisms for combining two different modality-dependent sets of representations in WM. It is assumed that this type of bound representation of verbal and visuospatial information might be used in guiding the first regression movement from the whodunit question. Therefore, the quality or availability of the bound representations could potentially have an influence on the efficiency of the relocation processes, reflected in DT; consequently, individual differences in the quality of the bound representations might affect the variability of DT.

We would like to note that the bound representations might be not solely based on presented information, particularly in the case of text reading. Recent studies (van Dijck and Fias, 2011; van Dijck et al., 2013) reported that each word in a temporal sequence of words have a respective spatial value (e.g., the first one has “left” and the last one has “right”) in WM and the words in the middle of the sequence have less spatial value. This finding indicates that words presented in a text receive spatial information not only from presented positions but also from presented order. The bound representations might be influenced by the position and order information. If this is the case, the bound representations for targets at the middle position would be more fragile due to the weaker spatial value, thus strongly exhibit individual differences of WM as reported in this paper. However, it is not clear whether spatial information from presented order includes the top—bottom contrast. This issue must be examined in future studies.

In the present study, we found a dynamic interaction between internal WM representations and external actions during text reading. Specifically, such an interaction occurs in long-range regression processes, demonstrating that internal WM representations have a strong influence at the first stage of the relocation process by driving the first regression movement toward the correct target position. The influence of WMC seems to operate through the domain-general WM system, perhaps through attentional control mechanisms for binding verbal and spatial information. By using eye-movement analyses and the whodunit task, we were able to build upon developments in previous work to produce a more natural reading situation. The micro activities, such as eye movements, that are usually difficult to capture were clearly affected by individual differences in WMC, and the effects are elucidated by the eye-tracking technique.

ACKNOWLEDGMENTS

Teppei Tanaka, Masashi Sugimoto, and Yuki Tanida were supported by a Research Fellowship of the Japan Society for the Promotion of Science (JSPS) for Young Scientists, and Satoru Saito by a Grant-in-Aid for Scientific Research (#23240036) from JSPS. We would like to thank Kohei Tsunemi who provided thoughtful feedback and valuable comments. We would also like to thank Ami Shinoda, Hanna Yasuda and Aya Kuzuguchi who helped us in material preparation and in data collection.

FOOTNOTES

1^Kurtosis and skewness of four indices in each positions were between −1 and 1, except RT in top position (kurtosis was 3.93, skewness was 1.17) and NF in middle position (kurtosis was 2.11, skewness was 1.15). We conducted ANOVAs on log transformed RTs and square-root transformed NFs in all positions. The same patterns (including subsidiary analyses) as ANOVAs without transformations were obtained. We reported analyses of the non-transformed data for all four indices.

REFERENCES

Baccino, T., and Pynte, J. (1994). Spatial coding and discourse models during text reading. Lang. Cogn. Process. 9, 143–155. doi: 10.1080/01690969408402114

Barrouillet, P., and Camos, V. (2012). As time goes by: temporal constraints in working memory. Curr. Dir. Psychol. Sci. 21, 413–419. doi: 10.1177/0963721412459513

Cowan, N. (2001). The magical number 4 in short-term memory: a reconsideration of mental storage capacity. Behav. Brain Sci. 24, 87–114. doi: 10.1017/s0140525x01003922

Cowan, N. (2010). The magical mystery four: how is working memory capacity limited and why? Curr. Dir. Psychol. Sci. 19, 51–57. doi: 10.1177/0963721409359277

Cowan, N., Saults, J. S., and Morey, C. C. (2006). Development of working memory for verbal-spatial associations. J. Mem. Lang. 55, 274–289. doi: 10.1016/j.jml.2006.04.002

Daneman, M., and Carpenter, P. A. (1980). Individual differences in working memory and reading. J. Verbal Learning Verbal Behav. 19, 450–466. doi: 10.1016/S0022-5371(80)90312-6

Fischer, M. (1999). Memory for word locations in reading. Memory 7, 79–116. doi: 10.1080/741943718

Friedman, N. P., and Miyake, A. (2005). Comparison of four scoring methods for the reading span test. Behav. Res. Methods 37, 581–590. doi: 10.3758/bf03192728

Inhoff, A. W., Weger, U. W., and Radach, R. (2005). “Sources of information for the programming of short- and long-range regressions during readin,” in Cognitive Processes in Eye Guidance, ed G. Underwood (New York: University press), 33–52.

Kane, M. J., Poole, B. J., Tuholski, S. W., and Engle, R. W. (2006). Working memory capacity and the top-down control of visual search: exploring the boundaries of “executive attention”. J. Exp. Psychol. Learn. Mem. Cogn. 32, 749–777. doi: 10.1037/0278-7393.32.4.749

Kennedy, A. (1992). “The spatial coding hypothesis,” in Eye Movements and Visual Cognition: Scene Perception and Reading, ed K. Rayner (New York: Springer), 379–396.

Kennedy, A., Brooks, R., Flynn, L.-A., and Prophet, C. (2003). “The reader’s spatial code,” in The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, eds J. Hyona, R. Radach and H. Deubel (Amsterdam: Elsevier, North-Holland), 193–212.

Kennedy, A., and Murray, W. S. (1987). Spatial coordinates and reading: comments on Monk (1985). Q. J. Exp. Psychol. 39A, 649–656. doi: 10.1080/14640748708401807

Langerock, N., Vergauwe, E., and Barrouillet, P. (2014). The maintenance of cross-domain associations in the episodic buffer. J. Exp. Psychol. Learn. Mem. Cogn. 40, 1096–1109. doi: 10.1037/a0035783

Maehara, Y., and Saito, S. (2007). The relationship between processing and storage in working memory span: not two sides of the same coin. J. Mem. Lang. 56, 212–228. doi: 10.1016/j.jml.2006.07.009

Oberauer, K., Farrell, S., Jarrold, C., Pasiecznik, K., and Greaves, M. (2012). Interference between maintenance and processing in working memory: the effect of item-distractor similarity in complex span. J. Exp. Psychol. Learn. Mem. Cogn. 38, 665–685. doi: 10.1037/a0026337

Osaka, M. (2002). Working Memory: The sketchpad in the Brain. Tokyo: Shinyosha.

Poole, B. J., and Kane, M. J. (2009). Working memory capacity predicts the executive control of visual search among distractors: the influence of sustained and selective attention. Q. J. Exp. Psychol. (Hove) 62, 1430–1454. doi: 10.1080/17470210802479329

Rawson, K. A., and Miyake, A. (2002). Does relocating information in text depend upon verbal or visuospatial abilities? An individual differences analysis. Psychon. Bull. Rev. 9, 801–806. doi: 10.3758/bf03196338

Rothkopf, E. Z. (1971). Incidental memory for location of information in text. J. Verbal Learning Verbal Behav. 10, 608–613. doi: 10.1016/s0022-5371(71)80066-x

Towse, J. N., Hitch, G. J., and Hutton, U. (1998). A re-evaluation of working memory capacity in children. J. Mem. Lang. 39, 195–217. doi: 10.1006/jmla.1998.2574

Treisman, A. M., and Gelade, G. (1980). A feature-integration theory of attention. Cogn. Psychol. 12, 97–136. doi: 10.1016/0010-0285(80)90005-5

van Dijck, J.-P., Abrahamse, E. L., Majerus, S., and Fias, W. (2013). Spatial attention interacts with serial-order retrieval from verbal working memory. Psychol. Sci. 24, 1854–1859. doi: 10.1177/0956797613479610

van Dijck, J.-P., and Fias, W. (2011). A working memory account for spatial-numerical associations. Cognition 119, 114–119. doi: 10.1016/j.cognition.2010.12.013

Weger, U. W., and Inhoff, A. W. (2007). Long-range regressions to previously read words are guided by spatial and verbal memory. Mem. Cognit. 35, 1293–1306. doi: 10.3758/bf03193602

Conflict of Interest Statement: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Received: 30 June 2014; Accepted: 10 September 2014; Published online: 29 September 2014.

Citation: Tanaka T, Sugimoto M, Tanida Y and Saito S (2014) The influences of working memory representations on long-range regression in text reading: an eye-tracking study. Front. Hum. Neurosci. 8:765. doi: 10.3389/fnhum.2014.00765

This article was submitted to the journal Frontiers in Human Neuroscience.

Copyright © 2014 Tanaka, Sugimoto, Tanida and Saito. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.