Cerebral Cortex Advance Access originally published online on December 5, 2006
Cerebral Cortex 2007 17(9):2151-2162; doi:10.1093/cercor/bhl123
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Nonvisual Codes and Nonvisual Brain Areas Support Visual Working Memory
1 Department of Psychology, 2 Neuroscience Training Program and Medical Scientist Training Program, University of Wisconsin-Madison, Madison, WI 53706, USA
Address correspondence to Bradley R. Postle, 1202 West Johnson Street, Madison, WI 53706, USA. Email: postle{at}wisc.edu.
| Abstract |
|---|
|
|
|---|
Systems models hold working memory to depend on specialized, domain-specific storage buffers. Here, however, we demonstrate that short-term retention of the identity or location of visually presented stimuli is disrupted by nonvisual secondary tasks performed in the dark—passive listening to nouns or endogenous generation of saccades, respectively. This indicates that the short-term retention of visual information relies on multiple mental codes, some of them nonvisual. Event-related functional magnetic resonance imaging (fMRI) reveals the neural correlates of these interference effects to be more complex and more regionally specific than previously described. Although nonspecific dual-task effects produce a generalized decrease of task-evoked fMRI response across many brain regions, the interference-specific effect is a relative increase of activity localized to regions associated with the secondary task in question: left hemisphere perisylvian cortex in the case of passive listening distraction and frontal oculomotor regions in the case of saccadic distraction. Within these regions, the neural interference effects are specific to voxels that show delay-period activity on unfilled memory trials. They also predict individual differences in the magnitude of the behavioral interference effect. These results indicate that nonvisual processes supported by nonvisual brain areas contribute importantly to "visual" working memory performance.
Key Words: FEF fMRI frontal eye field object spatial sylvian
| Introduction |
|---|
|
|
|---|
Working memory refers to the ability to retain information in an active state when it is not present in the environment, to transform it when necessary, and to use it to guide behavior. The multicomponent model of working memory, first proposed by Baddeley and Hitch (1974)
In the majority of selective interference research to date, secondary tasks and stimuli have been selected precisely because they featured the same domain and modality, and thus were expected to recruit the same mental processes, as did the primary memory tasks. (Two notable exceptions are nonvisual spatial secondary tasks that have demonstrated an important spatial component of visuospatial working memory [reviewed in Postle et al. 2006
] and phonological secondary tasks that demonstrate fact that visually presented digits, letters, and words are recoded into phonologically based representations [reviewed in Repovs and Baddeley 2006
].) In the present study, in contrast, we evaluated the effects of secondary distracting tasks that did not entail any visual stimulation and that did not overlap with the domains from which the memoranda were drawn. These secondary tasks, therefore, presented no obvious source of interference. Our study was motivated by the principle of multiple encoding (Wickens 1973
; Postle 2006
), which posits that humans will recode information and represent it in parallel in as many mental codes as are afforded by the stimulus. This principle accounts for the fact that the representational bases of working memory are not limited to the domain in which stimuli are perceived (Postle 2006
). The behavioral and neural correlates of 2 examples of this principle were tested in the experiments presented here.
One example that we studied was spatial working memory. It is well-established that one way that we represent visually perceived locations is with attention-based rehearsal, the covert allocation of attention to the location at which the stimulus appeared (Awh and Jonides 2001
). A second mechanism for the short-term retention of a location, however, is to prepare a motor plan for the acquisition of the target (e.g., with a saccade or a grasp) and to retain this motor plan as a "prospective memory" (e.g., Fuster 1995
; Postle and D'Esposito 2003
; Curtis et al. 2004
). Thus, the present experiments were designed, in part, to replicate the finding that spatial delayed recognition would be selectively sensitive to concurrent performance of endogenously guided saccades (Postle et al. 2006
).
The second example of multiple encoding that we studied here was working memory for visually presented objects, for which the well-established representational codes are those used by the perceptual apparatus of the ventral visual stream (e.g., Tresch et al. 1993
; Smith et al. 1995
; Della Sala et al. 1999
; Ranganath 2006
). In addition to visual codes, however, we have postulated that the short-term retention of object information also recruits verbal (and, perhaps, semantic) representations (Postle et al. 2005
). Specifically, because object recognition entails the association of visually perceived information with preexisting knowledge, we believe that the short-term retention of information about a visually presented object entails the retention of this semantic information, in addition to the retention of visual information. (We have hypothesized that this multiple encoding occurs automatically and obligatorily [Postle et al. 2005
], although we do not test this particular claim here). As a result, we predicted that object delayed recognition in the present study would be selectively sensitive to concurrent passive listening to concrete nouns. This predicted outcome would represent a replication and extension of previous results, in that a previous demonstration of selective disruption of object delayed recognition performance with a verbal secondary task used visually presented words for the verbal secondary task (Postle et al. 2006
). This procedure left open the possibility that the interference was visual, not verbal. For the present study, we did not have strong predictions, a priori, about whether the object–verbal interference would occur at the level of phonological representations (i.e., the "sounds" of the secondary words interfering with phonological representations of the object memoranda) or at the level of semantic representations (i.e., the "ideas" evoked by the secondary words interfering with semantic representations of the object memoranda). It would turn out, however, that the fMRI results suggested a post hoc account of the locus of the interference.
For both of these domains of memory, we used fMRI to examine the neural bases of secondary task interference produced by secondary tasks employing nonoverlapping modalities and drawing on nonoverlapping domains.
Overview of the Design
The logic of the study was to test the prediction that the data from 4 cells of a 2 (memory domain: location, object) x 2 (distraction: saccades, passive listening) fully crossed design would produce a crossover interaction. Such a "double dissociation" would permit strong inference that the predicted disruptive effects of saccades on location memory and of listening to nouns on object memory were selective. That is, it would rule out concerns that either of these instances of interference might be a nonspecific effect due merely to dividing attention between 2 concurrent tasks. This critical 2 x 2 arrangement was embedded in an overall design that had 3 levels per factor, because the factor of memory domain also had a "no memory" level, and the factor of distraction also had a "no-distraction" level. This was important for the fMRI study in particular because our underlying hypothesis for spatial working memory, for example, was that oculomotor codes contribute even to the simple, undistracted short-term retention of location information. Additionally, although the 2 x 2 contrast was necessary for demonstrating the specificity of the predicted effects, an interference effects itself must be defined and measured in relation to a no-distraction baseline. Thus, we used a multistep procedure, the essence of which was to first identify voxels that were active during the delay period of no-distraction trials, then to evaluate how the activity in these same "memory-delay" voxels varied as a function of concurrent distraction.
| Materials and Methods |
|---|
|
|
|---|
Subjects
Twenty-two healthy young adults (3 males, mean age = 22.6, standard deviation [SD] = 3.5) participated in a preliminary behavioral study, and 12 healthy young adults (5 males, mean age = 22.8, SD = 4.2) participated in the fMRI study. In both cases, subjects gave informed consent and reported neither history of neurological or psychiatric conditions nor recent use of psychoactive drugs. The methods described here were approved by the relevant Institutional Review Boards of the University of Wisconsin-Madison.
Apparatuses
A PC running Eprime software (Psychology Software Tools, Pittsburgh, PA) was used for all stimulus presentation and response collection. For the preliminary behavioral study, testing took place in a sound-attenuated booth in which all sources of illumination, except the monitor screen on which stimuli were presented, were occluded. The "brightness" setting on the monitor (Mitsubishi Diamond Plus 200) was set to 0 in order to prevent the gray–black glow that typically emanates from a computer monitor that is displaying a uniformly blank screen. The effect for the subject was to be sitting in a room in which nothing was visible except for the periodic appearance of instructions and stimuli. The subject sat with chin and forehead resting in a chin rest that was equipped with optics for infrared-based eye tracking (ASL Model 501, Bedford, MA). The chin rest was oriented such that, when positioned in it, the subject's natural "straight ahead" gaze would fall on the center of the display screen, with the eye approximately 63.5 cm from the screen. During the fMRI session, subjects viewed stimuli through eyepieces that displayed a rectangular screen subtending approximately 30 horizontal by 23 vertical degrees of visual angle (Avotec Silent Vision, Stuart, FL) and fitted with infrared-based eye-tracking cameras (SMI iView X, Teltow, Germany). Auditory cues were presented via nonpneumatic magnetic resonance–compatible headphones with 30 dB external noise attenuation (Resonance Technology Commander XG, Northridge, CA). Responses were recorded via a fiber optic button box connected to the PC running Eprime.
Behavioral Methods
Trials were drawn from each cell of the fully crossed 3 (memory domain: location, object, no memory) x 3 (secondary task: endogenously guided saccades, passive listening to concrete nouns, none) design, with the exception of the "no memory/no secondary task" cell. Each trial began with a 2-s presentation of instructions, indicating the memory domain. This was followed by a 2-s blank period, after which the target stimulus was presented for 1 s. Following this was a delay period, during which one of the 3 types of secondary tasks occurred. This was followed by a 1-s presentation of a memory probe stimulus, to which the subject was instructed to make a match/nonmatch response (right thumb/left thumb, respectively). For the preliminary behavioral study, the delay period lasted 11 s, 500 ms into which a luminance mask was flashed for a duration of 500 ms. Three seconds after the mask offset the secondary task period began, lasting until 500 ms prior to the onset of the memory probe. The procedure for the fMRI study was similar, except there was no mask, and the delay period was 7 s long. The intertrial interval (ITI) was 3 s for the preliminary behavioral study and 10 s for the fMRI study. With the exception of the saccadic secondary task and the ITI, subjects were trained to always keep their eyes in their natural straight ahead position whenever a stimulus was not present on the screen. (Although this corresponded to central fixation, there were never fixation cues, and care was taken not to suggest that subjects imagine a fixation cue).
The preliminary behavioral experiment was administered in 4 24-trial blocks. The fMRI experiment was administered in 8 16-trial blocks, with 2 trials of each type occurring in a randomly determined order in each block.
Memory Tasks
The 64 memory stimuli were organized around 16 abstract shapes (Attneave and Arnoult 1956
) normatively determined to be difficult to verbalize (Vanderplas and Garvin 1959
), subtending approximately 2 degrees of visual angle in their largest dimension. Each of these served as a prototype for 3 variants that each differed from the prototype by one salient feature (Postle and D'Esposito 1999
). On each trial, the target was pulled at random (without replacement) from the pool of 64. On "location memory" trials, the target position for each trial was determined pseudorandomly with 2 constraints: 1) across an experimental session, an equal number of location targets appeared in each quadrant of the screen and 2) no 2 targets were centered on precisely the same coordinates. Location probes were always drawn at random from one of the other 15 prototype groups, and the central coordinate of nonmatching probe locations was an average of 4 degrees of visual angle (SD = 1 degree, min = 2 degrees, max = 10.7 degrees) distant from the central coordinate of the corresponding target positions. That is, the average distance between the closest edges of a target and a nonmatching trial-final probe was roughly equivalent to the diameter of a stimulus. Instructions and training emphasized that a location-match probe stimulus would always have precisely the same center of mass as the corresponding target, whereas a location nonmatch would not overlap any area occupied by the target stimulus. On "object memory" trials, the stimuli always appeared in the center of the display, and nonmatching probes were always drawn from the same prototype group as the target. On location and object trials, the probe matched the target with P = 0.5. The stimulus on "no memory" trials was a white circle of 2 degrees of visual angle that was presented at the center of the screen as both target and probe. On these trials, subjects were instructed to respond with either thumb.
Secondary Tasks
Secondary task cues and stimuli were delivered via a loudspeaker positioned directly below the computer monitor for the preliminary behavioral study and via the headphones for the fMRI study. The onset and offset of saccadic secondary task periods were cued by a tone of 500 ms duration, the onset of which lagged the offset of the target by 500 ms and the offset of which preceded the onset of the probe by 500 ms. Subjects had been instructed that, upon hearing the first tone, they were to move their eyes in any manner that they chose until the second tone signaled the end of this secondary task period. The intent of this procedure was to minimize the likelihood that nonmotoric factors could explain the selective disruption that saccadic distraction was expected to produce. Because participants were seated in absolute darkness during the distraction portion of the delay period, and because the source of the auditory "start" and "stop" signals was 0 degree with respect to trunk and head position, there were no possible sources of perceptual distraction. Saccades were endogenously generated, and participants had never been trained to guide them with, for example, a visual array of targets. Therefore, the likelihood that participants guided their saccades with internally generated mental images was minimized to the greatest extent possible.
The "passive listening" secondary task began 500 ms into the delay, consisted of the serial presentation of 6 concrete nouns, one per second, spoken by a male voice, and ended 500 ms prior to the onset of the memory probe. Subjects were told that they did not have to "do anything" with the stimuli, but simply to maintain straight ahead eye position throughout the delay period.
No secondary task trials amounted to conventional delayed recognition with unfilled delay periods.
Tracking Eye Position
Great care was taken to rule out the possibility that endogenously guided eye movements made in the dark may have been targeting specific locations in space, as well as to ensure that fixation was maintained during all other trial types. Following the experiment, eye-tracking data were aligned to each block's starting time and segmented into the target, delay, probe, and ITI periods. Eye position during the delay period was analyzed with ILAB software (Gitelman 2002
) to evaluate compliance with the instructions for each type of secondary task. Although there was no fixation cross for the passive listening and "no secondary task" conditions, subjects were instructed to maintain straight ahead eye position during these delay periods. On these trials, eye position 500 ms after the offset of the target was designated to be the initial center of gaze for that trial's delay period, and trials for which delay-period center of gaze moved more than 2 degrees of visual angle from this initial center of gaze were designated as "fixation failure" trials. (How this influenced analyses is described below, in fMRI Methods, Data Analyses). "Saccade" trials, in contrast, were scored as "unacceptable" if delay-period eye position failed to cover each of the quadrants of the screen, and/or center of gaze was stationary for greater than 50% of the 6-s secondary task period. Additionally, eye movement patterns were analyzed to ensure that subjects did not make saccades in an orderly pattern, such as by tracing out a shape, or repeatedly moving to a specific location. This ensured that subjects were not using eye movements to represent properties of the target.
fMRI Acquisition and Preprocessing
Whole-brain images were acquired with a 3-T scanner (GE Signa VH/I). For all volunteers, we acquired high-resolution T1-weighted images (30 axial slices, 0.9375 x 0.9375 x 4 mm). We used a gradient-echo, echo-planar sequence (time repetition = 2000 ms, echo time = 50 ms) to acquire data sensitive to the blood oxygen level–dependent (BOLD) signal (Kwong et al. 1992
; Ogawa et al. 1992
) within a 64 x 64 matrix (30 axial slices coplanar with the T1 acquisition, 3.75 x 3.75 x 4 mm). Eight scans of the experimental task were obtained for each subject, each lasting 7 min 16 s (6 min 56 s of task preceded by 20 s of dummy pulses to achieve a steady state of tissue magnetization). Prior to scans of the experimental tasks, for each participant we obtained a scan from which we derived an estimate of the hemodynamic response function (HRF). During this scan, each participant performed a simple reaction-time task that required a bimanual button press once every 20 s in response to a brief change in shape of the fixation stimulus.
Raw BOLD image files went through the following preprocessing steps, in order: reconstruction (in-house software incorporating GE's EPIrecon code); sync interpolation in time to correct for the slice acquisition sequence (VoxBo—www.voxbo.org); correction for magnetic field inhomogeneities (in-house software); and rigid-body realignment to the first volume from the experimental task (VoxBo). Note that neither was spatial smoothing imposed nor were the data spatially transformed into a common atlas space prior to hypothesis testing. Rather, the data from each subject were analyzed in that subject's unsmoothed, native space (Postle, Zarahn et al. 2000
; Postle et al. 2003
). (Warping of data to Montreal Neurological Institute [MNI] space for display [Fig. 2] and tabulation [Table 1] of group-averaged results was done after the fact, for descriptive purposes only.) For the HRF-derivation run, a partial F-test associated with a Fourier basis covariate set was used to evaluate the significance of task-correlated activity in each voxel of the primary somatosensory and motor cortical regions of interest (ROIs). We obtained an HRF estimate by spatially averaging the time series of the suprathreshold voxels of these ROIs, filtering the resultant averaged fMRI time series to remove high (>0.244 Hz) and low (<0.05 Hz) frequencies, adjusting it to remove the effects of nuisance covariates (Friston et al. 1995
) and trial averaging. The HRF describes the fMRI response resulting from a brief impulse of neural activity (Boynton et al. 1996
) and can vary markedly across participants (Aguirre et al. 1998
; Handwerker et al. 2004
). Each subject's HRF was used to convolve independent variables entered into the modified general linear model (GLM) (Worsley and Friston 1995
) that we used to analyze the data from the scans of the experimental task.
|
Data Analyses
The analyses derived from GLMs that modeled task-related variance in the fMRI time series data with covariates consisting of delta functions positioned along the time line of the task to represent trial epochs of interest—instructions (time 0 s), target presentation (time 4 s), delay (time 8 s), and probe (time 12 s)—and convolved with the empirically derived HRF (Zarahn et al. 1997
; Postle, Zarahn et al. 2000
). Parameter estimates associated with each covariate of interest were extracted from the least-squares solution of the GLM of the fMRI time series. This approach yielded estimates of delay-period activity that were not contaminated by variance attributable to earlier portions of the trial (Zarahn et al. 1997
; Postle, Zarahn et al. 2000
). One GLM, the "full" GLM, modeled the data from all trials in this manner, and a second, the "eye position–corrected" GLM, excluded trials that were flagged by the eye position analysis.
Memory-Delay Analysis
This addressed the question how is memory-delay activity (i.e., delay-period activity during no secondary task trials) affected by concurrent distraction? It used the logic of a "functionally constrained search within a factorial design" (Friston et al. 2006
) that proceeded in 3 steps. First, whole-brain maps were generated from the full GLM for each single-subject data set with the contrasts (Delaylocation/no secondary task – baseline) and (Delayobject/no secondary task – baseline) and thresholded at the mapwise Bonferroni-corrected
of P < 0.05. (The full GLM was used for these contrasts to maximize sensitivity to identify critical voxels because it modeled all the trials in the condition of interest. This was important to do because each subject only performed 16 trials of each of these types, a relatively small number from which to generate a statistical map in an event-related analysis. The use of the full, rather than the eye position–corrected, GLM did not risk contaminating our results, because it was only used to identify voxels in which our hypotheses would be tested, but not to generate the quantitative estimates that were the bases of the hypothesis tests.) Second, maps were compared across subjects to identify anatomical regions in common (across subjects) that contained memory-delay activity (see Identifying Anatomical Regions, below). Regions identified in the majority (i.e., 5 or more) of the subjects were selected for subsequent analysis, and data from all suprathreshold voxels within each region were pooled to create functional ROIs. For single-subject data sets for which no suprathreshold voxels were identified in a selected region, the threshold was dropped to the regionwise Bonferroni-corrected
of P < 0.05 (for each [Delaydomain/none – baseline] contrast, there were a few single-subject data sets that did not show suprathreshold voxels at the regionwise level for one or more regions. This is not surprising when one considers that each subject performed only 16 trials of any one type. For the [Delaylocation/none – baseline] contrasts, there were 2 single-subject data sets that did not show suprathreshold voxels at the regionwise level for either right or left frontal eye fields [FEF], and so analyses for these ROIs only included 7 subjects. For the [Delayobject/none – baseline] contrast, there were 2 single-subject data sets that did not show suprathreshold voxels at the regionwise level for the left sylvian and 4 for the left superior temporal gyrus [STG] ROIs. Thus, to be able to include observations from all subjects for these 2 ROIs, the threshold for them was dropped to a critical t of 0 [one-tailed]). Third, from these memory-delay voxels the magnitudes of the delay-evoked responses across the 3 levels of secondary task were estimated with the contrasts (Delaydomain/no secondary task – baseline), (Delaydomain/saccades – baseline), and (Delaydomain/passive listening – baseline). These estimates were obtained with the eye position–corrected GLM, to ensure that quantitative estimates of delay-period activity were not contaminated by data from trials on which subjects did not comply with instructions.
Secondary Task Analysis
This addressed the question how are "secondary task–sensitive" voxels (i.e., voxels showing delay-period activity during no memory trials) affected by the demands of a concurrent memory task? It was performed following the same logic as the first analysis, with the exception that the first step used the contrasts (Delayno memory/saccades – baseline) and (Delayno memory/passive listening – baseline) to identify critical voxels.
Identifying Anatomical Regions
Anatomical regions were defined to encompass regions that are known to support different functions. In some cases, these spanned multiple Brodmann areas (BAs) if the functional distinctions between them was judged to be ill defined (as for, e.g., BAs 9 and 46 of middle frontal gyrus [MFG]) or too fine grained to be relevant for the purposes of our study (as for, e.g., BAs 18, 19, and 37 for inferior occipitotemporal cortex [IOTC]). Defining functional ROIs in this way allowed us to aggregate effects from functionally comparable regions across subjects and to perform inferential statistical tests at the group level. Anatomical regions were defined on the native-space anatomical scans of each subject in the following way: MFG, bounded by the inferior frontal sulcus, superior frontal sulcus (SFS), and precentral sulcus (PCS); FEF, beginning from the intersection of the PCS and SFS, the anterior bank of the PCS extending 6 mm lateral to this intersection, and the SFS extending 6 mm anterior to this intersection; supplementary eye fields (SEF), in the medial wall in the upper part of the PCS; STG, bounded by the sylvian fissure and superior temporal sulcus; sylvian, the cortex of the sylvian fissure extending from the frontal operculum to ventral supramarginal gyrus; IOTC, comprising the lingual, fusiform, and inferior temporal gyri and extending to the caudal terminus of the transverse occipital sulcus. Inferior parietal lobule (IPL), inferior and lateral to the intraparietal sulcus (IPS) and superior to the lateral occipital sulcus and sylvian fissure; IPS, the cortex in the banks of the IPS; superior parietal lobule (SPL), dorsal and medial to the IPS, posterior to the postcentral sulcus, and superior to the parietooccipital fissure; V1, the cortex in the banks of the calcarine sulcus; extrastriate cortex, corresponding to BAs 18 and 19 of the occipital lobe. ROIs that spanned the midline (SEF and SPL) were not divided by hemisphere.
Conjunction Analysis
This was performed to determine whether there may be additional areas, not engaged by single-task performance, but recruited in response to interfering secondary task performance. Such activity would be characteristic of some type of executive control, such as a "central executive" (D'Esposito et al. 1995
) or an inhibitory control process (Hester et al. 2004
), being recruited to manage the coordination of the primary and interfering secondary tasks. The conjunction analysis was designed to identify evidence for domain-general control, that is, voxels recruited by the concurrent performance of location memory and endogenously guided saccades that was dissociable from the nonspecific secondary task effect of concurrent location memory and passive listening, as well as by the concurrent performance of object memory with passive listening that was dissociable from the nonspecific secondary task effect of concurrent object memory with saccades. The analysis was performed as a spatial normalization-based group analysis. First, for each single-subject data set, the contrasts that would isolate the hypothesized "control" voxels were computed: For control of location memory, ([Delaylocation memory/saccades – Delayno memory/saccades] – [Delaylocation memory/passive listening – Delayno memory/passive listening]) and for control of object memory, ([Delayobject memory/passive listening – Delayno memory/passive listening] – [Delayobject memory/saccades – Delayno memory/saccades]). Second, the resultant unthresholded volumes of parameter estimates were warped to a template in MNI space and smoothed to a full-width half-maximum of 8 mm. Third, 2 group-average statistical maps were generated, for control of location memory and for control of object memory, and thresholded at P < 0.005 (uncorrected; degree of freedom = 8). The final step, which constituted the test for conjunction, was to evaluate whether there was any anatomical overlap in these 2 control maps.
| Results |
|---|
|
|
|---|
Behavioral Results
From the preliminary behavioral study, a 2 x 2 analysis of variance (ANOVA) indicated no main effects of memory domain or of distraction type, but a significant interaction (F1,21 = 24.31, P < 0.0001). Pairwise tests confirmed that this crossover interaction reflected code-specific interference: "Location/saccades" performance was significantly lower than that of location/no secondary task performance (t21 = 2.79, P = 0.01) and that "object/passive listening" performance was significantly lower that "object/no secondary task" performance (t21 = 2.18, P < 0.05) (Fig. 1a,b). Behavioral performance during the fMRI session followed the same trend (Fig. 1c). A 2 x 2 ANOVA of the fMRI behavioral performance also found no main effects of memory or distraction type but did find a significant interaction (F2,16 = 17.20, P < 0.0001). Pairwise tests showed the same trends as seen in the behavioral experiment: location/saccades performance was significantly lower than location/no secondary task performance (t8 = 2.60, P < 0.05), and object/passive listening performance trended toward being lower than object/no secondary task performance (t8 = 1.27, not significant [n.s.]).
|
fMRI Results
Data from 2 subjects were excluded from the analyses due to excessive noncompliance with eye movement/eye position instructions, and data from a third subject were excluded because of an error in recording eye position data.
Memory-Delay
Delay-period activity from no-distraction location memory trials was identified reliably across single-subject data sets in FEF, bilaterally, SEF, IPS, bilaterally, SPL, extrastriate cortex, bilaterally, primary visual cortex, and caudate nucleus, bilaterally. The analogous activity from object memory trials was found in MFG, bilaterally, left sylvian cortex, left STG, IPL, bilaterally, IOTC, bilaterally, and primary visual cortex. These patterns of activity, for both stimulus domains, are consistent with prior reports in the literature. The identification of object delay-period activity in the left hemisphere sylvian and STG regions, for example, has precedence in previous studies of working memory for unfamiliar faces (Rama et al. 2001
; Rama and Courtney 2005
; Roth et al. 2006
), colored disks (Imaruoka et al. 2005
), and abstract geometric shapes (Gibbs and D'Esposito 2006
), to name just a few object stimulus categories. (See Fig. 2 and Table 1 for results from spatially normalized group-average analyses.) The general trend, across both memory domains, was that a concurrent secondary task suppressed the delay-evoked response and that the magnitude of this suppression tended to be greater for concurrent passive listening than for concurrent saccades (Fig. 3). There were marked deviations from this trend, however, in regions commonly associated with the secondary tasks themselves.
|
|
On location memory trials, the only regions in which delay-evoked responses were numerically greater with concurrent saccades than with no secondary task were frontal areas associated with oculomotor control: the FEF, bilaterally, and the SEF (Figs 3a and 4a). The delay-evoked response in memory-delay voxels in each of these regions was significantly greater with concurrent saccades than with concurrent passive listening (right FEF t6 = 2.53, P < 0.04; left FEF t6 = 5.71, P < 0.001; and marginally so in SEF t8 = 2.11, P = 0.07). To further investigate this effect, we created an "anterior oculomotor" ROI by combining the FEF and SEF ROIs and compared it against the pooled location memory-delay voxels from all other brain regions. A comparison of the effect of the contrast (Delaylocation/no secondary task – Delaylocation/saccades) in the anterior oculomotor ROI versus the rest of the brain revealed a significant difference (t8 = 4.41, P < 0.005). Finally, we evaluated whether individual differences in the neural interference effect (i.e., [Delaylocation/no secondary task – Delaylocation/saccades]) in these anterior oculomotor regions would predict the magnitude of the behavioral interference effect (location/no secondary task – location/saccades). They did so in right FEF (r = –0.75, P = 0.05) and trended in this direction in left FEF (r = –0.56, n.s.) and SEF (r = –0.44, n.s.).
|
On object memory trials, in general, concurrent passive listening led to lower delay-evoked responses than did concurrent saccades. However, this pattern was reversed in the cortex of the left sylvian fissure and the left STG, both areas associated with auditory/phonological processing of language-based information in working memory (Buchsbaum et al. 2005
The similarity of the neural-selective interference effects to the behavioral-selective interference effects in the behavioral data was further illustrated with a 2 x 2 ANOVA of these neural effects, which compared location memory in the frontal oculomotor ROI with object memory in the left auditory/phonological ROI. It revealed a main effect of ROI (F1,8 = 10.02, P < 0.05), no main effect of secondary task (F1,8 = 0.34, n.s.), and a crossover interaction of the 2 (F1,8 = 7.67, P < 0.05) (Fig. 4).
"Distraction-Sensitive" Activity
We also evaluated the effects of the retention of different types of memory on voxels associated with processing of each distraction task (i.e., those identified with the contrasts [Delayno memory/saccades – baseline] and [Delayno memory/passive listening – baseline]). Although the distraction tasks activated voxels in many of the same regions as did the memory tasks (see Fig. 3), there was typically considerable nonoverlap at the voxel level. Thus, for example, many of the saccade-sensitive voxels of the FEF were different from the memory-delay voxels of the FEF. This is reflected in the fact that saccade-sensitive voxels in all brain areas decreased their activity with the addition of memory for either stimulus domain, and in no regions was the saccade-evoked effect more sensitive to one memory domain than the other. The same was true of passive listening–sensitive voxels (Fig. 3c,d). Thus, there was no evidence of a neural correlate of code-specific interference in distraction-sensitive ROIs.
Conjunction Analysis
Contrasts were computed to isolate dual-task effects attributable to the pairing of object memory with passive listening and of location memory with saccades. Their results yielded no regions of overlap, thereby failing to find evidence for domain-general control of dual-task performance (an effect that would have been consistent with engagement of a "central executive" [D'Esposito et al. 1995
] or an inhibitory control process [Hester et al. 2004
]). This null result is consistent with the results of previous studies employing similar experimental procedures (Klingberg 1998
; Adcock et al. 2000
; Bunge et al. 2000
, 2001
).
| Discussion |
|---|
|
|
|---|
The results from this study confirmed that endogenously guided saccades selectively disrupt the short-term retention of stimulus locations and that passive listening to nouns selectively disrupts the short-term retention of stimulus identity. These results are consistent with the multiple-encoding prediction that location working memory is supported by prospective motor codes and that object working memory is supported by verbal codes. The neural correlates of the behavioral double dissociation were also consistent with this interpretation: The neural interference effect—a relative increase of activity localized to regions associated with the secondary task in question—was observed in frontal oculomotor regions in the case of saccadic distraction of location memory; and it was observed in left hemisphere superior temporal and sylvian cortex in the case of passive listening distraction of object memory. We will first consider the logical bases for these interpretations and then consider their implications.
One challenge with secondary task interference studies is to discriminate patterns of interference that identify specific mental codes or processes from those that are nonspecific and that may simply result from divided attention or increased difficulty. This can be particularly troublesome in situations in which the secondary tasks are not carefully matched for difficulty and other factors. For example, is it problematic that one of the secondary tasks in the present study was active (making endogenously generated eye movements), whereas the other was passive? The answer is "no" because of the logic of the double dissociation. To argue that performing the secondary saccadic task was attention-demanding in a general sense, for example, one would need to show that this secondary task had disruptive effects on both memory tasks. Not only was this not the case but also the selective effect of passive listening on the object memory task means that the insensitivity of this latter task to concurrent saccades was not indicative of a general insensitivity to any kind of concurrent task. Rather, one is compelled to conclude that the selective patterns of interference were due to resources shared by the 2 tasks that interfered with one another. In the case of delayed recognition of locations, the fact that the interfering saccade task made no overt demands on sensory processing makes it most likely that the common resource was oculomotor control. The localization of the neural interference effect to frontal regions associated with oculomotor control is also consistent with this interpretation. In the case of delayed recognition of objects, the common resource might be verbal (our preferred interpretation) or it might be visual. According to the visual account, listening to concrete nouns might have evoked visual imagery, and these mental images would be expected to interfere with remembered object representations. There are 4 pieces of evidence, however, that make this unlikely. First, in a different study we have demonstrated an analogous pattern of selective disruption of object working memory with a secondary task requiring the concurrent syntactic analysis of abstract nouns and adjectives (Postle et al. 2005
). It is highly unlikely that this secondary task engaged visual imagery. Second, individual differences in the neural interference effect in left sylvian cortex, a region associated with phonological working memory (Buchsbaum et al. 2005
), showed a trend toward predicting the behavioral interference effect. The brain-behavior relation in right IOTC, however, was in the opposite direction, and these effects in these 2 ROIs differed significantly from one another. We interpret these results as evidence that although both of these regions were active during the object working memory task, their delay-period activity likely supported different functions, in that they were (significantly) differentially sensitive to concurrent passive listening. Third, although voxels in the right IOTC were strongly active during the object delay periods, they were not differentially sensitive to concurrent passive listening as compared with concurrent saccades. If passive listening engaged imagery, one would expect to see neural correlates of imagery in visual regions such as the right IOTC. Finally, the locus of the Delayobject memory, passive listening effect corresponds to a neural locus of the irrelevant speech effect (ISE) (Gisselgard et al. 2003
, 2004
), an unambiguously "verbal" effect that will be reviewed in detail further along in this discussion.
Turning now to the implications of these results, we believe that they require extension and/or refinement of current conceptions about working memory at 2 levels. Neurally, they specify a physiological and anatomical specificity of code-specific interference effects: they manifest as increases in delay-evoked signal that are restricted anatomically to memory-delay voxels in regions associated with the secondary tasks. Our results suggest that these code-specific neural interference effects are superimposed on general nonspecific decreases in evoked signal that are more generally characteristic of dual-task versus single-task conditions. Theoretically, they specify processes of recoding and parallel representation (i.e., of multiple encoding) that need to be explicitly accommodated in models of working memory. We will now consider these implications in more detail, first focusing on the neural implications.
The extant literature on the neural correlates of behavioral interference indicates that they are characterized by overlap of the cortical territories associated with performance of the 2 tasks (Klingberg and Roland 1997
; Klingberg 1998
; Gruber and von Cramon 2003
) and/or b) recruitment of "new" control areas (D'Esposito et al. 1995
; Hester et al. 2004
). Within the "overlap" literature, the experiments of Klingberg and Roland featured nonspecific interference, in that they examined the deleterious effects of having to perform 2 tasks concurrently, but they did not seek to produce code-specific interference of the kind featured in the present study. Their results indicated that one could predict behavioral interference with concurrent task performance if each of the 2 tasks recruited overlapping cortical territories (Klingberg and Roland 1997
; Klingberg 1998
) and that signal intensity within the region of overlap was unchanged (it was elevated for dual- vs. single-task performance, but not significantly so [Klingberg, 1998
]) during dual-task performance with respect to single-task performance (Klingberg 1998
). These are broadly consistent with what we report here.
The study of Gruber and von Cramon (2003)
, in contrast to those of Klingberg but in common with the present study, did employ the logic of domain-specific interference. It differed from the present study, however, in that its secondary tasks drew on the same sensory modalities and the same informational domains as did the primary tasks with which they were paired—articulatory suppression with delayed letter recognition and visuospatial suppression with delayed visuospatial recognition. The results of this study also differ from what we report here. Gruber and von Cramon (2003)
reported the neural correlates of domain-specific interference to be marked reductions in activity in the regions of overlap between primary and secondary task, with no appreciable effect of the pairing of the noninterfering secondary task on primary task performance (e.g., no effect of concurrent articulatory suppression on visuospatial delayed recognition). The present results, in contrast, show code-specific interference effects to be increases in the evoked response that are restricted to memory-delay voxels in regions associated with the secondary task, whereas nonspecific effects are in the opposite direction—a decrease in evoked signal in every brain area engaged by the task, regardless of overlap or nonoverlap of the 2 tasks in isolation. (Note that the code-specific increase is presumably superimposed on the nonspecific decrease, and so the value of the dual-task–evoked response might fall above [as it does for Delaylocation/saccades in frontal oculomotor control regions] or below [as it does for Delayobject/passive listening in left perisylvian regions] the single-task–evoked response.) At a general level, the 2 studies are compatible in that they each implicate cortical overlap and, by inference, competition for neural resources, as the physiological basis for domain-specific interference. At a more detailed level of description, however, they differ, in that the Gruber and von Cramon (2003)
results suggest that this competition produces a decrease in activity, whereas the present results suggest that it produces an increase. It may be that this discrepancy reflects the fact that the competition produced by the Gruber and von Cramon task could have occurred at multiple levels—transduction from physical energy into a neural code, perceptual processing, and short-term retention—whereas the competition produced by our task could only occur at the level of the short-term retention of target-related information. Indeed, we interpret the anatomical specificity of the neural interference effects in the present study as further evidence for multiple encoding, in that it reinforces the behavioral evidence for a contribution of oculomotor control to the short-term retention of visuospatial information and of verbal processing to the short-term retention of visuoobject information.
Another relevant comparison for the present study is neuroimaging studies of the ISE, which occurs when irrelevant sounds (including speech) are presented concurrent with or subsequent to presentation of to-be-remembered linguistic information (e.g., digits, letters, words). Although the precise factors underlying the ISE remain a matter of debate, there is general consensus that it is brought about by interference between mnemonic representations and distractors at the level of the phonological representation of the memoranda (Repovs and Baddeley 2006
). Thus, our interpretation of the interfering effect of passive listening on object delayed recognition can be cast as an ISE account. A previous study of the ISE using positron emission tomography showed it to be associated with a general decrease in all regions recruited by a verbal working memory task, an effect that was particularly salient in a region spanning STG and sylvian cortex, bilaterally (Gisselgard et al. 2003
). A follow-up study by the same group added memory load as an experimental factor, and found that the neural correlate of the ISE in left STG varied with memory load, with a relative increase in signal in this region in the high load condition (Gisselgard et al. 2004
). The centrality of the perisylvian cortex in these studies of the ISE in verbal working memory provides an independent corroboration of the face validity of our interpretation of the code-specific passive listening effecting our study: listening to words interfered with phonological representations of the object memoranda.
Theoretically, the present results highlight the importance of the principle of multiple encoding in working memory. For the short-term retention of location information, they provide independent behavioral evidence that the location delay-period activity observed in many studies (and, indeed, in many species) in the FEF (e.g., Postle, Berger et al. 2000
; Rowe et al. 2000
; Takeda and Funahashi 2002
, 2004
; Balan and Ferrera 2003a
, 2003b
; Curtis et al. 2004
) reflects, at least in part, a motoric code. For the short-term retention of object information, they offer 2 novel insights. The first is that the behavioral results confirm that the interference produced by a secondary verbal task occurs at a nonvisual level of representation. That is, the interference produced by the secondary verbal task was likely verbal, not visual, because the nouns were presented auditorily. (And we have already reviewed reasons to discount an imagery explanation of the passive listening effect.) Second, the fact that the critical anatomical locus of code-specific interference was left sylvian cortex, a region previously implicated in phonological representation in working memory (Buchsbaum et al. 2005
), further suggests that one representational locus of this interference is articulatory/phonological. (Whether there may also be a semantic component to this interference effect will require additional research.) We have also seen that this analysis corresponds well with neuroimaging correlates of the ISE in verbal working memory (Gisselgard et al. 2003
, 2004
).
How do these results and, more broadly, the principle of multiple encoding relate to current models of working memory? We will consider this question via the multiple-component model, unquestionably the most influential in contemporary cognitive psychology and cognitive neuroscience. The multiple-component model is a memory systems model, in that it posits specialized mechanisms whose principle function is to support a kind of memory. For example, it posits the existence of an "inner scribe" that represents locations and the paths between locations, a "visual cache" that represents visual object identity, and a "phonological loop" comprising a "phonological store" and an articulatory rehearsal process (Logie 1995
, 2003
; Baddeley and Logie 1999
; Repovs and Baddeley 2006
). From this perspective, one could portray the present results as adding mechanistic and neural specificity to the multiple-component model. To the inner scribe portion of the model, it specifies a mechanism—prospective oculomotor coding—and a neural substrate—the FEF. It can also be noted that a mechanism of prospective motor coding recruited in the service of spatial working memory can be portrayed as a process that is analogous to the manner in which visually presented verbal material is recoded into a phonological/articulatory code and retained in memory as a speech-based motor code (e.g., Page and Norris 1998
). To pursue the analogy, our saccadic distraction task might be viewed as analogous to articulatory suppression as an effective distractor of verbal working memory. Turning now to the short-term retention of object information, the present results can be interpreted to suggest that the function of the visual cache can be supplemented by the phonological loop, provided that the memoranda afford verbal recoding. This verbal recoding is hypothesized to be mediated via the association of visual stimuli with long-term knowledge, which, in turn, activates a lexical representation.
An alternative to this memory systems analysis, however, is to view the short-term retention of information as a property of each of the systems engaged by this task—oculomotor control, visual perception, and speech production. From this "emergent processes" perspective (Postle 2006
), what the present study has illustrated is ways in which nominally "nonmemory" systems (e.g., perceptual systems, motor systems, language systems) accomplish the short-term retention information in the service of behavioral goals.
| Acknowledgments |
|---|
We thank Olufunsho Faseyitan, Jared Hotaling, Christopher Jordan, Jamie Lapin, and George McPhail for assistance with programming, data collection, and analyses. Supported by National Institutes of Health MH064498 to B.R.P. Conflict of Interest: None declared.
| References |
|---|
|
|
|---|
Adcock RA, Constable RT, Gore JC, Goldman-Rakic P. Functional neuroanatomy of executive processes involved in dual-task performance. Proc Natl Acad Sci USA (2000) 97:3567–3572.
Aguirre GK, Zarahn E, D'Esposito M. The variability of human, BOLD hemodynamic responses. Neuroimage (1998) 8:360–369.[CrossRef][ISI][Medline]
Attneave F, Arnoult MD. Methodological considerations in the quantitative study of shape and pattern perception. Psychol Bull (1956) 53:221–227.
Awh E, Jonides J. Overlapping mechanisms of attention and spatial working memory. Trends Cogn Sci (2001) 5:119–126.[CrossRef][ISI][Medline]
Baddeley AD. Working memory. (1986) London: Oxford University Press.
Bower GH. Working memory. In: The psychology of learning and motivation (1974) New York: Academic Press. 47–89.
Baddeley AD, Logie RH. Working memory: the multiple-component model. In: Models of working memory—Miyake A, Shah P, eds. (1999) Cambridge (UK): Cambridge University Press. 28–61.
Balan PF, Ferrera VP. Effects of gaze shifts on maintenance of spatial memory in macaque frontal eye field. J Neurosci (2003a) 23:5446–5454.
Balan PF, Ferrera VP. Effects of spontaneous eye movements on spatial memory in macaque periarcuate cortex. J Neurosci (2003b) 23:11392–11401.
Boynton GM, Engel SA, Glover GH, Heeger DJ. Linear systems analysis of functional magnetic resonance imaging in human V1. J Neurosci (1996) 16:4207–4221.
Buchsbaum BR, Olsen RK, Koch P, Berman KF. Human dorsal and ventral auditory streams subserve rehearsal-based and echoic processes during verbal working memory. Neuron (2005) 48:687–697.[CrossRef][ISI][Medline]
Bunge SA, Klingberg T, Jacobsen RB, Gabrieli JDE. A resource model of the neural basis of executive working memory. Proc Natl Acad Sci USA (2000) 97:3573–3578.
Bunge SA, Ochsner KN, Desmond JE, Glover GH, Gabrieli JDE. Prefrontal regions involved in keeping information in and out of mind. Brain (2001) 124:2074–2086.
Curtis CE, Rao VY, D'Esposito M. Maintenance of spatial and motor codes during oculomotor delayed response tasks. J Neurosci (2004) 24:3944–3952.
D'Esposito M, Detre JA, Alsop DC, Shin RK, Atlas S, Grossman M. The neural basis of the central executive system of working memory. Nature (1995) 378:279–281.[CrossRef][Medline]
Della Sala S, Gray C, Baddeley A, Allamano N, Wilson L. Pattern span: a tool for unwelding visuo-spatial memory. Neuropsychologia (1999) 37:1189–1199.[CrossRef][ISI][Medline]
Friston KJ, Holmes AP, Worsley KJ, Poline JP, Frith CD, Frackowiak RSJ. Statistical parametric maps in functional imaging: a general linear approach. Human Brain Mapp (1995) 2:189–210.[Medline]
Friston KJ, Rotshtein P, Geng JJ, Sterzer P, Henson RN. A critique of functional localisers. Neuroimage (2006) 30:1077–1087.[CrossRef][ISI][Medline]
Fuster JM. Memory in the cerebral cortex. (1995) Cambridge (MA): MIT Press.
Gibbs SEB, D'Esposito M. A functional magnetic resonance imaging study of the effects of pergolide, a dopamine receptor agonist, on component processes of working memory. Neuroscience (2006) 139:359–371.[CrossRef][ISI][Medline]
Gisselgard J, Petersson KM, Baddeley AD, Ingvar M. The irrelevant speech effect: a PET study. Neuropsychologia (2003) 41:1899–1911.[CrossRef][ISI][Medline]
Gisselgard J, Petersson KM, Ingvar M. The irrelevant speech effect and working memory load. Neuroimage (2004) 22:1107–1116.[CrossRef][ISI][Medline]
Gitelman DR. ILAB: a program for post experimental eye movement analysis. Behav Res Methods Instrum Comput (2002) 34:605–612.[ISI][Medline]
Gruber O, von Cramon DY. The functional neuroanatomy of human working memory revisited: Evidence from 3-T fMRI studies using classical domain-specific interference tasks. Neuroimage (2003) 19:797–809.[CrossRef][ISI][Medline]
Handwerker DA, Ollinger JM, D'Esposito M. Variation of BOLD hemodynamic responses across subjects and brain regions and their effects on statistical analyses. Neuroimage (2004) 21:1639–1651.[CrossRef][ISI][Medline]
Hester R, Murphy K, Garavan H. Beyond common resources: the cortical basis for resolving task interference. Neuroimage (2004) 23:202–212.[CrossRef][ISI][Medline]
Imaruoka T, Saiki J, Miyauchi S. Maintaining coherence of dynamic objects requires coordination of neural systems extended from anterior frontal to posterior parietal brain cortices. Neuroimage (2005) 26:227–284.[CrossRef]
Klingberg T. Concurrent performance of two working memory tasks: potential mechanisms of interference. Cereb Cortex (1998) 8:593–601.
Klingberg T, Roland PE. Interference between two concurrent tasks is associated with activation of overlapping fields in the cortex. Cogn Brain Res (1997) 6:1–8.[CrossRef][Medline]
Kwong KK, Belliveau JW, Chesler DA, Goldberg IE, Weisskoff RM, Poncelet BP, Kennedy DN, Hoppel BE, Cohen MS, Turner R, et al. Dynamic magnetic resonance imaging of human brain activity during primary sensory stimulation. Proc Natl Acad Sci USA (1992) 89:5675–5679.
Logie RH. Visuo-spatial working memory. (1995) Hove (UK): Erlbaum.
Logie RH. Spatial and visual working memory: a mental workspace. In: Cognitive vision: the psychology of learning and motivation—Irwin D, Ross B, eds. (2003) San Diego (CA): Elsevier Science. 37–78.
Ogawa S, Tank DW, Menon R, Ellermann JM, Kim SG, Merkle H, Ugurbil K. Intrinsic signal changes accompanying sensory stimulation: functional brain mapping using MRI. Proc Natl Acad Sci USA (1992) 89:5951–5955.
Page MPA, Norris D. The primacy model: a new model of immediate serial recall. Ps



