In this chapter, we describe some methods to explore the automatic, unconscious processing of asynchronous events that are judged as being simultaneous. We first describe our conceptual framework, which was guided by clinical inquiries on patients with schizophrenia. We then describe how we applied the Simon effect, as a novel method of analyses. The code provided with the manuscript (see book’s GitHub repository) will allow the reader to generate his/her own data. The aim of the chapter is to provide all the necessary information for the reader to train and calculate effects like the Simon effect by including some example data files (see Annex). It must be noted at the onset that these calculations are not meant to replace more standardized measures like the evaluation of the threshold. They are only meant to complement them. Also, it is not the aim of this manuscript to provide a set method. We rather advocate that some flexibility in our methods can sometimes provide additional and useful information on the processes we explore. In the final section of this chapter, we provide an example of how and to which aim other methods can be used when utilizing identical tasks.
The flow of our mental activity in time is unidirectional, from the past to the future. This directionality is an intrinsic property of all cognitive functions, which appears to be a given fact, and which we seldom reflect upon or include as a parameter of our experimental approaches. Its importance is evident in clinical populations for whom the continuous flow of thought is altered. For example, clinical experience suggests that patients with schizophrenia suffer from a disruption of the feeling of time continuity. This disruption has been described by many psychiatrists based on patients’ reports (i.e., they are based on what the patients say within interviews with the psychiatrist; Andreasen, 1999; Fuchs, 2007; Minkowski, 1933; Vogeley & Kupke, 2007; Chapman, 1966). The drawback of these reports is that they are selected and interpreted subjectively by the psychiatrist. The impairments, thus, require objective experimentation and measurement. It is with the goal of objectively characterizing such a
Objectifying a disruption of the sense of time continuity requires an understanding of the mechanisms leading to the feeling of continuity to begin with. It is indeed far from clear whether our conscious access to sensory information is as continuous as experienced subjectively. Experimentally it is long known that two events are distinguished in time only if there is a minimal time asynchrony between them (Elliott & Giersch, 2016, for a review). For example, two visual events displayed in distinct spatial locations will have to be delayed by 30 to 50 ms to be distinguished in time (Brecher, 1932; Elliott, Shi, & Kelly, 2006; Pöppel, 1997; Wittmann, 2011). The fact that such an asynchrony is not zero leads to the concept of the temporal window, the interval of time within which the brain is hypothesized to process events as being co-temporal (Elliott et al., 2006; Elliott & Giersch, 2016; van Wassenhove, 2009; Wittmann, 2011). These results confirm that the conscious processing of information is not as continuous as experienced subjectively. As a matter of fact, if information is judged as being co-temporal within temporal windows, it is as if time stops for the period of the temporal window.
Other authors have also proposed that the processing of information is discrete rather than continuous (e.g., Pilz, Zimmermann, Scholz, & Herzog, 2013; Scharnowski et al., 2009; VanRullen, Zoefel, & Ilhan, 2014). In case of Pilz et al. and Scharnowski et al., this conclusion is derived from studies on visual integration rather than timing per se. These authors show that sensory stimuli require time to be processed. The consequence of this constraint in the processing time would be that information processing has an intrinsic inertia, which would not mirror external-event timing. Objects and scenes are indeed sensed and identified on the basis of multisensory signals, implying the need to integrate such sensory signals over space and sensory pathways. Even within a given modality like vision, bits of information have to be integrated before identification can occur (Boucart et al., 1994). This means that information is not processed instantaneously rather information is first decomposed and processed by specialized neurons. In the primary visual cortex, specialized pathways are used to process contour and surface information: neurons will respond selectively to orientation, code line-ends, and edges, or they will be activated by information related to color or texture. Whichever model is used to understand how information is bound together to identify forms and objects, time is required, because ambiguities regarding which information belongs to which object is often ambiguous (Grossberg & Pilly, 2008; Pugeault,
Finally, time is also needed to achieve neuronal synchronization. It is usually proposed that pools of neuron activations are formed to code all properties of objects. Such functional assemblies, however, require some time to be formed, as they rely on spikes synchronization (Varela, 1999). The perception of an object would not be available before such synchronization takes place, and, thus, accessing the experience of the object would need to be delayed. This may participate to inertia in information processing. The time required to experience each event, might, in turn, complicate the conscious temporal ordering of these events. In contrast, at initial stages, processing is parallel and accurate temporal properties of sensory information may be preserved at these initial, unconscious stages.
To check, therefore, whether temporal processing is more accurate at initial than at conscious stages, procedures are required to explore the implicit processing of information in time, in addition to explicit processing (see van Wassenhove, 2009, for additional arguments). ‘Explicit’ processing is defined here by the fact that, in a given task, participants are explicitly asked to make a conscious decision (e.g., decide whether or not stimuli are simultaneous; Coull & Nobre, 2008). Implicit processing is defined by no such required judgment by the participants. For example, implicit mechanisms play an important part in sensorimotor timing (Repp, 1999; van Wassenhove, 2009) and are not necessarily equated with explicit judgments (Martin, Giersch, Huron, & van Wassenhove, 2013; van Wassenhove, 2009). In other words, implicit processing relies on mechanisms that do not require conscious thought, but that nonetheless can influence what we experience.
2 Exploring Implicit Timing Mechanisms with the Simon Effect
There are several techniques that allow one to assess the properties of implicit mechanisms; we will describe only a subset of those below. This section merely describes new ways to explore implicit biases in participants’ responses during simultaneity/asynchrony judgments. It should be noted that here, ‘biases’ refer to the fact that when participants press on a response key, they not only base their choice on a conscious decision.
The method we employed is based on the use of the Simon effect. The Simon effect reflects the participants’ tendency to press the response button corresponding to the location where a stimulus is displayed on the screen.
The readers can train themselves calculating the Simon effect with the example data provided with the manuscript (see book’s GitHub repository: ‘resultsRaw.txt’).
Interestingly, for soas as short as 8 ms, the bias to the side of the first stimulus predicts the difficulties of the patients to explicitly detect asynchronies at larger soas. This correlation may suggest that the bias to the side of the 1st stimulus reveals an elementary mechanism at the core of our ability to predict and follow events over time (Giersch et al., 2015).
Difficulties at processing temporal information in patients with schizophrenia did not come as a surprise. As already emphasized, it has long been reported that the patients’ thought flow is disrupted (Fuchs, 2007; Vogeley & Kupke, 2007). Psychiatrists had proposed that patients with schizophrenia have a difficulty to look forward in time (Minkowski, 1933). Importantly, these descriptions were made before the discovery of antipsychotics (in the 1950s), suggesting they are independent from treatment in schizophrenia. More recently, it has been proposed that patients display a cognitive dysmetria (Andreasen, 1999); that is, a difficulty to organize thoughts, especially in time. With this background in mind, the reversal of the Simon effect observed at small asynchronies could reveal how visual stimuli are processed in time, and might contribute to a more general difficulty at organizing information in time. We made an attempt at interpreting the biases to answer to the side of the first or second stimulus, and proposed that healthy participants follow the events in time, and, thereby, have their attention on the second stimulus by the time of their response. In contrast, patients would stay stuck with the first stimulus in case of undetectable asynchronies. The patients would have difficulties to predict and follow sequences of events over a few ms, and this might be at the origin of their difficulties at explicitly detecting asynchronies (Foucher et al., 2007; Lalanne et al., 2012a, b). These difficulties become huge when distractors are present, or in case of multisensory signals (Giersch et al., 2009; Martin et al., 2013; Schmidt et al., 2011).
Alternative explanations (eye movements, difficulty to relate stimuli in space, interhemispheric transfer, basal Simon effect; reviewed in Giersch, Lalanne, van Assche, & Elliott, 2013) were ruled out: eye movements had been controlled for by having participants fixating at the center of the screen, effects have been recorded with intra- as well as with inter-hemispheric presentation,
3 Testing Implicit Timing Mechanisms with a Priming Task
To test whether the processing of visual stimuli over short intervals occurred automatically, we devised a new task, derived from earlier priming tasks (Elliott, Shi, & Sürer, 2007). In our task, priming stimuli were not related to the participants’ task, but nonetheless influenced performance. They were two empty frames displayed on the screen, one on the left and one on the right of the screen center. These two frames were subsequently filled in by the target stimulus that elicited the response. In a first task, only one frame was filled in with a white square. Participants were instructed to press on a response key to the side of the target, as quickly as possible (Figure 14.2). The main manipulation regarded the frames that preceded the target as the frames’ onsets were either simultaneous or asynchronous, but the asynchrony was short enough not to be perceived by participants (this was checked after the experiment). Participants were slightly (but significantly) faster in conditions where the target was displayed to the side of the second frame rather than to the side of the first one, when there was at least 75 to 100 ms between the frames and the target (Poncelet & Giersch, 2015).
We had similar effects when both frames were filled in with an soa of 100 ms and participants decided which frame was last filled in (Figure 14.3). With such an arrangement, we had to take into account whether frames and targets were displayed in the same direction or in the opposite direction. We showed that participants were faster when targets were displayed in the opposite direction relative to the primes. This was observed independent of whether participants had to press to the side of the first or second stimulus. We interpreted these effects as reflecting a shift of attention towards the second prime (Poncelet & Giersch, 2015). As a matter of fact, it is known that cueing the location of the first stimulus in a sequence of two facilitates temporal order judgment (Spence & Parise, 2010, for a review). The shift of attention towards the second prime would displace attention on the location of the first target when primes and targets are displayed in opposite directions. This cueing effect of the second prime would explain that participants are faster at making temporal judgments when the second prime is the location of the first target (i.e., when primes and targets are shown in opposite directions).
These results are important inasmuch they seem to be independent of direction perception. Else responses should have been faster when both primes and targets were displayed in the same direction. This was not the case, suggesting the results are related to time rather than direction (i.e., spatial perception). Second, we checked that the effects did not depend on the side of the response. We showed priming effects to be identical when participants had to decide about the side of the first target and when they had to decide about the side of the second one. These results, thus, suggest that participants are biased to shift their attention to the second stimulus and answer using the button on
The Simon effect, thus, represents a way to investigate the implicit processing of visual stimuli over time. However, there are several limits that should be kept in mind when using these procedures. First, the equipment should be checked carefully, and it should be verified with photocells that the delays between stimuli are as expected. Several parameters of the stimuli may influence the results, like the eccentricity of the stimuli, or their luminance. The typical Simon effect, which represents the bias to press on the side of a single stimulus, may also affect the results: if patients and controls differ on the ability to be biased by the location of the stimuli, then it may induce a group difference that would have nothing to do with a shift of attention in time. An impaired typical Simon effect should, thus, be controlled for. Since there are always two stimuli in the simultaneity/asynchrony discrimination task, the best control for the typical Simon effect is to present two stimuli on the right or two stimuli on the left side of the screen (instead of one on the left and one on the right). In this way, the typical effect should be a bias to respond on the side of the two stimuli, even when the task is to decide about their asynchrony. The critical analysis is to check that patients and controls do not differ on this typical Simon effect. Finally, it should be verified whether the tendency to press to the side of the first or second stimulus is related or not to a direction bias (e.g., a preference for a direction over the other one). The simplest way to check for this possibility is to verify that the pattern of results is the same whatever the response side (i.e., right response key for ‘simultaneous’ responses and left key for ‘asynchronous’ responses vs. the reverse).
It should be noted that a potential difficulty is related to eye movements. In the results described above on simultaneity/asynchrony discrimination, the participants were required to look towards the center of the screen and this was monitored continuously via an eyetracker. Whenever a saccade or fixation
4 Analyzing the Responses in the Simultaneity/Asynchrony Discrimination Task in Innovative Ways
We have described how the Simon effect can be applied in the simultaneity/asynchrony discrimination task to reveal nonconscious mechanisms of information processing in time. We hope that the readers see that there are many innovative ways of analyzing data in addition to the standard methods, provided one keeps in mind the limits of each method (i.e., the need to check for all possible confounding factors). We think that other methods might be applied to address additional questions.
In Poncelet and Giersch (2015), we proposed that patients with schizophrenia do not only have difficulties to automatically shift attention in time but also to predict the stimuli. We reasoned that prediction mechanisms should be involved to enable attention to be shifted from the first to the second stimulus, especially when the stimulus onsets are separated by very short delays. It must be reminded here that the results detailed above have shown that in healthy volunteers attention moves only after 75 to 100 ms towards the second stimulus in a sequence of two. This is consistent with the attention literature (see Poncelet & Giersch, 2015, for a thorough discussion on this point). This late attention shift implies that after the first stimulus onset, attention moves only with a delay towards the second stimulus. If the second stimulus is displayed for less than 20 ms after the first one, it means that attention moves only 80 ms after the second stimulus onset. Meanwhile attention is probably on the first stimulus to facilitate its processing. For attention to shift towards the second stimulus, it would, thus, help if the participant anticipated this second stimulus. This means that the participants would not simply follow events one after another but predict sequences of events, and prepare for their processing. This hypothesis could be checked using a trial-by-trial analysis (that is, an analysis of performance that verifies whether performance on a given trial depends of the properties of the trial presented previously). It has indeed been repeatedly shown that participants can better prepare a response if they expect it at a
This hypothesis can be evaluated by measuring the ability to detect asynchronies and taking into account what occurred during the trial before, not only the order of the stimuli (both trials left-right or right-left, or trials with an opposite stimuli direction) but also the relative asynchrony of the consecutive trials (trial N with a shorter, equal, or larger soa than trial N-1). These hypotheses can be understood within the general framework of predictive coding, whereby information is constantly predicted. Real sensory information is compared to predictions, enabling prediction errors to be detected (Friston, 2008). It has been proposed that ‘predictive coding’ might apply generally to brain functioning (Friston, 2008). In the context of the simultaneity/asynchrony discrimination task, trial N can be predicted on the basis of trial N-1. When stimuli are displayed on trial N, the similarity or discordance with the previous trial should, thus, be automatically detected (see Di Luca & Rhodes, 2016, for an application of predictive coding and Bayesian models to expectations in time, and Tschacher, Giersch, & Friston, 2017, for applications in mental health). In the simultaneity/asynchrony discrimination task, the detection of a difference between successive trials may incite participants to adapt their predictions, and predictions themselves could influence the response of the participants in two ways. The most trivial effect would be a facilitated detection of asynchronies in case of a similarity between trials N-1 and N (i.e., more asynchronous responses). Conversely, there would be a deterioration of performance, i.e., more ‘simultaneous’ responses, in case of a difference between trials N-1 and N. A deterioration of performance would be akin to an adaptation effect, but related to relative soa and not only direction (see e.g., the duration channel model for sub-second durations; Heron et al., 2012). Other effects may also be observed, related to the Simon effect. As discussed above, the prediction
We used the Simon effect to derive hypotheses regarding how participants follow visual stimuli in time at short time scales and how this is impaired in patients with schizophrenia. The same kind of method can be further adapted, e.g., to check for the implicit ability to predict very short soas in time. In other words, the Simon effect and related methods, are tools that can extend standard data analysis methods. As for any behavioral outcomes, the Simon effect does not provide direct insight into the mechanisms at play during a task, but it offers the possibility to make an inference. A major limitation of this type of inference is that participant’ responses are influenced by multiple aspects of the stimuli presented. Hence, the interpretation of the effects described above should be considered with caution. As it has been described at the beginning of the paper, the analysis of the Simon effect should be performed using multiple ways of testing, e.g., the priming task to validate the interpretation. Despite the limitations mentioned, these methods of data analysis can be used to complement standard rt and error analyses, and are powerful enough to yield heuristic hypotheses concerning both processing in healthy volunteers and impairments in patients.
‘SyncTest’ is the Matlab code for the experiment with synchronous vs. asynchronous stimuli. Some explanation on how the experiment is conducted, the instructions,
‘resultsRaw.txt’ corresponds to the raw data generated with the ‘SyncTest’ program.
‘Results_With_variable_Names.xls’ is the same file as the ‘resultsRaw.txt’ in excel but with variable names.
Both .txt and .xls data files can be used by importing them in a statistics software package to calculate percent errors by taking into account the order of the stimulus (left-right versus right-left).
Harvey C. E. Van der Burg & D. Alais (2014). Rapid temporal recalibration occurs crossmodally without stimulus specificity but is absent unimodally . Brain Research1585120–30.