User`s guide

E-Prime User’s Guide
Appendix B: Considerations in Research
Page A-35
Statistical Analysis of RT Data
While this brief review of single-trial RT research cannot include an extensive discussion of data
analysis, a few points deserve comment.
The typical analysis of single-trial RT data employs the analysis of variance (ANOVA) to compare
mean RT’s under various treatment conditions as defined by the levels of the independent
variables. For within-subjects variables, a repeated-measures ANOVA is employed. Sometimes,
both within- and between-subjects factors occur in the same experiment, resulting in a mixed
ANOVA. For the example experiment on letter identification, there are two independent variables
that define types of trials for the analysis. One is the location of the stimulus, which has six levels
(0, 1, 2, 4, 8, and 16°). The other is whether or not it was adjusted in size to correct for poor
acuity, which has two levels (adjusted or not). For this analysis, the mean RT for each of the
twelve conditions would be calculated for each subject, and those values would serve as the
data. The ANOVA then compares the means of those values based on all subjects to determine
statistical significance.
21
In addition to an analysis of RT’s, there should be a parallel analysis of error rates, expressed
either as percent correct or as percent error. (Since percent error is just 100 minus the percent
correct, these analyses yield the same result.) In general, error rates should parallel RT’s faster
conditions have lower error rates. If faster RT’s are associated with higher error rates, a speed-
accuracy trade-off should be suspected, and interpretation of RT differences should only be made
with extreme caution.
In almost all instances, RT analyses are based on correct trials only. It is good practice to
examine the overall error rate for each subject. While what constitutes an acceptable rate will
differ with different experiments, it is common practice to delete the data from any subject whose
error rates are clearly higher than the norm. In this case, it is likely that the subject either
misunderstood the instructions or was simply unable to perform the task. If at all possible, the
maximum error rate should be set in advance, so that there is no danger of deleting a subject’s
data because they do not conform to the expected results. Pilot testing should help in setting a
maximum error rate.
Another issue for analysis of RT data involves outliers, or extremely deviant RT’s that occur on
occasional trials. These usually involve extremely slow RT’s. Many researchers assume that
such extreme RT’s reflect momentary inattention or confusion, therefore they are properly omitted
from the analysis, prior to calculating the mean RT by condition for individual subjects. A
common criterion is to omit any trials whose RT is more than three standard deviations from the
mean for that condition. That can be done either based on the mean and standard deviation of
RT for all subjects, or for individual subjects. The latter is clearly indicated if there are large
differences in RT between subjects. More sophisticated schemes for treating outliers have been
suggested (Ratliff, 1993; Ulrich & Miller, 1994).
The repeated-measures ANOVA, which is almost always used for significance testing with RT
data, makes an assumption of “compound symmetry," or equality of covariances across all pairs
of conditions. That assumption is seldom met in real data. Most statistical packages compute
adjusted values of p based on either the Greenhouse-Geiser statistic or the newer, less
conservative Huynh-Feldt statistic. In general, these corrected values of p should be used in
assessing statistical significance.
21
A discussion of the controversy surrounding the merits of traditional null hypothesis testing is beyond the
scope of this discussion. See several articles in the January, 1997 issue of Psychological Science and
Chow (1998) for discussions of this topic.