User`s guide

E-Prime User’s Guide
Appendix B: Considerations in Research
Page A-22
prediction about how the DV will change, but there is the expectation that changes in the IV’s
studied will produce changes in the DV. If they do not, not much exploration has taken place.
How will data be analyzed?
The experiment and the data analysis should be co-designed. It is extremely important that the
methods of data analysis be known in advance of collecting the data. There are several reasons
why this is so. Since the point of the research is to make comparisons of DV’s at varying levels of
IV’s, it should be clear in advance what comparisons would be made and how they will be made
statistically. This can avoid nasty surprises later, such as discovering that a crucial variable was
not recorded, or (worse yet) that no appropriate statistical test is available. There is some
additional discussion of statistical analysis of single-trial RT data below.
Before collecting the data, it is useful to draw a graph of the data, to be clear as to what patterns
of RT’s would support or disconfirm the hypothesis. If there are other possible assumptions
about the effects of the IV(s), graph those as well. Such a graph will help clarify predictions.
Plotting the expected means along with expected standard error bars (perhaps derived from pilot
testing) can give a good perspective on what is expected to be seen, and what differences might
be significant. As a rule of thumb, a difference between the means of two standard errors is likely
to be significant. A statistical power analysis is useful as well, to help judge the likelihood of
getting a statistically significant difference between means based on the size of the differences
expected, the variability of the data, and the sample size.
How will the experimental tasks be presented?
A careful review of the pertinent literature is a natural starting place for any research, usually
focusing on previous theoretical and empirical developments. However, a review of Methods
sections of experiments using similar tasks is also likely to be rewarding. Such a review may
alert to considerations not thought of, saving much pilot testing. If there is literature or research
using similar tasks, it might be worthwhile to discuss the design with the authors and take
advantage of any insights not made a part of the formal report of Methods.
A host of considerations comes into play in the detailed design of an experiment and the analysis
of the data it produces. While some are specific to a restricted research domain, others are more
general. The discussion below of the minutia of single-trial, reaction-time research highlights
many of those considerations.
Implementing a Computerized Experiment
Once you have thoroughly thought out the question you wish to answer and how you plan on
answering it, you are ready to begin designing the experiment. Do not rush the previous planning
stage. It is critical to have properly prepared before attempting to design or implement an
experiment.
Constructing the experiment
Work from the inside out (or the bottom up). The best way to construct an experiment is to get a
few trials going before typing in full stimulus lists and instructions. We recommend leaving
instruction screens blank and specify a minimal list of stimuli; usually one from each level of a
single IV is sufficient. Once certain that the basic trial is running and that the data are stored
correctly, add the other IV’s, additional stimuli, instructions, and other details. It is fairly often the
case that in setting up an experiment, it becomes clear that further variables need to be specified
in stimulus lists. Going back to a long list of trials and adding those to each can be frustrating.