User`s guide

E-Prime User’s Guide
Appendix B: Considerations in Research
Page A-23
Thorough testing with a few trials of each type will usually catch such errors while they are easy
to correct. As an example, suppose that you have to type in 200 words to serve as stimuli, and
designate each word by frequency and length. If you then decided to add concreteness as an
additional IV, you must enter the concreteness rating for each of the 200 words. However, if you
first test the experiment with only four words, and discover that an additional IV is needed, only
four levels must be fixed.
Pilot testing
Once the experiment is set up, perform a couple levels of pilot testing. The first level is to sit
through the whole experiment alone. You may notice errors you did not spot before, or realize
that the experiment is too long and should be run over multiple sessions. Do not expect subjects
to undergo an experimental procedure that you are not willing to sit through yourself. This is
especially important if someone else sets up the experiment according to your specifications. As
the Cold War arms-control negotiators used to say, “Trust, but verify.” The second level of pilot
testing should be to have two or three people complete the experiment. These should be
laboratory assistants, colleagues, or others who might spot potential problems. Especially if
using students as pilot subjects, let them know that reporting anything that seems like a problem
is necessary.
Once the pilot data are collected, perform a full data analysis. Although it isn’t likely that so few
subjects will give the statistical power needed for “significance,” you can satisfy yourself that the
relevant variables are recorded and that you know how to proceed with the analysis. An
important aspect of analysis is to know how to structure the data for the program that is being
used for analysis. Note that most statistical programs will read in a tab-, comma-, or space-
delimited ASCII (or DOS) file, which should have the data for each subject on a single line. With
reaction-time research, it is common to use the mean RT’s for each condition as the data for
analysis, rather than single-trial data. That can be produced using the Analyze feature of the E-
DataAid application within E-Prime.
Formal data collection
Once formal data collection has begun with actual research subjects, it is a good idea to debrief
at least the first few subjects rather extensively when they finish the experiment. Check that they
understood the instructions. Ask whether they noticed anything that seemed unusual, and ask
them about strategies they may have adopted. Subjects sometimes read into an experiment all
sorts of demand characteristics that the experimenter never intended. Do not assume that the
subjects are going to tell about aspects of the experiment that bothered or puzzled them.
Therefore, explicitly ask whether there seemed to be anything “wrong.” Note also that the
colleagues or laboratory assistants who may have served as pilot subjects bring a special
expertise to bear, so they may spot problems a naïve subject would not. However, they may
also, for the very same reason, overlook problems in instructions and procedures that will bother
the naïve subject.
Also review both single-subject and mean data as the first few subjects complete the experiment.
Look for extremes of variability. In a single-trial, reaction time paradigm, look at a table of mean
RT’s by trial type, standard deviations and error rates. Extreme standard deviations or error rates
may indicate that a particular trial type is not being presented as expected, or that subjects are
not reacting as instructions suggested.