Do you have what it takes to design a smart experiment?

No worries, we’ve all been there. When you are a newbie to the field and don’t know your way around, it can be challenging to come up with a well-constructed experiment that follows established scientific protocols and is able to generate sound statistical data.
Experimental design (also referred to as DOE = design of experiments) is a rigorous method, regarded as the most valid and unequivocal standard for testing a hypothesis.
Sounds hard to live up to? Luckily, a little practice goes a long way. Once you get a handle on the process, you will be awarded with strikingly good results.
Jump right in with our  quick guide to smart experimental design.

Operationalization

Imagine this: You are into usability testing and would like to examine the performance of a new software interface. In the simplest of terms, your research interest centers on the following question:

“Is the new software interface better than the old one?”

Obviously, this sounds somewhat vague and little tangible – how would you actually test better? To start off, you need to operationalize your research question, that is, to define objective and countable aspects that you can analyze statistically. For example, you could operationalize better as

a. fewer touchpoints
b. less time spent with changing settings
c. faster document generation

Based on your operationalization, you now can phrase your hypothesis, which at bottom is a more technical description of your research question:

“The new software interface will result in

a) fewer touchpoints
b) reduced time spent with changing settings
c) faster document generation.”

Better already.

More formally, you have just defined an independent variable (version of the software interface) as well as three dependent variables (number of touchpoints, time spent on settings page, document generation time).

You could visualize your hypothesis like this:

visualize hypothesis
Now that you have your hypothesis in place and order, you are all set to start designing the actual experiment.

Respondent assignment

Basically, experimental design is all about how to assign respondents to the different experimental conditions in your experiment.

In case of your research question, you could test your hypothesis the following way:

1. Independent Groups. This design principle states that there are separate respondent groups for each level of your independent variable, implying that you need one group of respondents being tested on the old interface, and another group of respondents being tested on the new interface.
Perks:

  • As respondents participate in one condition only, you will not run into sequential effects (caused by practice or fatigue, for example). 

  •  Respondents are less likely to get fed up and drop out as they only take part in one of the tasks.

Downsides:

  • You have to recruit significantly more respondents compared to repeated designs.
  • Any difference between respondent groups might affect the outcomes. If you test the old software interface with elderly respondents whereas the new interface with young respondent only, the results will be biased and most likely prevent a proper conclusion. In fact, any variation in age, gender, cognitive skills or social background (so-called respondent variables) can impact the results (use properly matched respondent groups to overcome this issue).

2. Repeated design. Following this design principle all respondents are exposed to all levels of the independent variable, implying that you test all conditions using the same group of respondents. In case of your research question, all respondents test both the old and new software interface. Repeated designs are quite common in cognitive psychology, neuroscience and biophysiological studies.

Perks:

  • You have to recruit exactly half the number of respondents compared to independent groups. This is relevant whenever it’s hard to find respondents due to certain circumstances.
  • As the same respondents are tested on the various conditions, there can be no difference between the groups with respect to respondent variables.

Downsides:

  • Repeated designs are prone to sequence effects caused by practice, boredom or fatigue. Since respondents perform more than one task (testing old and new software interfaces), they have had some practice and might behave differently for reasons other than the independent variable.

Randomization

You can certainly overcome sequence effects by randomization – just vary the sequence of stimuli or conditions: Respondent A first tests the old software and then the new one while respondent B tests the new software first and then the old one.

At bottom, randomization means that any effects of practice, boredom or fatigue are present in all conditions of the independent variable. With only two conditions to check, this is quite easy to achieve. However, randomization becomes a bit more complex with the number of conditions you aim to test as there are a number of things to consider. If you would like to learn more about this procedure, reach out to our experts at iMotions.

Selecting sensors

Once you have defined independent and dependent variables and decided for an experimental design, you can go ahead and specify the sensors you would like to use.

In case of your research question, you could opt in for eye tracking to measure gaze and fixations while interacting with the software. In addition, GSR would certainly be helpful as it tracks respondents’ emotional arousal. Offering a deep glimpse into emotional valence, the recording of facial expressions could compliment your insights. And you certainly can’t go wrong with EEG – it is an excellent way to peek into people’s minds and assess engagement and attention levels based on the measurement of their brain activity.

Interested to find out more about experimental design? Read our blog post on how to run a biometric research study to make your research endeavor take wings.