Surveys and questionnaires are an excellent tool to capture self-reported behaviors and skills, mental or emotional states, or personality profiles of your respondents.

However, questionnaires are always just momentary snapshots and capture only certain aspects of a person’s behavior, thoughts, and emotions. The underlying principle is that the momentary aspects which are observable (and measurable) are driven by intrinsic factors that are hidden and unconscious to the respondent (and therefore cannot be measured directly).

These hidden factors are generally classified into two categories:

  1. Personality traits (e.g., “extravert”, “introvert”), which are supposed to be stable and consistent across multiple situations
  2. Physiological, emotional or mental states (e.g., “arousal”, “joy”, or “high workload”, respectively), which are supposed to vary depending on the task situation at hand.

A questionnaire that combines traits and states is Dr. Charles D. Spielberger’s “State-Trait Personality Inventory”.

There are a couple of things you should keep in mind when constructing, collecting and analyzing surveys. The 3 crucial aspects include:

1. Operationalize

Of course, you could ask people directly for their traits and states:

  • “Are you a creative person?”
  • “Have you been feeling stressed lately?”

However, due to social expectancies, peer pressure or avoidance tendencies you might not get their “true”, sincere response. Also, physiological states are generally unconscious and might only be verbalized with a certain amount of introspection.

To overcome these issues, break the overarching states and traits (e.g., “creativity”) into observable/measurable aspects that are easy to report, for example:

  • “How much time per week do you spend painting?”
  • “How much money per month do you spend for painting equipment?”
  • “How important is painting to you?”

This approach is generally referred to as “operationalization”. The art of questionnaire design is to operationalize properly, i.e., to identify the appropriate number of questions suitable to extract the maximum amount of information regarding the underlying states and traits of interest. For more information, visit the website of the Harvard University Program on Survey Research.

2. Avoid (double) negatives

I don’t like coffee.” – “Yes/No”.

As a coffee-loving person, which answer should you pick? Processing questions/statements like this takes significantly longer and causes higher amounts of misunderstandings and incorrect outcomes. People might just be puzzled or select the wrong answer.

Double negatives such as “It is not a good idea to not stick to work regulations” result in unreliable data because people are unsure about whether to put a “yes” or “no” – even if it is clear in their minds whether sticking to work regulations is a good idea.

Always try to phrase questions such that negatives are avoided. The initial statement could, for example, be rephrased to:

I dislike coffee.” – “Yes/No”.

Much easier to pick an answer now, isn´t it?


3. Counteract response biases

People tend to answer questions with certain response biases, i.e., we incline to consistently select one answer while avoiding others. Particularly when confronted with linear response scales (e.g., “awful” – “bad” – “so-so” – “good” – “awesome”), some of us show a trend to the extremes, avoiding the center of the scale.

This extreme trend can be counteracted by flipping the direction of the response scale from time to time, e.g., question block A might have the answer categories structured exactly as listed above (from “awful” to “awesome”), while question block B might have the answer categories flipped (from “awesome” to “awful”). On the other hand, people might exhibit a trend to the middle and generally select the center of the scale in order to avoid extremes.

This behavior is apparent quite often in clinical populations or when answering questions that address socially undesirable aspects. For discrete response scales (where only certain responses are possible), this trend can be attenuated by using an even number of responses. For example, one could skip the “so-so” answer above to avoid center trends. You can find more detailed info here.

Stay tuned for more interesting facts and tips worth knowing when it comes to the design of questionnaires. As any questionnaire can only capture a momentary snapshot and certain aspects of a person’s behavior, thoughts and emotions but never the total sum, it is important to repeat measurements to check for consistency, or to enrich the data with additional sources of insight, such as EEG, GSR, eye tracking or behavioral observation.