Issues in Questionnaire Design

Tobey Fumento

Survey Research Lab

University of Illinois at Urbana-Champaign

[to cite this:  Fumento, T. (nd). Issues in questionnaire design.   Evaluation Technical Assistance:  Dissemination Series.   Champaign, IL:  Transition Research Institute. University of Illinois]

Designing a questionnaire is a complicated process. The information contained in this paper outlines briefly some of the major concerns when constructing a questionnaire. This paper is not meant to be a through discussion of questionnaire design. Several references have been included which provide this information. This paper does point out frequently overlooked, yet important concerns for general questionnaire construction, as well as specific items dealing with potentially undesirable behavior and attitude and knowledge assessment. Examples are included for illustration purposes.

Survey Forms

 

Mail or Self-Administered

A thorough discussion of these types of questionnaires may be found in Asking Questions by Seymour Sudman and Bradbury and Mail and Telephone Surveys: The Total Design Method by Don A. Dillman.

A maximum of 12 pages is recommended for mail surveys; however, Sudman and Bradburn recommend only 2-4 pages for surveys of low salience to the population.

If you’re sending a cover letter, be sure to use letterhead stationary and have the principal investigator or someone of renown from the funding agency sign it to add prestige.

Your cover and advance letters should include a brief explanation of the study, an explanation of how you obtained the respondent’s name, why it is important that each "sampled" respondent cooperate, and a short assurance of confidentiality.

Your mailing envelope should include the words "Address Correction Requested." Don’t forget to include a stamped self-addressed envelope for the respondent to return the completed form.

It is a good idea to include your name and address on the questionnaire itself, in case the respondent loses the enclosed return envelope.

A few things to remember when planning a mail survey:

    1. Avoid mailing during holiday periods.
    2. Don’t have forms arrive on the first of the month.
    3. If mailing to businesses, don’t have forms arrive on the first day of the business week.
    4. If you plan only two questionnaire mailings, you might want a reminder postcard between mailings or a telephone follow-up. Don’t send a postcard if you have planned three questionnaire mailings.
    5. Be sure to allow at least two weeks between mailings.
    6. Change your cover letter for both the second and third mailings.
    7. If possible, change the questionnaire for follow-up mailings and include a new question.

When writing questionnaires that respondents will see, it is very important that you present a professional-looking document so that the respondents will know this is a serious research effort. Never send Xerox copies of loose pages held together with a staple in the upper corner. Always have the forms printed in a booklet, either saddle-stapled or glued, and use a heavier paper for the cover. Don’t forget to plan for printing time when setting your study schedule. Remember when laying out your questionnaire that it will be printed on both sides of a page and formed like a booklet, so that the final number of pages will be a multiple of 4. This format also allows for double-width layouts, if necessary.

Word questions to be read by a respondent as if you or an interviewer were speaking to the respondent. Don’t word them like school exams; remember to insert the work "please" in all instructions (e.g., "PLEASE SPECIFY").

 

Face-to-Face Interview

One advantage of face-to-face interviewing is the ability to use show cards. The use of show cards reduces the need to repeat answer choices when there is a series of questions using the same scale (e.g., excellent, good, fair, poor). The cards also provide the proper stimulus for selecting a number in a range from high to low and allow respondents to choose a letter or number for an answer rather than having to verbalize a response to a sensitive or threatening question. When designing a face-to-face instrument, you should look for opportunities to use show cards; keep in mind that you don’t want to burden the respondent with too many.

Of course, since respondents see the survey instrument, the same rules apply as in a self-administered form regarding printing and stapling.

 

Telephone Interview

When writing questions for a telephone interview, the important thing to remember is that respondents don’t see either the interviewer or the form. Therefore, you need to choose your words carefully so that they can be easily understood. Don’t present too many concepts for respondents to keep in their heads while you ask a series of questions referring to the concepts. Order the questions so that the format varies from scale-type questions to Yes/No or single-choice answers. You want to avoid boring your respondents or allowing them to get into a response pattern.

 

All Forms

Don’t skip around from topic to topic. Just as in writing a paper, prepare an outline and group your questions according to topic or subject matter. Write transitional statements between sections or changes in subject matter.

In general, you will begin your questionnaire with a question related to the subject of the research, one that is designed to capture the respondent’s interest without being threatening. The Can be an open-ended question that encourages respondents to express their thoughts about the subject matter and literally trains them to talk to you. However, it is often better to begin with closed questions that respondents can answer easily so that they can learn that the interview process will be fairly easy for them.

Questions can then be ordered from the least threatening issues to the most threatening. "Threatening" is, of course, a relative term, because questions about sexual practices and respondent incomes are considered "threatening."

In general, demographic questions are asked in the last part of the interview. However, it is sometimes necessary to obtain some demographic information early in the interview in order to determine how to proceed through sections of the questionnaire. It is not an absolute fixed rule that demographics are asked toward the end of the interview; you just need to be aware that they are considered sensitive and intrusive by some respondents. At the same time, of course, they are critical for analysis.

One important rule to follow is never to ask for more personal/demographic information than is required for analysis. If race is never going to be an analysis variable, don’t ask the question. The same thing is true for marital status, income, gender, etc. It is not necessary to include all of the demographic questions we have developed (see Appendix A). In fact, it would be a rare survey that should require all of them.

 

Do’s and Don’ts in Designing Questionnaires

    1. Be sure all answer choices are mutually exclusive.
    2. Check for ambiguity in wording the question – can respondents interpret the meaning of words differently?
    3. Don’t write your questions so that you "lead" the respondent to an obvious response.

      4. Example 1 (Leads the Respondent): Are you in favor of forcing state, county, and municipal employees to pay union dues to hold their government jobs?

      Example 2 (Leads the Respondent): Do you feel that eliminating taxes by law is an effective way to stop the government from picking your pocket every day?

    4. Watch the use of pronouns and articles – know when to use specific vs. nonspecific forms (e.g., "the" vs. "a").
    5. Watch personal pronouns (gender references) – never use only he (his) or she (her) unless you are referring to a specific gender.
    6. Don’t shorten questions that refer to previous questions; repeat the reference.
    7. Don’t let questions about knowledge provide the answer you are seeking (e.g., Are you aware that xyz program exists?). Obviously, the question indicates that it does exist.
    8. Don’t ask respondents to perform arithmetic computations; instead ask for factual information and make the calculations during the analysis.
    9. Write specific rather than general questions and word them so that they communicate uniform meaning.

      10. Example 1 (Too General): Did you vote in the primary election?

      Example 2 (More Specific): Did you vote in the 1988 Presidential election, the one that took place in the Spring of 1988?

    10. Avoid using double negatives – these are especially possible when using Agree/Disagree.

      11. Example 1 (Double Negative): Indicate if you agree or disagree. A chemical used in food production that has a negligible cancer risk should be prohibited even though it delays spoilage, prevents rancidity, or prolongs storage time.

    11. "Don’t know" is generally not offered as an answer choice; however, there may be times when "no opinion" should be offered.
    12. Offering a middle choice makes less difference to those who feel strongly about an issue, but if you offer a middle choice, more people will select it. One solution to the middle choice is to ask two questions – the first to identify the respondent’s position on an issue and the second to measure intensity.

Example 1 (Identify Respondent’s Position): Are you satisfied or dissatisfied with the prices of meals in your student union?

    1. Satisfied (Skip to 2)
    2. Dissatisfied (Skip to 3)
    3. Makes no difference

Example 2 (Additional Question to Measure Intensity): Are you ______

    1. Very satisfied
    2. Somewhat satisfied
    3. Slightly satisfied

Example 3 (Additional Question to Measure Intensity): Are you _______

    1. Very dissatisfied
    2. Somewhat dissatisfied
    3. Slightly dissatisfied
    1. Before the "official" pretest, try out your questionnaire on a few members of the target population.
    2. If there are many changes as a result of the "official" pretest, be sure to conduct a second pretest before beginning main study data collection.

References

Bradburn, N.M., Sudman, S., & Associates (1979). Improving Interview Method and Questionnaire Design. San Francisco: Jossey-Bass Publishers.

Converse, J.M., & Presser, S. (1986). Survey Questions: Handcrafting the Standardized Questionnaire. Beverly Hills, CA: Sage Publications.

Payne, S.L. (1951). The Art of Asking Questions. Princeton, NJ: Princeton University Press.

Sudman, S. & Bradburn, N.M. (1983). Asking Questions: A Practical Guide to Questionnaire Design. San Francisco: Jossey-Bass Publishers.

Asking Threatening Questions About Behavior
  1. Open questions are better than closed questions for obtaining information on frequencies of socially undesirable behavior.

    2. Example 1: On the average, about how many times did you go to school each week?

  2. Long questions are better than short questions for obtaining information on frequencies of socially undesirable behavior.
  3. The use of familiar words may increase the reporting of frequencies of socially undesirable behavior.

    4. Example 1: "Pot" for marijuana

  4. To obtain reports of threatening behavior, use data obtained from informants, if possible.
  5. Consider deliberate loading of the question to reduce both overstatements of socially desirable behavior and understatements of socially undesirable behavior. Do not depend on wording such as "Did you happen to …" to improve reporting of socially undesirable behavior. Such wording may actually increase threat.

    6. Example 1 (Loaded): Do you think it should be possible for a pregnant woman to obtain a legal abortion if she is married and does not want any more children?

    Example 2 (Unloaded): Do you believe that a woman should ever be allowed to have an abortion?

  6. For socially undesirable behavior, it is better, before asking about current behavior, to ask whether the respondent has ever engaged in the behavior. For socially desirable behavior, it is better to ask about current rather than usual behavior.
  7. Embed the threatening topic into a list of more and less threatening topics, to reduce the perceived importance of the topic to the respondent.
  8. Consider alternatives to standard questions, such as randomized response, card sorting, and sealed envelopes.
  9. Consider the use of diaries or asking the panel members questions several times to improve both reliability and validity.
  10. Avoid the use of reliability checks on the same questionnaire, since this will annoy respondents.
  11. Ask questions at the end of the interview to determine how threatening the topics were perceived to be by the respondent.

Measuring Attitudes: Formulating Questions

  1. Make sure that the attitude objects are clearly specified.
  2. Decide on the critical aspects of the attitude to be measured—affective, cognitive, and action. Do not assume that these must necessarily be consistent.
  3. Measure the strength of the attitude by building a strength dimension into the question itself; by asking a separate question or questions about strength; or by asking a series of independent questions, each of which reflects the general attitude.
  4. Avoid double-barreled and one-and-a-half barreled questions that introduce multiple concepts and do not have a single answer. Where possible, separate issues from individuals or sources connected with the issues.

    5. Example 1 (Bad): Do you favor legislation or marijuana for use in private homes but not in public places?

    Example 2 (Better): Do you favor or oppose legislation of marijuana for use in private homes?

  5. Consider the use of separate unipolar items if there is a possibility that a bipolar item might miss independent dimensions.

Example 1 (Bipolar): In the past few days were you…

    1. excited or bored
    2. joyful or depressed

Example 2 (Unipolar): In the past few days were you…

Yes No

Bored? 1 2

Depressed? 1 2

  1. Recognize that the presence or absence of an explicitly stated alternative can have dramatic effects on response. Specification of alternatives will standardize the question for respondents.

    2. Example 1 (Bad): In the past two years has your income increased or decreased?

    Example 2 (Better): In the past two years has your income increased, decreased, or remained the same?

  2. Pretest new attitude questions to determine how they are being interpreted by respondents.
  3. If general and specific attitude questions are related, ask the general question first.
  4. When asking questions of differing degrees of popularity involving the same underlying value, ask the least popular item first.
  5. In attempting to measure changes in attitude over time, ask exactly the same questions in all time periods, if at all possible.

Measuring Attitudes: Recording Responses
  1. Use open questions sparingly, for example, to develop further questions, to explore a topic in depth, and to obtain quotable material. Closed-ended questions are more difficult to construct but easier to analyze and less subject to interviewer and coder variance.
  2. Avoid interviewer field coding, if at all possible. If necessary, it is better to have field coding done by the respondent.

Example 1 (Bad): Interviewer says, "In general, how good is your health?"

    1. Excellent
    2. Good
    3. Fair
    4. Poor

Note: Interviewer codes what the respondent says and inevitably makes the decision. For example, "It’s not wonderful, but its’ O.K." This response could be coded by interviewer as "Good" or "Fair."

Example 2 (Better): Would you say your health, in general, is

    1. Excellent
    2. Good
    3. Fair
    4. Poor
  1. Start with the end of a scale that is least socially desirable. Otherwise, the respondent may choose a socially desirable answer without hearing or reading the entire set of responses.
  2. Do not use rating scales with more than four or five verbal points. For more detailed scales, use numerical scales.

    3. Example 1 (Bad): 1) Extremely likely 5) Slightly unlikely

    2) Very likely 6) Somewhat unlikely

    3) Somewhat likely 7) Very unlikely

    4) Slightly likely 8) Extremely unlikely

    Example 2 (Better): Extremely Likely Extremely Unlikely

    1 2 3 4 5 6 7

  3. Consider the use of analogies such as thermometers, ladders, telephone dials, and clocks for numerical scales with many points.
  4. Ranking of preferences for alternatives can be done only when respondents can see or remember all alternatives. In telephone interviews ranking should be limited to two or three alternatives at a time. In self-administered and face-to-face interviews where cards are used, respondents can rank no more than four or five alternatives. If many alternatives are present, respondents can rank the three most desirable and the three least desirable.
  5. Rankings can be obtained by a series of paired-comparison questions. Respondent fatigue, however, limits the number of alternatives that can be ranked.

    6. Example 1 (Paired comparison): Would you prefer to go shopping on a weekday or on the weekend?

    Note: See item 6 on ranking.

  6. When lists are used, complete information can be obtained only if each item is responded to with a "yes/no," "applies/does not apply," "true for me/not true for me," and the like, rather than with the instruction "Circle as many as apply."
  7. Complex ratings, even in two dimensions, can be accomplished by means of card-sorting procedures.

Questions for Measuring Knowledge

Example 1 (Bad): To what extent do you think (a-e) can be trusted to act honestly while in office?

    1. George Bush
    2. Paul Simon
    3. Charles Dickens
    4. Jesse Helms
    5. Ted Kennedy

Back to program evaluation technical assistance