Issues in Questionnaire Design
Survey Research Lab
University of Illinois at Urbana-Champaign
[to cite this: Fumento, T. (nd). Issues in questionnaire design. Evaluation Technical Assistance: Dissemination Series. Champaign, IL: Transition Research Institute. University of Illinois]
Designing a questionnaire is a complicated process. The information contained in this paper outlines briefly some of the major concerns when constructing a questionnaire. This paper is not meant to be a through discussion of questionnaire design. Several references have been included which provide this information. This paper does point out frequently overlooked, yet important concerns for general questionnaire construction, as well as specific items dealing with potentially undesirable behavior and attitude and knowledge assessment. Examples are included for illustration purposes.
Mail or Self-Administered
A thorough discussion of these types of questionnaires may be found in Asking Questions by Seymour Sudman and Bradbury and Mail and Telephone Surveys: The Total Design Method by Don A. Dillman.
A maximum of 12 pages is recommended for mail surveys; however, Sudman and Bradburn recommend only 2-4 pages for surveys of low salience to the population.
If youre sending a cover letter, be sure to use letterhead stationary and have the principal investigator or someone of renown from the funding agency sign it to add prestige.
Your cover and advance letters should include a brief explanation of the study, an explanation of how you obtained the respondents name, why it is important that each "sampled" respondent cooperate, and a short assurance of confidentiality.
Your mailing envelope should include the words "Address Correction Requested." Dont forget to include a stamped self-addressed envelope for the respondent to return the completed form.
It is a good idea to include your name and address on the questionnaire itself, in case the respondent loses the enclosed return envelope.
A few things to remember when planning a mail survey:
When writing questionnaires that respondents will see, it is very important that you present a professional-looking document so that the respondents will know this is a serious research effort. Never send Xerox copies of loose pages held together with a staple in the upper corner. Always have the forms printed in a booklet, either saddle-stapled or glued, and use a heavier paper for the cover. Dont forget to plan for printing time when setting your study schedule. Remember when laying out your questionnaire that it will be printed on both sides of a page and formed like a booklet, so that the final number of pages will be a multiple of 4. This format also allows for double-width layouts, if necessary.
Word questions to be read by a respondent as if you or an interviewer were speaking to the respondent. Dont word them like school exams; remember to insert the work "please" in all instructions (e.g., "PLEASE SPECIFY").
One advantage of face-to-face interviewing is the ability to use show cards. The use of show cards reduces the need to repeat answer choices when there is a series of questions using the same scale (e.g., excellent, good, fair, poor). The cards also provide the proper stimulus for selecting a number in a range from high to low and allow respondents to choose a letter or number for an answer rather than having to verbalize a response to a sensitive or threatening question. When designing a face-to-face instrument, you should look for opportunities to use show cards; keep in mind that you dont want to burden the respondent with too many.
Of course, since respondents see the survey instrument, the same rules apply as in a self-administered form regarding printing and stapling.
When writing questions for a telephone interview, the important thing to remember is that respondents dont see either the interviewer or the form. Therefore, you need to choose your words carefully so that they can be easily understood. Dont present too many concepts for respondents to keep in their heads while you ask a series of questions referring to the concepts. Order the questions so that the format varies from scale-type questions to Yes/No or single-choice answers. You want to avoid boring your respondents or allowing them to get into a response pattern.
Dont skip around from topic to topic. Just as in writing a paper, prepare an outline and group your questions according to topic or subject matter. Write transitional statements between sections or changes in subject matter.
In general, you will begin your questionnaire with a question related to the subject of the research, one that is designed to capture the respondents interest without being threatening. The Can be an open-ended question that encourages respondents to express their thoughts about the subject matter and literally trains them to talk to you. However, it is often better to begin with closed questions that respondents can answer easily so that they can learn that the interview process will be fairly easy for them.
Questions can then be ordered from the least threatening issues to the most threatening. "Threatening" is, of course, a relative term, because questions about sexual practices and respondent incomes are considered "threatening."
In general, demographic questions are asked in the last part of the interview. However, it is sometimes necessary to obtain some demographic information early in the interview in order to determine how to proceed through sections of the questionnaire. It is not an absolute fixed rule that demographics are asked toward the end of the interview; you just need to be aware that they are considered sensitive and intrusive by some respondents. At the same time, of course, they are critical for analysis.
One important rule to follow is never to ask for more personal/demographic information than is required for analysis. If race is never going to be an analysis variable, dont ask the question. The same thing is true for marital status, income, gender, etc. It is not necessary to include all of the demographic questions we have developed (see Appendix A). In fact, it would be a rare survey that should require all of them.
Dos and Donts in Designing Questionnaires
Example 1 (Leads the Respondent): Are you in favor of forcing state, county, and municipal employees to pay union dues to hold their government jobs?
Example 2 (Leads the Respondent): Do you feel that eliminating taxes by law is an effective way to stop the government from picking your pocket every day?
Example 1 (Too General): Did you vote in the primary election?
Example 2 (More Specific): Did you vote in the 1988 Presidential election, the one that took place in the Spring of 1988?
Example 1 (Double Negative): Indicate if you agree or disagree. A chemical used in food production that has a negligible cancer risk should be prohibited even though it delays spoilage, prevents rancidity, or prolongs storage time.
Example 1 (Identify Respondents Position): Are you satisfied or dissatisfied with the prices of meals in your student union?
- Satisfied (Skip to 2)
- Dissatisfied (Skip to 3)
- Makes no difference
Example 2 (Additional Question to Measure Intensity): Are you ______
- Very satisfied
- Somewhat satisfied
- Slightly satisfied
Example 3 (Additional Question to Measure Intensity): Are you _______
- Very dissatisfied
- Somewhat dissatisfied
- Slightly dissatisfied
Bradburn, N.M., Sudman, S., & Associates (1979). Improving Interview Method and Questionnaire Design. San Francisco: Jossey-Bass Publishers.
Converse, J.M., & Presser, S. (1986). Survey Questions: Handcrafting the Standardized Questionnaire. Beverly Hills, CA: Sage Publications.
Payne, S.L. (1951). The Art of Asking Questions. Princeton, NJ: Princeton University Press.
Sudman, S. & Bradburn, N.M. (1983). Asking Questions: A Practical Guide to Questionnaire Design. San Francisco: Jossey-Bass Publishers.
Asking Threatening Questions About Behavior
Example 1: On the average, about how many times did you go to school each week?
Example 1: "Pot" for marijuana
Example 1 (Loaded): Do you think it should be possible for a pregnant woman to obtain a legal abortion if she is married and does not want any more children?
Example 2 (Unloaded): Do you believe that a woman should ever be allowed to have an abortion?
Measuring Attitudes: Formulating Questions
Example 1 (Bad): Do you favor legislation or marijuana for use in private homes but not in public places?
Example 2 (Better): Do you favor or oppose legislation of marijuana for use in private homes?
Example 1 (Bipolar): In the past few days were you
- excited or bored
- joyful or depressed
Example 2 (Unipolar): In the past few days were you
Bored? 1 2
Depressed? 1 2
Example 1 (Bad): In the past two years has your income increased or decreased?
Example 2 (Better): In the past two years has your income increased, decreased, or remained the same?
Measuring Attitudes: Recording Responses
Example 1 (Bad): Interviewer says, "In general, how good is your health?"
Note: Interviewer codes what the respondent says and inevitably makes the decision. For example, "Its not wonderful, but its O.K." This response could be coded by interviewer as "Good" or "Fair."
Example 2 (Better): Would you say your health, in general, is
Example 1 (Bad): 1) Extremely likely 5) Slightly unlikely
2) Very likely 6) Somewhat unlikely
3) Somewhat likely 7) Very unlikely
4) Slightly likely 8) Extremely unlikely
Example 2 (Better): Extremely Likely Extremely Unlikely
1 2 3 4 5 6 7
Example 1 (Paired comparison): Would you prefer to go shopping on a weekday or on the weekend?
Note: See item 6 on ranking.
Questions for Measuring Knowledge
Example 1 (Bad): To what extent do you think (a-e) can be trusted to act honestly while in office?
- George Bush
- Paul Simon
- Charles Dickens
- Jesse Helms
- Ted Kennedy
Example 1 (Open ended): How many hours a day is the store open?
Back to program evaluation technical assistance