This course will cover the basic elements of designing and evaluating questionnaires. We will review the process of responding to questions, challenges and options for asking questions about behavioral frequencies, practical techniques for evaluating questions, mode specific questionnaire characteristics, and review methods of standardized and conversational interviewing.
Mode Choice: Implications for Layout
Loading...
From the course by University of Michigan
Questionnaire Design for Social Surveys
234 ratings
University of Michigan
234 ratings
Course 3 of 7 in the Specialization Survey Data Collection and Analytics
From the lesson
Unit 6: Putting It All Together
The questionnaire from start to finish; Things to put at the end; Mode Choice: Implementations for layout; Self-administered questionnaires
Meet the Instructors
Frederick Conrad, Ph.D.Research Professor, Survey Methodology
Institute for Social Research
Frauke Kreuter, Ph.D.Professor, Joint Program in Survey Methodology
Adjunct Research Professor, Institute for Social Research
[BLANK_AUDIO]
Choosing a mode, that can be hard thing.
So I'll talk in this segment a little bit about different data collection
modes, and some of the implications they have for layout of your questionnaire.
The choices you have are plentiful, and it is
good to think of different modes a little bit structurally.
So here, you have a table that shows you
the channel of presentation, whether it's auditory or visual,
the mode of response, whether it's oral, through a keypad, or written,
whether it's computer assisted or paper and pencil,
and whether there's an interview involved in
the administration or whether it's self- administered.
And here on the right, you have all
the different acronyms that methodologist's like to use.
And, you know, by now you have read so many
papers that I'm sure you've come across all these acronyms,
but mapping them back to these features might help you to think through
what, in terms of layout, you have to do for your questionnaire design.
So let's look at the effects that the mode of data collection can have.
Mode is a bundle of features.
Mode affects which sampling frame and which sampling method you can use.
Mode affects how you train and supervise your interviewers.
Mode affects how you contact and recruit your respondents.
And of course, if affects the method of
administration and, with that, the mode of response.
All of these features can affect error sources.
They are related to coverage, sampling, nonresponse, and measurement error.
The last one is the focus in this class,
but, as we showed you at the very beginning, it is really important that
you keep the entire total survey error
framework in mind when you design your questionnaire.
Sometimes you want to include a question in
your questionnaire that is also on your sampling frame
because then you have the chance to compare measurement
error elements, relative to what you have on the frame.
Sometimes it can be helpful to have,
you know, things that you collect during the
recruitment phase also be measured on the questionnaire.
For example, interview observations become increasingly of interest
in responsive design models or when you try to predict nonresponse,
but in order to evaluate the quality of these interview observations, you better
have on the questionnaire the same question
that you asked the interviewer to observe
so that you can later on, at least before the response,
decide on the quality or evaluate the quality of those interview observations.
The channel of presentation has a couple of interesting
pros and cons that we should keep in mind.
If the channel of presentation is auditory, there's no
literacy requirement, and depending on which county you
work in, or if you do a comparison
among race and ethnic groups or different countries,
that might very well affect your overall result, because you'll
automatically lose some cases that do not fit these literacy requirement.
If you are in a visual mode that does require reading.
So, you know, this is a negative part for visual but positive for auditory.
The problem with auditory modes is, though, that working memory
is much more taxed when you listen to question/answer categories.
You will likely see recency effects, and we talked about this
in the segment on ordering questions, order effect, and the like.
So that's something to keep in mind.
If it's auditory, you would probably want to have
the set of answer categories that is listed out
by an interviewer to be much shorter than the
set would be in a visual structure of the questionnaire.
Now, the mode of response has also
implications that are positive or negative.
Oral mode pretty much everyone can do.
Written and keypad requires literacy, and it requires computer skills.
Technology will help, sooner or the later, and, you know,
I wouldn't be surprised if there already some mobile device service
out there, where the respondents can give their answer in
an oral way, and it is automatically transferred into a text.
So, these tables of course get outdated very quickly.
Now what are the pros and cons for computer-assisted
versus paper-pencil in this different modes of data collection.
[BLANK_AUDIO]
Well, computer-assisted is fantastic to get the skip patterns right.
You click a box, you're automatically transferred to another segment in the
questionnaire, and there's no threat of
omission and commission, at least not accidentally.
You also have control over the sequence with which the questions are delivered.
And you have the possibility to randomize question and response options, and
with that mitigate order effects that we have seen throughout the course.
However, in a computer-assisted questionnaire you have no
control of who does the questionnaire, except, if this
were a computer-assisted portion in a survey administered by
an interviewer where the laptop is just turned around.
In a paper and pencil survey, there is a big
advantage from the part of the respondent that they can flip
through the questionnaire and sort of get a feel for how
long this is and whether they really want to do it.
But then, the same time, researchers rely
on the respondent to navigate through the questionnaire.
And they have no control on which
question is answered when, and therefore these things
might be subject to context effects that
were unplanned for or studied by the researcher.
A few more pros and cons for interviewers self-administered questionnaires.
So the first row is interviewer administered questionnaire.
They tend to, well, I'm actually not sure if we can say this so generally,
but they tend to have better data quality because the
interviewer has the chance to motivate the respondent, probe, and clarify.
Now, they also have interviewer effects and it is, I
mean, we've discussed about this in the tentative question section,
considerable social desirability biases when you have sensitive question.
But for a survey that doesn't have sensitive question or where
really motivation is important, there's
nothing like an interviewer administered questionnaire.
Self-administered questionnaires on the other hand, they do have the advantage of
having no interviewer effect and no social desirability bias.
If the interviewer is present and is still self-administered, then the interviewer
can motivate, but we have a reduced
social desirability bias and reduced interviewer effects.
These days, if you look in the lower right corner here, Web surveys tried to
experiment with ways to pick up when the
respondent has a question or might need clarification,
and there's an attempt to have then the
computer automatically react and provide additional explanations and feedback.
Some of the research that Fred is doing will show you that,
and we have extra segments for you to see how this might work.
Now the question is how to choose modes.
One way to determine this is how is the respondent samples reached and contacted,
right?
This might determine which mode you have to use or which mode
you might not want to use depending on your type of population.
So in CATI and IVR survey's respondents have to have
a telephone number, it can be landline or a cell phone.
In personal interviews respondents
must have address to be reached.
In Web surveys they must have internet access, and you must have the respondents'
e-mail addresses or mailing addresses to send them the information for the Web survey.
Now, of course, there are combinations of these modes possible, and we'll come to that,
but you could envision a situation where
you do a television survey for the recruitment,
and then I come back to the respondent
and ask the respondent for their e-mail address.
But nevertheless, they need an e-mail address or you
could tell them a Web page and log in information
over the telephone, require the respondent to have at least internet access.
Which mode is also effected by the sensitivity of the topic?
If you have sensitive topic, then you would prefer a self-administered mode.
If the topics are very easy, you also can do easily a self-administered mode.
Complex questionnaires, long questionnaires
preferably administer in person.
If the topics are very difficult, then
the interview administered mode's are also advisable.
Most important though, budget and the timeframe that you have.
If it's quick and dirty, it should be cheap, then
most people end up with a Web survey.
Let me point out, though, one more thing,
the respondent must have Internet access.
Some surveys do provide the respondent with Internet access
to make sure that they don't lose those that don't.
So there you see, increasingly the field
moves towards a mix-and-match of these modes.
So let's look at examples for these mixed mode design,
the idea of mixing different survey modes in the same survey design.
This could, for example, be mail first and
then follow up with telephone and face-to- face interview.
The US Census Bureau has that in place.
The pros, you maximize response rate, and you reduce the cost.
You can maximize quality and efficiency.
The cons,
well, you might have mode effects.
People react to the mode
of administration, oral or visual presentation,
in different ways, and, likewise, the response option might have an effect.
Also different people are recruited by different modes,
they answer, as I said, differently by the different modes,
and these differences in the answer reflect true
differences, the composition or response errors, you don't know.
So it's good to look at many of these mixed modes
that is out there now, to see what can you build
into your design so that you can at least measure some
of these effects or disentangle them as you have the data.
Here's another question that was posted online.
[BLANK_AUDIO]
And the general answer to these types of questions is,
you should always try to maximize quality in any given mode.
The research is not, has not solidified yet to clear answers,
you know, what is best in which situation and what not.
But keeping it the same across all modes, seemed unfortunate because you end
up with the smallest denominator, instead of maximizing the potential
for each particular survey mode, and getting the
best data quality possible in each of the modes.
In the next segment we'll talk about a few peculiarities of
self-administered questionnaires, also with respect
of maximizing the potential of those.
Explore our Catalog
Join for free and get personalized recommendations, updates and offers.
Coursera provides universal access to the world’s best education,
partnering with top universities and organizations to offer courses online.
© 2018 Coursera Inc. All rights reserved.
