This course will cover the basic elements of designing and evaluating questionnaires. We will review the process of responding to questions, challenges and options for asking questions about behavioral frequencies, practical techniques for evaluating questions, mode specific questionnaire characteristics, and review methods of standardized and conversational interviewing.
Concept Specification
Loading...
From the course by University of Michigan
Questionnaire Design for Social Surveys
257 ratings
University of Michigan
257 ratings
Course 3 of 7 in the Specialization Survey Data Collection and Analytics
From the lesson
Introduction and Unit 1: Overview of Standardized Interviewing
Introduction; Different types of questions, Measurement error in questions: Bias and variance; Standardized and conversational interviewing; From specifying a concept to asking questions
Meet the Instructors
Frederick Conrad, Ph.D.Research Professor, Survey Methodology
Institute for Social Research
Frauke Kreuter, Ph.D.Professor, Joint Program in Survey Methodology
Adjunct Research Professor, Institute for Social Research
[BLANK_AUDIO]
So this segment is about concept specification.
Remember we already had segments on
interviewers, standardised interviewing and conversation interviewing.
Pointing out how interviewers can contribute to variance and bias.
And for the rest of the course, we pretty much
will talk about items and how they affect these two elements.
But the first step on the item side, is of course, finding the right item.
And to launch us into this, I want to cite Norbert
Schwarz a former faculty member here at the University of Michigan.
Who stated a common problem, and I'll let you read out this quote here for a moment.
[BLANK_AUDIO]
Now, the point that Norbert makes here is very common.
I've consulted in a number of research settings,
and with people who are ...
who have some vague idea of what they try to measure.
But getting to that in the form of a question can be very, very difficult.
And so that's what we're talking about here in this segment.
So when you plan your survey, the first thing that you
do is you have to think about what is your research question?
And how do you envision the results?
This is key.
You need both of these things in your
head and discuss them with the substantive researcher.
Because otherwise you might be missing crucial items that
belong to you or should belong into your survey.
The second step, once you have that mapped out is to translate
larger concepts that should be measured into
questions, which we will do in a second.
Then you need to check the fit and appraise the question for common pitfalls.
So let's start with the first.
Here's an example.
This is a study that media research was doing or better yet
the Concil of Research Excellence that oversees
some of what Nielsen media research does.
They wanted to know if the households that Nielson
samples when they do their television measures, from which
they don't get any tuning or viewing data, whether they
have patters that are similar or meaningfully different from households
that cooperated with the Nielsen panel.
That is important because if they are missing there
will be biases in the viewing measures. Alright,
so, you can envision in the background here a survey going out
both to the Nielsen participants and those that declined to join that panel,
and finding out their viewing patterns.
You can translate this concept or this research
question into an analysis graph or result graph.
Here you see a chart that compares Intab,
those are the ones that Nielsen was able to do
measures on, to non-intabs
in surveys often called nonrespondents,
and the total measure.
The numbers are entirely made up.
The study is not.
We were here at the University of Michigan, jointly with
the University of Maryland, involved in helping with this research project.
But what was measured here is average hours of TV viewing through the survey.
So that's one measure of TV viewing and the intabs
and non-tabs should later on be compared on that measure.
This is a very simple analysis, if you have more complex
analysis, or even multivariate models, many more variables will be involved.
But picturing how this result should look like will help you in the design.
The next piece that will help you in
the design is translating complex concepts into questions.
So here's an example from de Vaus, Surveys in Social Research.
That shows how a very abstract concept, Social
Capital, can be divided up in measurable components.
So, the suggestion here is to break Social Capital
up in the dimensions that form Social Capital, and for
anyone listening here who is not a social scientist
or sociologist and not familiar with the term Social Capital:
What is meant here is, unlike real capital --money-- or human capital,
which is usually called the knowledge that a person inherits, social capital has to
do with the people around you, that you can draw on for various
things, so it's in a way question of the quantity and quality,
of the relationships around you.
So there you have already the first dimensions
that form this abstract concept of social capital.
Now in this example here, the quality of the relations
are, you know, divided up in two sub-dimensions, trust and reciprocity.
And the structure of the relations,
they are divided into sub-dimensions like
spatial location, size, the type, the structure.
The type itself can then further be sub-divided into formal and informal.
And you can see how this breaks out even further.
Eventually we reach the family and friends
and so on.
If you want to see the full picture, I
refer you to the book that's referenced here.
But, you can see how the dividing this up will help you to eventually reach
a level where you will be able to
formulate a particular question and get meaningful answers.
It won't be very helpful to ask respondents:
"How much social capital do you have?"
They might not know what it is, let alone how to quantify that.
So that's your job as a researcher.
Here are a couple of bigger concepts that are hard to specify and well, you
know, I want to challenge you or that
you know, are common survey problems.
Satisfaction with an employer, satisfaction with the current government.
Class evaluations are in a similar vein.
Satisfaction with a particular course.
A concept like eating healthy, household consumption,
energy consumption, or whatever your topic might be.
Think about the dimension and sub-dimension for a
second that we would need to evaluate satisfaction.
With the course that you take Coursera.
[BLANK_AUDIO]
Such concept specification could look like this.
You have satisfaction with the course as overall abstract concept.
There are two aspects that dominate in a course.
It's the professor, and it's the content of the course.
You know, you might like content.
You might hate the content.
You might love the content.
You might, you know, be less thrilled about the professor.
On the content, you can evaluate and ask respondents, or
yourself, for that matter, if it was the right amount.
If the topic was the right topic, included all the elements that you had hoped for.
If you look at the professor, you can
think about how personal the professor is, or
was the in Coursera courses may be a
little less noticeable than in on site classes.
But certainly.
Your assessment of knowledge that was
conveyed, teaching clarity and many other aspects.
Don't get fooled here by the absence of labels on the different sub-dimensions.
This is only due to the space of the slide here.
You yourself can fill in the relevant aspects.
Once you've done that, you have to, of course, check the fit.
Do you really have everything that you need?
Ask yourself.
Which variables do you want to combine to measure satisfaction?
What are the statistical requirements on those variables?
What kind of scale points would you might need?
Do you need to group them?
Do you need to.
Spread that more out.
You know, those kind of things.
What you do you want to do with afterwards?
But then also, importantly, for what subgroups would you want to report?
I have also been involved in surveys that,
were very much interested in reporting content via gender.
But failed to ask the respondent what, the respondent's gender is so.
On the last minute was that error noticed
and we added back on, into the questionnaire.
Seems such an easy question, very easy to forget.
You also want ask yourself if you want to explain differences between different
demographic groups or other groups and what variables you will use to do so.
If you do want to do that, you have ask them in your questionnaire.
So this will add up a lot of background questions
presumably to the topic of interest that you actually have.
But, be careful that it doesn't get too much.
You want to remove everything that you don't
need because you do want to minimize [UNKNOWN].
And that's why we can't emphasize enough that you have
to have an analysis plan before you start planning your survey.
If you have that, that will really decrease the likelihood
that critical items are left out of the final questionnaire.
It will decrease the likelihood that the questionnaire includes
everything in the kitchen sink and creates excessive response burden.
It will help identify which topics are
most important and need space in the instrument.
All these things are really key into making good questionnaire.
Finally, you want to appraise your questions with common pitfalls.
You want to make sure that you don't trapped into
those and we will have a special segment just with those.
But before that it's important that you do know about the general response process.
And the issues related to response behavior in general.
So thank you.
Explore our Catalog
Join for free and get personalized recommendations, updates and offers.
Coursera provides universal access to the world’s best education,
partnering with top universities and organizations to offer courses online.
© 2019 Coursera Inc. All rights reserved.
