Survey Research

Constructing Surveys
Measuring Responses
Important Considerations for Survey Items
Collecting Survey Data
Evaluating Surveys and Survey Data
Sampling
Probability Sampling
Nonprobability Sampling
 Describe survey research.

Survey research obtains data about opinions,

attitudes, preferences, and behaviors using
questionnaires or interviews.
The survey approach allows researchers to study
private experience, which cannot be directly
observed.
Example:
Student Perspectives on Online Education
Effectiveness

Survey Research
 What are the advantages of the survey approach?

1. We can efficiently collect large amounts of

data.

2. Anonymous surveys can increase the

accuracy of answers to sensitive questions.

3. Surveys can allow us to draw inferences

about the causes of behavior and can
complement laboratory and field experiments.

Survey Research
 What is the most important limitation of the
survey approach?

1.The survey approach does not allow us to

test hypotheses about causal relationships
because we do not manipulate independent
variables and control extraneous variables.

Example:
Exploring the connection between personality
traits and effective self-leadership : A quantitative
study on how personality affects performance

Survey Research
 What are the major steps in
constructing surveys?

1. Identify specific research objectives.

2. Decide on the degree of imposition of units
(degree of response restriction).
3. Decide how you will analyze the survey
data.

Constructing Surveys
Survey Responding as a Psychological Process
Before looking at specific principles of survey construction, it will help to consider survey
responding as a psychological process.

A Cognitive Model
Figure 7.1 presents a model of the cognitive processes that people engage in when
responding to a survey item (Sudman, Bradburn, & Schwarz, 1996)[1]. Respondents must
interpret the question, retrieve relevant information from memory, form a tentative judgment,
convert the tentative judgment into one of the response options provided (e.g., a rating on a
1-to-7 scale), and finally edit their response as necessary.
Consider, for example, this questionnaire item:

How many alcoholic drinks do you consume in a typical

day?
_____ a lot more than average
_____ somewhat more than average
_____ average
_____ somewhat fewer than average
_____ a lot fewer than average
Although this item at first seems straightforward, it poses several
difficulties for respondents. First, they must interpret the question. For
example, they must decide whether “alcoholic drinks” include beer and
wine (as opposed to just hard liquor) and whether a “typical day” is a
typical weekday, typical weekend day, or both. Even though Chang and
Krosnick (2003)[2] found that asking about “typical” behavior has been
shown to be more valid than asking about “past” behavior, their study
compared “typical week” to “past week” and may be different when
considering typical weekdays or weekend days). Once respondents
have interpreted the question, they must retrieve relevant information
from memory to answer it. But what information should they retrieve,
and how should they go about retrieving it? They might think vaguely
about some recent occasions on which they drank alcohol, they might
carefully try to recall and count the number of alcoholic drinks they
consumed last week, or they might retrieve some existing beliefs that
they have about themselves (e.g., “I am not much of a drinker”).
Then they must use this information to arrive at a tentative judgment
about how many alcoholic drinks they consume in a typical day. For
example, this mental calculation might mean dividing the number of
alcoholic drinks they consumed last week by seven to come up with an
average number per day. Then they must format this tentative answer
in terms of the response options actually provided. In this case, the
options pose additional problems of interpretation. For example, what
does “average” mean, and what would count as “somewhat more” than
average? Finally, they must decide whether they want to report the
response they have come up with or whether they want to edit it in
some way. For example, if they believe that they drink a lot more than
average, they might not want to report that for fear of looking bad in the
eyes of the researcher, so instead, they may opt to select the
“somewhat more than average” response option.
 Describe the major question types.

Closed questions (structured questions)

can be answered using a limited number of
alternatives and have a high imposition of units.
For example, “How many songs did your
roommate illegally download this month?”

Constructing Surveys
High Imposition of Units:
Conversely, high imposition of units occurs
when researchers limit the range of possible
responses.
This might involve using closed-ended
questions with specific response options (e.g.,
yes/no, Likert scales, numerical ratings). In
such cases, participants’ choices are
constrained by the provided options.
 Describe the major question types.

Open-ended questions (open questions)

require that participants respond with more than
a yes or 1-10 rating and have a low imposition
of units.
For example, “Why did your choose your major?”

Constructing Surveys
Low Imposition of Units:
Ø When there is low imposition of units, it means
there is minimal restriction on the responses from
participants. Researchers allow subjects to
provide a wide range of answers without imposing
strict limitations.
For example, using open-ended questions in a
survey allows participants to express their thoughts
freely. Responses are not confined to predefined
categories or specific numerical scales.
 How do researchers analyze data from each
question type?

The number or percent of responses can be

reported for closed questions.
Open-ended questions can be analyzed using
content analysis, like Yepez’s INTERSECT, in
which responses are assigned to categories
using objective rules.

Constructing Surveys
 What are three concerns when constructing
questions?

1. Keep items simple and unambiguous, and

avoid double negatives.
2. Avoid double-barreled (compound)
questions that require responses about
two or more unrelated ideas.
3. Use exhaustive response choices.

Constructing Surveys
Example of double negative question:
“Would you not disagree with the statement
that our service is not unsatisfactory?”
Issue: The double negative makes it unclear
whether the respondent agrees or disagrees
with the statement.
A clearer version would be: “Do you find our
service satisfactory?”
Example of double barreled question:
“How satisfied are you with the
quality of our product and the
customer service?”
• Issue: The respondent might be
satisfied with one aspect (e.g.,
product quality) but dissatisfied with
the other (e.g., customer service).
The combined question doesn’t allow
them to express this nuance.
 Describe a nominal scale.

A nominal scale assigns items to two or more

distinct categories that can be named using a
shared feature, but does not measure their
magnitude.
For example, you can sort professors into
exciting and dull categories.

Measuring Responses
Nominal Scale: (Categorization)
Definition: Nominal data assigns names or labels to each data point
without imposing any inherent order or ranking.
Examples:
• City of birth, Gender, Ethnicity, Car brands, Marital status

Characteristics:
ü Categories are mutually exclusive.
ü No meaningful order exists between the categories.
ü You can categorize data, but ranking or measuring intervals is not
possible.
ü Nominal scales are suitable for non-ordered categories.
 Describe an ordinal scale.

An ordinal scale measures the magnitude of

the dependent variable using ranks, but does
not assign precise values.
For example, marathon contestants may finish
from first place to last place.

Measuring Responses
Ordinal Scale: (more on ranking)
Definition: Ordinal data groups data based on a ranking system. It allows
you to categorize and rank data, but the intervals between rankings are not
precisely defined.
Examples:
Top 5 Olympic medallists (ranking without precise intervals)
Language ability (e.g., beginner, intermediate, fluent)
Likert-type questions (e.g., very dissatisfied to very satisfied)
Characteristics:
ü Categories have an order.
ü Intervals between rankings are not uniform.
ü You can compare relative positions but not measure exact differences.
ü Ordinal scales are useful when you need to capture ordered information
without precise measurement.
 Describe an interval scale.

An interval scale measures the magnitude

of the DV using equal intervals between values
with no absolute zero point.
For example, Fahrenheit or Centigrade
temperatures, and Sarnoff and Zimbardo’s
(1961) 0-100 scale.

Measuring Responses
Temperature Example:
A classic example of interval data
is temperature measured in degrees
Fahrenheit. The difference between
100°F and 90°F is the same as the
difference between 60°F and 70°F.
However, there is no true zero
temperature (where there is no heat
at all).
IQ Test Scores example:
Another instance of interval data is IQ
test scores. The intervals between
different IQ scores are equal, but an
IQ of zero does not imply a complete
lack of intelligence.
 Describe a ratio scale.

A ratio scale measures the magnitude of

the dependent variable using equal intervals
between values and an absolute zero.
This scale allows us to state that a 2-meter
board is twice as long as a 1-meter board.
For example, distance in meters.

Measuring Responses
On a ratio scale
üa zero means there’s a total absence
of the variable of interest.
For example,
the number of children in a household
or years of work experience are ratio
variables: A respondent can have no
children in their household or zero
years of work experience.
Examples of ratio scale:
• Height: The height of an individual in
centimeters or inches.
• Weight: The weight of an object or
person in kilograms or pounds.
• Age: The age of individuals in years.
• Money: Monetary values, such as
income or savings.
 How should we select measurement scales?

The best type of scale depends on the

variable you are studying and the level
of precision you desire.
Since psychological variables like traits, attitudes,
and preferences represent a continuous
dimension, several levels
of measurement “fit” equally well.

Measuring Responses
 Remember that continuous dimensions allow for precise
measurements and can be analyzed using statistical techniques
appropriate for continuous data

Temperature: Temperature is a continuous variable because it can take

any value within a range (e.g., from -10°C to 30°C). Researchers can
measure temperature with precision, such as 25.5°C or 28.3°C.
Height: Height is another continuous dimension. People’s heights can vary
continuously, and measurements can be precise (e.g., 165.2 cm or 6 feet 1
inch).
Weight: Weight is also continuous. It can be measured in grams,
kilograms, or pounds, and there are infinite possible values within a given
range.
Time: Time is continuous because it flows without interruption.
Researchers can measure time down to fractions of seconds (e.g., 2.5
seconds or 0.1 milliseconds).
Age: Age is typically measured in years, but it is still continuous. A person’s
age can be 25.3 years or 42.7 years.
 How should we select measurement scales?

When working with variables like sociability,

psychologists often select the highest scale
since it provides more information and allows
analysis using more powerful statistics.

Measuring Responses
 What should you consider when creating survey
items?

Subjects decide to refuse to answer surveys

during the start or first few questions.
Engage subjects from the start by asking
interesting questions they will not mind
answering.

Important Considerations for Survey Items

 What should you consider when creating survey
items?

The first survey question should be:

1. relevant to the survey’s central topic
2. easy to answer
3. interesting
4. answerable by most respondents
5. closed format

Important Considerations for Survey Items

 What should you consider when creating survey
items?

Whenever possible, use commonly used

response options.
Avoid value-laden questions that might
make a response seem embarrassing.

A value-laden question refers to a question that

is influenced by personal opinions or strong
views.

Important Considerations for Survey Items

 What is a response style?

Response styles are tendencies to respond

to questions or test items without regard to their
actual wording.
People differ in their willingness to answer,
position preference, and yea-saying and nay-
saying.

Important Considerations for Survey Items

 Explain the willingness to answer response style.

Willingness to answer is the tendency to

guess or omit items when unsure.

Important Considerations for Survey Items

 Explain the position preference response style.

Position preference is selecting an answer

based on its position.
For example, students choosing “c" on multiple-
choice exams.

Important Considerations for Survey Items

 What is manifest content?

Manifest content is the plain meaning of the

words printed on the page.
While we expect subjects to respond to the
manifest content of questionnaires, they may
ignore it when answering questions about their
feelings or attitudes.

Important Considerations for Survey Items

 What are yea-saying and nay-saying?

Yea-saying is agreeing with an item

regardless of its manifest content.
Nay-saying is disagreeing with an item
regardless of its manifest content.

Important Considerations for Survey Items

 What are context effects?

Context effects are changes in question

interpretation due to their position within a survey.
This problem is especially likely when two
questions are related and not separated by
buffer items (unrelated questions).

Important Considerations for Survey Items

 Explain the social desirability response set.

The social desirability response set is

representing ourselves in a socially appropriate
fashion when responding to a question’s latent
content (underlying meaning).

For example, you may dress formally for a job

interview instead of wearing your favorite jeans.

Collecting Survey Data

 Compare structured and unstructured interviews.

In structured interviews, questions are asked

the same way each time. This provides more
usable, quantifiable data.
In unstructured interviews, the interviewer can
explore interesting topics as they arise. These
data may not be usable for content analysis.

Collecting Survey Data

 What is the relationship of a sample to its
population?

A population consists of all people, animals, or

objects that share at least one characteristic.
A sample is a subset of the population of interest
(the population we are studying).

Sampling
Probability Sampling:
• is a method where each member of the population has a pre-specified and equal chance of being
selected as part of the sample.
• Basis of Selection:
• Randomly: The selection process is random and ensures that every individual has an equal
opportunity to be chosen.
• Research Purpose:
• Probability sampling is commonly used for conclusive research, where the goal is to make statistical
inferences about the entire population based on the surveyed sample.
Methods:
• Simple Random Sampling: Randomly selecting individuals from the population.
• Stratified Sampling: Dividing the population into strata (subgroups) and then randomly selecting from
each stratum.
• Cluster Sampling: Randomly selecting clusters (groups) of individuals.
• Systematic Sampling: Selecting every nth individual from a list.
Result:
• The results obtained through probability sampling are unbiased and can be generalized to the entire
population.
Non-Probability Sampling: is a method where the selection of individuals is not based on
random criteria. Instead, it relies on factors like availability, geographical proximity, or expert
judgment.
Basis of Selection:
Arbitrarily: The researcher chooses individuals subjectively, without a specific probability
attached to each unit.
Research Purpose:
Non-probability sampling is often used for exploratory research or when population
parameters are unknown or individually unidentifiable.
Methods:
Convenience Sampling: Selecting individuals who are easily accessible.
Quota Sampling: Setting quotas for different groups and selecting individuals accordingly.
Judgment or Purposive Sampling: Handpicking individuals based on the researcher’s
judgment.
Result:
Conclusions drawn from non-probability sampling cannot be generalized to the entire
population due to potential bias.
 What are two advantages of probability sampling
over nonprobability sampling?

1. A probability sample is more likely to represent

the population (external validity) than a
nonprobability sample.
2. We know the exact odds of members of the
population being included in our sample. This
tells us whom the sample represents.

Probability Sampling
 Which are the main probability sampling
methods?

The four main probability sampling methods are:

§ simple random sampling
§ systematic random sampling
§ stratified random sampling
§ cluster sampling

Probability Sampling
 Which are the main nonprobability sampling
methods?

The four main nonprobability sampling methods

include:
§ quota sampling
§ convenience sampling
§ purposive sampling
§ snowball sampling

Nonprobability Sampling