Research Methods:

Validity
Objectives:
• To understand what validity means
• To understand the different types of validity and the differences between them
• To understand the differences between validity and reliability
WHY ARE WE LEARNING THIS NOW?
• VALIDITY and RELIABILITY are central to our understanding of the
strengths and weaknesses of research methods.
• We will be encountering these terms from the outset so it is vital that
you have a clear understanding of what these terms mean.

So here we go…
What is it all about?
 VALIDITY
INTERNAL VALIDITY EXTERNAL VALIDITY
Does the test measure Can we generalise our findings to
what we intended to other situations / populations/
measure? contexts
***sub-type known as ecological
validity – can our findings be
generalised to other REAL LIFE (e.g.
experiment environment)***
LEARN THESE DEFINITIONS!!!
INTERNAL VALIDITY- Does the test measure what it intends to
measure- WHY MIGHT THE FOLLOWING EXAMPLES NOT
MEASURE WHAT THEY INTEND TO?
For example:
• A questionnaire was carried out by teachers in school on under 16 year olds
investigating if they smoke, and if they do how much do they smoke.
• Under 16s are likely to LIE- this is called SOCIAL DESIRABILITY BIAS
• An observation was carried out in a factory by the owner to investigate whether
altering the brightness of the lighting affected the work rate of employees. The
employees were told they were being observed.
• If they know they are being watched by the owner they will work as hard as they can-
THE HAWTHORNE EFFECT/ DEMAND CHARACTERISTCS
• An experiment was carried out on temperature and concentration. Participants had
to do a comprehension test in cold conditions, again in moderate conditions and
finally in hot conditions.
• By the time they have done it a THIRD time they may guess what is expected-
DEMAND CHARACTERISTICS
Demand characteristics VS. Hawthorne
effect
PARTICIPANT REACTIVITY:
• Hawthorne effect: changes in participant behaviour due to
the knowledge they are being observed.
• Demand characteristics: the participants' efforts to validate a
researcher's hypotheses.

• *Social desirability bias:*

• A tendency to answer in ways that make them look good in the
eyes of others, regardless of the accuracy of their answers.
Ways to assess internal validity
Face validity:
• the extent to which a test looks like it is measuring what it was
intended to measure.
• For example, whether the questions on a stress questionnaire are
obviously related to the stress.
• Face validity only requires intuitive measurement.
Ways to assess internal validity
Concurrent Validity
• Comparing the findings from a study with previously validated
findings on the same topic.
• For example, a new intelligence test could be administered to a
group, and you can compare the results to their performance on a
well-established pre-existing IQ test.
• Close agreement (i.e. a correlation of +0.80) would indicate the new
test is a valid one.
What are our threats to INTERNAL
VALIDITY?
• A lack of control over EXTRANEOUS VARIABLES (such as some
participants being more able than others/ situational variables such
as temperature or noise)
• An extraneous variable is any variable that you're not investigating
that can potentially affect the dependent variable of your research
study.
• A confounding variable is a type of extraneous variable that not only
affects the dependent variable, but is also related to the independent
variable.
What are our threats to INTERNAL
VALIDITY?
• DEMAND CHARACTERISTICS- what does this mean (mentioned on
last slide)?
• participants' efforts to validate a researcher's hypotheses.
• INVESTIGATOR EFFECTS- what do you THINK this may mean?
• Where a researcher (consciously or unconsciously) acts in a way to
support their prediction. This can be a particular problem when
observing events that can be interpreted in more than one way.
• Such as:…
• Researcher Bias: A researcher can affect the results reported from a
piece of research by interpreting the data in a biased way.
• Non-verbal communication > nodding
What are our threats to INTERNAL
VALIDITY?
DESIGN ISSUES:
• i.e. a lack of control over INDIVIDUAL DIFFERENCES e.g. independent groups design: an
experimental design whereby two groups are exposed to different experimental
conditions (between group comparison)
• ORDER EFFECTS- e.g. repeated measures design: an experimental design where the same
participants take part in each condition of the experiment (within group
comparison)- people get bored, tired/fatigued or practice effect
• Fatigue: a decrease in performance of a task due to repetition
• Practice effect: an improvement in performance on a task due to repetition, for example,
because of familiarity with the task
• *Order effects refer to confounding influences of an experiment where the materials of an
experimental condition are presented in different orders to different subjects. Sequence
effects confound an experiment by varying the sequence of differing experimental conditions
to different subjects.*
• Last lesson I did an observational experiment with you, what design was that? Independent
groups or repeated measures?
Independent VS. dependent variable
• Independent (IV):
• the characteristic of an experiment that is manipulated or changed by
researchers, not by other variables in the experiment
• For example, in an experiment looking at the effects of studying on test scores,
studying would be the independent variable
• Dependent (DV):
• What you measure in the experiment and what is affected during the experiment.
• The dependent variable responds to the independent variable. It is called
dependent because it "depends" on the independent variable.
• SO… The independent variable is the cause. Its value is independent
of other variables in your study. The dependent variable is the effect.
Activity- you need a piece of lined paper

My hypothesis is that the front

row will write quicker than the
other 2 rows in lessons.
• You have ONE MINUTE to write the word BIBLIOGRAPHY over and
over until you are told the time is up. I will then take an average from
the boys and the girls in the class.
Over to you…
• What did you think when you read the hypothesis?
• Did you write your normal speed or did you want to prove a point?
• What is this CALLED in Psychology?
• Does this affect the validity of my study?

• Demand characteristics- participants figure out the aim of the

study and go all out to either prove or disprove the hypothesis so
you are NOT MEASURING WHAT YOU INTEND TO MEASURE!!!
• Therefore, the study lacks INTERNAL VALIDITY
Over to you… what are the threats to
internal validity in these studies?
1. An employer is keen to understand whether her employees like her style of
management or not. She decides to interview each of them one by one and asks
them whether there are happy working at the company.
2. An experiment was carried out on the effect that hydration has on concentration.
Participants were asked to read a difficult book, and every 15 minutes they had to
drink another pint of water. The amount of pages read every 15 minute interval was
recorded.
1. Social desirability bias: as the interviewer is their manager, they are more likely to lie
and tell the manager what they deem is socially desirable in that situation. The better
way to conduct this experiment would be to have a third party interviewer with no prior
relationships to the employees, manager or company.
2. Order effects: Lack of control over individual differences such as, practice effect may
hinder the validity of the study, some people may be faster readers than others. The
longer the experiment lasts the higher the likelihood of participants experiencing
fatigue and experiencing a decrease in their performance. To improve this study by
using an independent measures design, to be able to compare the differences between
Quick recap…
• What is meant by the term INTERNAL VALIDITY?
• Are you measuring what you intend to measure

• What are the threats to INTERNAL VALIDITY?

• Order effects, demand characteristics, investigator effects, lack of
control of extraneous variables, Hawthorne effect etc
EXTERNAL VALIDITY
• EXTERNAL VALIDITY:
• Can I generalise the findings to other situations or populations/ contexts etc (e.g. would this work in other
cultures, situations > realism, contexts, ages, genders).
SUB CATEGORIES!!!
• ECOLOGICAL VALIDITY:
• Can I generalise the findings to real life (e.g. experiment environment > is it realistic, does it simulate real life
circumstances. Does the experience simulate real life?).
• POPULATION VALIDITY:
• Can I generalise the findings to other groups of people (e.g. can I extrapolate the date from this sample size
and generalise it to a larger group, i.e. all women, men, all people etc.)
• TEMPORAL VALIDITY:
• Can I generalise the findings to other points in time/ history (e.g. . If I do a study at 8pm on a Friday, will I see
the same outcomes if I repeated the study at 10am on a Tuesday. Can I say that the same findings would be
apparent in 50 years time. Are results from 40 years ago still being found in 2023)

What are our threats to EXTERNAL VALIDITY? Let us consider the following
Ecological validity
• ECOLOGICAL VALIDITY refers to the ability to generalise research
findings from one setting to other settings e.g. everyday life.
• A common misconception is that lab studies lack ecological validity
and this is not strictly true.
• If the study involves an everyday task (e.g. a memory test that uses
items of a shopping list) it has what is called ‘MUNDANE REALISM’
and therefore can be generalised.
• Mundane realism: the degree to which the materials and procedures
involved in an experiment are similar to events that occur in the real
world.
Temporal validity
• TEMPORAL VALIDITY concerns the ability to generalise the research
beyond the particular time period when the study was carried out.
• For example, Asch’s study of conformity was carried out in a very
conformist period of US history (1950s) and it has been argued that
his findings are not valid in society today. > different ERAS
• Temporal validity also refers to the time of day a study is carried out
e.g., doing a repeated measures study at different time of day may
result in different findings for multiple reasons. This study would lack
temporal validity
What are our threats to EXTERNAL
VALIDITY?
• Not carrying out a study in a realistic environment…
• If we are to use LABORATORY EXPERIMENTS we need to ensure that that they have REALISM.
• This means they should be engaging and appear realistic if we want people to behave naturally
• What type of validity would this lack?
• ECOLOGICAL VALIDITY
• Not using a sample that is representative…
• We need to consider our TARGET POPULATION and our SAMPLING TECHNIQUES- volunteer,
opportunity, random.
• What validity would this lack?
• POPULATION VALIDITY
• Not carrying out a study which is applicable to the times
• Considering the or a time of day a study is conducted
• Understanding that time in which we live in, or a time period of an original study some social
norms, language, technology, attitudes etc., may have changed
• What validity would this lack?
OVER TO YOU… Why might these studies
lack EXTERNAL VALIDITY?
1. A researcher wanted to carry out a study into how hours of revision effects exam
grades. She approached students at NHSG and using a volunteer sampling
technique, she questioned 50 students on their revision programme and their exam
grades.
2. A researcher wanted to investigate how quickly people would help when
someone else was in need. She gained an opportunity sample of people in Croydon
on one Monday at noon, and placed them in a room where a confederate dropped a
pile of papers all over the floor. (STRETCH, THINK ABOUT INTERNAL VALIDITY ALSO)
1. POPULATION VALIDITY: Only those who like to revise would come forward
2. 2. Who would be in Croydon on a Monday at noon? Only at work so the sample
is POPULATION BIASED. Plus, putting them in room at faking an accident means
people would probably not behave as they normally would on the street.
Therefore this study also lacks TEMPORAL and ECOLOGICAL VALIDITY. This
could lead to DEMAND CHARACTERISTICS and/or SOCIAL DESIRABLITY BIASES
Improving validity: Experiments
• Validity can be improved in many ways:
• E.g. using a control group allows the researcher to be able to better assess whether changes in the
DV were due to the effect of the IV
• -i.e. studying the effectiveness of therapy: if the control group who did not have therapy see no
change, whilst the other group did; then the researcher can be confident in saying that therapy
had an impact. IV > DV)
• Standardised procedures minimise the impact participant reactivity and investigator effects may
have on the validity
• Using single-blind and double-blind procedures - single-blind procedures reduces the effect of
demand characteristics. A Double-blind study a third party conducts the investigation WITHOUT
knowing its purpose. This reduces demand characteristics and investigator effects, this improving
validity
• E.g. In a single-blind study, patients do not know which study group they are in (for example
whether they are taking the experimental drug or a placebo). In a double-blind study, neither the
patients nor the researchers/doctors know which study group the patients are in.
Improving validity: Questionnaires
• Many questionnaires and psychological tests incorporate a lie scale
within their questions, this is done to assess the consistency of the
respondents response and to control for the effects of social
desirability bias
• E.g. asking the same question in different ways/format
• Validity may be further enhanced by assuring all respondents data is
anonymous
• This will encourage truthfulness as they can not be identified
Improving validity: Observations
• Observational research can have high ecological validity due to
minimal intervention of the researcher, especially if the observer
remains undetected – covert observations
• Covert observations improve validity as the behaviour from those
being observed is more likely to ne authentic and natural and not
subject to participant reactivity or social desirability bias
• Additionally, behavioural categories that are too broad, overlapping or
ambiguous can negatively impact validity due to subjectivity/
researcher bias
Improving validity: Qualitative Research
• Qualitative methods are usually regarded as having higher ecological validity than
quantitative research, due to it requiring less interpretation due to the depth of information
associated with case studies and interviews (better reflect a participants reality)
• However, researchers may still demonstrate the interpretive validity of their conclusions
(Degree to which the research participants viewpoints, thoughts, intentions are accurately
understood by researcher)
• This is the extent to which a researchers interpretation of events matches that of the
participants
• This can be demonstrated by the coherence of the researchers narrative and inclusion of
direct quotes from the participants within the report
• Validity can be further enhanced via triangulation:
• The use of many different sources as evidence (double checking)
• E.g. analysing data compiled from interviews with friends, family, checking personal diaries,
conducting observations etc. (like cross-referencing)
OVER TO YOU…
• Explain the term ecological validity and illustrate your answer with an
example of a study that lacks external validity (4 marks)

A Psychologist wanted to investigate the extent to which people believe in

ghosts and devised a questionnaire as a way of assessing this. There were 20
questions in total.
•• Explain what is meant by validity. Refer to the investigation above in
your answer (3 marks)

•• Explain TWO ways in which the psychologist could have improved the
validity of the investigation above (2+2 marks)
 Another example of where the difference between VALIDITY and RELIABILITY are
illustrated… what aspect of this information links to RELIABILITY and what aspect
links to VALIDITY?
VALIDITY (internal)= Are you measuring what you intend to measure
RELIABILITY= Consistency
In 1991, a series of weapons inspectors from the United Nations
visited Iraq on a number of occasions to check whether the leader
Saddam Hussain was making nuclear weapons. On every occasion
they said they believed that he was.
As a result, the USA and UK invaded Iraq and Saddam Hussain was
executed. Once he was removed, it turned out there were no
nuclear weapons after all.
The weapons inspectors were RELIABLE in their findings but in the end their
conclusions were not VALID!
they were there to measure whether there were weapons and they did not
measure what they intended to measure!
To finish off… consider this MADE UP
scenario

3
You aren’t measuring what
1 you intend to measure (i.e.
intelligence) BUT the
2
findings will be RELIABLE as
3 the circumference will be
the same every time its
4 measured i.e. consistent.
Validity OR Reliability? (or a bit of both!)
• A curtain fitter turns up to work every day on time…
• RELIABLE
• A curtain fitter uses a tape measure to measure the curtains before fitting
them…
• VALID
• A curtain fitter turns up to work every day on time but generally guesses an
approximate length of the curtains he is fitting…
• RELIABLE BUT NOT VALID
• A curtain fitter goes to work when he feels like it, but he uses a tape measure to
measure the curtains before fitting them.
• NOT RELIABLE BUT VALID
 OBJECTIVES AND OUTCOMES CHECK…
• We need to be able to describe the terms
INTERNAL VALIDITY
• EXTERNAL VALIDITY What do these
terms MEAN?
• RELIABILITY
• We need to identify threats to VALIDITY and
RELIABILITY.
What THREATS to we have
to VALIDITY (internal and
external)? To RELIABILITY?