A Guide To Evaluating Road Safety Education Programs For Young Adults

A guide to evaluating road safety
education programs for young adults

Contents
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1. Identify the goals and objectives of your program. . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Choose the best methods of evaluation for your program. . . . . . . . . . . . . . . . . . . . . 5
3. Choose how you will collect the information . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4. Carry out the evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
5. Report on your findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Appendix A: Evaluation planning template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Appendix B: Case study of a young driver program . . . . . . . . . . . . . . . . . . . . . . . . . 23
Appendix C: Different types of evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Appendix D: Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Appendix E: Examples of useful surveys and questionnaires. . . . . . . . . . . . . . . . . . . 32
Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Department of Transport and Main Roads, A guide to evaluating road safety education programs for young adults, November 2009
Introduction
The purpose of this guide is to assist community organisations, and other providers of road
safety education programs for young novice drivers, to evaluate their programs. It includes
guidance, tools and templates for how to plan, undertake and report on a program evaluation.
Why was the guide developed?

Road safety education seeks to bring about change in knowledge, attitude or skills related to
being safe and making sure others are safe on the road. It can be delivered in many ways. A key
role of road safety education is to raise awareness of road safety as a personally relevant issue.
There is a pressing need for this guide as many road safety education programs have not
been evaluated. In addition, many providers do not have the expertise to self-evaluate their
programs or the resources to employ expert consultants to undertake such a task.
The Department of Transport and Main Roads is concerned that the content of some programs
may not be consistent with ‘best practice’ in road safety education. In particular, some
programs may contain content that research has shown to result in negative road safety
outcomes (for example, skid control training in off-road areas) which have been linked to an
increased crash risk.
The department’s aim in developing this guide is to assist providers of road safety education
programs to review the effectiveness of their existing programs so they can be improved and
refocussed where necessary. This will ensure that their road safety objectives are actually
being achieved. This guide may also assist those developing new programs to build in
evaluation techniques. Evaluation will help to ensure that your program is based on the best
available evidence, works in the way you think it does, and changes what you want to change.
Evaluation results can also help you to seek funding to continue or expand your program.
For more information on best practice in road safety education, it is recommended that you
refer to a recent literature review of best practice, Driver education for senior school students
(Years 10–12) and novice drivers, which was developed by the Department of Transport and
Main Roads in conjunction with the Centre for Accident Research and Road Safety – Queensland
(CARRS-Q). It should be of particular interest to those trying to develop or maintain high quality
road safety education programs.
What is in the guide and how to use it

This guide is designed as a self-help document that assists you to plan and carry out an
evaluation of your program. You can work through it at your own pace.
The guide will help you to:
• identify the goals and objectives of your program
• choose the best methods of evaluation for your program
• choose how to collect the information
• carry out the evaluation
• report on your findings.
1 Department of Transport and Main Roads, A guide to evaluating road safety education programs for young adults, November 2009
Ready to evaluate yet?
Working through the guide and completing the template may help you decide when you are
ready to start the evaluation process. Perhaps you already feel confident that you have the skills
and information needed to undertake an evaluation of your program? Some people may refer
to this document as a ‘buyers guide’ before developing a program or seeking the services of an
evaluator. Common pitfalls to evaluation are also discussed throughout.
The tools identified in this guide are:
• a template to help you plan an evaluation (see Appendix A). Wherever you see the
symbol, that’s when you need to refer to the template. As you will probably want to fill in
the template as you move through the guide, it is best to first print a copy of the template so
you can have it in front of you as you work through this guide. You can then start using the
template by filling in your program’s name and basic details at Step 1. An electronic version
of the template can also be downloaded from www.transport.qld.gov.au/sde.
• a case study (see Appendix B) – the example provided summarises the evaluation process for
the passenger safety component of a road safety education program to improve the safety of
teenagers.
• information about more complex types of evaluation (see Appendix C).
• links to sources of additional information, for example, web resources, relevant research
(see Appendix D).
• examples of survey forms and questionnaires that could be used in program evaluations
(see Appendix E).
• a glossary of key definitions, terms and concepts used in the guide.
This symbol lets you know it’s time to refer to or fill in the template.
Department of Transport and Main Roads, A guide to evaluating road safety education programs for young adults, November 2009 2
The diagram below shows the recommended steps in an evaluation1 and refers you to the
relevant pages where these steps are covered.
Step 1
Identify the program goals and objectives
Page 4
Choose the best method of evaluation

Step 2 • define the objectives of the evaluation
Pages 5–6 • plan your evaluation
• key questions for your evaluation
Choose how you will collect the information

Step 3 • possible data collection methods
Pages 7–11
• ethical considerations
Carry out the evaluation

Step 4 • collect data/information
Pages 12–13 • analyse the data/information
• interpret the results
Step 5 Report on your findings

Pages 14–15
Adapted from Sentinella (2004)
1 Sentinella, J. (2004). Guidelines for evaluating road safety education interventions . In Proceedings of 69th
Road Safety Congress: Protecting Vulnerable Road Users, 1-3 March 2004. Retrieved 20 January 2009 from:
http://www.rospa.com/RoadSafety/conferences/congress2004/proceedings/sentinella.pdf.
1. Identify the goals and objectives
of your program
An evaluation measures the extent to which a program has met its goals and objectives. The first
step is to identify the goals and objectives of your program.
A goal is a general statement about the desired outcome of the program. For example, your goal
may be ‘To improve the safety of young drivers in your area’.
An objective is a measurable outcome of the program that relates to the goal. For example, an
objective of your program may be ‘To reduce the number of young people travelling as passengers
of P-plate drivers at night in your town’.
A program usually will have a single goal, but may have multiple objectives.
Below is an example of a goal and objective for a program which were identified by considering
some key questions. Thinking about these questions will help you to identify the goal and
objective/s of your program.
Key questions to help you identify your program’s goals and

objective/s
To begin with, you need to identify your program’s goal – this is a general statement about what
your program is aiming to achieve. After you have identified your goal, the next step is to identify
your program’s objective/s.
The diagram below explains this process and also poses some key questions for you to consider.
Turn to the second page of your template and fill in this information (Step 1).
Identify your program’s goal.

This should be a statement about what your program was designed to achieve. For example:
‘To reduce the number of crashes involving young people on their way to a party.’
Identify your program’s objective/s.

Use answers to the following questions as a guide.
WHO? (e.g. just young males)
WHAT?
Knowledge (e.g. of graduated licensing scheme laws) Describe as clearly as
Attitude (e.g. toward speeding) possible what your program
was designed to achieve.
Behaviour (e.g. talking on the phone and driving)
The questions relating
to who, what, when and
WHEN? (e.g. driving at night on the way to a party)
where will help prompt you.
WHERE? (e.g. local streets)
Now specify your program’s objective/s as a statement.

‘To reduce the number of young males in my suburb who crash because
they are distracted by talking on a mobile phone while driving.’
Remember that your program may have been designed with more than one objective in mind. If so,
make separate statements about each.
2. Choose the best methods of evaluation
for your program
Choosing which method of evaluation to use depends on:

• the question/s you want to answer about your program
• what information you need to find out
• what resources you have available to carry out the evaluation.
Process evaluation
If, for example, you want to know the answers to such questions as, ‘Are you (the course
provider) doing what you said you would?’, or ‘Is the program reaching the target audience?’,
then the type of evaluation that would provide you with answers to these types of questions is
a process evaluation. This type of evaluation tells you about how valuable the content is and
also how effectively the program was delivered.
Outcome evaluation
However, if you want to know whether the program has really made a difference to participants’
road user behaviour, then you would conduct an outcome evaluation. This type of evaluation
tells you about change by answering questions such as ‘Is the participant less likely to
drink and drive after completing the course?’ or ‘Has the participant’s attitude to speeding
changed?’. Generally, an outcome evaluation will require more resources, such as time and
access to people with statistics knowledge, than a process evaluation.
It is, of course, possible that you want to look at both process and outcome issues in evaluating
your program. Whether you conduct a process or outcome evaluation depends on the
evaluation objectives you would like to focus on. The table below provides examples of the
findings likely to be captured by each evaluation method.
Process evaluation Outcome evaluation

• Did the program reach the intended • Did the program improve the participants’
audience? road safety knowledge?
• What worked, or didn’t work, when you • What did the participants learn after the
ran the program? program?
• Did all participants complete the program • Did the participants’ attitudes to road
or did some drop out? safety change?
• What did it take to keep participants • Did the participants’ perceptions of road
involved in the program? safety behaviours change (such as their
awareness of risks)?
• Did you keep to the agreed timelines?
• Did the participants’ road user behaviour
• What processes helped to keep to the
change?
agreed timelines?
• Was there a decrease in crash rates?
• Were activities delivered as you
described? If not, what changed? • Was there a decrease in injury and
hospitalisation rates?
• How many attended the training program?
Define evaluation goals
The goal of your evaluation will relate to why you are undertaking the evaluation. For example,
this might be:
To gather information to help:
• demonstrate the program was effective in producing behaviour change
• improve the program
• support requests to expand the program
• apply for funding.
Write your evaluation goal in the template (Step 2).
Define evaluation objectives

Identify the objectives of your evaluation, which will also relate to the objectives of your program.
Remember, objectives are specific and measurable. They are clear, realistic, achievable and have
a time-frame included.
To help identify your evaluation objectives, make sure you consider the following questions:
• What information is needed?
• How much of that information can be collected with your current resources?
• How reliable will that information be?
• Does it contribute to your evaluation goal?
• Who do you want to influence by the evaluation (for example, program participants, general
public, project team, stakeholders, management, funding bodies, policy and research
community)?
Now list your evaluation objectives in the template (Step 2).
Different types of evaluation

There are many different ways of collecting evaluation information. The method covered in
this guide is one of the simplest. This is known as a before and after evaluation, where the
knowledge, attitudes and perhaps behavioural intentions of participants are measured before
the program, and then after the program has been completed. For most educational and
awareness programs provided at a local level, this type of evaluation is sufficient. More complex
evaluation approaches are briefly summarised and provided for your information in Appendix C.
Avoiding bias
It is important to avoid bias in selecting your evaluation sample and in conducting your
evaluation. A biased evaluation (for example, asking leading questions) will give a biased
result. The web pages listed in Appendix D provide good advice on evaluation planning that will
minimise bias. Appendix C also provides more information on sampling and avoiding bias.
3. Choose how you will collect the information
Once you have defined your evaluation objectives you will need to decide how you will collect
the information required. This section provides information on some of the most common data
collection methods used in evaluating road safety education programs and their pros and cons.
Possible data collection methods

Questionnaires
A questionnaire is a list of questions which has space for answers and is usually completed by
the respondent using a pen and paper. Different types of questions will be asked depending on
whether you are conducting a process or outcome evaluation.
Process evaluation—Ask participants questions such as: Did you receive the handout? Did you
remember a discussion on the number of passengers allowed? Ask facilitators questions such
as: Did you start the discussion on passenger restrictions? Was this discussion interactive?
Outcome evaluation—Ask participants questions such as: Imagine you are
driving on a typical 60km/hr road, are you more or less likely to drive above Who might provide the
60km/hr after completing the course? information?
• The participant
See Appendix A for other examples.
• The program presenter
Individual interviews • Other stakeholders
(e.g. parents)
This includes one-on-one question and answers and is usually done face-to-face
• An independent observer.
but can be done on the phone. Questions which require more than a ‘yes’ or ‘no’
answer may get more detailed responses—these are called open questions (for
example, what activities encouraged student participation?)*
Discussion groups
This involves a group of four to 12 people who discuss their opinions/answers to questions
asked by a facilitator (using open questions).
Observe behaviour
This involves watching participants’ behaviour (for example, count the number of students at a
school who leave wearing a bicycle helmet).
Observations of the program

This involves watching how the program was delivered. For example, were the participants
actively involved in the sessions? Did the facilitator provide the handout or ask a particular
question? Did the participants then answer the question? How many people turned up?
*Note that responses to open questions require higher level skills in analysis and reporting
than closed questions.
What are the positives and negatives of different approaches?
Positives Negatives
Questionnaires • It's easier to compare • Some small changes in wording
standardised responses and might change answers
analyse the data • Answers to earlier questions might
• Closed questions can be affect responses to later questions
easier to answer (e.g. yes/ • Might not get the full story – does
no, multiple choice) not allow for unusual or unexpected
• Can be anonymous responses
• Can be least expensive to • Open questions can be more difficult
administer to analyse than closed questions
Individual interviews • Don’t require written literacy • Interviewer could be biased or

skills or internet access may not be skilled in this area
• Allow participants to express (e.g. they may be unable to interact
issues in their own words well with people or unable to 'think
on their feet')
• Better chance to ask follow
up questions • Respondents might not be able to
express themselves well verbally
• Can be used if there are
only a small number of • Can be time consuming and,
participants (fewer than therefore, expensive
four) • Require note-taking and/or
recording
• Can be difficult to analyse and
compare responses as individual
responses are not standard
• There could be some bias if not
all program participants are
interviewed
Focus groups • Can be more cost effective • The discussion of the group can be
than interviews influenced by personalities (e.g.
some people might not want to
speak up and others can take over
the discussion). It can sometimes
be difficult to get a group of people
together
• Requires a skilled facilitator
Observations • Ability to view events as they • Information can be difficult to

are actually happening (and categorise and interpret
no need to rely on human • Can influence usual process of
memory) activities
• Extensive notes will be • Can be labour intensive and
available after observations expensive
are complete
Official records • Generally unbiased • May be difficult to access

• May not measure what you want to
measure
• Information may not be up-to-date
• Not always ethical to use official
records
Types of data
Quantitative data
Quantitative data is information that can be counted or expressed in numbers. This data is
generally represented visually in graphs, tables and charts. Examples include:
• questionnaire response – strongly agree, agree, neutral, disagree or strongly disagree
• height or weight
• speed
• number of students who completed the program.
Qualitative data
Qualitative research involves analysis of data that can be observed. Examples include:
• focus groups
• individual interviews
• direct observation
• transcripts
• role plays
• videos.
Designing a questionnaire
There are a number of previously developed questionnaires that can be used to evaluate
your program which are possibly a more cost effective and reliable approach than developing
your own (see Appendix E). However, ultimately the questions you choose to ask participants
depends on your evaluation objective. Below are a few of the important things to consider if you
are developing your own questionnaire.
• Keep the questionnaire concise. You should have a strong reason for including each
question.
• Get feedback. Your questionnaire should be proof-read and easy to understand. You might
get feedback from a family member, colleague etc.
• The response items must fit the question. For example, if your question asks someone to
choose the right answer to a problem, the responses can’t be ‘agree’ or ‘disagree’.
• Make sure every possible answer is included in the response list.
• Keep the pattern of questions in a logical order. That is, keep all the questions on one
topic together (for example, all the demographic questions in one section, all the speeding
questions in another and all the mobile phone questions in another section).
• Be mindful of leading questions. For example, don’t have a sentence about how dangerous
it is to speed and then ask if the participant speeds.
Some resources you might like to use if you are developing your own questionnaire are on the
following page.
Practical assessment, research and evaluation
This link provides some hints for designing questionnaires.
http://pareonline.net/getvn.asp?v=5&n=3
The Great Lakes Epicenter News

This link provides more hints for designing a questionnaire.
http://www.glitc.org/epicenter/publications/Files/news/10-Newsletter%20Spring%202002.pdf
Remember
• When, where and how you ask questions can influence the kind of responses you receive.
• Ensure participants know the survey is confidential so they don’t just try to please you
with their answers – this won’t end up being meaningful.
• If you do a follow-up survey months after the program has finished, you increase your
chances of getting a better idea of any longer term change in attitude or behaviour.
Tips for conducting individual interviews and focus groups

Individual interviews
• Be prepared – have a number of questions ready for the interview.
• Help the participant to feel comfortable – explain the process and let them know that their
comments will be confidential.
• Avoid asking leading questions.
• Listen to the participant and ask further questions based on their responses.
• More information on conducting interviews can be found at
http://www.managementhelp.org/evaluatn/intrview.htm
Focus groups
• Ideal for between four-12 participants.
• If possible have a skilled moderator (leader) conduct the session.
• Make sure you have planned how you will run the focus group – for example, craft the
questions so they flow like a normal conversation.
• Limit the questions to around six, starting broad and then narrowing down your focus.
• If possible, record the focus group session and/or have an observer take notes – especially
taking note of who is making specific comments.
• Ensure you have the right mix of people in the group. For example, are you after both males
and females, older and younger?
• Be objective when dealing with participants and don’t judge them.
• Don’t let one person dominate the discussion.
• Periods of silence are fine as they allow people time to gather their thoughts.
• Choose a quiet room free from distractions, and have participants facing one another,
possibly in a semi-circle.
• Observers need to know what the aim of the focus group is so they can remain focused on
the objectives of the sessions.
Focus group process
• Welcome people to the focus group. Build rapport, and let people know there are
no right or wrong answers. Explain what you are hoping to achieve and that all
information is confidential.
• Ask questions, and let the participants be involved in the direction of the discussions.
• Close the session by giving an overview of the session, major points and thanking the
participants.
• More information on how to conduct focus groups can be found at
http://edis.ifas.ufl.edu/PD036
Tips for using official records

• Information on analysing the data from interviews and focus groups can be found at
http://info.emeraldinsight.com/research/guides/interviews.htm?part=5
Ethical considerations in evaluation

• Gain approval from relevant authorities before commencing your evaluation.
• Make participants aware they are being evaluated – do they know their rights and risks?
• Be sensitive to individuals’ beliefs, culture, language and bias in the questions that you ask,
and the way in which you survey (for example, discrimination based on gender, disability,
ethnicity, etc).
• Report information in confidence so that no individual is ever identified.
• Put safeguards in place for privacy during data collection, analysis and reporting.
• Ensure participants understand how their information will be used.
Please be aware that research conducted in schools must adhere to guidelines designed to
protect young people. The websites below will provide you with further information on these
requirements.
Education Queensland: http://education.qld.gov.au/corporate/research/
Catholic Education: http://www.bne.catholic.edu.au/asp/index.asp?pgid=10730&cid=5258&id=84
Next steps
Having read through considerations in choosing methods for your evaluation—including types
of data collection, the costs and benefits, possible survey tools and ethical considerations—it is
now time to identify the methods you will use.
You are now ready to fill in the tables at Step 3 in the template.
4. Carry out the evaluation
It is now time for you to collect and analyse the data. The steps you need to take will depend on
what type of data collection method you use. If you conduct interviews or discussion groups,
then you will need to type up exactly what was said and then summarise this information.
If you only took notes, then you will need to summarise your notes. However, if you observed
the behaviour of young people or observed the program being delivered and have counted the
number of times something happened, then you will need to calculate totals and averages to
summarise your results.
If you used a questionnaire, you will need to set up a spreadsheet with ‘Participant number’
as the heading of the first column. Assign each person who filled out a questionnaire with a
number, starting with one, through to as many who filled in the questionnaire. It is useful to
write the participant number on the questionnaire in case you need to double-check responses.
Write each question number at the top of each subsequent column. Then type in each person’s
answer on each row. The analysis will involve calculating the average value for items where a
rating is given on a scale, and then calculating the percentage of participants who chose each
option for multiple choice questions. For example, ‘80% of young drivers reported they would
be less likely to carry more than one passenger after completing the program. The percentage
who said this was greater for females, than males (90% versus 70%)’.
Below is an example of how a spreadsheet might be set up.
Participant Speeding in Speeding in Speeding

number 60km 100km offences
Question 1 Question 2 Question 3
The key tasks in carrying out the evaluation are:

• collecting data/information
• analysing the data/information
• interpreting the results.
You can tick these tasks off in the template (Step 4) as you complete them.
Selecting an evaluator?
Anyone can evaluate your program. You might have the skills to conduct the evaluation
yourself, or perhaps someone within your organisation, who was not involved in developing
or delivering the program. However, it is possible that you may need assistance from outside
your organisation to conduct part or all of the evaluation. This could include working with road
safety partners, your local TAFE or university, or perhaps a consultant.
The table below highlights a range of advantages, disadvantages and considerations for both
of these approaches.
Advantages Disadvantages Considerations

Someone inside • May be more cost • Evaluators may bring • How can you report
your organisation effective their own bias that any bias was
• Evaluators may • Evaluators may minimised?
have a greater lack the necessary • Can you clearly show
understanding of evaluation skills that the evaluation
the program was conducted
effectively?
Someone outside • Should be unbiased • Can be expensive • What will the

your organisation • Should bring expert • May take longer evaluators provide in
evaluation skills the final report?
• Should be more • Are the evaluators
credible experts in
evaluation?
• Will it provide value
for money?
5. Report on your findings
It is important to provide a report on your findings so that feedback on the program can be
obtained. It will also increase your understanding of the effectiveness of the program and how
to continually improve outcomes.
What might the report include?

A typical evaluation report is likely to include the components below.
Executive summary A summary of the report, outlining the main points from each section—
or Abstract this often is no longer than one to two pages.
An abstract is a short summary covering only the key points. This is
often less than half a page.
Background or Describes the background to the program (for example, when and
Introduction why the program was developed, how long the program has been
offered, aims and objectives of the program). It may also include a brief
summary of past evaluations and similar programs.
Aims States the objectives of the evaluation, why it was conducted, what is
covered in the evaluation, and the evaluation measures.
Methods Outlines how the program was evaluated—including the design
and methods used to collect data, the materials used (for A good report will be:
example, survey questionnaires), how the sample was drawn • specific to the
(including sample sizes and response rates), and the methods program goals
used to analyse the data • relevant to the
Results Presents the results of the evaluation measures—often target audience
summarised under headings with tables, graphs and diagrams • considered early.
to illustrate the results.
Discussion Includes discussion of the results and the implications. This
might include to what extent the program met its objectives, what
improvements could be made to better reach objectives in the future,
any constraints on research design used or difficulties encountered
and identification of any issues which may have affected the results.
Recommendations Makes recommendations for action (for example, modifications to the
program, further evaluation) linked to the outcomes of the evaluation
as reported.
Appendices Could include copies of the evaluation materials (for example, copies
of survey questionnaires, copy of program outline).
Your report might also include any references that you used in conducting the evaluation or
writing the report (for example, another evaluation on which you based your evaluation).
The following link provides a helpful template for report writing:
http://som.flinders.edu.au/FUSA/SACHRU/PDF/EvalReportTemplate.pdf
An example of a recent James Cook University evaluation report on a road safety
education program conducted in Queensland secondary schools can be found at
www.transport.qld.gov.au/sde.
A guide to evaluating
Department road
of Transport andsafety
Main education programs
Roads, A guide for young
to evaluating safetyDepartment
adults,
road of Transport
education programs and Main
for young Roads,
adults, 20092009
November 14
Identify the key people for whom the evaluation was designed
Below are some examples of the key people who may be interested in your findings.
(Note – your answer should directly relate to the goal of your evaluation.) They might be one of
three groups:
• those who are involved in the activities of the program
(for example, presenters, supporters)
• those who are directly affected by the program (for example, young people, their parents)
• those who might use the evaluation findings (for example, other designers, funders).
Write on the template who your evaluation is aimed at (Step 5).
Identify how you might present your end product

How you present your end product will depend on who the key readers are and what they are
likely to take notice of. The following methods can be used to communicate your findings:
• short paper based—postcards, newsletters, brochures
• electronic—electronic summaries, websites
• in person—workshops, seminars, DVDs
• longer paper-based—briefing documents, formal reports.
Learning from the feedback

It is important to use the evaluation findings to improve your program. Consequently, you might
take each of the following steps:
• Go back to your program goals and remind yourself of what you set out to do. Make the
necessary changes to your program that better align with those goals.
• Revise your program objectives, or make other changes to the way the program is delivered,
or to the program message.
• Seek further feedback from others, either internal or external to your organisation, about
what to include in your program/changes to make.
• Consider establishing ongoing evaluation mechanisms to track how effectively your
program is running.
Even if the evaluation didn’t turn out as expected, the information is always useful in helping
you revise the program. The process is about continual improvement and the evaluation
process then starts again!
Appendix A: Evaluation planning template
How to use this template

This template will help you plan an evaluation of your road safety education program. It is
best to print a copy of the template so that you can have it in front of you as you move through
the guide.
As you work through the template you will see that it is cross-referenced to the pages of the
guide and the appendices that will help you complete each step.
Step 1: Identify the program goals and objectives

Write in your program name
Provide information on your program (for example, target group, how it is

delivered, costs, who delivers the program, history of the program etc).
Following from the description on page 4, it’s time for you to write the goal and objectives of
your program.
Identify your program goal .

To help make it specific, use answers to the following questions to guide you.
Identify your program objectives.

Use answers to the following questions as a guide.
WHO?
WHAT? (it may not be all of these)

Knowledge You might have several
Attitude objectives. For each
Behaviour objective you have to repeat
this thinking process.
WHEN?
WHERE?
Now specify the program objective as a single statement …
Step 2: Choose the best method of evaluation
Define evaluation goals

The goal of your evaluation relates to why you are undertaking the evaluation.
Write in the box below what the goal of your evaluation is (from page 9). For example,
‘To find out how much the safety of participants improved’.
Define evaluation objectives

Identify the objectives of your evaluation (refer to page 4).
The objectives of your evaluation relate to the objectives of your program.
Objectives (you can have a few objectives, keep them clear and simple and linked to
measurable outcomes)
• Example of process evaluation: Understand if the program participants received the
materials at the beginning of the session
• Example of outcome evaluation: Identify if there was a reduction in young people who
travel as passengers of P-plate drivers at night in your town
•
Identify your budget and timeline for each evaluation objective. You might need some more
space for each of the evaluation objectives that you have.
The details to help you fill in this form are on pages 4–12.
19
Step 3: Choose how you will collect the information
PROCESS EVALUATION FORM
OBJECTIVE (1): Write objective 1 in here (you will need a separate sheet for each objective):
Example: ‘Understand if the program participants received the materials at the beginning of the session’
Questions What might you measure and how Data collection considerations Time and resource considerations
(this might include the data source, (this includes a time line, who will collect
length of the survey, simplicity of the data, where the data will be collected
questions, language appropriate etc) from, work hours, budget)
Did the participants Question on survey: Participants to answer one item Short question
receive the Example: Did you receive the fact sheet (the question is part of a bigger survey (this question should take about one minute
materials at the ‘the law and mobiles’ at the start of given at the end of the program) and is included in the survey)
beginning? the session? Costs: facilitator’s time used to collect
survey, person to enter findings into
spreadsheet, person to analyse meaning of
all responses put together
OUTCOME EVALUATION FORM
OBJECTIVE (2): Write objective 2 in here (you will need a separate sheet for each objective):
Example: ‘Examine participants’ change in speeding behaviour’
Questions What might you measure and how Data collection considerations Time and resource considerations
(this might include the data source, (this includes a time line, who will collect
length of the survey, simplicity of the data, where the data will be collected
questions, language appropriate etc) from, work hours, budget)
Did participants Question on survey: Participants to answer one item Short question
reduce the times Example: Imagine you are driving on a (the question is part of a bigger survey (this question should take about one minute
they were speeding typical 60 km/hr road. You are not running given both before the program starts and is included in survey)
after the program late and traffic is light and free flowing. and at the end of the program)
Costs: facilitator’s time used to collect
compared with Please estimate the speed you usually survey, person to enter findings into
before the program? drive in this situation. …..km/hr spreadsheet, person to analyse meaning of
all responses put together
This page gives you another example, with a different objective (again see pages 4–12).
20
Step 4: Carry out the evaluation
Checklist of tasks
Evaluation task Tick off when task completed
Collect data/information
Analyse the data/information
Interpret the results
Step 5: Report on your findings

Decide who your report is for
Write in the box below who your evaluation is for (from page 15). For example, those who
are involved in the activities of the program (such as presenters, or supporters), those who
are directly affected by the program (young people, their parents), those who might use the
evaluation findings (other designers, funding bodies).
Decide what type of reporting methods you will use
• short paper based—postcards, newsletters, brochures,
• electronic—electronic summaries, websites
• in person—workshops, seminars, DVDs
• longer paper-based—briefing documents, formal reports
Prepare the evaluation report

Consider what components you will need in your report:
• Executive summary
• Background or Introduction
• Aims
• Methods
• Results
• Discussion
• Recommendations
• References
• Appendices
• Use the resources listed in Appendix D on report writing to help you.
Consider what the evaluation results mean for your program

• Do the goals and objectives need to be modified?
• Does the content of the program need to be changed?
• Do you need help from other people or organisations to help modify the program?
• Do you need to plan further evaluations?
Appendix B: Case study of a young driver
program
The Skills for Prevention Injury in Youth (SPIY) program, specifically addresses passenger
safety and has been involved in a continuous evaluation process over a number of years.
This case study outlines the methods used to design, review and evaluate the program.
Establishing the need for a young driver program

The first part in the needs
What was the problem? analysis was to answer
‘What is the problem?’. To
The program developers thought that teens who were passengers of unsafe do this official data and local
drivers had an increased chance of being injured. data such as discussions and
surveys can be used.
Gathering the evidence
After examining official statistics such as hospital records, the program
developers found that some teens were being injured as passengers, often quite The official evidence showed
seriously. how widespread the problem
was, including the gender and
Local evidence was collected age of those being injured.
Surveys were used to identify whether this behaviour was a problem in the area
where the program was to be delivered. Surveys were given to a sample of school
students in the local community who were targeted by the program. Questions
The local evidence showed
were asked that gathered information about injuries, the number of passengers
more detail about the
riding with unsafe drivers, and protective steps taken to minimise risk.
situations of injury and
School permission was obtained using the appropriate forms provided by unsafe passenger behaviour.
Education Queensland. The survey was completed individually by students
during class time and collected at the end by the researchers. The exact way this
was done was negotiated with the school.
Focus groups were held to gather more information about passenger behaviour,
driver behaviour, and factors that might influence someone to get into a car
with an unsafe driver. The groups also discussed what young people thought an
unsafe driver was.
The range of questions asked of the students included:
• How do people your age get hurt? Questions that were asked
• What sort of things can happen? Can you think of someone who has been
hurt and give me an example?
• How often does that sort of thing happen?
• What happened just before they got hurt?
• What do you think makes young people take risks by getting into a car with
someone who was drinking?
• What might stop them from taking risks?
The answers provided a more detailed understanding of how young people might get hurt as
a passenger, and information about the kinds of situations in which young people became
passengers of unsafe drivers.
Permission was again sought from the schools to run the focus groups and each
individual student and their parents gave written permission before they were Seek the necessary permission
involved, as per the guidelines of Education Queensland. Students were then
selected and the recorded focus groups were run in a way that was suitable for
the school, including timings.
Permission was also sought to hold focus groups with health and student
wellbeing teachers who might be involved with the program. Sessions were held
at a convenient time for both the school and teachers.
Again, general and specific questions were asked such as:
• What are some of the risky things that young people do that get them injured?
• What was already being done, what had worked and what had not
worked before?
• What kinds of things should we be doing to prevent young people from taking
these sorts of risks?
The second part in the needs
• What are some of the ways to get the message across? analysis was to answer
• What resources need to be included? ‘What has already been
done, what worked and
• What are the appropriate messages for students and how might they
what didn’t work?’.
be delivered?
In terms of delivery, schools have an advantage in that they bring together

a large number of young people in a single space. However, they are not the
only place in which to deliver programs. In the planning of this case study, the
developers looked at whether schools would be an appropriate option and talked
with many Education Queensland staff and teachers. Further information can be
found at http://education.qld.gov.au/eq/. Teachers might not know what has
been demonstrated to work previously, but they can still help you understand
what might be more effective in their schools.
Review of existing material

Scientific literature was examined and the internet used to research other
programs that already existed in the area of passenger behaviour. Information
about other programs designed to change behaviour that had been delivered in
schools were explored to uncover strategies in programs that resulted in positive
behaviour change. The program design team also talked with curriculum staff to
see what had been done and what guidelines existed. It is important to note this
existing material might change from time to time, so it’s always worth checking
to see if you have the latest information before you start developing the program.
Developing the program

An important step was putting
The program was then developed using the information gathered from all the
together information gathered
research as a guide. The team conducted more interviews with teachers to from the needs analysis into a
understand whether material would be deliverable and appropriate for the age well-planned program.
targeted. Finally, facts were checked with experts in the area and materials
were proof-read.
Process evaluation
There were a range of process evaluation methods employed to discover if the program
was delivered as intended. Information was collected from students by surveys and during
discussions. In addition, information was collected from teachers through interviews. An
independent observer also watched a number of sessions to make sure they were being
delivered as intended.
The table below provides suggestions to help you undertake a process evaluation.
What was the key Information

How was it asked?
question? source
Did the students Student Do you remember a lesson on passengers (yes/no)?
receive the questionnaire
message?
Teacher Did you cover the lesson on passengers (most, some,
interview few, none)?
Observer rating Please rate how well you think the activities in the
lesson were delivered (scale provided).
Were the Observer rating Please rate how well you think objectives of the
objectives of the program were met (scale provided).
lesson met?
Was the message Observer rating Please rate how well you think the discussions were
delivered with conducted (scale provided).
discussions?
Teacher What were some of the challenges in participating in
questionnaire the discussions?
Was the program Student Did you learn anything in the lesson on passengers
well received? questionnaire (yes/no)?
and discussion Overall, did you find the program enjoyable, boring,
group interesting (rate on a scale of one to 10)?
What did you learn from the program and how did it
change your behaviour?
Adapted from Buckley and Sheehan (2008). Health Education Research2
2 Buckley, L. and Sheehan, M. (2008). A process evaluation of an injury prevention school-based programme for
adolescents. Health Education Research In press. Retrieved 29 January 2009 from:
http://eprints.qut.edu.au/15320/
Outcome evaluation
To understand whether the program influenced the students’ behaviour, an outcome evaluation
survey was conducted. Students were surveyed before and after they completed the program
and again six months after that. Students at a comparison school where the program was
not run were also surveyed. This allowed differences between schools to be looked at to
understand whether it was the program or the school that made the difference to the students’
behaviour. Also, it allowed behaviour change in the students who completed the program, to
be understood.
Students were asked questions about their injury experience, behaviour and attitudes.
An example of a question from the survey is provided below. Some of these questions are
included in Appendix E under ‘Passenger behaviour and attitudes’, developed by Ulleberg
and Rundmo (2002)3.
How often do you ride as a passenger in a friend’s car?
Never (please circle one number) Very often
1 2 3 4 5 6 7
Report writing
Reports were written with different stakeholders in mind and included commentary on:
• the original goals of the program and how they were met
• challenges faced in meeting the goals
• details of the useful resources available in the community
• how much it cost to deliver the program.
What’s next?
The program continued to be refined with comments from students and teachers and the
observer rankings in mind. Another outcome evaluation will be undertaken to see if the
program reduced the risk of participants being passengers of unsafe drivers.
3 Ulleberg P. and Rundmo,T.(2002) Risk-taking attitudes among young drivers: The psychometric qualities and
dimensionality of an instrument to measure young drivers’ risk-taking attitudes. Scandinavian Journal of
Psychology. Vol. 43, 227-237.
Appendix C: Different types of evaluation
Before and after evaluation

There are many different ways of collecting evaluation information. The method described
in the guide and shown in the case study is one of the simplest. This is known as a before
and after evaluation where the knowledge, attitudes and perhaps behavioural intentions of
people are measured before participation in a program, then again after the program has been
completed. The responses or scores obtained before the program are then compared with those
obtained after the program to look for changes. For example, the results table below shows
pre and post program knowledge scores for Year 11 students who completed a road safety
awareness program. The results show that student knowledge scores were higher after the
program than before, suggesting that attending the program may have improved the students’
road safety knowledge.
Percentage of correct
answers
Knowledge Pre Post

program program
Age group most likely to be killed in road crash (17-24 yr) 74 94

Licence group most likely to have a crash (learner (L), provisional
67 81
(P) or elderly)
Major causes of road death (speed, drink driving, not wearing a
74 87
seatbelt, fatigue)
Permissible blood alcohol concentration (BAC) for 21 year old
82 91
P-licence holder (0.00%)
Most common hours for driver fatigue (10 pm-6 am, 2-4 pm) 37 56
For most road safety education program developers and providers, before and after type
evaluations should be adequate for process and outcome evaluation purposes.
Control groups and evaluation

The program evaluation that produced the table of results shown above had no control group,
so the improvement in knowledge could also have been due to something other than program
attendance. A control group is made up of an equivalent number of people who match the
program participants as closely as possible but who do not take part in the program. However,
they do complete the same before and after evaluation measures. A control group is sometimes
called a comparison group as it is compared to the treatment or experimental group (i.e. the
group taking part in the program or intervention).
The use of a control group allows you to compare the results for people who participated in the
program with similar people, for example, Year 11 students at another secondary school in the
region who did not participate. If a difference in road safety knowledge is found between the
control group and those who attended the program, then participation in the program is likely
to be the factor contributing to the difference.
An evaluation with a sound control group provides better results than one with no control
group. However, evaluations using control groups are usually more expensive to conduct.
Cross-sectional and longitudinal studies
Most evaluations are known as cross-sectional which means that they are conducted at
a particular point in time, such as the before and after evaluation previously described.
Alternatively, some evaluations are longitudinal, where the effects of a program or intervention
are measured over time. An example of a longitudinal study is one which tracks the traffic
convictions and crashes of drivers over a five year period, to compare the patterns of those
who completed or did not complete driver education classes at secondary school. Longitudinal
studies are more expensive and complicated to conduct than cross-sectional studies and
should only be attempted by experienced, professional evaluators.
Other types of evaluation studies

Many professional evaluations in the road safety field (conducted by university researchers,
statisticians or research psychologists) are known as quasi-experimental. This type of
evaluation is an experiment where one group (the experimental group) is compared with
a matched control group as closely as possible with all the known sources of possible
error or bias carefully balanced out. This often involves random allocation of people to the
experimental group and the control group to help balance out bias. These types of studies
are often large, complicated and expensive and should only be attempted by experienced,
professional evaluators.
Some road safety evaluation studies look for correlations, or associations, between groups
of road users without the use of a quasi-experimental approach. For example, a study that
reported a relationship where young male drivers who completed skid-control training had
more crashes in the last five years than those who did not, is correlational. In this instance,
the evaluators merely surveyed a large group of drivers in the community asking them if they
had completed such training and compared the reported crash rates for those who said yes
with those who said no. Correlational studies are not as reliable as true experiments as many
unknown factors could contribute to the reported relationship. However, they can be useful in
identifying patterns for further research. Conducting a good correlational study can be difficult
and should only be attempted by experienced, professional evaluators.
Another type of evaluation is a community based study where the effects of a safety
intervention on a whole community, rather than on individuals, are assessed. For example,
a Canadian study compared injury and fatality rates in isolated communities where first aid
training was provided, to that of similar communities where such training was not provided.
The results showed lower injuries and deaths for the communities where first aid courses had
been completed. Like correlational studies, community based evaluations are not as reliable
as true experiments as many unknown factors could contribute to the reported relationship.
However, they can be useful in identifying patterns for further research. Good community
based studies can be difficult to conduct and should only be attempted by experienced,
professional evaluators.
Sampling and sample size
Choosing who to include in your evaluation is known as sampling. For example, you might
select two secondary schools in your area to include in an evaluation of a Year 11 road safety
awareness program. The Year 11 students who will participate in the program, or be assigned
to a control group, is your sample. A good sample is as closely matched and representative of
the target group as possible. Ideally, the schools chosen should have similar characteristics,
such as a similar number of students, with a similar numbers of boys and girls with similar
demographic characteristics. Choosing a small, all girls private school in an affluent area, to
compare with a large coeducational government school from a less affluent area, would not
represent good sampling.
The sample should also be large enough to provide valid and reliable results. An evaluation
based on 100 students is more reliable than one based only on 10. While there are complicated
formulas for working out minimum sample sizes, a rule of thumb is not to use samples below 50
for evaluation purposes. If in doubt, use the links provided in Appendix D or consult an expert.
Please keep in mind there are likely to be costs associated with consulting an expert such as a
statistician or research psychologist.
Appendix D: Resources
Here are some links to road safety, evaluation and report writing materials that you may
find useful.
National links
Transport Accident Commission (TAC Victoria)
• Guidelines for evaluation
http://www.tac.vic.gov.au/jsp/corporate/homepage/home.jsp
• Guidelines for identifying road safety problems
http://www.tacsafety.com.au/jsp/content/NavigationController.do?areaID=13&tierID=2&
navID=42E71F8E7F00000100849D457A5A9D9A&navLink=null&pageID=1525
• Guidelines for preparing a high quality funding application
navID=AA20BA247F00000100193229D9DB4CB9&navLink=null&pageID=1538
• Report writing
navID=A30882D97F00000100DC524609D052CD&navLink=null&pageID=1592
Department of Planning and Community Development

• A detailed step-by-step guide to conducting evaluations of community projects
http://www.dpcd.vic.gov.au/Web14/dvc/rwpgslib.nsf/GraphicFiles/Evaluation+Step-by-
Step+Guide/$file/Evaluation+Step-by-Step+Guide.pdf
Australasian Evaluation Society

• http://www.aes.asn.au/
South Australian Community Health Research Unity

• How to write a report and example template
http://som.flinders.edu.au/FUSA/SACHRU/PDF/EvalReportTemplate.pdf
Learning Centre, University of NSW

• Report writing: FAQs
http://www.lc.unsw.edu.au/onlib/pdf/report%20.pdf
International links
American Evaluation Association
• http://eval.org/resources.asp
W.K. Kellogg Foundation, U.S.A.

• Summary information and links about an evaluation plan
http://www.wkkf.org/Default.aspx?tabid=90&CID=281&ItemID=2810015&NID=2820015&
LanguageID=0
• Report writing
http://www.wkkf.org/Default.aspx?tabid=90&CID=281&ItemID=2810021&NID=2820021&
LanguageID=0
Royal Society for Prevention of Accidents, U.K. (RoSPA)

• Guidelines for evaluating road safety education interventions
http://www.rospa.com/
The Community Toolbox, Kansas University, U.S.A.

• Developing an evaluation plan
http://ctb.ku.edu/tools//sub_section_main_1352.htm
• Evaluating findings
http://ctb.ku.edu/tools//sub_section_main_1048.htm
• Report writing
http://www.cdc.gov/mmwr/preview/mmwrhtml/ss5302a1.htm
Appendix E: Examples of useful surveys
This appendix contains examples of useful surveys and questionnaires that have been used
in road safety evaluations in Australia and overseas. They are arranged into groupings as
shown in the list below. You may find it easier to use some of these in your evaluation than to
develop new surveys of your own. The surveys listed below have been developed and used by
experts as valid and reliable measurement tools. In short, they measure what they claim to,
and do so consistently.
The example surveys cover:
• demographics and background
• driver risk taking behaviour
• managing risk
• perceptions of driving and likelihood of detection
• exposure
• crash involvement and offence history
• knowledge
• passenger behaviour and attitudes
• intermediate factors: attitudes and intentions
• alternative travel modes.
Additionally, it is advisable to include some instruction for the survey respondents to follow,
such as the examples provided below.
Example instructions
When answering all the questions it is important to remember that:
• there are questions on both sides of the page
• carefully read the directions for each question
• please answer carefully and honestly
• most questions can be answered by circling a number
• for a few questions you might have to write your answer on the line provided
• please ask if you have any questions
• do not write your name on the questionnaire.
Demographics and background
1. What is your current age? (please write number of years)
years
2. Gender (please circle one number)
Male 1
Female 2
3. Licence type (please circle one number)
No licence 1
Learner 2
Open 3
Provisional (prior to 2007 system) 4
P2 provisional 5
P1 provisional 6
Probationary 7
Restricted 8
4. Generally, where do you do most of your driving? (please circle one number)
Only city/suburban roads 1
Mainly city/suburban roads 2
City/suburban roads and country roads equally 3
Mainly country roads 4
Only country roads 5
5. The car you usually drive is: (please enter details)
Make
Model
Year
Driver risk-taking behaviour
Driver Behaviour Questionnaire (DBQ)
The DBQ is a short, self-administered, multiple-choice questionnaire originally developed in
the United Kingdom as part of research into driver behaviour and road crashes. It is also known
as the Manchester DBQ which acknowledges where it was first developed.
The original DBQ, was developed by Dr James Reason and colleagues in 19904. Dr Reason is a
research psychologist well known in the human factors and accident fields. The DBQ has been
used extensively in driver research around the world5.
No one is perfect. Even the best drivers make mistakes, do foolish things, or bend the rules at some
time or another. Some of these behaviours are trivial, but some are potentially dangerous. For each item
below you are asked to indicate how often, if at all, this kind of thing has happened to you.
(please circle one number for each)
Nearly all the time

Base your judgments on what you remember of your driving over,
Occasionally
Hardly ever
Quite often
say, the last year.
Frequently
Never
Attempt to overtake someone who you hadn't noticed to be signaling a

0 1 2 3 4 5
right turn
Stay in a lane that you know will be closed ahead, until the last minute 0 1 2 3 4 5
Miss 'stop' or 'give way' signs and narrowly avoid colliding with traffic
0 1 2 3 4 5
having right of way
Pull out of a junction so far that the driver with right of way has to stop
0 1 2 3 4 5
and let you out
Fail to notice that pedestrians are crossing when turning into a side street
0 1 2 3 4 5
from a main road
Drive especially close to the car in front as a signal to its driver to go

0 1 2 3 4 5
faster or get out of the way
Sound your horn to indicate your annoyance to another driver 0 1 2 3 4 5
Queuing to turn left onto a main road, you pay such close attention to the
0 1 2 3 4 5
mainstream of traffic that you nearly hit the car in front
Cross a junction knowing that the traffic lights have already turned
0 1 2 3 4 5
against you
4 Reason, J., Manstead, A., Stradling, S., Baxter, J., and Campbell, K. (1990). Errors and violations: a real
distinction? Ergonomics, 33, 1315-1332.
5 Lajunen, T., and Summala, H. (2003). Can we trust self-reports of driving? Effects of impression management
on driver behaviour questionnaire responses. Transportation Research, Part F, 6, 97-107.
On turning left, nearly hit a cyclist who has come up on your inside 0 1 2 3 4 5
Disregard the speed limit on the motorway 0 1 2 3 4 5
Fail to check your rear-view mirror before pulling out, changing lanes, etc 0 1 2 3 4 5
Become angered by a certain type of driver and indicate your hostility by

0 1 2 3 4 5
whatever means you can
Become impatient with a slow driver in an outer lane and overtake them on the
0 1 2 3 4 5
inside
Race away from the traffic lights with the intention of beating the driver next to
0 1 2 3 4 5
you
Brake too quickly on a slippery road, or steer the wrong way and skid 0 1 2 3 4 5
Drive the car even though you suspect you may be over the legal blood-alcohol
0 1 2 3 4 5
limit
Become angered by another driver and give chase with the intention of giving
0 1 2 3 4 5
him/her a piece of your mind
Below are some questions related to speeding previously used by researchers at the Centre for Accident
Research and Road Safety – Queensland (CARRS-Q), Queensland University of Technology6.
Imagine you are driving on a typical 60 km/h road. You are not running late and
traffic is light and free flowing. Please estimate the speed you usually drive in this
situation. km/h
Imagine you are driving on a typical 100 km/h road. You are not running late and
traffic is light and free flowing. Please estimate the speed you usually drive in this
situation. km/h
Below is an example question related to drink driving
Have you driven after drinking when you were over the legal limit for your licence Yes
type in the last 12 months? No
Below is an example question related to drug driving
Have you driven while under the influence of one or more recreational drugs in Yes
the last 12 months (for example, marijuana, speed, meth/amphetamine, ecstasy,
cocaine, heroin)? No
6 Fleiter, Judy J. and Watson, Barry C. (2005) The speed paradox: the misalignment between driver attitudes
and speeding behaviour. In: Australasian Road Safety Research, Policing and Education Conference, 14-16 November,
Wellington, New Zealand.
Below are some questions related to aggressive driving from the driver anger scale7 used by
researchers at the Centre for Accident Research and Road Safety – Queensland (CARRS-Q), Queensland
University of Technology.
Extremely angry
Not at all angry
A little angry
Fairly angry
Very angry
Explain how you would respond to the following situations
Someone in front of you does not move off straight away when the traffic lights turn green 1 2 3 4 5
Someone is driving too fast for the road conditions 1 2 3 4 5
A pedestrian walks slowly across the middle of the street, slowing you down 1 2 3 4 5
Someone is driving too slowly in the outside lane, and is holding up traffic 1 2 3 4 5
Someone is driving very close to your rear bumper 1 2 3 4 5
Someone is weaving in and out of the traffic 1 2 3 4 5
Someone cuts in right in front of you on the motorway 1 2 3 4 5
Someone is driving more slowly than is reasonable for the traffic flow 1 2 3 4 5
A slow vehicle on a winding road will not pull over and let people pass 1 2 3 4 5
You see a police car watching traffic from a hidden position 1 2 3 4 5
Someone backs out right in front of you without looking 1 2 3 4 5
Someone runs a red light or a stop sign 1 2 3 4 5
Someone beeps their horn at you and your driving 1 2 3 4 5
Someone coming towards you does not dim their headlights at night 1 2 3 4 5
At night, someone is driving right behind you with bright lights on 1 2 3 4 5
You spot a speed camera site ahead 1 2 3 4 5
Someone is slow in parking and holds up traffic 1 2 3 4 5
Someone speeds up as you try and pass them 1 2 3 4 5
You are stuck in a traffic jam 1 2 3 4 5
Someone pulls out right in front of you when there is no one behind you 1 2 3 4 5
Someone makes an obscene gesture towards you about your driving 1 2 3 4 5
A police car is driving in traffic close to you 1 2 3 4 5
Someone is driving well above the speed limit 1 2 3 4 5
Someone shouts at you about your driving 1 2 3 4 5
A cyclist is riding in the middle of the lane, slowing traffic 1 2 3 4 5
A police officer pulls you over 1 2 3 4 5
7 O’Brien, S. Tay, R. and Watson, B. (2003). ‘An exploration of Australian driver anger.’ In Australasian Road Safety
Research, Policing and Education Conference, 24-26 September, Sydney.
Managing risk
Have you personally done anything in the last 12 months to reduce
(please circle one number)
your chances of being injured in a car crash?
Yes 1
No 2
Don’t know 3
If yes, what have you done? (please circle one number)
Took driving lessons 1
Had parent give extra driving practice/help 2
Not driven at night 3
Not carried passengers of the same age 4
Bought a safer car 5
Went on a first aid course 6
Others (specify) 7
Perceptions of driving and detection

Below is a measure called the driver skill inventory, developed by researchers in Finland8.
It measures perceived driving skill.

Please estimate how skillful you are in each of the following
aspects of driving, using the following scale. Below average Above average
Fluent driving (management of your car in heavy traffic) 0 1 2 3 4
Performance in a critical situation 0 1 2 3 4
Perceiving hazards in traffic 0 1 2 3 4
Driving in a strange city 0 1 2 3 4
Paying attention to pedestrians and bicyclists 0 1 2 3 4
Driving on a slippery road 0 1 2 3 4
Conforming to the traffic rules 0 1 2 3 4
Managing the car through a slide 0 1 2 3 4
Previewing traffic situations ahead 0 1 2 3 4
Driving carefully 0 1 2 3 4
Control of the traffic situations 0 1 2 3 4
Fluent lane-changing in heavy traffic 0 1 2 3 4
Fast reactions 0 1 2 3 4
Making firm decisions 0 1 2 3 4
Paying attention to other road users 0 1 2 3 4
Driving fast if necessary 0 1 2 3 4
Driving in the dark 0 1 2 3 4
Controlling the vehicle 0 1 2 3 4
Avoiding competition in traffic 0 1 2 3 4
Keeping sufficient following distance 0 1 2 3 4
Adjusting your speed for the conditions 0 1 2 3 4
Overtaking 0 1 2 3 4
‘Relinquishing’ legitimate rights when necessary 0 1 2 3 4
Conforming to the speed limits 0 1 2 3 4
Avoiding unnecessary risks 0 1 2 3 4
Tolerating other drivers’ blunders calmly 0 1 2 3 4
Obeying the traffic lights carefully 0 1 2 3 4
Below is a measure developed by Australian researchers9. It measures perceptions of getting caught.
Please estimate the chance of the following things (please circle one number for each)
happening some time while you are driving in the

next two weeks. Very unlikely Very likely
Seeing a speed camera 1 2 3 4 5 6 7
Being stopped for a breath test 1 2 3 4 5 6 7
Having my speed checked by the police 1 2 3 4 5 6 7
Being stopped for a random drug test 1 2 3 4 5 6 7
8 Lajunen, T, and Summala H. (1997). Effects of driving experience, personality, and driver’s skill safety orientation
on speed regulation and accidents. In T. Rothengatter and E. Carbonell Vaya (Eds.), Traffic and transport
psychology: Theory and application (pp. 283–294). Amsterdam: Pergamon
9 Senserrick, T and Swinburne, G. (2001) Evaluation of an insight driver-training program for young drivers. Monash
University Accident Research Centre: Melbourne.
Exposure
Below are some questions about exposure to driving.
On average, how many hours per week do you drive?
hours per week (write number)
How long have you held a licence (not including learner licence)?
years months
Doesn’t apply (have learner licence/no licence)
On average, how many hours per week do you drive at night?
On average, how many hours per week do you drive with a passenger about your age?
Crash involvement and offence history
Below is a measure developed by Fleiter and Watson6. It measures previous crash involvement
and offences.
(write number)
How many speeding tickets you have received in the past three years?
How many warnings for speeding (but no ticket) you have received in past
three years?
How many crashes (whether you were at fault or not) have you been in while
driving in the past three years?
Road rules knowledge
The Department of Transport and Main Roads provides an online selection of road rule
questions. These questions can help people prepare for their learner licence test.
Go to: https://www.service.transport.qld.gov.au/rrtexternal/SelectExam.jsp
Passenger behaviour and attitudes

Below are measures developed by researchers in Scandinavia3. They measure attitudes
towards being a passenger and passenger behaviour.
These items refer to riding with a friend who you most frequently catch a ride with.
How often do you ride as a passenger in a friend’s car?

Never Very often
1 2 3 4 5 6 7
How much stress did you feel as a passenger in that friend’s car?
No Stress Very much stress
1 2 3 4 5 6 7
How often did this friend take the following risks in traffic?
Never Very often
Speeding 1 2 3 4 5 6 7
Dangerous overtaking 1 2 3 4 5 6 7
Close following 1 2 3 4 5 6 7
Running red lights 1 2 3 4 5 6 7
Running yellow lights 1 2 3 4 5 6 7
How often did you address your friend’s driving when they were doing each of the
following?
Never Seldom Sometimes Often Very often
Speeding 1 2 3 4 5
Dangerous overtaking 1 2 3 4 5
How often did you refrain from addressing your friend’s driving when they were doing
each of the following?
Never Seldom Sometimes Often Very often
Speeding 1 2 3 4 5
Dangerous overtaking 1 2 3 4 5
Please respond to how much you agree with each of the following.
Strongly disagree Strongly agree

It is only wishful thinking to believe that one can
1 2 3 4 5
influence others to drive more slowly
More and more, I feel helpless to prevent reckless

1 2 3 4 5
driving
There is very little I can do to prevent others from

1 2 3 4 5
driving recklessly
It cannot be my duty to influence how others drive 1 2 3 4 5
I might get into the car with friends who I know are
1 2 3 4 5
unsafe drivers
I would get into the car with a reckless driver if I had

1 2 3 4 5
no other way to get home
I might get in the car with an unsafe driver if my

1 2 3 4 5
friends did
I would rather walk a hundred miles than get into a

1 2 3 4 5
car with an unsafe driver
A driver who is speeding is a more attractive person

1 2 3 4 5
than a driver who always follow the rules
I would be very unpopular if I asked the person I

1 2 3 4 5
was driving with to drive more carefully
Boys prefer girls who dare to get into a car when

1 2 3 4 5
they are speeding
If I should ask my friends to drive more carefully, it

1 2 3 4 5
would be perceived as an unnecessary hassle
How often do you experience the following?

Never Very often
Feeling unsafe of being hurt in a road crash 1 2 3 4 5 6 7
Feeling worried and concerned of being

1 2 3 4 5 6 7
hurt in a road crash
Intermediate factors: attitudes and intentions
Below is a measure developed by Australian researchers6. It measures intended
speeding behaviour.
How fast do you intend to drive in the next month, if running late or not?
When not running late When running late

(Circle one option per line (Circle one option per line
in this section) in this section)
Just occasionally
Just occasionally
Most occasions
Most occasions
Nearly always
Nearly always
Sometimes
Sometimes
Always
Always
Never
In the next month, when driving on urban roads (50 and 60 km/hr), how often will you: Never
Exceed the speed limit by less

1 2 3 4 5 6 1 2 3 4 5 6
than 10 km/hr?
Drive 10-20 km/hr over the

1 2 3 4 5 6 1 2 3 4 5 6
speed limit?
Drive more than 20 km/hr over

1 2 3 4 5 6 1 2 3 4 5 6
the speed limit?
In the next month, when driving on open roads (100 and 110 km/hr),how often will you:
Exceed the speed limit by less

1 2 3 4 5 6 1 2 3 4 5 6
than 10 km/hr?
Drive at 10-20 km/hr over the

1 2 3 4 5 6 1 2 3 4 5 6
speed limit?
Drive more than 20 km/hr over

1 2 3 4 5 6 1 2 3 4 5 6
the speed limit?
Alternative travel mode
Below is a measure developed by Dr Barry Watson, Centre for Accident Research and Road
Safety – Queensland (CARRS-Q), Queensland University of Technology. It measures attitudes
towards alternative transport options to driving.
How much do you agree or disagree with each statement?

Strongly disagree Strongly agree
You find it possible to do most things by
1 2 3 4 5 6 7
using public transport
You can generally get a lift from family or

1 2 3 4 5 6 7
friends when you need one
There is not much public transport available

1 2 3 4 5 6 7
in the area where you live
You can’t always rely on your family or

1 2 3 4 5 6 7
friends for lifts
You could get by without driving if you really

1 2 3 4 5 6 7
had to
Below is a measure developed by British researchers10. It measures use of transport types.
How often within the past two years have you used each of the following types of
transport for any kind of journey?
More than About once About once Several About once
Never
once a week a week a month times a year a year
Car 1 2 3 4 5 6
Bus 1 2 3 4 5 6
Taxi 1 2 3 4 5 6
Train 1 2 3 4 5 6
Bicycle 1 2 3 4 5 6
Walking* 1 2 3 4 5 6
*at least 10 minutes
10 S.Anderson and S.G.Stradling (2004). Attitudes towards car use and modal shift in Scotland. Scottish Executive
Social Research.
Glossary
Here is a summary of some key definitions, terms and concepts used in this guide. Some of
these may also be explained in the main document.
A predisposition or tendency to respond positively or negatively towards

Attitude ideas or concepts.
A way of doing something that has been shown to be successful in

Best practice producing measurable improvement in areas such as cost, quality,
performance or safety
A leaning or prejudice towards a position or conclusion. In evaluations, it

Bias can also relate to samples of participants that are not representative of
the target population.
The people selected and treated the same way as those participating
Control group in a program or intervention but who do not complete the program or
participate in the intervention (also known as a comparison group).
A description of the features of a population or people within

Demographics a population.
A system of moral principles, rules and standards of conduct, to ensure

Ethics that people are not abused, mistreated or unfairly taken advantage of.
Most government and professional bodies have codes of ethical conduct.
The measurement of the extent to which a program has met its goals
Evaluation and objectives.
Being subjected to an action or an influence—in road safety people

Exposure could be exposed to fatigue, drink driving, or speeding behaviour.
Sometimes called exposure-to-risk.
A general statement about the desired outcome of the program.

Goal For example, your goal may be ‘To improve the safety of young drivers
in your area’.
A measurable outcome of the program that relates to the goal.

For example, an objective of your program may be ‘to reduce the number
Objective of young people travelling as passengers of P-plate drivers at night in
your town’.
An evaluation that tells you whether the program has influenced or

Outcome changed the participant’s road user behaviour or knowledge. This type
evaluation of evaluation tells you about change rather than about the process of
delivering the program.
The process by which people interpret and organise what they hear,
Perception see and feel to produce a meaningful experience of the world.
An organised sequence or series of presentations that includes
resources to assist with the planning, delivery, support and evaluation
Program of a program. This may include session plans, audio visual resources,
booklets and guidance for presenters. A training course is an example of
a program.
An evaluation that tells you about the value of a program’s content to

Process participants and also how effectively the program was delivered.
evaluation It does not tell you about changes in knowledge or behaviour resulting
from the program.
Information that can be observed, and described, but not counted or

Qualitative
expressed in numbers. Examples include information gathered from
data interviews and focus groups.
Information that can be counted or expressed in numbers. It is generally

Quantitative
shown visually in graphs, tables and charts. Speed is an example of
data quantitative data.
The ability of a test or survey to measure variables and produce the

Reliability same results when used under the same conditions.
Materials such as booklets, DVDs or kits that can be used within

Resources programs to provide guidance to program developers and presenters
or assist the learning of participants.
Sample size The number of people selected to participate in an evaluation.
Sampling The process of selecting the participants of an evaluation.
Treatment People who complete a program or participate in an intervention

group (also known as an experimental group).
The ability of a test or survey to measure what is intended (for example

Validity knowledge, attitude, behaviour).
A guide to evaluating road safety education programs for young adults, Department of Transport and Main Roads, 2009

A Guide To Evaluating Road Safety Education Programs For Young Adults

Uploaded by

Copyright:

Available Formats

A Guide To Evaluating Road Safety Education Programs For Young Adults

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

A Guide To Evaluating Road Safety Education Programs For Young Adults

Uploaded by

Copyright:

Available Formats

A guide to evaluating road safety

education programs for young adults

1. Identify the goals and objectives of your program. . . . . . . . . . . . . . . . . . . . . . . . . 4

2. Choose the best methods of evaluation for your program. . . . . . . . . . . . . . . . . . . . . 5

3. Choose how you will collect the information . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4. Carry out the evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5. Report on your findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Appendix A: Evaluation planning template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Appendix B: Case study of a young driver program . . . . . . . . . . . . . . . . . . . . . . . . . 23

Appendix C: Different types of evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Appendix E: Examples of useful surveys and questionnaires. . . . . . . . . . . . . . . . . . . 32

Why was the guide developed?

What is in the guide and how to use it

Choose the best method of evaluation

Choose how you will collect the information

Carry out the evaluation

Step 5 Report on your findings

Adapted from Sentinella (2004)

Key questions to help you identify your program’s goals and

Identify your program’s goal.

Identify your program’s objective/s.

WHO? (e.g. just young males)

Now specify your program’s objective/s as a statement.

Choosing which method of evaluation to use depends on:

Process evaluation Outcome evaluation

Write your evaluation goal in the template (Step 2).

Define evaluation objectives

Now list your evaluation objectives in the template (Step 2).

Different types of evaluation

Possible data collection methods

Observations of the program

Individual interviews • Don’t require written literacy • Interviewer could be biased or

Observations • Ability to view events as they • Information can be difficult to

Official records • Generally unbiased • May be difficult to access

The Great Lakes Epicenter News

Tips for conducting individual interviews and focus groups

Tips for using official records

Ethical considerations in evaluation

Participant Speeding in Speeding in Speeding

The key tasks in carrying out the evaluation are:

Advantages Disadvantages Considerations

Someone outside • Should be unbiased • Can be expensive • What will the

What might the report include?

Write on the template who your evaluation is aimed at (Step 5).

Identify how you might present your end product

Learning from the feedback

How to use this template

Step 1: Identify the program goals and objectives

Provide information on your program (for example, target group, how it is

Identify your program goal .

Identify your program objectives.

WHAT? (it may not be all of these)

Now specify the program objective as a single statement …

Define evaluation goals

Define evaluation objectives

Evaluation task Tick off when task completed

Analyse the data/information

Interpret the results