Graduate Attributes 101

Greg Wohl
(using Marilyn Lightstone’s presentation)
Mechanical Engineering
July 15, 2019

1
 What we will cover in this workshop
1. Some background on the CEAB 3-4
2. Importance of attaining your P.Eng. 5-7
designation
3. What are “Graduate attributes”? 8-13
4. Curriculum mapping 14-17
5. How to write a learning outcome 18-29
statement
6. Indicator measurements 30-38
7. GA Documentation 39-44
2
 Canadian Engineering
Accreditation Board (CEAB)
• CEAB - a standing committee to “Engineers
Canada”
• Engineers Canada - a national federation of all
the provincial licensing organizations (i.e. PEO)
• Accreditation - performed to ensure that
graduates of an engineering program are
academically qualified to be licensed as a
professional engineer
• Accreditation is critical to our success!
3
 What does the CEAB look at?
(Six key areas)
1. Graduate attributes – we will come back to this
2. Continual improvement – will come back to this
3. Students – policies, procedures, quality,
counselling, …
4. Curriculum content and quality – inputs based
assessment (counts AU – accreditation units)
5. Program Environment (faculty, labs, libraries,
financial resources …)
6. Additional criteria (program options, weakest
link, program name, …)

4
 Critical to our success
• Become a P.Eng.
• Must have your P.Eng. to participate in design
courses!
• Faculty has workshops to help you prepare for
the law and ethics exam.

5
 McMaster Accreditation Status
• All programs underwent CEAB review in 2015
when GA/CI was in early stages of
development
• All programs received 6 year of accreditation
• Next review is in 2021
• Graduate Attributes assessment and
continuous improvement is ON-GOING effort.
• CEAB will have higher expectations on GA/CI
at the next review.
6
Sample of PEO Current Expectations:*
• It is expected that suitable committee and
reporting structures are in place to assure the
sustainable development and measurement of
graduate attributes.
• All faculty members of the relevant academic
unit are expected to be aware of and engaged in
outcomes-based assessment.

*from Accreditation Criteria and Procedures 2016

7
 3
WHAT ARE
“GRADUATE ATTRIBUTES”?

8
What are the 12 “graduate attributes” (GAs)?
Attributes/skills that graduates of a program must have include:
1. Knowledge base for engineering
2. Problem analysis
3. Investigation
4. Design
5. Use of engineering Tools
6. Individual and Team Work
7. Communication Skills
8. Professionalism
9. Impact on society and environment
10. Ethics and equity
11. Economics and Project management
12. Life-Long learning
9
 Background/Language –
Outputs Based Assessment
• We need to some way to assess the degree
to which our students have attained these
attributes.
• We start by defining “indicators” of the
attributes.
• Each attribute has a number of ‘indicators’
associated with it.
• The indicators describe the various
elements of an attribute.
• Indicators must be “measurable”.
10
 Example – Attribute/Indicators
Attribute: 1. A knowledge base for engineering
Description of the attribute (from CEAB):
Demonstrated competence in university level
mathematics, natural sciences, engineering
fundamentals, and specialized engineering knowledge
appropriate to the program.

Four Indicators for “Knowledge base for engineering”:

1.1 Competence in Mathematics
1.2 Competence in Natural Sciences
1.3 Competence in Engineering Fundamentals
1.4 Competence in Specialized Engineering knowledge
11
 Another Example: Attribute/Indicators
Attribute #6: Individual and team work
Description of the attribute (from CEAB):
An ability to work effectively as a member and leader in
teams, preferably in a multi-disciplinary setting.

Three Indicators for “Individual and team work”:

6.1 Manages time and processes effectively, prioritizing competing

demands to achieve personal and team goals and objectives.

6.2 Develops and implements processes and methodologies to manage

the effectiveness of a team both in terms of the quality of the work
produced by the team as well as the inter-personal relationships
within the team.

6.3 Works in a group, taking a leadership role as appropriate and

relinquishing the leadership role as appropriate.
12
Structure of Attributes/Indicators:
(we are measuring learning outcomes
that relate to the ‘indicators’)

• Graduate Attribute #1
– Indicator 1.1
– Indicator 1.2
• Grad Attribute #2
– Indicator 2.1
– Indicator 2.2
– Indicator 2.3
• …
• Graduate Attribute #12
– Indicator 12.1
– Indicator 12.2
13
 4
CURRICULUM MAPPING

14
 Curriculum Mapping
• You will need to assess which indicators are relevant to
the courses that you teach.
• You will need to come up with a list of “learning
outcomes” for your course (more on this later).
• Need to complete a “Course Information Sheet” (or
possibly an excel file provided by dept administrator)
• For each indicator, you need to state if:
– Not relevant
– I – Introduced in your course
– D – Developed in your course
– A – Applied in your course
– (We previously used a scale of 0 (not relevant), 1, 2, 3)

15
 Curriculum Mapping
I D A mapping of indicators (from CEAB website 2019):
https://www.engineerscanada.ca/sites/default/files/draft_program_visitor_guide_v1.25.pdf
p. 6 of the visitor guide
• Introductory: … the students learn the working vocabulary of the area of content,
along with some of the major underlying concepts. Many of the terms need
defining and the ideas are often presented in a somewhat simplified way.

• Development: …the students use their working vocabulary and major

fundamental concepts to begin to probe more deeply, to read the literature, and
to deepen their exploration into concepts.
At this level, students can begin to appreciate that any field of study is a complex
mixture of sub-disciplines with many different levels of organization and analysis.

• Application: … the students approach mastery in the area of content.

They explore deeply into the discipline and experience the controversies, debate
and uncertainties that characterize the leading edges of any field.
An advanced student can be expected to be able to relate course material across
different courses, to begin to synthesize and integrate and achieve fresh insights.
Students at this level are working with the knowledge very differently, perhaps
even creating new knowledge through independent investigation.

16
 Curriculum Mapping
• All faculty members provide a description of
how relevant the indicators are to their
courses
• This is put together into a large excel file to
see an overall mapping for the program
• Helps to identify any “gaps” that may exist
• You will likely be asked to measure the extent
to which the students have mastered the
relevant indicators for your course.
17
 5
LEARNING OUTCOMES

18
Indicators/Course Learning Outcomes
• Indicators can be broad statements that apply
across departments.
Example “Competence in Mathematics”
• Learning outcome statements are more
precise and contain detailed information
about what a student will learn.
• We are measuring how well the students
mastered the learning outcomes from your
course that pertain to the indicators.

19
Example #1: Indicator and Associated
Learning Outcome
• Indicator:
– 1.1 Competence in Mathematics
• Example of an associated learning outcome
for a specific course:
– “A demonstrated ability to solve a linear
homogeneous differential equation with
associated boundary conditions.”

20
Example #2: Indicator and Associated
Learning Outcome
• Indicator:
– 1.4 Competence in specialized engineering
knowledge
• Example of an associated learning outcome
for a specific course:
– “A demonstrated ability to describe the basis of
the heat and momentum transfer analogy and
perform heat transfer calculations on thermal
boundary layers.”

21
 Example #3 – Indicator is already a
learning outcome
• Indicator:
– 8.1 Demonstrates an understanding of the role of
the engineer in society, especially in protection of
the public and public interest.
• This is an indicator for the attribute: “Professionalism”
• This indicator is already specific enough to be a learning
outcome for a course (example: ENGINEER 1P03 –
“Introduction to Professional Engineering”)

22
 Measurement Goals:
1. To determine the extent to which our
students are attaining specific learning
outcomes associated with the ‘indicator’ that
we are measuring.
2. To use this information to improve our
program in subsequent years. Need to satisfy
the “continuous improvement” requirement
of CEAB.

23
 High level description of
measurement procedure
• Want to get a sense of how well our students are
attaining the learning outcomes associated with the
indicator and determine what they are struggling with.
• We use tests, assignments, presentations, reports in
our measurement process.
• We use rubrics to measure the student learning
outcomes.
• Four levels of grading are used:
1. Does not meet expectations
2. Marginal
3. Meets expectations
4. Exceeds expectations
24
 What is a learning outcome?
Simply stated, a learning outcome is:
1. What faculty members want students to
know at the end of the course
AND
2. What faculty members want students to be
able to do at the end of the course

Reference: Pauline Smiley, Fleming College, Symposium on Learning

Outcomes Assessment, Toronto, 2012.
25
 Characteristics of Learning Outcomes
1. They specify an action by the students that is
observable.
2. They specify an action by the students that is
measurable.
3. They specify an action that is done by the
students (rather than the faculty members).

Reference: Pauline Smiley, Fleming College, Symposium on Learning

Outcomes Assessment, Toronto, 2012.
26
 Writing learning outcome statements
• Must include verbs!
• Example of a bad learning outcome
statement:
– “Differential equations”
• Example of a good learning outcome
statement:
– “A demonstrated ability to solve a linear
homogeneous differential equation with
associated boundary conditions.”
– Note: verbs are included and detail is given.
27
 Aside: Bloom’s Taxonomy
• Taxonomy is: Remember, Understand, Apply,
Analyze, Evaluate, Create.
• It provides useful words for developing learning
outcome statements.
• How does Bloom’s fit into our process?
– It tends to confuse us.
– Don’t get caught up in using Bloom’s to determine
your learning outcomes.
– Upshot: Don’t worry about Bloom’s! – Just think of it
as providing useful verbs!

28
REMEMBER UNDERSTAND APPLY ANALYZE EVALUATE CREATE

Count Associate Add Analyze Appraise Categorize

Define Compute Apply Arrange Assess Combine
Describe Convert Calculate Breakdown Compare Compile
Draw Defend Change Combine Conclude Compose
Identify Discuss Classify Design Contrast Create
Label Distinguish Complete Detect Criticize Drive
List Estimate Compute Develop Critique Design
Match Explain Demonstrate Diagram Determine Devise
Name Extend Discover Differentiate Grade Generate
Outline Extrapolate Divide Discriminate Interpret Group
Point Generalize Examine Illustrate Judge Integrate
Quote Give examples Graph Infer Justify Modify
Read Infer Interpolate Outline Measure Order
Recall Paraphrase Manipulate Point out Rank Organize,
Recite Predict Modify Relate Rate Plan
Recognize Rewrite Operate Select Support Prescribe
Record Summarize Prepare Separate Test Propose
Repeat Produce Subdivide Rearrange
Reproduce Show Utilize Reconstruct
Select Solve Related
State Subtract Reorganize
Write Translate Revise,
Use Rewrite
Summarize
Transform29
Specify
 6
INDICATOR MEASUREMENTS

30
 Overall Measurement Procedure
1. Decide on which student work will be used for
measurement (i.e., tests, exam, presentations,
project reports…)
2. Develop a rubric to describe desired student
learning outcomes (more detail on this to come)
3. While you are marking, keep track of how the
student did by ticking the appropriate box
4. Analyze results to provide information for
continuous improvement (i.e., identify learning
outcomes that the students are struggling with)
5. Document measurement results
31
Example: Measurement of the Indicator “Competence
in Specialized Engineering Knowledge” using
MECH ENG 4S03 (Incompressible Flows)

1. Student work used for measurement:

• Final exam
2. Development of rubric (example to follow):
• Think about what you wanted the students to learn
• Link those learning outcomes to the exam questions
• Decide on what the students needed to be able to do
to demonstrate that they “met expectations”
• Then define learning outcomes for “exceed
expectations”, “marginal”, “does not meet
expectations”
32
 Example Rubric – MECH ENG 4S03 (Incompressible Flow)
Topic Below Marginal Meets Exceeds
(exam questions Expectations Expectations Expectations
used)

Topic #1: -Does not -Able to use the - Can explain the -“meets
Heat and understand the correlations. math. basis of expectations”
momentum concept of the - Understands the analogy. plus:
transfer analogy analogy that there is an - Can determine - Can explain
(Question 4 of analogy, but appropriate why analogy
final exam) cannot explain correlation to does not hold if
the math behind solve for heat there is pressure
it. transfer or drag gradient

Comments on Topic #1 performance:

33
 Example Rubric – MECH ENG 4S03 (Incompressible Flow)
Topic Below Marginal Meets Exceeds
(exam questions Expectations Expectations Expectations
used)
Topic #2: -Cannot use -Can draw -Can draw -”meets
Boundary layers correlations velocity profile boundary layer expectations”
(Question 3 of correctly -Can calculate velocity profile plus:
final exam) -Unable to local shear and - can calculate - Can explain
explain total drag local shear and (based on
separation -Doesn’t total drag physics in near
understand -Can say wall region) why
separation whether flow sep. cannot
will separate or occur for fav.
not pressure grad.

Comments on Topic #2 performance:

34
 POOR Example Rubric
Topic Below Marginal Meets Exceeds
(exam questions Expectations Expectations Expectations
used)
Topic #3: Student does Student has Is able to answer Does really well
(Question 4 of not answer problems with question 4 on question 4
final exam) question or does question 4. satisfactorily
poorly on
question 4

[< 4.5 / 10] [4.5 – 6 / 10] [6-8 / 10] [>8 / 10]

Comments on Topic #3 performance:

35
 Example Rubric – MECH ENG 4S03 (Incompressible Flow)
Topic Below Marginal Meets Exceeds
(exam questions Expectations Expectations Expectations
used)

- Keep adding rows until you have covered topics you wish to measure
- remember to leave blank space for ‘ticks’ when measuring
- remember to leave area to write in comments while you are marking

- If all the elements of the exam or test pertain to the indicator being measured (i.e.,
‘Competence in specialized engineering knowledge’ as in this example), then add a row with
the overall mark distribution on the exam or test:

Overall exam % that did not % that were % that met % that exceeded
performance meet marginal expectations expectations
expectations
(grade of roughly (grade of roughly (top students:
(i.e., grade of 48% to 59%) 60% to 79%) 80% and above)
less than 48% on
exam)
36
 Measurement Logistics:
As you mark a question that is on the rubric, tick off the
appropriate box. Add comments as appropriate.
Topic #1: -Does not -Able to use the - Can explain -“meets
Heat and understand the correlations. the math. basis expectations”
momentum concept of the - Understands of the analogy. plus:
transfer analogy analogy that there is an -Can choose - Can explain
(Question 4 of analogy, but appropriate why analogy
final exam) cannot explain correlation to does not hold if
the math solve for heat there is
behind it. transfer or drag pressure
/// gradient
//// //// //// //// ////
//// //// //// ///

Comments on Topic #1 performance:

-Students were generally good at using correct correlation
-Some had trouble explaining the mathematics behind the analogy (need
to spend more lecture time on that next year)
37
So what do we do with this data? 30% “marginal” or below
60

50
Percentage (%)

40

30

20

10

0
FEAS - 3.12-FY1
Overall examFEASTopic
- 3.12-FY2
#1 FEASTopic
- 3.12-FY5
#2 FEAS - 3.12-FY6
Topic #3
distribution
1 - Not Demonstrated 2 - Marginal 3 - Meets Expectations 4 - Outstanding

Topic #1 – many are exceeding expectations (could reduce lecture time on it)
Topic #3 – more than 30% of class is marginal or below (struggling with this)
*38 Plot is from Brian Frank from Queen’s
 7
GRADUATE ATTRIBUTE
DOCUMENTATION

39
Documentation of Measurement Results
• Need to write a short document summarizing
results. It should include:
– Rubric used for measurement
– Corresponding exam or test
– Distributions for each learning outcome area
– Identified areas for continuous improvement
– Sample exam papers with performance in each area
(below expectations, marginal, …)
– Suggestions for how to improve measurement
procedure (if any)
• VENA: Faculty of Engineering has invested in
database software to “store” these data.
40
 Follow-up
• Continual Improvement:
– Incorporate areas identified as needing improvement
into your lectures the next time you teach this course
• Keep track of the changes made to your course
since we will likely need to incorporate them into
the next CEAB report.
• Subsequent measurement of the same learning
outcomes should hopefully show improvement in
those areas where we found the students were
struggling.

41
 Continual Improvement*
• At the program level, CEAB will be assessing:
– Improvement process:
• Suitable committee structure
• Engagement of relevant stakeholders
• Well-defined timetable
– Stakeholder engagement:
• Consultation with broadly-based set of stakeholders
(internal and external to the program and institution)
– Improvement Actions:
• Expectation of curriculum or other program
improvements
• improved achievement of graduate attributes
• Improvements in assessment process

*From “Accreditation Criteria and Procedures 2016 – Appendix 10” 42

 If you need help with this:
• Each department has a graduate attributes (GA)
committee
• Committee responsible for:
– Developing a GA measurement plan for the department
– Organizing stakeholder meetings
– Communicating expectations to faculty members
– Assisting faculty members with GA measurements
– Reviewing GA measurement reports
– Keeping track of continuous improvement at course level
(via measurement reports)
– Recommending program improvements based on program
level GA measures
• e.g., which attributes do not meet the threshold or are not well
represented in the curriculum.
43
 Graduate Attributes – Key Documents:
Marilyn’s website: http://mechfaculty.mcmaster.ca/~lightsm/gradatt/

- List of indicators (revised in 2013)

- This workshop presentation
- Sample measurement report
- Report cover page with checklist of contents

Key Documents from CEAB website:

https://engineerscanada.ca/sites/default/files/accreditation-criteria-
procedures-2016-final.pdf

https://engineerscanada.ca/sites/default/files/draft_program_visitor_
guide_v1.25.pdf

44
APPENDIX
LIST of INDICATORS

45
 Appendix – Indicator List
1. A knowledge base for engineering
Demonstrated competence in university level mathematics, natural sciences,
engineering fundamentals, and specialized engineering knowledge appropriate to the
program.

– Competence in Mathematics
– Competence in Natural Sciences
– Competence in Engineering Fundamentals
– Competence in Specialized Engineering knowledge

2. Problem analysis
An ability to use appropriate knowledge and skills to identify, formulate, analyze, and
solve complex engineering problems in order to reach substantiated conclusions.
– Demonstrates an ability to identify reasonable assumptions (including identification of
uncertainties and imprecise information) that could or should be made before a solution path
is proposed.
– Demonstrates an ability to identify a range of suitable engineering fundamentals (including
mathematical techniques) that would be potentially useful for analyzing a technical problem.
– Obtains substantiated conclusions as a result of a problem solution including recognizing the
limitations of the solutions.

46
3. Investigation
An ability to conduct investigations of complex problems by methods that include appropriate
experiments, analysis and interpretation of data, and synthesis of information in order to reach
valid conclusions.

– Recognizes and discusses applicable theory knowledge base

– Selects appropriate model and methods and identifies assumptions and constraints.
– Estimates outcomes, uncertainties and determines appropriate data to collect.

4. Design
An ability to design solutions for complex, open-ended engineering problems and to design
systems, components or processes that meet specified needs with appropriate attention to
health and safety risks, applicable standards, economic, environmental, cultural and societal
considerations.

– Recognizes and follows an engineering design process.

(This means an iterative activity that might include recognizing the goal, specifying the constraints
and desired outcomes, proposing solutions, evaluating alternatives, deciding on a solution, and
implementing.)
– Recognizes and follows engineering design principles including appropriate consideration of
environmental, social and economic aspects as well as health and safety issues.
– Proposes solutions to open-ended problems.
– Employs appropriate techniques for generation of creative ideas such as brainstorming and
structured inventive thinking.
– Includes appropriate health and safety considerations
– Determines and employs applicable standards and codes of practice.

47
5. Use of engineering tools
An ability to create, select, apply, adapt, and extend appropriate techniques,
resources, and modern engineering tools to a range of engineering activities, from
simple to complex, with an understanding of the associated limitations.

– Evaluates and selects appropriate modern tools.

– Demonstrates an ability to use modern/state of the art tools.
– Creates, adapts, modifies and extends tools and techniques as appropriate to solve problems.

6. Individual and team work

An ability to work effectively as a member and leader in teams, preferably in a multi-
disciplinary setting.

– Manages time and processes effectively, prioritizing competing demands to achieve personal
and team goals and objectives.
– Develops and implements processes and methodologies to manage the effectiveness of a
team both in terms of the quality of the work produced by the team as well as the inter-
personal relationships within the team.
– Works in a group, taking a leadership role as appropriate and relinquishing the leadership role
as appropriate.

48
7. Communication skills
An ability to communicate complex engineering concepts within the profession and
with society at large. Such abilities include reading, writing, speaking and listening, and
the ability to comprehend and write effective reports and design documentation, and to
give and effectively respond to clear instructions.

– Demonstrates an ability to respond to technical and non-technical instructions and

questions.
– Presents instructions and information clearly and concisely as appropriate to the audience
– Constructs effective oral or written arguments as appropriate to the circumstances

8. Professionalism
An understanding of the roles and responsibilities of the professional engineer in
society, especially the primary role of protection of the public and the public interest.

– Demonstrates an understanding of the role of the engineer in society, especially

in protection of the public and public interest.
– Demonstrates an understanding of legal requirements governing engineering
activities (including but not limited to personnel, health, safety, and risk issues).
– Shows an awareness of the PEO and the role of licensing.

49
9. Impact of engineering on society and the environment
An ability to analyze social and environmental aspects of engineering activities. Such
abilities include an understanding of the interactions that engineering has with the
economic, social, health, safety, legal, and cultural aspects of society; the uncertainties
in the prediction of such interactions; and the concepts of sustainable design and
development and environmental stewardship.

– Identifies and quantifies the full range of short-term, long-term, local and global impacts of
their engineering projects on society, including: economic aspects; social, cultural, and human
health aspects, and; ecosystem integrity aspects.
– Addresses uncertainties in the prediction of interactions on society and the environment in a
structured and transparent manner.
– Assesses possible options and design configurations from a sustainability engineering
perspective, which emphasizes environmental stewardship, life-cycle analysis, and long-term
decision-making principles.

10. Ethics and equity

An ability to apply professional ethics, accountability, and equity.

– Applies the engineering code of ethics, understanding of the stakeholders: the individual, the
employer, and the public.
– Applies ethical frameworks and reasoning in situations where there may be conflicting interests
among the stakeholders
– Applies knowledge of law and principles of equity to ensure equitable treatment of others.

50
11. Economics and project management
An ability to appropriately incorporate economics and business practices
including project, risk and change management into the practice of
engineering, and to understand their limitations.

– Applies economic principles in decision making

– Plans and effectively manages time, resources, and scope
– Understands the business processes for implementing engineering ideas
– Identifies, characterizes, assesses, and manages risks to project success

12. Life-long learning

The ability to identify and to address their own educational needs in a
changing world, sufficiently to maintain their competence and contribute to
the advancement of knowledge.

– Critically evaluates and applies knowledge, methods and skills procured

through self directed and self identified sources, including those that lie
outside the nominal course curriculum.
– Shows an awareness of the wide range of engineering societies, literature,
conferences, and other information sources.

51