Anesthesiology: 2017 ABA Exams Report

The American Board of
Anesthesiology
2017 ABA Exams Report

Society of Academic Anesthesiology Associations
DAVID O. WARNER, M.D.

Chair, ABA Assessments & OSCE Committees
Mayo Clinic
Rochester, MN
QUESTION EDITORS
ITE/BASIC/ADVANCED/PART 1 EXAMS
76 JUNIOR EDITORS
10 were selected in 2016
Write questions (18/year)
Revise questions based on feedback
43 SENIOR EDITORS & COMMITTEE MEMBERS

Edit questions
Mentor junior editors
2
QUESTION DEVELOPMENT
• Question generation for the examinations:

BASIC, ADVANCED, Part 1 and ITE
– Approximately 1,400 questions generated
– Three senior editors meetings and 8 webinars to
review items
– ITE and BASIC Exam forms reviewed by exams
committees in August and September 2016
– ADVANCED form was reviewed in January 2017
3
2018 IN-TRAINING EXAM (ITE)
• Internet-based, secure, proctored exam

delivered via vendor
– Exam may be delivered any time from 12 a.m. ET on
Thursday, Feb. 15, to 11:59 p.m. ET on Tuesday,
Feb. 20, with multiple administrations
• Every computer used must pass complete

systems check; administration guides delivered
in December
• Questions may include graphics, including still

shots of a monitor screen and ultrasound images
4
ITE-PAIN MEDICINE (ITE-PM)
• In 2017, 86 programs administered the exam to

nearly 320 fellows
• 2018 ITE-PM will be administered at fellowship

program sites on Friday, March 16
• ITEs for Critical Care Medicine and Pediatric

Anesthesiology are expected to be added in
2019
5
IN-TRAINING EXAM PERFORMANCE
BY TRAINING LEVEL & EXAM YEAR
6
RELATION OF SCORES
2017 ITE TO 2017 JUNE BASIC EXAMINATION
BASIC Examination Scaled Score
ITE Scaled Score

The correlation between the 2017 ITE scaled scores
and 2017 June BASIC Exam scaled Scores is 0.622, p < 0.001 7
RELATION OF SCORES
2017 ITE TO 2017 JUNE BASIC EXAM
Scaled 2017 June BASIC Scaled BASIC

N
Score Score Mean (S.D.) Pass Rate
≤25 217 201 (44) 64%
26-30 524 228 (41) 85%
31-35 508 250 (43) 94%
36-40 306 283 (45) 99%
41-45 89 325 (47) 100%
≥46 20 341 (61) 100%
8
RELATION OF SCORES
2017 ITE TO 2017 JULY ADVANCED EXAMINATION
ADVANCED Exam Scaled Score
ITE Scaled Score

The correlation between the 2017 ITE scaled scores
and 2017 July ADVANCED Exam scaled Scores is 0.540, p < 0.001. 9
RELATION OF SCORES
2017 ITE TO 2017 JULY ADVANCED EXAM
2017 ADVANCED
Scaled ADVANCED
N Scaled Score Mean
Score Pass Rate
(S.D.)
≤25 15 197 (46) 73%
26-30 126 210 (41) 81%
31-35 392 232 (41) 94%
36-40 562 249 (37) 98%
41-45 310 272 (36) 100%
≥46 120 305 (41) 100%
10
EXAMINATION SCORING
• Standard-setting study conducted every five

years
• Following exam administration, preliminary item

analysis conducted (difficulty & discrimination)
• Key validation for items

– negative discrimination
– no clear correct answer
11
2017 JUNE BASIC EXAM RESULTS
• Candidates were assigned to examine on Friday

or Saturday
• Key validation eliminated nine items from two
forms
• 88.4% of candidates passed
Mean Scaled Standard Pass

N Reliability
Score Deviation Rate
1,696 247.6 54.0 88.4% 0.80
12
2017 JULY ADVANCED EXAM
RESULTS
• Candidates were assigned to examine on Friday
or Saturday
• Key validation eliminated 10 items from two forms
• 95.2% of candidates passed
Mean Scaled Standard Pass

N Reliability
Score Deviation Rate
1,610 248.9 47.4 95.2% 0.75
13
PART 2/APPLIED EXAMINATION
SUCCESS RATES
14
RELATION OF SCORES
2017 SOE/PART 2 EXAM TO FIRST ATTEMPT ON
2016 ADVANCED/PART 1 EXAM
The correlation between the 2017 SOE/Part 2 Exam scaled scores and first attempt on
2016 ADVANCED/Part 1 Exam scaled scores is 0.266, p < 0.001 15
STAGED EXAMINATIONS
• Staged exams launched in 2014
to replace the traditional Part 1
and Part 2 Exams
– Candidates who completed residency
training on or after Oct. 1, 2016, take the
staged exams – BASIC, ADVANCED
and APPLIED Exams
• In 2018, the Objective Structured Clinical

Examination (OSCE) component will be added to
the APPLIED Exam, completing the staged exams
rollout
16
APPLIED EXAM: WHY ADD OSCES?
• OSCES are intended to assess two major domains
that are difficult to assess in written or oral exam
formats, such as
– Communication and Professionalism
– Technical skills related to patient care
• Evidence that these

domains are important
in physician performance
after training
17
OSCE FORMAT
• Candidates will participate in a seven-station

circuit that will evaluate their proficiency in
seven of the nine skills from the OSCE Content
Outline:
– Six Communication & Professionalism skills
– Three Technical skills
• Each station will be eight minutes long with

four minutes between stations to review their
next scenario
18
OSCE FORMAT
• In some stations the candidate will interact

with a standardized patient actor as part
of the scenario
• In others, the candidate

will interact directly with
an examiner, but
examiners will not
be in most exam rooms
19
WHY ARE WE DOING THIS?
• Assessment
– To better assess competencies important for
diplomates to possess; better align with expansion of
competencies reflected by the ACGME milestones
– Desire to improve the discrimination and external
validity of our overall assessment process and
assess whether there is significant collinearity with
the SOE
• Training
– To drive improved training in these domains within
residency programs
20
PREPARATION FOR OSCES?
• Extensive exam prep/training should not
be necessary
– Communication & Professionalism
scenarios: Based on OSCE formats that medical
school grads have experienced – enter a room and
talk with a standardized patient/clinician
– Technical Skills scenarios: Mimic common

clinical teaching settings (e.g., explaining what the
candidate sees on a monitor, showing faculty
relevant ultrasound anatomy, interpreting an echo)
21
HOW WILL OSCES BE SCORED?
• Scoring method similar to that used for the

current Structured Oral Examination
• Standards for passing the OSCE

component will account for the novel
nature of the exam
22
OSCE RESOURCES
• Published these resources on the ABA
website to help programs help residents
prepare for the first OSCE administration in
2018:
– OSCE Content Outline
– OSCE administration schedule
– OSCE example scenarios with links to TEE
resources
– OSCE overview video
23
The American Board of
Anesthesiology
QUESTIONS?
COMMUNICATIONS CENTER MAIL CORRESPONDENCE
Phone: (866) 999-7501 ABA Secretary
Fax: (866) 999-7503 4208 Six Forks Rd, Suite 1500
Email: [email protected] Raleigh, NC 27609-5765
FOLLOW US:

Anesthesiology: 2017 ABA Exams Report

Uploaded by

Copyright:

Available Formats

Anesthesiology: 2017 ABA Exams Report

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Anesthesiology: 2017 ABA Exams Report

Uploaded by

Copyright:

Available Formats

The American Board of

2017 ABA Exams Report

DAVID O. WARNER, M.D.

43 SENIOR EDITORS & COMMITTEE MEMBERS

• Question generation for the examinations:

• Internet-based, secure, proctored exam

• Every computer used must pass complete

• Questions may include graphics, including still

• In 2017, 86 programs administered the exam to

• 2018 ITE-PM will be administered at fellowship

• ITEs for Critical Care Medicine and Pediatric

ITE Scaled Score

Scaled 2017 June BASIC Scaled BASIC

ITE Scaled Score

• Standard-setting study conducted every five

• Following exam administration, preliminary item

• Key validation for items

• Candidates were assigned to examine on Friday

Mean Scaled Standard Pass

1,696 247.6 54.0 88.4% 0.80

Mean Scaled Standard Pass

1,610 248.9 47.4 95.2% 0.75

• In 2018, the Objective Structured Clinical

• Evidence that these

• Candidates will participate in a seven-station

• Each station will be eight minutes long with

• In some stations the candidate will interact

• In others, the candidate

– Technical Skills scenarios: Mimic common

• Scoring method similar to that used for the

• Standards for passing the OSCE

You might also like