How Surgeons Make Decisions When the Evidence Is Inconclusive

MATERIALS AND METHODS

Using an institutional review board-approved protocol, we asked the 400 surgeons of the Science of Variation Group to complete a survey about decision making in the face of inconclusive evidence, and 337 participated. The Science of Variation Group is an international collaboration of fully trained surgeon observers that studies variation in the definition, interpretation, classification, and treatment of human illness. Collaborative authorship, scientific curiosity, and camaraderie are the only incentives for participation.

Evaluation

The observers were first asked to enter their demographic and professional information: sex, country or world region of practice, years in independent practice, supervision of trainees, and surgical subspecialty. Next, the observers were given the following context: "The American Academy of Orthopaedic Surgeons Evidence-Based Guidelines have been largely inconclusive for lack of evidence. It is difficult to show a difference in a well-designed prospective randomized, controlled trial-most will show little or no difference between treatments. Therefore it is important to decidebefore starting the study-what our fallback will be. How do we decide between treatment options when the data are either insufficient or otherwise inconclusive?"

In this context, participants were asked to rate the importance of 7 factors when deciding between operative treatment and palliative treatment (eg, the natural history of the disease) and 12 factors when deciding between 2 operative treatments (Table 1), with a com-ment section for listing additional factors. The ratings were based on a 5-point Likert scale between very important and very unimportant. The statements were developed by brainstorming. One author created a list, and the other authors edited until all authors felt that the list covered all potential fallback options.

Statistical analysis

Categorical data were presented as frequencies and percentages. The statements were ranked from highest to lowest by adding the percentages of the very important and somewhat important (Figs. 1, 2). The write-in answers were grouped by subject. In addition, the Likert scale was translated to an ordinal scale from 2 (very important) to -2 (very unimportant), and the mean on each scale across the entire sample was calculated. We analyzed the influence of nationality, years in practice, fractures treated per year, and specialization on preferred fallbacks. The subcategory "years in practice" was dichotomized to less than or equal to 10 years and more than 10 years of experience to facilitate analysis. For continuous variables, we used a Mann Whitney U test to compare 2 groups and a Kruskal-Wallis test for multiple groups. We evaluated differences between subgroups with the Mann Whitney U test.

Observer demographics

The demographics for the 337 respondents are listed in Table 2.

RESULTS

Statement rating

According to the percentages of statements rated very important or somewhat important, the most popular fallbacks when evidence cannot demonstrate that a given treatment is better than following the natural course of the illness are noted in Figure 1. The top fallbacks when evidence shows no difference between 2 surgeries are noted in Figure 2.

United States versus Europe

Using the average values on the numeric conversion of the Likert scale, Europeans rated "works in my hands," "burns fewer bridges," and "cheapest/most resourceful" of significantly greater importance and "what others are doing," "highest reimbursement," and "shorter procedure" of significantly lower importance than surgeons in the United States (Table 1).

Years in practice

Observers with 10 or fewer years in independent practice rated "what my mentor taught me," "what others are doing," and "highest reimbursement" of significantly lower importance compared to observers with more than 10 years in independent practice (Table 3).

Orthopedic specialty

General orthopedists rated "what my mentor taught me" of greater importance than orthopedic traumatolo-gists and hand and wrist surgeons. In addition, general orthopedists rated "what others are doing" of greater importance than shoulder and elbow surgeons and hand and wrist surgeons (Table 4).

Write-in answers

The most common write-in answers were "best available outcome/evidence-based" (14 surgeons), "common sense and risk for patients" (5 surgeons), and "shared decision making or patient's opinion" (4 surgeons).

DISCUSSION

Because evidence-based medicine is an amalgamation of individual clinical expertise and best available evidence, the question arises, what is the basis for provider recommendations when the best evidence is inconclusive? We found that the most popular factors that surgeons use to make recommendations when evidence is inconclusive relate primarily to the surgeon's perspective (eg, "works in my hands," "familiarity with the treatment," "what my mentor taught me") rather than the patient's perspective (eg "doing something vs doing nothing," "patients are requesting the procedure"). Exceptions include "fewer complications" and "quicker recovery," which benefit both the surgeon and the patient. Highest reimbursement was also rated relatively unimportant, particularly in Europe but across all countries and regions. This study should be interpreted in light of the fact that the 337 participating surgeons may not be repre-sentative of the average surgeon, because many surgeons in the surveyed group are in academic practice. Also, important options such as "I share the decision with the patient" were not offered because it was our intention to study the recommendation of the surgeon before accounting for the patient's preferences. Finally, there is evidence that incentives such as reimbursement can have a subconscious influence that may not be accounted for by this survey. 8 That health care providers fall back to their personal preferences based on experience is no surprise. 9 On the other hand, it is notable that factors related to quality, safety, and efficiency such as "cheapest/ most resourceful," "shorter procedure," and "what others are doing" (in the sense of diminished unwarranted variation) were rated relatively unimportant. The fact that Europeans rated "cheapest/more resourceful" significantly more important than Americans may reflect the prevalence of national health care in Europe, leading to a greater awareness of the management of limited resources. In contrast, surgeons from the United States rated "what others are doing," "highest reimbursement," and "shorter procedure" more important than European surgeons. It is not clear whether these factors relate most to quality and efficiency or marketing and profitability of health care in a for-profit system, or both.

Less experienced surgeons placed significantly less importance on "what my mentor taught me," "what others are doing," and "highest reimbursement." This might reflect a change in mindset as the emphasis is placed on evidence and as we continue to address the rising costs of health care.

The write-in answers revealed that surgeons prefer to fall back to the "best available outcome/evidencebased" even when the scenario is that the evidence is inconclusive. Patient-centered care/shared decision making was also mentioned, which is entirely applicable. The involvement of patients in decision making is particularly important when the evidence is inconclusive. Decision aids (independent structured guides, either written, video, or web-based) have been shown to decrease decision conflict and, for some illnesses, use of resources. 10 -13 These merit additional study.

In other words, rather than studying the surgeon's recommendation before accounting for the patient's preferences, it might have been preferable for our survey to include the option of following the patient's preference when evidence is inconclusive. On the other hand, we have an obligation to consider resources, safety, simplicity, consistency, efficiency, practicality, optimism, and patient self-management as important goals in and of themselves, and this is part of the expertise that we share with our patients. Patients look to their surgeons for expertise regarding the optimal fallback options when evidence is inconclusive. Perhaps-on the basis of the results of this survey study-surgeons will be motivated to develop consensus regarding the fallback principles that best support optimal health.

TABLE 1 . Geographic Difference and FactorsThe Importance That a Given Treatment Is Better Than the Course of the Illness Without Treatment