developing information to improve safety

Assessing and Responding in

Real Time to Online Anti-vaccine
Sentiment during a Flu Pandemic
Neil Seeman, Alton Ing and Carlos Rizo

P
Abstract rior to the 2009 influenza A (H1N1) pandemic,
The perceived safety of vaccination is an important explana- public health experts recognized that communities
tory factor for vaccine uptake and, consequently, for rates throughout the globe were deficient in pandemic
of illness and death. The objectives of this study were (1) to planning (Mareinniss et al. 2009) and could benefit
evaluate Canadian attitudes around the safety of the H1N1 from strategies to increase vaccination rates. In any epidemic,
vaccine during the fall 2009 influenza pandemic and (2) to high vaccination uptake is essential in order to limit transmis-
consider how public health communications can leverage the sion, protect groups at high risk, reduce the number of severe
Internet to counteract, in real time, anti-vaccine sentiment. outcomes and prevent an overload of health services use.
We surveyed a random sample of 175,257 Canadian Inadequate information about the protective effects of a demon-
web users from October 27 to November 19, 2009, about strably safe flu vaccine reduces immunization rates, contributing
their perceptions of the safety of the HINI vaccine. In an to a more rapid spread and wider distribution of an epidemic.
independent analysis, we also assessed the popularity of Healthcare workers are at particular risk, and, accordingly, in
online flu vaccine-related information using a tool developed some jurisdictions such as Ontario, it is a hospital board–level
for this purpose. A total of 27,382 unique online participants responsibility to ensure rapid-response emergency preparedness
answered the survey (15.6% response rate). Of the respond- plans are in place to protect the safety of hospital workers in the
ents, 23.4% considered the vaccine safe, 41.4% thought it event of an infectious outbreak (Seeman et al. 2008).
was unsafe and 35.2% reported ambivalence over its safety. Systematic reviews show that vaccines prevent infection,
Websites and blog posts with anti-vaccine sentiment complication and death, especially when provided to groups
remained popular during the course of the pandemic. at high risk (Jefferson et al. 2008). Why, then, do many people
Current public health communication and education choose not to be vaccinated? Reasons include a lack of familiarity
strategies about the flu vaccine can be complemented by with the epidemiological facts, a lack of support or notification
web analytics that identify, track and neutralize anti-vaccine from the healthcare system and unfounded fears about vaccine
sentiment on the Internet, thus increasing perceived vaccine safety (Baeyens 2010; Maurer et al. 2010). Common fears are
safety. Counter-marketing strategies can be transparent and that a new vaccine has been rushed to production with insuf-
collaborative, engaging online “influencers” who spread ficient prior research, that it has not been adequately tested and
misinformation. that long-term studies are needed (Seale et al. 2010). General

8 Healthcare Quarterly Vol.13 Special Issue September 2010

 Neil Seeman et al. Assessing and Responding in Real Time to Online Anti-vaccine Sentiment during a Flu Pandemic

anti-vaccination sentiment has been growing worldwide due perceptions of the safety of the H1NI vaccine after Health
to the well-publicized but unsubstantiated link between flu Canada approved the vaccine. In step B, we determined which
immunization and autism; between hepatitis B vaccination vaccine safety Internet sites were most trusted by the public by
and multiple sclerosis in France; and between convulsions and deploying a dynamic “Internet robot” that informed us about
sudden death and human papillomavirus immunization in (1) which uniform resource locators (URLs) regarding “myths
Austria, Germany and Spain (Alvarez-Pasquin et al. 2009). and facts” about the H1N1 vaccine were being most widely
Efforts to offset the arguments of the anti-vaccine movement, shared and discussed among English-language Internet users and
to calm public fears and to provide accurate information require (2) which websites, blogs and links were being shared on social
sustained, effective public health communication. Concerns media sites. Both step A (the survey) and step B (the Internet
about safety and side effects need to be addressed; as well, trans- robot) were independent, and the results should be interpreted
parency is required about the vaccine development process. Was as such. Both steps of our process are described below.
this successfully accomplished in Canada? A poll conducted
between October 1 and 5, 2009, by Harris-Decima revealed Step A: Random Online Survey of Internet Users’
that only a third of Canadians intended to get vaccinated, 11% Perceptions of Vaccine Safety
described themselves as very concerned about H1N1 and 25% For the survey of Internet users’ perceptions of vaccine safety,
reported being somewhat concerned (Harris-Decima 2009). we used the RIWI Time Trender service (http://riwi.com),
The federal health minister noted that the biggest challenge to which applies a patent-pending Internet intercept method
preventing the spread of the virus was communicating the need that provides access to immediate respondent data based on
for vaccination. Health Canada’s information about vaccine a random sampling of Internet users. Response to the survey
safety was broadly disseminated on posters, on buses and was randomized by accessing thousands of “nonsense” domain
subways, in multi-language newspapers and on social media such names (URLs) that reach hundreds of thousands of random
as Facebook. The information was posted on government and Internet users. A nonsense domain is a URL that has no
hospital websites across Canada, and Health Canada’s website English-language meaning (e.g., www.jhwje.ca) and is not
was prominently hyper-linked via mainstream Canadian media being used for commercial or other purposes. Thus, the method
news sites. This seemed like a logical communications strategy captures potential respondents navigating the Internet who type
given that the news media have been a leading source of public in nonsense domains by random accident (i.e., mistypes). The
health information (Gollust and Lantz 2009). Internet intercept method is not like email spam; it is more akin
The challenge with social risk communication in the age of to an online “random digit dialing” survey since all Internet
the Internet is the increasing fragmentation of media (Sunstein users have a relatively equal probability of inadvertently landing
2007). Today, Canadians access health information not through on the web page where the survey is posted. Only Canadian
print newspapers, radio or cable television but predominantly Internet users were able to respond (geographically identified,
through the Internet. For at least five years, the first place people anonymously, by their Internet protocol, or IP, address).
seek health-related information has been the web (Hesse et al. The survey asked, “Is the H1N1 flu vaccine safe?” Answer
2005). Unfortunately, some Internet sites and postings, light options were limited to “yes,” “no,” “don’t know” and “skip.”
on facts and packed with emotionally laden anecdotes, worsen Respondents were also asked their age (under 18, 18–29, 30–49,
concerns regarding vaccination safety (Maurer et al. 2010; 50–64 and over 64) and their sex. They were able to answer
Wolfe et al. 2002). only once, either in English or French. The survey contained
Given the degree to which the public accesses vaccine-related a privacy policy explaining that collected information would
information online, we wanted to track whether online postings not identify individuals, businesses or households. Respondents
about the H1N1 flu vaccine were undermining ongoing commu- were advised that information would be kept anonymous and
nications efforts by public health authorities during the fall of that they had the choice not to respond.
2009. We also wanted to know whether anti-vaccine sentiment
escalated after Health Canada’s approval of the vaccine and, if Step B: H1N1 Myth and Fact Internet Aggregator
so, to suggest Internet communications strategies (Rizo et al. At the same time that we initiated the survey (October 27,
2005) at the national, regional and hospital levels that could 2009), we launched a software tool to count how often flu
assess, monitor and, ideally, counteract such sentiment. vaccine–related information websites were being shared on blogs
and social media sites such as Digg, YouTube, Facebook and
Methodology Twitter. We wanted to identify which websites containing infor-
Two parallel, independent steps were initiated to address our mation on myths and facts about the H1N1 vaccine were the
objectives. In step A, we surveyed a random sample of Canadian most viewed, read and shared on the web. We used standardized
web users from October 27 to November 19, 2009, about their English search strings to identify which websites were discussing

Healthcare Quarterly Vol.13 Special Issue September 2010 9

Assessing and Responding in Real Time to Online Anti-vaccine Sentiment during a Flu Pandemic Neil Seeman et al.

the safety of the vaccine. Using a structured algorithm, we were 4. Categorization of chat level. The chat level for each search
able to track, on a daily basis, which of these websites were rising result was categorized into one of four levels: high (60 dB
in popularity (i.e., were being shared with increasing frequency and above), medium (40–59 dB), low (20–39 dB) and none
among web users). This is different from counting website “hits,” (below 20 dB).
which do not track whether the individuals visiting the site take 5. Daily dynamic scoring. The change in chat level (over 24
the additional steps of creating a short form of the URL (i.e., by hours) was calculated in order to rank “trending” search
using popular website “shorteners” such as http://www.tinyurl. results. On a daily basis, the chat levels for new and existing
com or http://www.bit.ly) and then emailing, texting or other- search results were updated. In this manner, we visualized the
wise sharing the shortened link (e.g., via Facebook or Twitter) change in chat level for each search result.
with others. We aggregated and displayed this information in
real time on a publicly accessible area called the Flu Chat Lab Results
at http://www.myhealthinnovation.com. The dynamic aggrega- Step A Findings: Daily Tracking Survey of Canadian
tion of this Flu Chat Lab content is now accessible and ongoing Web Users about Perceived Vaccine Safety
at http://lab.innovationcell.com (Figure 1). There were 27,382 unique respondents (i.e., from unique
The computer-programmed Flu Chat Lab aggregation computing devices) who completed the survey, out of 175,257
technique involved the following five steps: separate Canadians exposed to the survey. This translated to
a response rate of 15.6%. The remainder of the respondents
1. ‘Chatter’ collection. We built a selection of relevant English- (84.4%) chose to hit “skip” (signaling their unwillingness to
language search strings (available upon request). The goal of complete the survey) or closed their web browsers. We verified
the search strings was to identify postings on the web that that the target of our survey only covered Canadian IP addresses,
contained self-reports about the perceived truth or falsity of across all provinces and territories. Each day, an average of 1,141
information concerning the H1N1 flu vaccine. Canadian web users completed the survey.
2. Data collection. All the search string queries were submitted Table 1 shows the relative response rate of Canadian Internet
on a daily basis into Google
Search, and search results were
collected in a database, with
duplicate URLs removed. The Figure 1. Screenshot of the Flu Chat Lab
first 64 top-ranked search results
(for each search string) were
collected daily.
3. Scoring. Each unique search
re s u l t w a s m e a s u re d f o r
“mentions” – that is, the degree
to which the URL was shared
by global Internet users across
the web – to establish a “chat
level” in “decibels” (dB). The
aggregator tool counted the
number of mentions of every
search result in the database and
assigned it a chat level. The chat
level was calculated in units of
decibels (dB) as 20 LOG10 (9
+ Mentions). This method of
measuring chat level is analogous
to the measurement by audio
engineers of intensity, loudness
and power. The logarithmic scale
allowed us to visualize mentions
as “audio intensity” on a linear
scale.

10 Healthcare Quarterly Vol.13 Special Issue September 2010

 Neil Seeman et al. Assessing and Responding in Real Time to Online Anti-vaccine Sentiment during a Flu Pandemic

vaccine safe, while

Table 1. Survey response by region compared with percentage of Canadian population 41.4% thought it was
unsafe and 35.2%
Percentage from Each Region Region Population as reported ambivalence
Region (N = 27,382) Percentage of Population* over its safety. Over
Alberta 12.3 10.9 the 24 days surveyed,
the percentage of
British Columbia 12.0 13.2 those who said the
vaccine was not safe
Manitoba 3.4 3.6
peaked at 45.3% (on
New Brunswick 2.0 2.2 November 18, 2009).
The general trend line
Newfoundland and Labrador 1.2 1.5
for those who felt the
Nova Scotia 3.4 2.7 vaccine was safe stayed
relatively static, with
Northwest Territories 0.1 0.1 a low of 21.0% (on
Nunavut 0.1 0.1 November 17) and
a high of 28.4% (on
Ontario 41.2 38.7 October 29), two days
Prince Edward Island 0.5 0.4
after Health Canada
had approved the
Quebec 20.4 23.2 H1N1 vaccine and
public health commu-
Saskatchewan 3.2 3.1
nications efforts were
Yukon 0.1 0.1 most visible in online
and print media.
*Source: Statistics Canada (2009a).
Step B Findings:
Aggregating
Perceived
Table 2. Survey response by sex compared to percentage of Internet users Web-Posted Myths and Facts
about the Flu Vaccine
Response Rate (%) Sex as Percentage of Our Flu Chat Lab aggregator
Sex (N = 27,382) Canadian Internet Users*
showed that, from October 27,
Male 38.8 50.2 2009, to the date of analysis (April
6, 2010), websites containing
Female 61.2 49.8
anti-vaccine sentiment remained
popular. The number of search
*Source: comScore Networks Inc. (2008).
results we collected as of the time of
writing was 17,392. The distribu-
tion of search results about vaccine
users to the survey, comparing the response rate in each juris- safety by chat level is described in Table 4. Appendix 1 identifies
diction to its proportion in the national population. Table 1 the top 20 search results discussing the safety of the vaccine.
shows responses by region compared with the percentage of Twelve of the 20 URLs (60%) in Appendix 1 (http://www.
the Canadian population, and Table 2 shows responses by sex longwoods.com/content/ 21923) contain anti-vaccine senti-
compared with the Canadian population. Table 3 shows the ment (numbers 1, 2, 3, 5, 7, 9, 10, 14, 15, 16, 17 and 19).
survey responses by age compared with the relative frequency Three of 20 (15%) are government sources (numbers 8, 12 and
of Internet use by age group (from the most comparable data 20). Our methodology indicated that each of these 20 URLs
source available). had been shared and viewed over 1,300 times (i.e., passed from
Our survey findings, illustrated in Figure 2, show that one Internet user to another via social networks such as
an average of 23.4% of Canadians surveyed considered the Facebook, Twitter, YouTube or Digg). The top-ranked URL in

Healthcare Quarterly Vol.13 Special Issue September 2010 11

Assessing and Responding in Real Time to Online Anti-vaccine Sentiment during a Flu Pandemic Neil Seeman et al.

tion; however, with Canadian Internet

Table 3. Survey response by age compared to relative Internet usage usage being 70% or over for ages 15–64
(over 80% for those 15–54 and 70% for
Response Rate (%) Relative Internet Usage, from Most those 55–64 and rising (Statistics Canada
Age (y) (N = 27,382) Comparable Data Source Available*
2009b), our survey accessed at least as
<18 41.3 23.8 (for ages 12–17) wide a net of potential respondents as
computer-assisted telephone interviewing.
18–29 37.8 22.3 (for ages 18–29)
A growing percentage of Canadians do
30–49 13.7 21.8 (for ages 30–44) not own a telephone landline or they
block telemarketers and polling compa-
50–64 3.2 19.4 (for ages 45–59) nies from contacting them. At 15.6%, the
65+ 3.9 12.7 (for ages 60+) relatively low response rate could suggest
important differences between those who
*Source: Zamaria and Fletcher (2008). responded and the majority who did not.
For example, the respondents might have
had more time available to complete the
survey than non-respondents. However,
Table 4. Distribution of URL postings about vaccine safety by the survey took only 10–60 seconds to
“chat level”
complete. The large sample size (N =
27,382) reduces concerns about coverage
Chat Level Count Percentage of Postings
bias. A response rate of 15.6% is reflec-
High (60 dB) 55 0.32 tive of web surveys generally, response rate
being a challenge for all online surveys,
Medium (40–59 dB) 475 2.73
even when pre-recruited panels are used
Low (20–39 dB) 2,139 12.3 (Couper and Miller 2008). We tried to
correct for a low response rate by using
None/minimal (0–19 dB) 14,723 84.7 large, geographically representative, daily,
Total 17,392 100.1 random sampling.
With regard to the findings of our Flu
URL = uniform resource locator. Chat Lab (step B), whose findings are
independent and should not be correlated
with those in step A, the dominant bias
Appendix 1 was shared and then viewed by the person to whom is the English-language nature of the postings aggregated. We
the link was sent over 9,600 times, as of the time of this writing. were limited by query limits imposed by Google Search in terms
As of April 6, 2009, the Flu Chat Lab showed that the video of the number of search results for each search string. Further,
“Girl Gets ‘Flu’ Shot and Now Can Only Walk Backwards” Google Search was the only search engine we used. However,
(number 2) had been shared 8,773 times and viewed 2,386,817 Google Search is by far the most popular search engine in
times, as indicated by YouTube. Canada, with the most number of web pages indexed compared
with any other search engine in the world.
Discussion It is not always possible to interpret the meaning of numbers
For our daily tracking survey (step A), we attempted to obtain a of mentions of websites since interest can be generated by
random sample of Internet users. Based on Tables 1, 2 and 3, we stories that are amusing, celebrity-focused or popular for other
feel we succeeded in approximating geographical representation reasons. Nevertheless, we did establish that anti-vaccine sites
across Canada, but Tables 2 and 3 reveal that our sample was were generating wide interest among Canadians. There is no
not representative in terms of sex and age, which may limit the direct evidence that viewing anti-vaccine sites led to the percep-
generalizability of our findings. The online nature of our survey tion shared by 41.4% of Canadians surveyed that the H1N1
skewed the response toward a younger demographic. Women vaccine was not safe.
were overrepresented, possibly due to the fact that women
are usually the primary caregivers and would thus be more Future Policy Directions
concerned about vaccine safety. One could question whether Prior research has suggested that health-related blogs with
the universe of Internet users is representative of the popula- clinically relevant and accurate chronic illness information are

12 Healthcare Quarterly Vol.13 Special Issue September 2010

 Neil Seeman et al. Assessing and Responding in Real Time to Online Anti-vaccine Sentiment during a Flu Pandemic

Figure 2. Percentage of Canadians, each day, saying “yes,” “no” or “I don’t know” to the question,
“Do you think the H1N1 flu vaccine is safe?”

50.0%

45.0%

40.0%

35.0%

30.0%

25.0%

20.0%

15.0% yes

10.0% no

5.0% I don’t know

0.0%
Oct. 27

Oct. 28

Oct. 29

Oct. 30

Oct. 31

Nov. 1

Nov. 2

Nov. 3

Nov. 4

Nov. 5

Nov. 6

Nov. 7

Nov. 8

Nov. 9

Nov. 10

Nov. 11

Nov. 12

Nov. 13

Nov. 14

Nov. 15

Nov. 16

Nov. 17

Nov. 18

Nov. 19
frequently the most viewed and, increasingly, the most trusted health problems and needs. Highly shared information can
by Internet users (Seeman 2009). Nonetheless, our findings in help to guide the improved visual design, features and language
step B show that sustained anti-vaccine sentiment continues to formatting for online health tools that target subpopulations
be viewed and shared actively on the web. Our findings from step of interest. Applying both qualitative and quantitative analyses
A show that Canadian Internet users, even after the approval of to blogs and exchanges on social networks can potentially tap
the H1N1 vaccine in Canada, were skeptical, over the course of into the perceptions of large numbers of people with respect to
24 days, about the vaccine’s safety. This suggests that public health many health issues other than vaccinations (e.g., satisfaction
authorities may need to use “counter-marketing” strategies. with healthcare services, pathways to care and outcomes and
So-called counter-marketing is a growing social marketing overall experiences of care).
strategy that has been effective in tobacco control (countering Using tools similar to the ones described here, hospitals,
the messages of tobacco companies) (Evans and McCormack public health agencies, health regions and health ministries can
2008). An effective counter-marketing strategy can proactively learn about the extent and causes of the public’s anti-vaccine
identify and expose misinformation and anecdotal evidence that sentiments and devise methods to effectively neutralize them.
“tugs at the heart strings” in near real time (Davies et al. 2002). For example, an independent evaluation unit staffed with expert
Our novel approaches to determining public attitudes to clinical reviewers and social media experts could create a running
healthcare issues using real-time Internet data gathering can be search string methodology (in both official languages and in other
applied more broadly to understand public sentiment, at low languages reflective of Canada’s diverse population) akin to our
cost and with rapidity, on a broad range of policy issues. Step A, approach in step B. This approach would identify, in real time,
for example, has already been used to determine public attitudes which websites were disseminating popular anti-vaccine–related
among Canadians toward providing social supports (e.g., help information. Sites with anti-vaccine sentiment that were growing
with the laundry or other household chores) for chronically ill in intensity could be flagged. These findings could be stored in a
neighbours (Seeman and Brown in press). Step B can be used to secure database accessible by website editors working with public
determine the readiness with which people share online infor- health officials at the national, provincial and local levels, and by
mation with their peers about taboo subjects such as mental web editors working with hospitals and health regions.

Healthcare Quarterly Vol.13 Special Issue September 2010 13

Assessing and Responding in Real Time to Online Anti-vaccine Sentiment during a Flu Pandemic Neil Seeman et al.

While we have not provided direct evidence that Internet region and hospital could engage in meaningful dialogue with
viewing was responsible for the low uptake of the H1N1 locally influential dissenters. Those with disproportionate influ-
vaccine, Betsch et al. (2010) have shown that accessing vaccine- ence in a particular geographical region could be identified by the
critical websites for five to 10 minutes increases the percep- evaluation unit using detection methods geo-located to towns,
tion of risk of vaccinating and decreases the perception of risk cities and provinces. Local authorities could choose with whom
of not vaccinating. Intentions to vaccinate are diminished by to engage in dialogue in order to achieve maximum impact.
such viewing. Vaccine-critical websites therefore potentially
contribute to changes in risk perception, which, in turn, can Conclusion
affect the public’s willingness to get vaccinated. Future research The web contains much flu-related anti-vaccine sentiment that
should validate the extent to which website information does, in is potentially dangerous to the perceptions of risk and a willing-
fact, influence perceptions of vaccine safety, public willingness ness to get vaccinated. This challenge, we feel, can potentially be
to get vaccinated and other areas of patient safety. Given the mitigated using real-time web analytics. Current public health
amount of resources that companies, charitable organizations communication and education strategies can be complemented
and healthcare organizations are currently investing in viral by web analytics that identify and track anti-vaccine sentiment
advertising on the web, particularly on social networks, it is on the Internet. A collaborative counter-marketing model can be
likely to have some impact (Seeman 2008); but the exact extent supported by the type of real-time daily Internet tracking survey
of this impact is hard to assess beyond traditional metrics such described in this article (i.e., step A), making it possible – at a
as website hits and trends in site usage (e.g., Alexa.com). community, city, province, region, or country level – to monitor
There is some literature suggesting that people seek out the success of the collaborative counter-marketing strategy.
information that confirms their existing attitudes (Sunstein When, in a future pandemic, scarce public health resources need
2007); therefore, the impact of misinformation about the to be shifted rapidly to regions where anti-vaccination senti-
vaccine might be less than we think. Given the thousands of ment runs high, a method of collaborative counter-marketing,
websites being created every second, competition for public as described here, can provide the public with accurate risk
attention online is extremely challenging. Therefore, measures information, which should help to boost vaccination rates and
of engagement of the target population, rather than simple thereby enhance public safety.
website hits, are more valuable to assessing the impact of any
online healthcare intervention or information tool. The degree Acknowledgements
to which the public shares websites (our process in step B), The RIWI Corporation conducted the online survey for step
rather than website hits, is one such measure of engagement. A of this article without compensation, in the public interest.
Other measures of value include the degree to which poten- Bob Seeman, a director of RIWI, is the brother of Neil Seeman.
tial users of a website can become aware of its existence. Such The Health Strategy Innovation Cell is supported by a grant
measures might include the Google “footprint” for the website from the Ontario Ministry of Health and Long-Term Care.
of interest (“geo-located” to URLs for the target populations);
“in-links” to the website; and the extent to which the site is References
visible on Google when the target populations actively search Alvarez-Pasquín, A.J., H. Heijbel, J. Yarwood, P. Van Damme and
VACSATC partners. 2009. “VACSATC (Vaccine Safety: Attitudes,
for related information (e.g., cancer and depressed for people Training and Communication): Why Such a Project?” Euro Surveillance
suffering from these overlapping conditions). 14(16): 1-4.
Baeyens, J.P. 2010. “Ensuring the Willingness to Vaccinate and Be
Our Approach: Collaborative Counter- Vaccinated.” Expert Review of Vaccines 9(3 Suppl.): 11–4.
Marketing to Address Anti-vaccine Sentiment Betsch, C., F. Renkewitz, T. Betsch and C. Ulshöfer. 2010. “The
What we call a “collaborative counter-marketing model” Influence of Vaccine-Critical Websites on Perceiving Vaccination Risks.
involves engaging in publicly viewable web discussions with the Journal of Health Psychology 15(3): 446–55.
authors of the anti-vaccine postings. This could be accomplished comScore Networks, Inc. 2008. Canadians Online. Reston, VA:
through an independent evaluation unit’s ongoing postings or Author. Retrieved April 6, 2010. <http://ir.comscore.com>.
annotations to the content posted by vaccine dissenters. This Couper, M.P. and P.V. Miller. 2008. “Web Survey Methods.” Public
approach would show constructive, transparent engagement Opinion Quarterly 72(5): 831–35. Retrieved April 9, 2010. <http://
poq.oxfordjournals.org/cgi/content/full/72/5/831>.
and provide demonstrable evidence that a counter-marketing
strategy is meant to provoke dialogue, not shut down dissent. Davies, P., S. Chapman and J. Leask. 2002. “Antivaccination Activists
on the World Wide Web.” Archives of Disease in Childhood 87(1): 22–5.
The public perception of shutting down dissent could, in itself,
Evans, W.D. and L. McCormack. 2008. “Applying Social Marketing
raise skepticism and anti-vaccine sentiment that could accumu- in Health Care: Communicating Evidence to Change Consumer
late on blogs and online forums. Under our approach, each Behavior.” Medical Decision Making 28(5): 781–92.

14 Healthcare Quarterly Vol.13 Special Issue September 2010

 Neil Seeman et al. Assessing and Responding in Real Time to Online Anti-vaccine Sentiment during a Flu Pandemic

Gollust, S.E. and P.M. Lantz. 2009. “Communicating Population Wolfe, R.M., L.K. Sharp and M.S. Lipsky. 2002. “Content and Design
Health: Print News Media Coverage of Type 2 Diabetes.” Social Science Attributes of Antivaccination Web Sites. Journal of the American
and Medicine 69(7): 1091–98. Medical Association 287(24): 3245–48.
Harris-Decima. 2009. “Concern over H1N1 Jumps Significantly.” Zamaria, C. and F. Fletcher. 2008. Canadian Internet Project (Based
Ottawa, ON: Author. Retrieved April 9, 2010. <http://www.harris- on 2007 Data). Retrieved April 9, 2010. Toronto, ON: Author.
decima.com/releases/2009/11/03/365-concern-over-h1n1-jumps-si]>. <http://www.ciponline.ca/en/docs/2008/CIP07%20CANADA%20
Hesse, B.W., D.E. Nelson, G.L. Kreps, R.T. Croyle, N.K. Arora, B.K. ONLINE-HIGHLIGHTS.pdf>.
Rimer et al. 2005. “Trust and Sources of Health Information: The
Impact of the Internet and Its Implications for Health Care Providers:
Findings from the First Health Information National Trends Survey.” About the Authors
Archives of Internal Medicine 165(22): 2618–24. Neil Seeman, JD, MPH, is a member of the Health Strategy
Innovation Cell, Massey College, at the University of Toronto,
Jefferson, T., A. Rivetti, A. Harnden, C. Di Pietrantonj and V. and the Department of Health Services Management, Ryerson
Demicheli. 2008. “Vaccines for Preventing Influenza in Healthy University, in Toronto, Ontario. You can contact him by phone
Children. Cochrane Database of Systematic Reviews 2: CD004879. at 416-205-9984, by fax at 416-352-5071 or by email at neil.
DOI: 10.1002/14651858.CD004879.pub3. [email protected].
Mareiniss, D.P., J.M. Hirshon and B.C. Thibodeau. 2009. “Disaster
Planning: Potential Effects of an Influenza Pandemic on Community Alton Ing, BaSc, MaSc, is a member of the Health Strategy
Healthcare Resources.” American Journal of Disaster Medicine 4(3): Innovation Cell, Massey College, at the University of Toronto.
163–71. Carlos Rizo, MD, is a member of the Health Strategy
Maurer, J., L. Uscher-Pines and K.M. Harris. 2010. “Perceived Innovation Cell, Massey College, at the University of Toronto.
Seriousness of Seasonal and A(H1N1) Influenzas, Attitudes toward
Vaccination, and Vaccine Uptake among U.S. Adults:
Does the Source of Information Matter?” Preventive
Medicine 51(2):185–87.
Rizo, C.A, D. Lupea, H. Baybourdy, M. Anderson,
T. Closson and A.R. Jadad. 2005. “What Internet
Patient and staff safety
Services Would Patients Like from Hospitals during
an Epidemic? Lessons from the SARS Outbreak in
Toronto.” Journal of Medical Internet Research 7(4):
are inseparably linked
e46.
Seale, H., A.E. Heywood, M.L. McLaws, K.F. Ward, At Vancouver Coastal Health, we
believe that safe and healthy staff
C.P. Lowbridge, D. Van et al. 2010. “Why Do I Need ensure safe patient care.
It? I Am Not at Risk! Public Perceptions towards the
Pandemic (H1N1) 2009 Vaccine.” BMC Infectious We are committed to active involvement
Disease 19(10): 99. in targeting improvements to the
Seeman, N. 2008. “Web 2.0 and Chronic Illness: New health and well being of employees,
the healthiness of the workplace and
Horizons, New Opportunities.” Electronic Healthcare the sustainability of the health care
6(3): 104–10. system. We encourage a safety culture
Seeman, N. 2009. “Inside the Health Blogosphere: of open communication that invites
Quality, Governance and the New Innovation participation and knowledge exchange.
And we extend our health and safety
Leaders.” Electronic Healthcare 7(3): 101–08. efforts beyond the workplace by
Seeman, N. and A.D. Brown. In press. “We Are Not engaging with our community partners
Bowling Alone.” Queen’s Quarterly 117(2). and educational institutions, resulting
in benefits that contribute to healthy
Seeman, N., G.R. Baker and A.D. Brown. 2008. workplaces and to patient safety.
“Emergency Planning in Ontario’s Acute Care
Hospitals: A Survey of Board Chairs.” Healthcare Vancouver Coastal Health. Delivering
Policy 3(3): 65–74. primary through quaternary care and Mary M., Practice Initiatives Lead
services to residents of Metro Vancouver Vancouver Coastal Health
Sunstein, C.R. 2007. Republic.com 2.0. Princeton, NJ: and across British Columbia.
Princeton University Press.
Statistics Canada. 2009a. Population by Year, by
Province and Territory. Ottawa, ON: Author. Retrieved Care. Teaching. Research.
April 9, 2010. <http://www40.statcan.gc.ca/l01/
cst01/demo02a-eng.htm>.
Statistics Canada. 2009b. Rates of Internet Use, by Age
www.vch.ca
Group, Selected Years (Based on 2007 Data). Ottawa,
ON: Author. Retrieved April 9, 2010. <http://www.
statcan.gc.ca/pub/11-008-x/2009002/c-g/10910/c-
g001-eng.htm>.

Healthcare Quarterly Vol.13 Special Issue September 2010 15

 developing information to improve safety

Assessing and Responding in Real Time to Online

Anti-vaccine Sentiment during a Flu Pandemic
Neil Seeman, Alton Ing and Carlos Rizo

Appendix 1. Most discussed 20 search results about the H1N1’s vaccine clinical safety (as of April 6, 2010)

Rank out of Top 20 ‘Chat Level’ Web Posting Describing URL Content Source Site and Context
(N = 17,392) (dB)

1 80.0 “I was exposed to 2 people on 2 diff occasions, both http://www.youtube.com

had swine flu and im fine. all u need is vit C and fish [Comment by bobthe24 on YouTube]
oil. and oh by the way, i work in a convenience store
and am exposed to hundreds of people per day and
their filthy money.”

2 79 “Girl gets ‘Flu’ Shot & Now Can Only Walk Backwards http://www.youtube.com
… Experts claim serious side effects of flu shots [Video posted by freshtildeathonline on
amount to about one in a million – well this is what YouTube]
that one in a million looks like. This has freaked me
out enough to bypass flu shots in the future.”

3 77 “The current strain, A(H1N1), is a new variation of an http://articles.mercola.com

H1N1 virus – which causes seasonal flu outbreaks in [Article titled “Critical Alert: The Swine Flu
humans – that also contains genetic material of bird Pandemic – Fact or Fiction?”]
and pig versions of the flu. Symptoms include: Fever
of more than 100; Coughing; Runny nose and/or sore
throat; Joint aches; Severe headache; Vomiting and/
or diarrhea; Lethargy; Lack of appetite …”

4 76 “Now some flu experts are challenging the medical http://theatlantic.com

orthodoxy and arguing that for those most in need of [Article titled “Does the Vaccine Matter? – The
protection, flu shots and antiviral drugs may provide Atlantic (November 2009)]
little to none … while vaccines for, say, whooping
cough and polio clearly and dramatically reduced
death rates from those diseases, the impact of flu
vaccine has been harder to determine.”

5 73 “It was previously revealed that some batches of http://www.prisonplanet.com and aggregated
the vaccine will contain mercury, a toxin linked with at http://jimcorr.com
autism and neurological disorders. The vaccine will [Article titled “CDC Warns Neurologists to
also contain the dangerous ingredient squalene, Watch for Nerve Disease following Swine Flu
which has been directly linked with cases of Gulf War Shots”]
Syndrome and a host of other debilitating diseases.”

6 71 “Pregnant mothers are more [at] risk for H1N1 and http://informationisbeautiful.net
are more likely to be hospitalized.” [Article titled “Is the H1N1 Swine Flu Vaccine
Safe? What if I’m Pregnant?”]

Healthcare Quarterly Vol.13 Special Issue October 2010 1

Assessing and Responding in Real Time to Online Anti-vaccine Sentiment during a Flu Pandemic Neil Seeman et al.

Rank out of Top 20 ‘Chat Level’ Web Posting Describing URL Content Source Site and Context
(N = 17,392) (dB)

7 70 “Flu Vaccine DOUBLES Risk of Getting H1N1 … It http://articles.mercola.com

is also important to view this in the face of the new [Article titled “Swine Flu – One of the Most
unpublished Canadian study of 12 million people Massive Cover-ups in American History”]
that found getting the seasonal flu vaccine, as
recommended by the CDC and NIH, doubles one’s
risk of developing the H1N1 infection. It would also
make the infection much more serious. So much for
expert advice from the government.”

8 70 “It is safe for children and pregnant women to http://flu.gov

receive a flu vaccine that contains thimerosal.” [Assistant Surgeon General Dr. Anne Schuchat
dispels myths about the H1N1 flu virus on The
Doctors]

9 68 “Pregnant women, children, and those whose http://articles.mercola.com

immune system are already compromised are also [Comment by “Dr. Mercola” called “Warning:
at greater risk to suffer severe side effects from the Swine Flu Shot Linked to Killer Nerve Disease”]
vaccine, as the adjuvants in the vaccine are designed
to reduce your immune function.”

10 68 “Vaccinations do not optimize immune function and http://articles.mercola.com

there is way too little information about what else [Comment by “Dr. Mercola” called “Alert:
they do in the human body that may contribute to Special Swine Flu Update”]
poor health. My main point is that there are ways
to prevent illness and maintain health that are so
much more natural, easier, less expensive and more
effective than routine use of multiple vaccines in
childhood and throughout life.”

11 68 “Getting vaccinated is by far the most effective thing http://www.newscientist.com

you can do to protect yourself and your family from [Article titled “Swine Flu: Eight Myths That
swine flu.” Could Endanger Your Life”]

12 66 “Flu should never be dismissed as ‘just the flu.’ It’s http://cdc.gov

a serious disease, and while most cases are mild, [CDC’s Questions and Answers: “Vaccine
some can be deadly. So far this flu season, most flu against 2009 H1N1 Influenza Virus”]
activity has been caused by the 2009 H1N1 virus,
which was first identified in April 2009 and caused
the first flu pandemic in 40 years.”

13 66 “Your immune system recognizes squalene as an oil http://articles.mercola.com

molecule native to your body. It is found throughout [Comment by “Dr. Mercola” called “Squalene:
your nervous system and brain. In fact, you can The Swine Flu Vaccine’s Dirty Little Secret
consume squalene in olive oil and not only will your Exposed”]
immune system recognize it, you will also reap the
benefits of its antioxidant properties. The difference
between ‘good’ and ‘bad’ squalene is the route by
which it enters your body. Injection is an abnormal
route of entry which incites your immune system
to attack all the squalene in your body, not just the
vaccine adjuvant. Your immune system will attempt
to destroy the molecule wherever it finds it, including
in places where it occurs naturally, and where it is
vital to the health of your nervous system.

2 Healthcare Quarterly Vol.13 Special Issue October 2010

 Neil Seeman et al. Assessing and Responding in Real Time to Online Anti-vaccine Sentiment during a Flu Pandemic

Rank out of Top 20 ‘Chat Level’ Web Posting Describing URL Content Source Site and Context
(N = 17,392) (dB)

14 65 “Study after study, and master studies that http://articles.mercola.com

compile the results from several studies to get a [Comment by “Dr. Mercola” called “Flu Vaccine
more objective result, keep coming to the same Exposed”]
conclusion: Flu vaccines DO NOT WORK, and in
many cases do more harm than good.”

15 65 “The risks of these, and the current fast-tracked http://articles.mercola.com

swine flu vaccine, are truly unknown at this time. [Comment by “Dr. Mercola” called “CBS Reveals
There is NO Incentive for the CDC or Vaccine That Swine Flu Cases Seriously Overestimated”]
Manufacturers to Care About Safety.”

16 65 “None of the vaccines have been subjected to http://www.naturalnews.com

real-world testing for any meaningful duration. [Article titled “Ten Swine Flu Lies Told by the
The ‘safety’ of these vaccines is nothing more than Mainstream Media”]
wishful thinking.”

17 64 “Adjuvants are used in vaccines in order to create http://articles.mercola.com

a sufficiently strong immunological response. [Comment by “Dr. Mercola” called “Expert
Adjuvants augment your immunological response. Pediatrician Exposes Vaccine Myths”]
But there is clear evidence that adjuvants, like
aluminum and squalene impair your immune system.

18 63 “If you have any of the conditions that put you at http://www.webmd.com
increased risk of severe H1N1 swine flu – pregnancy, [Article titled “Swine Flu FAQ”]
asthma, lung disease, diabetes, heart disease,
neurologic disease, immune suppression, or other
chronic conditions – it may be riskier for you to get
the flu than for other people.”

19 63 “There is absolutely no scientific evidence http://www.naturalnews.com

demonstrating that the swine flu vaccine offers real- [Article titled “Vaccine Revolt! Swine Flu
world protection against the circulating H1N1 virus. Vaccine Support Crumbles”]
While there is evidence showing that it produces
antibodies, as people who have studied vaccine
effectiveness already know, the mere existence of
vaccine-induced antibodies doesn’t directly translate
into functional, real-world immunity! Vaccines are
often, in effect, fabricated immunity that often don’t
provide practical immune defense in the real world.”

20 63 “The H1N1 vaccine is expected to be as safe as http://www.azdhs.gov

seasonal flu vaccine. H1N1 vaccine is made in the [“Fact sheet” from the Arizona government’s
same way and in the same licensed facilities used news release on H1N1]
to make seasonal flu vaccines. People with a severe
allergy to chicken eggs or to another substance in
the vaccine, or who have had a serious reaction to
flu vaccine should not be vaccinated.”

Healthcare Quarterly Vol.13 Special Issue October 2010 3