Proceedings of the Seventeenth International AAAI Conference on Web and Social Media (ICWSM 2023)

What Are You Anxious About? Examining Subjects of Anxiety during the
COVID-19 Pandemic
Lucia L. Chen1 , Steven R. Wilson2 , Sophie Lohmann3 , Daniela V. Negraia3,4
1
Department of Health Policy, Stanford University, United States
2
School of Engineering and Computer Science, Oakland University, United States
3
Max Planck Institute for Demographic Research, Rostock, Germany
4
Department of Sociology, Oxford University, United Kingdom
[email protected], [email protected], [email protected], [email protected]

Abstract infected with COVID-19, and distress caused by informa-

tion from social media were reported by 12.5%, 37.8%, and
COVID-19 poses disproportionate mental health conse-
36.4% of participants, respectively (Roy et al. 2020).
quences to the public during different phases of the pandemic.
We use a computational approach to capture the specific as- Anxiety is conceptualized as a multi-system response to
pects that trigger an online community’s anxiety about the perceived risks, experienced as a feeling of unease, worry,
pandemic and investigate how these aspects change over time. or fear (Wilkinson 2001). Experiencing occasional anxiety
First, we identified nine subjects of anxiety (SOAs) in a sam- about a situation is a normal part of life. However, prolonged
ple of Reddit posts (N =86) from r/COVID19 support using or frequent anxiety may lead to general anxiety disorders,
thematic analysis. Then, we quantified Reddit users’ anxi- heightened depressive symptoms, reduction of sleep quality
ety by training algorithms on a manually annotated sample (Huang and Zhao 2020), and other mental health risks. Psy-
(N =793) to automatically label the SOAs in a larger chrono-
chological studies often use scales to assess levels of anx-
logical sample (N =6,535). The nine SOAs align with items
in various recently developed pandemic anxiety measurement iety. Studies that examine COVID-19 anxiety mainly fall
scales. We observed that Reddit users’ concerns about health into two categories: those that treat anxiety as a clinical en-
risks remained high in the first eight months of the pan- tity with a focus on the symptoms (Ahorsu et al. 2020) and
demic. These concerns diminished dramatically despite the those that examine the specific aspects of the pandemic that
surge of cases occurring later. In general, users’ language dis- trigger anxiety (McElroy et al. 2020; Taylor et al. 2020). Al-
closing the SOAs became less intense as the pandemic pro- though methodologies that use a survey approach with scale
gressed. However, worries about mental health and the future measurements can provide detailed descriptions of individ-
increased steadily throughout the period covered in this study. ual respondents, their implementation is expensive and time-
People also tended to use more intense language to describe consuming.
mental health concerns than health risks or death concerns.
Our results suggest that this online group’s mental health con- In an effort to collect data in a timely and cost-effective
dition does not necessarily improve despite COVID-19 grad- manner, researchers have been exploring avenues to approx-
ually weakening as a health threat due to appropriate coun- imate the anxiety of the public using proxy signals in social
termeasures. Our system lays the groundwork for popula- media records (Guntuku et al. 2019). Our study does not aim
tion health and epidemiology scholars to examine aspects that to replace the measurement using scales because some of the
provoke pandemic anxiety in a timely fashion. behavioral, affective, and cognitive characteristics measured
with their items may rarely be reflected in the social me-
Introduction dia text. Rather, our automatic system may provide a proxy
value for the scientific and public health community to un-
The novel Coronavirus Disease (COVID-19) pandemic has derstand better how the public responds to large-scale health
created a global crisis. Unlike other recent pandemics (e.g., threats promptly.
H1N1 or type-A influenza), the COVID-19 pandemic has re-
Our objective is to use data from Reddit to understand the
sulted in strict and extensive lockdown measures for large
aspects that provoke people’s anxiety during early phases of
swathes of the global population, such as complete lock-
the pandemic and how these aspects have changed as the
downs, 14-day quarantine periods, closing of national bor-
pandemic has developed. Reddit is an online platform in
ders, and disruption of international travel. A series of life
which a network of communities is organized around users’
interruptions resulting from the COVID-19 pandemic are re-
interests. Reddit users engage with these communities, also
lated to heightened anxiety and depression in many people
known as subreddits, by submitting text posts, links, im-
(Smith et al. 2020). In a 2020 study, over 80% of respon-
ages and comments. In line with the platform’s conven-
dents reported that their day-to-day thoughts were occupied
tions, for the remainder of this paper we will use the pre-
by topics related to the COVID-19 pandemic (Roy et al.
fix “r/” to denote specific subreddit communities (e.g., r/-
2020). Sleep difficulties, extreme anxiety about becoming
COVID19 support; a subreddit that is specifically for peo-
Copyright © 2023, Association for the Advancement of Artificial ple sharing advice and coping mechanisms for COVID-19).
Intelligence (www.aaai.org). All rights reserved. Focusing on a sample of Reddit users who have posted in r/-

137
COVID19 support, we propose the following research ques- way III, Woodard, and Zubrod 2020; Taylor et al. 2020). A
tions (RQs): machine-annotated pipeline allows us to track how these as-
• RQ1: What are the subjects of anxiety (SOAs) expressed pects change across time. Fourth, by evaluating our results
in r/COVID-19 support? How do the SOAs change over on new incoming data, we highlight the importance for re-
time? searchers to update their knowledge about social media con-
tent while constructing models to infer mental health sta-
• RQ2: How does the intensity of anxiety-related language tus. Our approach can help public health practitioners un-
change over time? derstand how the public responds and adjusts to a pandemic
• RQ3: Which SOAs were described by users with high in real-time.
anxiety-related language?
To answer these questions, we first conduct a thematic Prior Work
analysis to identify the subjects of anxiety (SOAs) men-
tioned in posts from r/COVID19 support. Thematic analy- Disclosure of Anxiety on Social Media
sis is a qualitative data analysis method for identifying top- Seeking support can be helpful for people with anxiety
ics and concepts that occur repeatedly in the data. Follow- (Roohafza et al. 2014). Support is especially important dur-
ing the thematic analysis, we identified nine SOAs in the ing a pandemic when stay-at-home orders curtail the possi-
Reddit posts. Then we define detailed annotation guidelines bility of in-person consultations between practitioners and
for identifying these SOAs in 793 posts (12% of the data) clients. However, people often find it difficult to do so for
extracted from r/COVID19 support. Each post could con- many reasons. Some people may hesitate to seek profes-
tain more than one SOA. At the time we developed our an- sional help and doubt the efficacy of medical resources be-
notation guidelines, there were no published guidelines on cause they attribute mental illnesses to personal weakness
measuring COVID-19-related anxiety. Later on, we found rather than illness (Yap, Wright, and Jorm 2011).
that the SOAs in our annotation guidelines overlapped with Fortunately, help-seeking behavior can be facilitated by
items in the recently developed COVID-19 anxiety mea- various forms of social media. Social media platforms such
surement scales (C-19ASS) (18 items) (Nikčević and Spada as Reddit that encourage anonymity allow people to talk
2020), COVID-19 Concern Questionnaire (CCQ)(6-items), about their mental health issues without fear of being judged
COVID-19 Experiences Questionnaire (CEQ) (14 items) or identified. During the COVID-19 pandemic, social me-
(Conway III, Woodard, and Zubrod 2020) and the COVID dia platforms, among other services, connected people who
Stress Scales (33-items) (Taylor et al. 2020), providing fur- were struggling with social isolation. These platforms facil-
ther support for the widespread existence SOAs which we itate the sharing of useful information during user interac-
identified (see table 1). tions which can help them to cope with anxiety (Wiederhold
Next, we turn our attention to the intensity of the lan- 2020).
guage used to describe each SOA. In the example “I am Audiences in support communities often respond soon af-
super anxious about...”, “super anxious” indicates that the ter a post is published, providing timely support to those
author has a high level of anxiety towards the SOA. Im- seeking help (Pfefferbaum and North 2020). Additionally,
portantly, however, the level of anxiety we annotated only helping other people deal with their struggles is not only
reflects the author’s anxiety towards the specific topic that beneficial for the recipient but also for the support provider
they wrote about in that particular post. We do not have in- (Brown et al. 2003). Prior work shows that online support
formation on their anxiety over time or about other topics communities provide mental health benefits to people seek-
and our analysis therefore does not reflect a holistic, clini- ing help from the platform (Bargh, McKenna, and Fitzsi-
cal definition of anxiety. Finally, using the annotated posts, mons 2002). The linguistic attributes of social media con-
we train multi-label classifiers based on both support vector tent also reflect users’ depressive symptoms (Chancellor and
machines (SVMs) and deep learning architectures. We then De Choudhury 2020), addiction (Murnane and Counts 2014)
use the SVM classifier to annotate the remaining posts in r/- and other mental health concerns.
COVID19 support (N = 5747), then analyze the trends and During a pandemic, social media users often describe
patterns of the machine-annotated labels. their struggles in plain text. Anxiety is often expressed in
social media posts and recent studies have taken steps to
Contributions document it. For example, Jones and Silver (2020) anno-
This study makes several methodological and substantive tated anxiety expressed in social media posts according to
contributions to the study of mental health using social me- anxiety words measured by the Linguistic Inquiry and Word
dia data. First, by examining data from an online social me- Count (LIWC) program. In LIWC, words that reflect anx-
dia platform where users can maintain their anonymity, we iety include “afraid”, “scared”, “worried” and 111 other
had the opportunity to study naturalistic reports about peo- words (Boyd et al. 2022). Shen and Rudzicz (2017) used
ple’s personal life experiences during a global pandemic. textual features (e.g., topic modeling, word embeddings)
Second, we developed an annotation scheme for aspects to distinguish posts in anxiety-related Reddit communities
that trigger anxiety among social media users. Third, we from other Reddit communities. Recently, researchers de-
developed a machine-annotated pipeline to identify these veloped tools that facilitate early recognition of hot-spots of
aspects, adding new insights to the extant knowledge de- declining mental health by detecting anxiety language from
rived from survey studies (Nikčević and Spada 2020; Con- tweets (Guntuku et al. 2020). During COVID-19, compu-

138
 Scale-item SOA SOAs annotation guidelines
The Coronavirus Anxiety Syndrome Scale (CASS) -18 items
Avoided using public transport travel Author expresses feeling anxious about being in-
fected during the course of traveling, taking trains,
flights or public transportation. Feeling anxious
about how to travel when the city is in lockdown and
frustrated about not being able to travel.
Checked myself for symptoms of coron- health, infected Author shows signs of fear or anxious about self, a
avirus (COVID-19). friend/family member will be infected with COVID-
19. Or redditor listed symptoms and asked whether
they are infected. Or redditor listed symptoms and
thought they were infected.
Concerned about not having adhered strictly guideline Author shows signs of fear or anxious that family
to social distancing guidelines for coron- members, friends and strangers do not follow pre-
avirus (COVID-19). ventative measurements in the supermarket, public
transportation and any other enclosed environment,
feeling anxious about having to use a face mask for-
ever. Feeling anxious about what should be the ap-
propriate preventative measure in a given situation.
Researched symptoms of coronavirus health, infected Author shows signs of fear or feeling anxious that
(COVID-19) at the cost of off-line so- self, a friend or family member will be infected with
cial activities such as spending time with COVID-19. Post author researched and listed symp-
friends/family. toms, asking whether they are infected.
Paying close attention to others displaying health, infected Author shows signs of fear or anxiety that self,
possible symptoms of coronavirus (COVID- a friend or family member will be infected with
19). COVID-19. users researched and listed symptoms,
asking whether they are infected.
Imagined what could happen to my fam- death Author shows signs of fear or anxiety about that self,
ily members if they contracted coronavirus a friend, family member or someone they know will
(COVID-19). die of COVID-19 or become critically ill.
COVID-19 Concern Questionnaire - 6 items
Worried I or loved ones will get COVID-19. death Author shows signs of fear or anxiety about that self,
a friend, family member, or someone they know will
die of COVID-19 or become critically ill.
Practiced social distancing due to anxiety quarantine, social Author shows anxiety or fear about COVID-19 re-
about getting COVID-19. distancing and other strictions, such as when to quarantine, must quar-
COVID-19 restric- antine, doesn’t know when quarantine will finish, or
tions conflicts with family members in a quarantine situa-
tion. Struggles with quarantine and not seeing loved
ones.
Felt threatened, fear of, anxious about death Author shows signs of fear or anxiety that self, a
COVID-19. friend, family member, or someone they know will
die of COVID-19 or become critically ill.
COVID-19 Experiences Questionnaire - 14 items
Showed COVID-19 symptoms. health risk Author shows signs of fear or anxiety about being in-
fected with COVID-19. Author lists symptoms and
asks whether they are infected. Author lists symp-
toms and thinks they are infected. Author was in di-
rect contact with people who tested positive and is
now afraid of being infected. Author went to public
places or travelled with public transport and is afraid
of infection.

Lost job-related income, financial insecurity. financial/career Author expresses fear or anxiety about their finan-
cial, academic, or career situation. Consider if the
Author is seeking financial/ career advice. Consider
if the Author is anxious about COVID-19 interrupt-
ing school or career advancement.
Depressed, negative impact on psychological mental health Author shows signs of fear or anxiety about the men-
health. tal health of self, friends or family. Author focuses
on expressing their anxiety. Author seems to strug-
gle with mental health issues.

Table 1: Comparing COVID-19 anxiety measurement scale with SOA annotation. This table shows that our SOAs annotation
overlaps with many items in multiple COVID-19 anxiety measurement scales. Scale-item: items in the COVID-19 scales.
SOA: SOA in our annotation task. SOAs annotation guidelines: annotation guidelines for subjects of anxiety.

139
tational approaches revealed that language related to “eco- the analysis of social media text. Recently conducted content
nomic stress”, “isolation” and “home” increased signifi- analyses on Reddit posts often included 100-200 annotation
cantly among many Reddit mental health support groups documents (Maxwell et al. 2020). In this work, we randomly
(e.g., r/addiction, r/alcoholism, r/adhd, r/anxiety, r/autism, selected 90 posts from r/COVID-19 support between Febru-
r/BipolarReddit, r/bpd and r/depression) (Low et al. 2020). ary and August 2020 (around 10 posts per month). Posts
without content and those that only contained URLs or im-
Data ages were removed, resulting in 86 posts for annotation, with
Social media platforms feature different functionalities, and an average word count of 161. Since we have roughly 10
these functionalities influence users’ motivation to post on posts per month for the initial thematic analysis for subjects
the platform. Reddit is not only a platform for sharing news of anxiety, it is possible that some users’ sources of anxiety
and interests; it also comprises many mental health support may have been missed.
communities that allow people to focus on sharing, talk- Initially, the first author read and re-read a post to iden-
ing, and fostering group connections (Chen 2021). Reddit tify potential SOAs. The annotation includes a brief descrip-
is more suitable for identifying themes in discussions and tion of various SOAs identified in the text. Then the first
conversations than Facebook and Twitter. The anonymous author summarized the SOAs into several main groups and
nature of Reddit allows people to talk about their mental subgroups (see Figure 1). The groups were then forwarded
health issues without the fear of being stigmatized. In this to the third and fourth authors. The second level of analysis
study, we focus on a community (“r/COVID19 support”) involved both the third and fourth authors reviewing some of
for people interested in support navigating the COVID-19 these initial coding examples and the SOA groups. They par-
pandemic. “r/COVID19 support” is the only subreddit dedi- ticularly considered how to retain the initial codes’ diversity
cated to this specific purpose (news sharing was forbidden), while producing overarching elements and sub-groups. Dis-
which was created on 12th February 2020 and had 29.9k agreement of the codes were resolved through discussion.
users by April 2021.
We collected the data from the Reddit API using The
Manual Annotation of Subjects of Anxiety
Python Reddit API Wrapper (PRAW) (Boe 2016). The data After we identified the SOAs with thematic analysis, three
collection process took place between April 2020 and April authors of the present study compiled the annotation guide-
2021. Our dataset contains all available posts (N =6,540, lines (https://bit.ly/3mUYMGR). Next, we used stratified
1.92 posts per user) between the inception of the “r/- random sampling to select 100 posts from each month (be-
COVID19 support” community (12th February 2020) and tween February and Oct 2020). We pooled the data from
the end period of the present analysis (28th April 2021). This February and March 2020 because there were only 85 posts
dataset covers 13 months of the COVID-19 pandemic. Posts in February 2020. After removing posts that did not con-
that were flagged as “deleted” or “removed” by the post au- tain content or contain hyperlinks only, we obtained 793
thors were removed from the sample. posts for the manual annotation task. Two annotators cross-
annotated each post, and a third annotator resolved any an-
Ethics and Privacy notation conflicts. Three of the authors and a student helper
The data used in this study were publicly available when collaborated on this task, and the annotation agreement
collected from https://www.reddit.com/. Data were secured (Cronbach’s alpha) was 0.82.
on firewalled servers to ensure data protection, and re-
Identifying Anxiety-related Language
searchers could download the data only on local machines.
Researchers were not allowed to share data and had no inter- Annotating the intensity of users’ anxiety-related language.
action with the users. To protect the privacy and anonymity We further annotated the intensity of users’ anxiety-related
of the users in our dataset, we paraphrased the quotes pub- language present in each post. We defined three intensity
lished in this paper. The annotated dataset will not be re- levels: Level 0: user did not express anxiety or fear, for ex-
leased but the model pipeline is published on https://github. ample, “I came across this article from New York Times.
com/luciasalar/pandemic anxiety and annotation guidelines Thoughts on accuracy?” Level 1: user expressed some level
will be shared. of anxiety or fear but no urgency or desperation, for exam-
ple, “I found that I’m always thinking about COVID related
Methodology things. It seems so normal to me to feel this way now.” Level
2: user expressed extreme anxiety or fear, such as “I am so,
Thematic Analysis for Subjects of Anxiety so, so scared”, or “My anxiety is going through the roof”.
Identifying Users’ Subjects of Anxiety (SOA). We adopted The annotation agreement for the anxiety-related language
thematic analysis (Braun and Clarke 2006) to answer our was Cronbach’s alpha = 0.72.
first research question, “What are the SOAs expressed in Rater Credibility and Expertise Three out of the four an-
r/COVID-19 support?”. We refer to studies with a focus on notators are authors of this paper who have psychological
qualitative analysis to determine the number of posts we expertise and research experience in both quantitative and
need to analyze. In general, interview-based qualitative anal- qualitative studies of mental health. The fourth annotator
ysis researchers suggest that large interview studies often was a Master’s-level student in demography who received
contain 50-60 interviews (Britten 1995). In our case, there training from the study authors prior to completing the an-
are no existing guidelines on the number of participants for notation task.

140
Machine Annotation subsequent data. In the main model, we used models trained
We used the annotated data (N = 793) to construct a multi- on a dataset from Feb 2020 - Oct 2020 to predict data col-
label support vector machine (SVM) classifier, composed of lected after Oct 2020. We employed two approaches to ex-
an ensemble of binary classifiers, to automatically identify amine the model validation on chronological data. 1) “Last
the SOAs present in a given Reddit post. month holdout”. We constructed a set of models trained on
Combining Labels. We found that the posts with a lan- data from Feb 2020 - Sept 2020. Stratified five-fold cross-
guage intensity level 0 (no signs of anxiety) only accounted validation and a grid search of hyperparameters was car-
for about 10% of the data (N = 80). 80% (N = 64) of ried out for the classification algorithms. Then, we tested
the level 0 posts did not mention any SOA. This included the models on a holdout set in Oct 2020. 2) “Human evalua-
posts that only had a title but no content, or only emoji in tion on Incoming data”. We were also aware that new SOAs
the content. Level 0 posts were mainly noise for the anxiety might have emerged after we completed the thematic anal-
language analysis task, but we did not construct a separate ysis for SOAs in Aug 2020. To examine the model’s valid-
classifier to remove level 0 posts for this analysis because ity on new data, especially data after 2020, we conducted
the number of posts was too small for a training sample. human evaluation on 50 posts randomly selected during 1st
Instead, we aggregated level 0 with level 1 (posts express- Mar 2021 and 30th Apr 2021. The first author conducted an-
ing mild anxiety), then built a binary classifier for anxiety- other round of thematic analysis and SOA annotation on the
related language intensity, i.e., the classifier is trained to pre- 50 posts.
dict whether a given post’s intensity should be categorized as
level 2 (posts expressing extreme anxiety of fear) or not. Comparison to Deep Learning Models
Feature Selection. We adopted features (e.g., sentiment Some works on using social media data to infer mental ill-
and topics) that are commonly used in classifying mental ill- nesses symptoms adopted deep learning techniques (Shen
ness symptoms (Chancellor and De Choudhury 2020; Chen et al. 2018). Therefore, we also experiment with a fine-
et al. 2020). Features involved in the SOAs and language tuning DistilBERT model for our task (Sanh et al. 2019). For
intensity including: 1) N-gram word representation (tf-idf this task, the annotated dataset is divided into train, valida-
count vector), where n ∈ N := {1, 2, 3}. The preprocess- tion, and test sets with a proportion of 0.7:0.1:0.2. We used
ing of n grams included removing stopwords, lowercasing, Huggingface Transformers (Wolf et al. 2019) combined with
ignoring terms that appear in more than 50% of the docu- hyperparameter search using the Ray Tune Python library
ments or less than 0.25% of the documents. 2) Sentiment. (Liaw et al. 2018).
We used the Valence Aware Dictionary and Sentiment Rea-
soner (VADER), which is a rule-based sentiment analysis Results
tool specifically for detecting sentiment expressed in social
media text (Hutto and Gilbert 2014), to generate a positive,
Summary of Statistics
negative and neutral score for each text. 3) 15 topics ex- We collected 6535 posts from January 2020 to April
tracted with LDA topic modeling. We experimented with 2021 from r/COVID19 support. Posts that were removed or
10, 15 and 20 topics, and found that 15 topics yielded the deleted by the author by the time we collected the data were
highest coherence score(U MASS measure (Mimno et al. not included in the dataset. Table 2 shows the number of
2011)). Topic coherence measures the degree of semantic posts each month. We found a dramatic surge of posts in
similarity between high scoring words in the topic. 4) The March 2020 followed by a tapering off of posts over the
Flesch reading ease score. The reading ease score indicates following months. Users may have become less enthusias-
the readability of a text by computing the average length of tic about COVID-19 discussions or moved these discussions
the sentences and the average number of syllables per word. to other forums.
A lower score indicates more complicated and long words
are used in the text, making it more difficult for readers to Identifying the Subjects of Anxiety
process the text (Coleman and Liau 1975). We identified nine groups of subjects of anxiety (SOA) in
We adopt a “one vs rest” approach to train an ensemble the thematic analysis. The authors and student helper co-
of nine binary classifiers. In the main model, we split the annotated 793 posts according to the annotation guideline
data into a train (80%, N = 634) and test set (20%, N = 159) for Subjects of Anxiety (SOA). Among these, 645 contained
in a stratified fashion. Stratified five-fold cross-validation in at least one SOA, 148 (18%) did not match any of our anno-
the training set was used to optimize the hyperparameters in tation topics, and 379 (50%) posts contained more than one
the model training and a grid search of hyperparameters was SOA. All of the SOAs could be directed either towards self
carried out for the classification algorithms. For the base- or towards friends or family members.
line models, we constructed dummy classifiers that generate The nine groups of SOA include worry, fear, or anxi-
predictions by respecting the training set’s class distribution ety (N indicates the number of posts labeled as positive in
(See Sklearn DummyClassifer) (Pedregosa et al. 2011). the corresponding category): 1) about COVID-19 leading
to financial difficulties or stagnation in one’s career/school
Model Validation on Chronological Data progress (Career/Finance, N = 98); 2) of being infected
We assumed that the aspects that provoke the public’s anxi- with COVID-19 in general, including fears that others (e.g.,
ety would change over time, therefore, it’s important to val- friends or family members) will get infected (Health Risks,
idate that models trained on historic data perform well on N = 382); 3) of being infected with COVID-19 because

141
 Y 2020 2021
M 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4
P 12 1023 745 481 333 538 417 400 409 409 470 466 329 206 297
P/U 1.09 1.56 1.42 1.38 1.44 1.41 1.40 1.39 1.31 1.29 1.38 1.49 1.53 1.44 1.47

Table 2: Statistics for collected posts. Y: year M: month. P: number of posts collected in r/COVID19 support in the correspond-
ing month. P/U: number of posts per user.

of going to work in an environment where protection mea- their family members were essential workers and that they
sures would be insufficient, or physical distancing would worried about being infected in the working environment,
not be possible (Work Risks, N = 52); 4) that stranger- for example:
s/friends/family members did not follow guidelines for pre-
ventative measures (Disrespecting Health-Related Guide- My mom is a lung doctor and she was working with
lines, N = 164); 5) of traveling using public transporta- patients in the ICU. I’m really scared.
tion (Travel Risks, N = 48); 6) of oneself or loved ones Some users were unhappy about poor working conditions.
dying due to COVID-19, or becoming critically ill (Death They were also worried about losing their jobs or missing
anxiety, N = 66); 7) about when to quarantine if a person is job opportunities if they did not continue to work despite
tested positive and general lockdown for the public, what the potential risks.
guidelines for quarantine/social distancing are, when quar-
antine/mandated social distancing will finish, or conflicts I saw an ad on Facebook saying they’re looking for
with familiar members in a quarantine/social distancing sit- employees at a small grocery store nearby, and I
uation, or struggling with quarantine/social distancing and wanted to apply. But I’m a little nervous about expos-
not seeing loved ones (COVID-19 Restrictions, N = 164). ing myself like this. I’m currently unemployed, and my
The final two subjects captured descriptions 8) of deterio- savings will only last a few months.
rating mental health, or worry that one’s mental health may Worry about COVID-19 Restrictions. As the pandemic
worsen (Mental Health, N = 188); 9) of losing hope for the progressed, around April 2020, issues related to COVID-
future (Future, N = 98). Below are the categories and ex- 19 restrictions (e.g., quarantine, social distancing, and lock-
amples of our annotation (these examples are paraphrased to downs) emerged as a major concern. Reddit users expressed
protect user privacy). feelings of distress and loss about not being able to meet
Worry about Health Risk. At the beginning of the pan- their loved ones for months:
demic, due to the lack of knowledge about how the virus It’s been more than two months of social isolation,
spread, users were worried about infection risk partially be- and all I want to do now is be physically with my part-
cause they did not know the appropriate preventative mea- ner. I don’t know when I’ll be able to see her again be-
sures. We identified several sub-groups under this SOA. cause she’s immunocompromised. That’s a bummer...
Users often sought information about whether they were us-
ing the right preventative approach (see Figure 1), such as At the same time, restrictions related to the COVID-19
whether they should go to the hospital for a COVID-19 un- pandemic have forced people to spend more time together at
related treatment, or whether they could be infected with home. In some cases, users’ posts suggested that they were
COVID-19 by unpacking parcels, for example: experiencing greater conflict with loved ones than before the
pandemic. We found these posts occurred more often at the
Can I get the virus by unpacking boxes if they are beginning of the pandemic.
shipped today and come Tuesday?
I’m really frustrated because the pandemic is causing
Some users also listed symptoms and asked if those were so much conflict between me and my wife. My wife is
symptoms of COVID-19. We observed that people with pre- getting incredibly angry at me with little or no provo-
existing health conditions were more likely to have this con- cation. Little things are blown up into massive argu-
cern. ments.
My normal body temperature is 96, but I have been
up to 99.5 on and off for a week. I have mild asthma. I Worry about Career/Finance. As quarantine and social
have sleep apnea and I have a slight dry cough. Now distancing were needed to stop the spread of the virus, many
I’m afraid I might have it. users, especially essential workers, mentioned that they were
worried about financial or career situations. Some users
Worry about Work Risk. Worry about work risk is a mentioned that they postponed their study plans due to a lack
subgroup of health risk (see Figure 1) (all work risk posts of motivation to carry out their studies online or financial
were also labeled as health risk). People worried about be- difficulties. We combined worry about academic progress
ing infected at work and were often struggling with mul- with worry about one’s career into one category because the
tiple problems, such as customers or patients disrespecting academic progress is closely related to one’s career, for ex-
health-related guidelines. Users often revealed that they or ample:

142
 Will social life
going to hospital for goes back to less career family and
people from a
COVID-unrelated normal? opportunities love ones
different future self
unpacking treatment in the near
household
parcels future
entering their business reopening death
house increase health risk
doing groceries if they have income
health risk hopelessness
developed reduced
poor working lost job
COVID-19
conditions mental insomnia
symptoms
asking for advice socializing health
on preventative with people finance/career/academia
approach customers not work risk
following anxiety depression
health
guideline postpone study
friends or family socialize plan
with people who do not live
in the same household unable to meet
people live in the same new friends
unable to meet
household do not
disrespect people who their romantic frequent
agree on one set of
health-related don't wear partner COVID-19 arguments
guidelines
guidelines masks restrictions with family
unable to meet memebers
taking public not able to
travel risk friends and
transportation travel due strangers doing family unable to socialize with
to travel non-essential activities
people in school
ban

Figure 1: Subgroups for manually annotated SOAs in Thematic Analysis. This graph shows the subgroups in each SOA. Dotted
lines indicate SOAs that have overlapping subgroups and SOAs which are subcategories of others.

I had dreams of completing my college degree, which I I need to return home sometime this year, but I’m al-
would have probably finished in 1.5 years. It will most most sure I missed my window of opportunity. Am I
likely take at least another two years to finish now. I fucked for the rest of the year? How can I go home?
had hoped to teach music or start my own business in :( Please help.
the near future, but that is no longer viable.
Users posted about being worried to use public transporta-
Many users reported struggling with financial situations tion in general, and particularly about air travel:
because they were unable to find a part-time job. I am going to fly out to visit family for a week in the
I’ll be starting college soon, and I’m as broke as ev- first week of July and I am extremely anxious about it.
eryone else. I really need this money if I’m going to Worry about Death. At the beginning of the pandemic,
survive. most of the concerns about the pandemic centered around
Worry about Disrespecting Health-related Guidelines. infection risks or feeling anxious about elderly family mem-
We found that many users developed their own set of health- bers becoming severely ill, especially when the number of
related guidelines. Users reported feeling uncomfortable or reported cases was surging, for example:
distressed that friends, family, or strangers did not respect Not only am I terrified of dying, but I’m also worried
their health-related guidelines. For example, they disagreed about my parents and grandfathers. Every day on so-
with others that they lived with over whether they should or cial media, I see the number of deaths and mourn be-
should not visit or socialize with non-household members. cause so many people have lost loved ones...
My parents are going on walks with friends (across Worry about the Future. Users also expressed anxiety
the street but still make me nervous). about the immediate and future economic impacts of the
I’ve set many boundaries with my roommate. First COVID-19 pandemic. These concerns were related to career
was no visitors at all. She got around it by smuggling opportunities, being furloughed or laid off and interruption
someone in. Then visitors only in her room, with her of school, for example:
sanitizing everything afterward. She broke that bla- I’m a student at college. I just received acceptance to
tantly. a university that said that all classes would be held
online for the full semester. My wife will begin her
Worry about Traveling and Public Transportation. new career from home as well. Everything that used
Starting in March 2020, many countries imposed travel re- to be nice in my life makes me miserable.
strictions or travel bans in an attempt to reduce the spread
of the virus. Users posted that these travel restrictions were Worry about Mental Health. In October 2020, about
a source of anxiety especially for those who couldn’t visit seven months after the start of the pandemic, we observed
their partners or family. For example: that a higher share of users expressed hopelessness, and

143
concerns about mental health deterioration becoming more
severe. Concerns about mental health remained high until
April 2021, when we finished the data collection for this
study.
I don’t know how much longer I can continue. Since
March, I’ve been in and out of severe anxiety and de-
pression episodes when this began. I have health anx-
iety and this is my worst nightmare...

Classifying Subjects of Anxiety

We constructed an ensemble model comprising nine binary
classifiers for the SOAs using SVM and a fine-tuned Dis-
tilBERT model (main models). Table 3 shows the results of
the SOA classifiers and language intensity classifier. In the
SVM main model, the per-class F1 scores were above 0.60
for half of the classes. However, categories with extremely Figure 2: Changing Patterns of machine-annotated SOAs
imbalanced negative to positive classes, such as work (149: (Aggregated by Month). SOA: percentage of document
10), death (146:13), travel (149:10), and future (139:20) had mention specific SOA, the dashed blue line in the graph in-
poorer model performance but still surpassed the baseline dicates COVID-19 cases reported in the U.S.
dummy classifier. The DistilBERT model performed better
in the health risk, travel and future categories but performed
poorer than the SVM in the rest of the categories. We also many SOAs after Feb 2021, this does not necessarily mean
compared the main models’ performance with logistic re- that people worried less. Instead, people may have shifted to
gression and random forest 3. The random forest algorithm worries about the vaccine.
showed the poorest performance among all the algorithms
we tried, whereas the logistic regression models performed How Did Subjects of Anxiety Change Over Time?
worse than SVM and neural network models. Figure 2 presents the percentage of posts mentioning the
Chronological data analysis. Since the SVM model had nine SOAs for each study month using the SVM machine-
a more satisfactory result than DistilBERT in the majority annotated labels. To compare the anxiety index with re-
of the categories, we focus on evaluating the SVM model in ported COVID-19 cases, we retrieved COVID-19 case
the chronological data analysis. Results from the “last month numbers from the website OurWorldinData (Max Roxer
holdout (SVM)” models were similar to the SVM main mod- and colleagues 2021). According to website traffic tracked
els, except for “disrespecting health-related guidelines” and from Statista (https://bit.ly/3DJf9wX), Reddit traffic mainly
“death”. Our results suggest that although the SOA annota- comes from the U.S. The blue dotted line in Figure 2 indi-
tion guideline was defined according to the thematic anal- cates the number of COVID-19 cases reported in the U.S.
ysis conducted on data between Feb 2020 and Aug 2020, We found that feeling anxious about infection risk (health
the automatic SOA annotation models perform satisfactorily risk) was the most prevalent concern throughout the study
on data in Sept and Oct 2020. However, models trained on period, followed by struggling with COVID-19 restrictions
infrequently occurring variables are less stable. (restrict).

Model Validation in 2021 Dominant Topics Figure 2 shows that “Health risk”
(worry about being infected with COVID-19) and “COVID-
To examine the model performance on data from 2021, we 19 restrictions” (struggle with quarantine and social isola-
conducted a thematic analysis on data in March and April tion) were mentioned most often in the Reddit posts through-
2021. We found several new SOAs that were related to vac- out the time we examined. “Health risk” was mentioned
cines. Users were worried about vaccine delay, low vacci- most frequently in r/COVID19 support during the entire pe-
nation rates, vaccine side effects, and vaccine efficacy. We riod examined in this study (see Figure 2), followed by
also identified new sub-groups in the SOAs. For example, “COVID-19 restrictions”. Worries about “Health risk” re-
for health risks, users were worried that it would be unsafe mained high in the first eight months after the pandemic
to socialize or travel even after vaccination. Other infection started, but it declined dramatically from the ninth month
risks included COVID reinfection and vaccine breakthrough when there was a large wave of COVID-19 cases in Europe
infection. Table 3 shows the result evaluation for 2021 data. and the U.S. from October 2020 to January 2021, likely due
The performance of language intensity remains similar to to people visiting their loved ones during the holidays.
data in 2020. However, classifiers with fewer positive class
training data (e.g., travel, death) were again not performing Trends: Shift of Dominant Topic We observed a dra-
well in the new dataset. Most importantly, our classifiers of matic downward shift in worries “Health risk” and “Disre-
course were not able to identify SOAs that had not been rep- spect for Health-related Guidelines” as the timeline moved
resented in our codings (e.g., vaccine-related worries). Al- to 2021. The shift may be due to three reasons: 1) users’
though our classification results show a decreasing trend of SOAs changed to topics that are not represented in our cod-

144
 SVM DistilBERT LR RF baseline last month 2021 Data
ratio P R F1 P R F1 P R F1 P R F1 P R F1 P R F1 ratio P R F1
int 36% 70 71 71 64 68 66 70 67 68 70 62 61 52 52 52 79 73 74 33% 76 75 75
fin 13% 75 45 56 57 85 68 88 35 50 0 0 0 8 10 9 67 75 71 4% 30 50 40
res 23% 68 55 61 63 50 56 67 42 52 90 24 38 26 29 28 64 55 59 10% 57 80 67
hea 48% 72 78 75 77 85 80 71 77 74 70 74 72 46 39 42 82 74 78 20% 40 60 48
gui 21% 79 45 58 100 6 11 70 48 57 100 6 11 22 24 23 55 34 41 4% 0 0 0
wor 6% 50 30 38 7 83 14 50 10 17 0 0 0 9 10 9 50 50 50 2% 0 0 0
men 22% 72 66 69 56 68 62 72 51 60 78 20 30 16 17 16 65 65 65 12% 36 67 47
dea 8% 42 39 40 0 0 0 0 0 0 0 0 22 15 18 0 0 0 0 2% 0 0 0
tra 6% 50 10 17 60 60 60 100 10 18 0 0 0 0 0 0 100 40 57 8% 0 0 0
fut 13% 55 30 39 83 50 63 80 20 32 100 5 10 8 10 9 57 33 32 12% 25 17 20

Table 3: Model Performance. P:precision, R:recall and F1-score of positive class on subjects of anxiety. LR: logistics regression.
RF: random forest. Macro average is reported for language intensity. ratio: ratio of positive class in test set. fin: career or
finance, res: COVID-19 restrictions, hea: health risks (general), gui: disrespecting health-related guidelines, wor: work risks,
men: mental health, int: anxiety-related language intensity.

int fin res hea gui wor men dea tra How Did Anxiety-related Language Change Over
fin .1*
res .2* -.1 Time?
hea .3* -.1 -.1* Figure 3 shows the weekly time series of the mean anxiety-
gui .1* .1 .1 .5*
wor .1 .2* -.1* .3* .2* related language intensity from machine annotations. Users’
men .4* 0 .2* -.1* .1 0 anxiety-related language intensity took a downward trend
dea .2* -.1 -.1* .1* 0 0 .1* about half a year after the pandemic started (August 2020).
tra .1 0 .1 0 0 -.1 -.1* -.1 The decrease became more dramatic in February 2021,
fut .1 .2* .1* -.2* -.1 -.1* .2* 0 0 which correlates with several developed countries starting
their vaccination programs (see Figure 2). Note that unlike
Table 4: Correlations Between machine-annotated SOAs. SOAs whose measurement quality shifted over time, our ma-
Pearson correlations, *: p < .01. The p-values are corrected chine annotation on language intensity maintained good per-
for multiple inference using Holm’s method. formance in the new incoming data (see Table 3).
Overall, we found that over time, Reddit users’ language
in describing the SOAs became less intense. This pattern
ings (see section: Model Validation 2021). 2) users’ need for may suggest that despite the pandemic situation not improv-
sharing or obtaining a specific type of information during ing, users developed strategies to cope with anxiety issues
the pandemic decreasing over time due to information fa- after a while. Alternatively, they may have shared their wor-
tigue Skulmowski and Standl (2021) or they had already ob- ries in other spaces, such as in personal conversations in-
tained the information or coping strategies that they needed stead of a Reddit forum.
and were thus no longer seeking support on Reddit. 3) users’ Anxiety Language: High Intensity in Mental Health.
risk evaluations changed such that they no longer regarded Since the language describing the SOAs reflects the intensity
Covid-19 risks as equally threatening. of these anxieties, we further conducted correlation analyses
to understand which SOA was described with more intense
Trends: Worry about Mental Health and the Future. anxiety-related language. Table 4 shows that language inten-
Although we see that worries about “Health risk”, “COVID- sity was mostly correlated with worries about deteriorating
19 restrictions” and “Disrespect for Health-related Guide- mental health (r = 0.42, p < 0.001), followed by worries
lines” experienced a downward shift at the beginning of about death (r = 0.24, p < 0.001) and becoming infected
2021, the number of posts mentioning “mental health” and (r = 0.24, p < 0.001). Feeling anxious about the future was
“future” remained similar throughout the period we inves- closely linked to feeling anxious about career advancement
tigated. Unlike most of the SOAs that showed a downward and COVID-19 restrictions.
trend at the beginning of 2021, feeling anxious about the fu-
ture slowly but steadily increased during the pandemic. In Feature Importance
the last month holdout model, the model performance of the We ranked the coefficients of the linear kernel SVM models
“mental health” and “future” categories have been affected and extracted the top 10 important features for each classi-
by the shift of SOAs, but the effect is less prominent com- fier. Table 5 shows the important features in each classifier.
pared with other categories such as “COVID-19 restrictions” In general, the top features for each classifier reflect the be-
and “work risk”. This is likely due to the fact that mental havior or emotions we looked for during the annotation pro-
health and future concerns are still mentioned very often in cess. For example, the finance or career classifier included
the 2021 posts. employment-related words and topics, which were slightly

145
 top 10 features (ranked by coefficients)
intensity sad, because, option, shopping, droplet,
negemo(LIWC), younger, terrify, anxiety roof
finance job, lose job, hire, employment (LDA),
work (LIWC), achieve(LIWC), lose, money
(LIWC), future, reason why
restrict social, quarantine, toll, social activities (LDA),
honest, uni, friend because, proceed, know
virus, sex
guide plane, wear, destroy, wear mask, seriously,
toronto, right thing, parties, request, refuse
work come work, work, start work, employee, job,
homework, need money, blow, new job, apply
mental therapy, sane, depression, anxiety, silent, af-
ford, alone, headline, everything, mental
death dying, hospital, die, cancer, death (LIWC),
need hospital, story, death, recover, need hear
Figure 3: Intensity of machine-annotated anxiety-related travel flight, fly, trip, plane, travel, florida, metro, air-
language in COVID19 Support Dataset (Weekly).value: port, uber, hotel
mean predicted value of anxiety-related language future hopeless, vaccine, economy, absolute best,
taken away, feel hopeless, end, oxford, forever,
world
overlapped with the top features of the work risk classi-
fier. Top features for disrespecting health-related guidelines Table 5: Feature Importance. intensity: anxiety-related lan-
reflected disrespectful behaviors to the COVID19 guide- guage intensity; LIWC: LIWC categories, LDA: LDA topic
lines (e.g., “(not) wear mask”, “social”). The anxiety-related modeling.
language classifier had words indicating negative emotions
ranked as top predictive features.
may imply that users’ language surrounding anxiety towards
Discussion people disrespecting health-related guidelines had changed.
In this study, we use a thematic analysis to summarize the We also see a similar pattern in the evaluation of 2021 data
themes and sub-themes of aspects that trigger anxiety dur- (see Table 3). We highlight the importance of updating our
ing a pandemic (see Figure 1). Then we used a computa- knowledge of how the textual content evolves while using
tional approach to identify Reddit users’ subjects of anx- social media text to construct automatic systems to infer
iety (SOAs) expressed in r/COVID-19 support during the people’s mental health status.
COVID-19 pandemic. To our knowledge, this is the first
study that combines quantitative and qualitative analyses of COVID-19 Restrictions Compromise Mental
social media data to provide insight into how users’ focus of Health
anxiety changed during the pandemic. The percentage of posts mentioning mental health concerns
remains similar even though there was a shift of topic as the
Content Shift in Social Media Discussion timeline entered 2021. We found that posts worrying about
Our trend analysis only focuses on categories that have rea- COVID-19 restrictions were significantly correlated with
sonable F1 scores from the SVM model. We found that users’ mental health expressed in the same posts (r = 0.21,
health risks, COVID-19 restrictions, and other people disre- p < 0.01). In our thematic analysis, we found that people
specting health-related guidelines were the most prominent who worried about COVID-19 restrictions often felt inse-
subjects that people worried about. Worries about health cure over how long and when to quarantine/distance, wor-
risks reduced dramatically starting Jan 2021, coinciding ried about meeting with friends, family and loved ones, or
with when several countries, especially developed countries, felt threatened by family members or roommates not stick-
started to plan vaccine rollouts. In our analysis of 2021 ing to quarantine/distancing regulations as closely as they
data, we found that worries about vaccine safety and effi- did. Research is beginning to show that that prolonged iso-
cacy had become dominant SOAs. On the one hand, we sus- lation and limited physical space during isolation during the
pect that people may have been less anxious about infection pandemic compromised mental health (Pancani et al. 2021).
risk due to vaccination programs. On the other hand, people
may shift their major subject of anxiety from health risks of Adjustment to the “New Normal”
COVID-19 to vaccine safety. We observe that users tended to use less intense language to
Nevertheless, through our investigation on the last month describe the SOAs as the pandemic proceeded, even though
hold-out set, we found that precision on “disrespecting the number of confirmed cases did not decrease. This may
health-related guidelines” was much lower on the October suggest that after coping with ongoing stressors for a long
data compared with data from the previous month, which time, people may start to adapt to the new reality. Further,

146
the sudden drop of anxiety-related language intensity around around 90% of users are under the age of 35, with a mean
February 2021 correlates with the starting point of vacci- age of 25 years (Duggan and Smith 2013). We recognize that
nation programs in the US and many European countries. studying the mental health implications of this pandemic on
Although speculative, this co-occurrence could indicate that all age groups - spanning the entire life course - is impor-
vaccination programs may have reduced users’ pandemic tant and invite future studies to replicate these findings using
anxiety. younger and older samples.
Despite the above limitations, our method can be adopted
Application to analyze subjects of anxiety and its language intensity
Annotation of social media posts is not meant to replace self- across various social media user subgroups. By combining
report measurement of anxiety levels or anxiety subjects. the results of many social media subgroups, we could gain a
The types of information assessed with self-report measures general idea of the subjects of anxiety among certain popu-
do not necessarily exist in the social media text because Red- lations, especially young people in certain countries.
dit users may be mindful of self-presentation as their posts
are part of their larger “Reddit identity”. Conclusion
However, the methods proposed in this paper can in-
In this paper, we employed a computational approach to
form health providers about the mental health state of users
measure, describe, and understand what people worried
in support-seeking discussion forums during a pandemic.
about during the COVID-19 pandemic and how these pat-
Samples in our paper are not representative of social me-
terns changed throughout the pandemic. We combined quan-
dia platforms in general, but it is possible to adopt similar
titative techniques with a content analysis approach to reveal
approaches to identify pandemic-related anxiety themes in
the subjects of anxiety in a COVID-19 support community
other social media platforms and support forums. Health
from Reddit. We found that oneself or a loved one becom-
care providers can gauge the impact of the pandemic on
ing infected with COVID-19 was the most frequent source
the public’s mental health and better prepare resources ac-
of anxiety and that feeling anxious about COVID-19 restric-
cording to the estimated trends and widespread concerns ex-
tions was frequently mentioned in conjunction with con-
pressed across various social media platforms and discus-
cerns about one’s deteriorating mental health. We believe the
sion forums.
present study contributes novel insights into the pandemic’s
Limitations and Future Directions impacts on population health.
The thematic analysis for subjects of anxiety included only
about ten posts per month (in total, 86). Thus, some of References
the subjects of anxiety may have been missed because ten Ahorsu, D. K.; Lin, C.-Y.; Imani, V.; Saffari, M.; Griffiths, M. D.;
posts only account for a small percentage of the posts in and Pakpour, A. H. 2020. The fear of COVID-19 scale: develop-
each month, especially when r/COVID19 support became ment and initial validation. International journal of mental health
and addiction, 1–9.
more popular in some months. Future studies can con-
sider expanding the number of annotation examples in the Bargh, J. A.; McKenna, K. Y.; and Fitzsimons, G. M. 2002. Can
initial stage for identifying themes, although such annota- you see the real me? Activation and expression of the “true self”
on the Internet. Journal of social issues, 58(1): 33–48.
tion is very resource expensive because annotation must be
conducted with trained personnel with relevant background Boe, B. 2016. The Python Reddit API Wrapper 4. https://github.
knowledge instead of relying on crowdsourcing. In addition, com/praw-dev/praw/. Accessed: 2021-08-31.
we found that a small percentage of posts (8%) did not men- Boyd, R. L.; Ashokkumar, A.; Seraj, S.; and Pennebaker, J. W.
tion any SOAs and did not contain any anxiety language. 2022. The development and psychometric properties of LIWC-22.
As motivated in the Methods section, we did not construct Austin, TX: University of Texas at Austin, 1–47.
separate classifiers to remove the noise from our dataset in Braun, V.; and Clarke, V. 2006. Using thematic analysis in psy-
this work. Future work should increase the annotated sam- chology. Qualitative research in psychology, 3(2): 77–101.
ple size and construct a classifier to remove posts only for Britten, N. 1995. Qualitative research: qualitative interviews in
information sharing. medical research. Bmj, 311(6999): 251–253.
Like in many other social media studies, users from so- Brown, S. L.; Nesse, R. M.; Vinokur, A. D.; and Smith, D. M. 2003.
cial media platforms are not a representative sample of the Providing social support may be more beneficial than receiving it:
general population. Many people choose not to browse or Results from a prospective study of mortality. Psychological sci-
post on social media platforms for many reasons. The sam- ence, 14(4): 320–327.
ple in this study may not be representative of social media Chancellor, S.; and De Choudhury, M. 2020. Methods in predic-
users in general as well. Users from r/COVID19 support tive techniques for mental health status on social media: a critical
may also have more heightened anxiety than the general review. NPJ digital medicine, 3(1): 1–11.
population. Nevertheless, Reddit does not contain informa- Chen, L. 2021. Monitoring depressive symptoms using social me-
tion about users’ socio-demographic characteristics and re- dia data. PhD dissertation, The University of Edinburgh.
fined analyses where we include control variables for rele- Chen, L.; Magdy, W.; Whalley, H.; and Wolters, M. K. 2020. Ex-
vant factors like users’ age, gender, country, or ethnic group amining the role of mood patterns in predicting self-reported de-
were therefore not possible. Survey studies examining the pressive symptoms. In 12th ACM Conference on Web Science,
demographic characteristics of Reddit users estimated that 164–173.

147
Coleman, M.; and Liau, T. L. 1975. A computer readability for- Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion,
mula designed for machine scoring. Journal of Applied Psychol- B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg,
ogy, 60(2): 283. V.; Vanderplas, J.; Passos, A.; Cournapeau, D.; Brucher, M.; Perrot,
Conway III, L. G.; Woodard, S. R.; and Zubrod, A. 2020. Social M.; and Duchesnay, E. 2011. Scikit-learn: Machine Learning in
psychological measurements of COVID-19: Coronavirus perceived Python. Journal of Machine Learning Research, 12: 2825–2830.
threat, government response, impacts, and experiences question- Pfefferbaum, B.; and North, C. S. 2020. Mental health and the
naires. Covid-19 pandemic. New England Journal of Medicine, 383(6):
Duggan, M.; and Smith, A. 2013. 6% of online adults are reddit 510–512.
users. Pew Internet & American Life Project, 3: 1–10. Roohafza, H. R.; Afshar, H.; Keshteli, A. H.; Mohammadi, N.;
Guntuku, S. C.; Preotiuc-Pietro, D.; Eichstaedt, J. C.; and Ungar, Feizi, A.; Taslimi, M.; and Adibi, P. 2014. What’s the role of per-
L. H. 2019. What twitter profile and posted images reveal about ceived social support and coping styles in depression and anxiety?
depression and anxiety. In Proceedings of the International AAAI Journal of research in medical sciences: the official journal of Is-
Conference on Web and Social Media, volume 13, 236–246. fahan University of Medical Sciences, 19(10): 944.
Guntuku, S. C.; Sherman, G.; Stokes, D. C.; Agarwal, A. K.; Roy, D.; Tripathy, S.; Kar, S. K.; Sharma, N.; Verma, S. K.; and
Seltzer, E.; Merchant, R. M.; and Ungar, L. H. 2020. Tracking Kaushal, V. 2020. Study of knowledge, attitude, anxiety & per-
mental health and symptom mentions on twitter during covid-19. ceived mental healthcare need in Indian population during COVID-
Journal of general internal medicine, 35(9): 2798–2800. 19 pandemic. Asian Journal of Psychiatry, 102083.
Huang, Y.; and Zhao, N. 2020. Generalized anxiety disorder, de- Sanh, V.; Debut, L.; Chaumond, J.; and Wolf, T. 2019. DistilBERT,
pressive symptoms and sleep quality during COVID-19 outbreak a distilled version of BERT: smaller, faster, cheaper and lighter.
in China: a web-based cross-sectional survey. Psychiatry research, arXiv preprint arXiv:1910.01108.
112954. Shen, J. H.; and Rudzicz, F. 2017. Detecting anxiety through reddit.
Hutto, C.; and Gilbert, E. 2014. Vader: A parsimonious rule-based In Proceedings of the Fourth Workshop on Computational Linguis-
model for sentiment analysis of social media text. In Proceedings tics and Clinical Psychology—From Linguistic Signal to Clinical
of the International AAAI Conference on Web and Social Media, Reality, 58–65.
volume 8. Shen, T.; Jia, J.; Shen, G.; Feng, F.; He, X.; Luan, H.; Tang, J.;
Jones, N. M.; and Silver, R. C. 2020. This is not a drill: Anxiety Tiropanis, T.; Chua, T. S.; and Hall, W. 2018. Cross-domain de-
on Twitter following the 2018 Hawaii false missile alert. American pression detection via harvesting social media. International Joint
Psychologist, 75(5): 683. Conferences on Artificial Intelligence.
Liaw, R.; Liang, E.; Nishihara, R.; Moritz, P.; Gonzalez, J. E.; and Skulmowski, A.; and Standl, B. 2021. COVID-19 information fa-
Stoica, I. 2018. Tune: A Research Platform for Distributed Model tigue? A case study of a German university website during two
Selection and Training. arXiv preprint arXiv:1807.05118. waves of the pandemic. Human Behavior and Emerging Technolo-
gies.
Low, D. M.; Rumker, L.; Talkar, T.; Torous, J.; Cecchi, G.; and
Ghosh, S. S. 2020. Natural language processing reveals vulnera- Smith, L.; Jacob, L.; Yakkundi, A.; McDermott, D.; Armstrong,
ble mental health support groups and heightened health anxiety on N. C.; Barnett, Y.; López-Sánchez, G. F.; Martin, S.; Butler, L.;
reddit during covid-19: Observational study. Journal of medical and Tully, M. A. 2020. Correlates of symptoms of anxiety and de-
Internet research, 22(10): e22635. pression and mental wellbeing associated with COVID-19: a cross-
sectional study of UK-based respondents. Psychiatry research,
Max Roxer and colleagues. 2021. Coronavirus Pandemic (COVID-
291: 113138.
19) – the data. https://ourworldindata.org/coronavirus-data. Ac-
cessed: 2021-05-13. Taylor, S.; Landry, C. A.; Paluszek, M. M.; Fergus, T. A.; McKay,
D.; and Asmundson, G. J. 2020. Development and initial validation
Maxwell, D.; Robinson, S. R.; Williams, J. R.; and Keaton, C.
of the COVID Stress Scales. Journal of Anxiety Disorders, 72:
2020. ” A Short Story of a Lonely Guy”: A Qualitative Thematic
102232.
Analysis of Involuntary Celibacy Using Reddit. Sexuality & Cul-
ture, 24(6). Wiederhold, B. K. 2020. Using social media to our advantage:
alleviating anxiety during a pandemic. Cyberpsychology, Behavior,
McElroy, E.; Patalay, P.; Moltrecht, B.; Shevlin, M.; Shum, A.;
and Social Networking, 23(4): 197–198.
Creswell, C.; and Waite, P. 2020. Demographic and health fac-
tors associated with pandemic anxiety in the context of COVID-19. Wilkinson, I. 2001. Anxiety in a risk society. Psychology Press.
British Journal of Health Psychology, 25(4): 934–944. Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi,
Mimno, D.; Wallach, H.; Talley, E.; Leenders, M.; and McCallum, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; et al. 2019. Hug-
A. 2011. Optimizing semantic coherence in topic models. In Pro- gingface’s transformers: State-of-the-art natural language process-
ceedings of the 2011 conference on empirical methods in natural ing. arXiv preprint arXiv:1910.03771.
language processing, 262–272. Yap, M. B.; Wright, A.; and Jorm, A. F. 2011. The influence of
Murnane, E. L.; and Counts, S. 2014. Unraveling abstinence and stigma on young people’s help-seeking intentions and beliefs about
relapse: smoking cessation reflected in social media. In Proceed- the helpfulness of various sources of help. Social Psychiatry and
ings of the SIGCHI conference on human factors in computing sys- Psychiatric Epidemiology, 46(12): 1257–1265.
tems, 1345–1354.
Nikčević, A. V.; and Spada, M. M. 2020. The COVID-19 Anxiety
Syndrome Scale: development and psychometric properties. Psy-
chiatry research, 292: 113322.
Pancani, L.; Marinucci, M.; Aureli, N.; and Riva, P. 2021. Forced
Social Isolation and Mental Health: A Study on 1,006 Italians Un-
der COVID-19 Lockdown. Frontiers in Psychology, 12: 1540.

148