NAACL 2022 Findings

FedNLP: Benchmarking Federated Learning Methods

for Natural Language Processing Tasks
Bill Yuchen Lin1∗, Chaoyang He1∗ , Zihang Zeng1 , Hulin Wang1 ,
Yufen Huang1 , Christophe Dupuy2 , Rahul Gupta2 ,
Mahdi Soltanolkotabi1 , Xiang Ren1∗ , Salman Avestimehr1∗
University of Southern California1 Amazon Alexa AI2
{yuchen.lin,chaoyang.he,saltanol,xiangren,avestime}@usc.edu [email protected]

Abstract
Text Classification Question
Increasing concerns and regulations about data Answering
privacy and sparsity necessitate the study Text Generation
arXiv:2104.08815v3 [cs.CL] 6 May 2022

of privacy-preserving, decentralized learn- Language Transformer

ing methods for natural language processing Sequence Tagging Modeling LMs
(NLP) tasks. Federated learning (FL) pro-
vides promising approaches for a large num-
ber of clients (e.g., personal devices or or-
FedNLP
ganizations) to collaboratively learn a shared
global model to benefit all clients while al-
lowing users to keep their data locally. De-
spite interest in studying FL methods for NLP
Federated Learning
tasks, a systematic comparison and analysis is ↑ Upload the updates of a local model
lacking in the literature. Herein, we present ↓ Download the updated global model
the FedNLP1 , a benchmarking framework for Private Local Data (never exposed)
evaluating federated learning methods on four Federated models for an NLP task.
common formulations of NLP tasks: text clas-
sification, sequence tagging, question answer-
ing, and seq2seq generation. We propose Figure 1: The FedNLP benchmarking framework.
a universal interface between Transformer-
based language models (e.g., BERT, BART)
and FL methods under various non-IID parti- Alations
globalabout
modeldata
forprivacy
an NLP(e.g.,
task.GPDR (Regula-
tioning strategies. Our extensive experiments tion, 2016)) emerging data from realistic users
with FedNLP provide empirical comparisons have been much more fragmented and distributed,
between FL methods and help us better under- forming decentralized private datasets of multiple
stand the inherent challenges of this direction.
“data silos” (a data silo can be viewed as an in-
The comprehensive analysis points to intrigu-
ing and exciting future research aimed at de- dividual dataset) — across different clients (e.g.,
veloping FL methods for NLP tasks. organizations or personal devices).
To respect the privacy of the users and abide
1 Introduction by these regulations, we must assume that users’
Fine-tuning large pre-trained language models data in a silo are not allowed to transfer to a cen-
(LMs) such as BERT (Devlin et al., 2019) of- tralized server or other clients. For example, a
ten leads to state-of-the-art performance in many client cannot share its private user data (e.g., docu-
realistic NLP applications (e.g., text classifica- ments, conversations, questions asked on the web-
tion, named entity recognition, question answer- site/app) with other clients. This is a common
ing, summarization, etc.), when large-scale, cen- concern for organizations such as hospitals, finan-
tralized training datasets are available. How- cial institutions, or legal firms, as well as personal
ever, due to the increasing concerns and regu- computing devices such as smartphones, virtual
∗
assistants (e.g., Amazon Alexa, Google Assistant,
Bill and Chaoyang contributed equally; Xiang and
Salman are equal advisors for this work. etc.), or a personal computer. However, from a
1
https://github.com/FedML-AI/FedNLP machine learning perspective, models trained on
a centralized dataset that combine the data from clients, which often happen in real-world NLP
all organizations or devices usually result in bet- applications. As for the base NLP models, we
ter performance in the NLP domain. Therefore, it use the Transformer architecture (Vaswani et al.,
is of vital importance to study NLP problems in 2017) as the backbone and support a wide range of
such a realistic yet more challenging scenario — pre-trained LMs such as DistilBERT (Sanh et al.,
i.e., training data are distributed across different 2019), BERT (Devlin et al., 2019), BART (Lewis
clients and cannot be shared for privacy concerns. et al., 2020), etc. To conduct extensive experi-
The nascent field of federated learning (et al, ments, we need to support the experiments with
2019; Li et al., 2020a) (FL) aims to enable many multiple options on dimensions such as (1) task
individual clients to train their models jointly formulations, (2) NLP models, (3) FL algorithms,
while keeping their local data decentralized and and (4) non-IID partitions. Therefore, we propose
completely private from other users or a central- FedNLP, a modular framework with universal in-
ized server. A common training schema of FL terfaces among the above four components, which
methods is that each client sends its model param- is thus more extensible for supporting future re-
eters to the server, which updates and sends back search in FL for NLP.
the global model to all clients in each round. Since We aim to unblock the research of FL for NLP
the raw data of one client has never been exposed with the following two-fold contributions:
to others, FL is promising as an effective way to • Evaluation and analysis. We system-
address the above challenges, particularly in the atically compare popular federated learning
NLP domain, where many user-generated text data algorithms for mainstream NLP task formu-
contain sensitive and/or personal information. lations under multiple non-IID data parti-
Despite the growing progress in the FL domain, tions, which thus provides the first compre-
research into and application for NLP has been hensive understanding. Our analysis reveals
rather limited. There are indeed several recent that there is a considerably large gap between
works on using FL methods for processing med- centralized and decentralized training in var-
ical information extraction tasks (Sui et al., 2020). ious settings. We also analyze the efficiency
However, such prior work usually has its exper- of different FL methods and model sizes.
imental setup and specific task, making it diffi- With our analysis, we highlight several direc-
cult to fairly compare these FL methods and an- tions to advance FL for NLP.
alyze their performance in other NLP tasks. We • Resource. The implementation of our ex-
argue that future research in this promising direc- periments also forms a general open-source
tion (FL for NLP) would highly benefit from a uni- framework named FedNLP, which is capable
versal benchmarking platform for systematically of evaluating, analyzing, and developing FL
comparing different FL methods for NLP. To the methods for NLP. We also provide decentral-
best of our knowledge, such a benchmarking plat- ized NLP datasets of various task formula-
form is still absent from the literature. tions created by various non-IID partitioning
strategies for future research.
Therefore, our goal in this paper is to provide
comprehensive comparisons between popular FL The remainder of this paper is structured as fol-
methods (e.g., FedAvg (McMahan et al., 2017a), lows. We introduce the background knowledge
FedOPT (Reddi et al., 2021), FedProx (Li et al., of federated learning and several typical FL al-
2020b)) for four mainstream formulations of NLP gorithms in §2. Then, we present the proposed
tasks: text classification, sequence tagging, ques- non-IID partitioning strategies to create synthetic
tion answering, and seq2seq generation. Although datasets for different task formulations in §3. Our
there are few available realistic FL datasets for results, analysis, and findings are in §4. Finally,
NLP due to privacy concerns, we manage to use we discuss related work (§5) and conclusions (§6).
existing NLP datasets to create various non-IID
2 Federated Learning for NLP
data partitions over clients. These non-IID parti-
tions simulate various kinds of distribution shifts In this section, we first introduce the background
(e.g., label, features, quantities, etc.) over the knowledge of federated learning (FL) in the con-
text of NLP tasks. Then, we illustrate a unified we have S (t) = {1, 2, . . . , M }. Consequently, we
FL framework that we used to study typical FL can learn a global model to benefit all clients while
algorithms. Based on this, we build our research preserving their data privacy.
framework, a general pipeline for benchmarking
2.2 Our Unified Framework for FL
and developing FL methods for NLP.

2.1 Federated Learning Concepts Algorithm 1: F ED O PT (Reddi et al.,

Federated learning (FL) is a machine learning 2021)): A Generic FedAvg Algorithm
paradigm where multiple entities (clients) collab- Input: Initial model x(0) , C LIENT O PT,
S ERVERO PT
orate in solving a machine learning problem un- 1 for t ∈ {0, 1, . . . , T − 1} do
der the coordination of a central server or service 2 Sample a subset S (t) of clients
provider. Each client’s raw data is stored locally 3 for client i ∈ S (t) in parallel do
(t,0)
and not exchanged or transferred; instead, focused 4 Initialize local model xi = x(t)
5 for k = 0, . . . , τi − 1 do
updates intended for immediate aggregation are 6 Compute local stochastic gradient
used to achieve the learning objectives (Kairouz (t,k)
gi (xi )
et al., 2019). Therefore, federated learning has 7 Perform local update xi
(t,k+1)
=
(t,k) (t,k)
been seen as a promising direction to decrease the C LIENT O PT (xi , gi (xi ), η, t)
risk of attack and leakage, reduce the difficulty 8 Compute local model changes
(t) (t,τ ) (t,0)
and cost of data movement, and meet the privacy- ∆i = xi i − xi
9 Aggregate local changes
related data storage regulations. (t) P
∆(t) = i∈S (t) pi ∆i / i∈S (t) pi
P
In the basic conception of federated learning, 10 Update global model
we would like to minimize the objective function, x(t+1) = S ERVERO PT (x(t) , −∆(t) , ηs , t)
F (x) = Ei∼P [Fi (x)],
(1)
where Fi (x) = Eξ∼Di [fi (x, ξ)]. In this work, we propose to use FedOPT (Reddi
x ∈ Rd represents the parameter for the global et al., 2021), a generalized version of FedAvg, to
model, Fi : Rd → R denotes the local objective build the FedNLP platform. As the pseudo-code
function at client i, and P denotes a distribution presented in Algorithm 1, the algorithm is parame-
on the collection of clients I. The local loss func- terized by two gradient-based optimizers: C LIEN -
T O PT and S ERVERO PT with client learning rate
tions fi (x, ξ) are often the same across all clients,
but the local data distribution Di will often vary, η and server learning rate ηs , respectively. While
capturing data heterogeneity. C LIENT O PT is used to update the local models,
Federated averaging (FedAvg) (McMahan S ERVERO PT treats the negative of aggregated lo-
et al., 2017a) is a common algorithm to solve (1) cal changes −∆(t) as a pseudo-gradient and ap-
by dividing the training process into rounds. At plies it to the global model. This optimization
the beginning of the t-th round (t ≥ 0), the server framework generalizes to many aggregation-based
broadcasts the current global model x(t) to a co- FL algorithms and simplifies the system design.
hort of participants: a random subset of clients To make our research general, we explore dif-
from S (t) which includes M clients in total. Then, ferent combinations of S EVERO PT and C LIEN -
T O PT . The original FedAvg algorithm implicitly
each sampled client in the round’s cohort performs
τi local SGD updates on its own local dataset and sets S EVERO PT and C LIENT O PT to be SGD, with
(t) (t,τ )
sends the local model changes ∆i = xi i −x(t) a fixed server learning rate ηs of 1.0. FedProx (Li
to the server. Finally, the server uses the aggre- et al., 2020b), tackling statistical heterogeneity by
(t)
gated ∆i to update the global model: x(t+1) = restricting the local model updates to be closer to
P
p∆
(t) the initial (global) model, can be easily incorpo-
i∈S (t) i i
x(t) + P
p . where pi is the relative weight rated into this framework by adding L2 regular-
i∈S (t) i
of client i. The above procedure will repeat un- ization for better stability in training. Moreover,
til the algorithm converges. In the cross-silo set- given that AdamW (Loshchilov and Hutter, 2019)
ting where all clients participate in training on ev- is widely used in NLP, we set it for ClientOpt
ery round (each cohort is the entire population), and let the ServerOpt be SGD with momentum
to reduce the burden of tuning. Task Txt.Cls. Seq.Tag. QA Seq2Seq
Dataset 20News Onto. MRQA Giga.
2.3 The Proposed FedNLP Framework
# Training 11.3k 50k 53.9k 10k
To support our research in this paper and other fu- # Test 7.5k 5k 3k 2k
ture work in the area of federated learning for NLP, # Labels 20 37* N/A N/A
we build a general research framework named Metrics Acc. F-1 F-1 ROUGE
FedNLP, based on the above universal optimiza-
tion framework. We here briefly highlight its Table 1: Statistics of the selected datasets for our ex-
unique features and leave the details in the fol- periments. *37 is the size of the tag vocabulary.
lowing content and a detailed design is shown in
App. F. First, FedNLP is the very first frame-
• QA: MRQA (Fisch et al., 2019) is a bench-
work that connects multiple FL algorithms with
mark consisting of 6 popular datasets2 :
Transformer-based models, to our best knowledge.
SQuAD (Rajpurkar et al., 2016) (8529/431),
Also, we implement a flexible suite of interfaces to
NewsQA (Trischler et al., 2017) (11877/613),
support different types of NLP tasks and models,
TriviaQA (Joshi et al., 2017) (4120/176) ,
as well as different non-IID partitioning strategies
SearchQA (Dunn et al., 2017) (9972/499)
(Sec. 3.2). To study security and privacy guaran-
, HotpotQA (Yang et al., 2018b) , and
tees, we incorporate state-of-the-art secure aggre-
NQ (Kwiatkowski et al., 2019) (9617/795).
gation algorithms such as LightSecAgg (see F.5).
• Seq2Seq Generation: Gigaword (DBL,
2012) is a news corpus with headlines that are
3 Benchmarking Setup with FedNLP
often used for testing seq2seq models as a sum-
In this section, we introduce the creation of our marization task. Other tasks such as dialogue
benchmark datasets from a set of chosen NLP response generation and machine translation can
tasks with different non-IID partition methods. We also be adapted to this format.
evaluate various FL methods on these datasets. We show the basic statistics of the above
datasets in Table 1. Note that our FedNLP as a
3.1 Task Formulations, Datasets, and Models
research platform supports a much wider range of
There are numerous NLP applications, but most specific tasks of each formulation, while we only
of them can be categorized based on four main- introduce the ones used in our experiments here
stream formulations: text classification (TC), se- with typical settings. Moreover, our contribution
quence tagging (ST), question answering (QA), is more of a general FL+NLP benchmarking plat-
and seq2seq generation (SS). The formal def- form instead of particular datasets and partitions.
inition of each formulation is detailed in Ap-
pendix §B. To cover all formulations while keep- Base NLP Models. Fine-tuning pre-trained
ing our experiments in a reasonable scope, we se- LMs has been the de facto method for NLP re-
lect one representative task for each formulation: search, so we focus on testing Transformer-based
• Text Classification: 20Newsgroup (Lang, architectures in FedNLP. Specifically, we choose
1995) is a news classification dataset with an- to use BART (Lewis et al., 2020), a text-to-text
notations for 20 labels. We showcase our Transformer model similar to the T5 model (Raf-
FedNLP with this dataset as it has a larger out- fel et al., 2020), for seq2seq tasks.
put space (20 labels) than sentiment-analysis
3.2 Non-IID Partitioning Strategies
datasets, which is an important factor for the
label-distribution shift scenarios. . The existing datasets have been used for central-
• Sequence Tagging: OntoNotes (Pradhan ized training in NLP. As our focus here is to test
et al., 2013) (5.0) is a corpus where sentences decentralized learning methods, we need to dis-
have annotations for the entity spans and types. tribute the existing datasets to a set of clients. It
We use it for the named entity recognition task, is the non-IIDness of the client distribution that
which is fundamental to information extraction 2
We only use part of the data to demonstrate and verify
and other applications. our hypothesis; we show the train/test split in brackets.
makes federated learning a challenging problem. 100 clients JSD

Thus, we extend the common practice widely used

100 clients
in prior works to the NLP domain for generating
synthetic FL benchmarks (Li et al., 2021a). We
𝛼=1 𝛼=5 𝛼 = 10 𝛼 = 100
first introduce how we control the label distribu-
tion shift for TC and ST, then the quantity dis- Figure 2: The J-S divergence matrix between
tribution shift, and finally how we model the dis- 100 clients on the 20News dataset when α ∈
tribution shift in terms of input features for non- {1, 5, 10, 100}. Each sub-figure is a 100x100 symmet-
ric matrix. The intensity of a cell (i, j)’s color here
classification NLP tasks (e.g., summarization).
represents the distance between the label distribution
Non-IID Label Distributions. Here we present of Client i and j. It is expected that when α is smaller,
the partition over clients is more non-IID in terms of
how we synthesize the data partitions such that
their label distributions.
clients share the same (or very similar) number
of examples, but have different label distribu-
ilar number of examples. A concrete example is
tions from each other. We assume that on ev-
shown in Figure 8 (Appendix).
ery client training, examples are drawn indepen-
dently with labels following a categorical distri-
bution over L classes parameterized by a vec- Controlling non-IID Features. Although
tor q (qi ≥ 0, i ∈ [1, L] and kqk1 = 1). To syn- straightforward and effective, the above label-
thesize a population of non-identical clients, we based Dirichlet allocation method has a major
draw q ∼ DirL (αp) from a Dirichlet distribu- limitation — it is only suitable for text classifi-
tion, where p characterizes a prior class distribu- cation tasks where the outputs can be modeled
tion over L classes, and α > 0 is a concentra- as category-based random variables. To create
tion parameter controlling the identicalness among synthetic partitions for other non-classification
clients. For each client Cj , we draw a qj as its la- NLP tasks and model distribution shifts, we
bel distribution and then sample examples without thus propose a partition method based on feature
replacement from the global dataset according to clustering. Specifically, we use Sentence-
qj . With α → ∞, all clients have identical dis- BERT (Reimers and Gurevych, 2019) to encode
tributions to the prior (i.e., uniform distribution); each example to a dense vector by their text then
with α → 0, on the other extreme, each client we apply K-Means clustering to get the cluster
holds examples from only one class chosen at ran- label of each example; finally, we use these cluster
dom. In Fig. 2, we show heatmaps for visualizing labels (as if they were classification tasks) to
the distribution differences between each client. follow the steps in modeling label distribution
Figure 3 shows an example of the concrete label shift. There are two obvious benefits of this
distributions for all clients with different α. We clustering-based Dirichlet partition method: 1) It
can see that when α is smaller, the overall label enables us to easily synthesize the FL datasets for
distribution shift becomes larger. non-classification tasks (i.e., ST, QA, SS) as they
do not have discrete labels as output space; 2) The
Controlling non-IID Quantity. It is also com- BERT-based clustering results naturally imply
mon that different clients have very different data different sub-topics of a dataset, and thus feature
quantities while sharing similar label distribution. shift can be seen as a shift of latent labels — we
We thus also provide a quantity-level Dirichlet al- can reuse the same method for the label-based
location z ∼ DirN (β) where N is the number of Dirichlet partition method.
clients. Then, we can allocate examples in a global
dataset to all clients according to the distribution z Natural Factors For datasets like MRQA, we
— i.e., |Di | = zi |DG |. If we would like to model consider a cross-silo setting where each client is
both quantity and label distribution shift, it is also associated with a particular sub-dataset (out of the
easy to combine both factors. Note that one could six datasets of the same format), forming a natu-
assume it is a uniform distribution z ∼ U (N ), (or ral distribution shift based on the inherent factors
β → ∞) if we expect all clients to share a sim- such as data source and annotating style.
 Task Dataset Partition Clients FedAvg FedProx FedOPT # Rounds
Text Classification 20news α =1 (label shift) 100 0.5142 0.5143 0.5349 22
Sequence Tagging OntoNotes α =0.1 (label shift) 30 0.7382 0.6731 0.7918 17
Question Answering MRQA natural factor 6 0.2707 0.2706 0.3280 13
Seq2Seq Generation Gigaword α =0.1 (feature shift) 100 0.3192 0.3169 0.3037 13

Table 2: The comparisons between different FL methods under the same setting on different NLP tasks. The
number of workers per round are 10, expect for the MRQA task, which uses 6.

20 labels Ratio submitted supplementary materials.

Our experiments cover both cross-device and
cross-silo settings. As shown in Table 2, in the
100 clients

cross-device setting, we use uniform sampling to

select 10 clients for each round when the client
number in a dataset is very large (e.g., 100). For
the cross-silo setting, each round will select the
same number of clients (we use 6 for the QA task).
The local epoch number is set to 1 for all experi-
𝛼=1 𝛼=5 𝛼 = 10 𝛼 = 100 ments. To make our results reproducible, we use
wandb.ai to store all experiment logs and hyper-
Figure 3: Visualizing the non-IID label distributions parameters as well as running scripts.
on 20News with α being {1, 5, 10, 100}. Each sub-
figure is a 100x20 matrix, where 100 is the number of Q1: How do popular FL methods perform
clients, and 20 is the number of labels. The intensity of differently under the same setting?
a cell here represents the ratio of a particular label in the
local data of a client. When α is smaller (1, 5, 10), each We compare the three typical FL methods under
client has a relatively unique label distribution, thus the the same setting (i.e., data partition, communica-
differences between clients are larger; when α = 100,
tion rounds, etc.) for each task formulation. As
every client has a nearly uniform label distribution.
shown in Table 2, we report the results of FedAvg,
FedProx, and FedOPT. We can see that overall Fe-
4 Experimental Results and Analysis dOPT performs better than the other two methods,
with the only exception being in the seq2seq gen-
In this section, we aim to analyze typical federated
eration task. FedAvg and FedProx perform sim-
learning methods (introduced in our benchmark
ilarly with marginal differences, but FedAvg out-
datasets with multiple dimensions with the base
performs FedProx in sequence tagging. These two
NLP models listed previously. We put more im-
exceptions are surprising findings, as many prior
plementation details and additional results in Ap-
works in the FL community show that FedOPT is
pendix. We organize our extensive experimental
generally better than FedProx and FedAvg on vi-
results and findings from the analysis as a collec-
sion tasks and datasets.
tion of research questions with answers.
We conjecture that such inconsistent perfor-
Experimental Setup and Hyper-parameters. mance across tasks suggests the difference in
We use DistilBERT and BART-base for most of terms of the loss functions has a great impact on
our experiments, as the former is a distilled ver- FL performance. Seq2seq and sequence tagging
sion of the BERT model and has a 7x speed tasks usually have more complex loss landscapes
improvement over BERT-base on mobile devices than text classification, as they are both typical
— a common scenario for FL applications; the structured prediction tasks, while the text classi-
BART-base model is the most suitable option con- fication has a much smaller output space. From
sidering the trade-off between performance and Fig. 4, we see that the FedOPT outperforms the
computation cost. We leave our implementation other two methods at the beginning while gradu-
details and the selected hyper-parameters in the ally becoming worse over time.
 20news Ontonotes MRQA Gigaword

Figure 4: The learning curves of the three FL Methods on four different task formulations. The metrics used for
these tasks are accuracy, span-F1, token-F1, and ROUGE respectively; The x-axis is the number of rounds.

0.7
uniform Frozen Layers # Tunable Paras. Cent. FedOpt.
0.6 label ( =1)
label ( =10) None 67.0M 86.86 55.11
0.5 label ( =5) E 43.1M 86.19 54.86
quantity ( =1) E + L0 36.0M 86.54 52.91
0.4
E + L0→1 29.0M 86.52 53.92
0.3 E + L0→2 21.9M 85.71 52.01
E + L0→3 14.8M 85.47 30.68
0.2
E + L0→4 7.7M 82.76 16.63
0.1 E + L0→5 0.6M 63.83 12.97
0 5 10 15 20
Table 3: Performance (Acc.%) on 20news (TC) when
Figure 5: Testing FedOPT with DistilBERT for different parts of DistilBERT are frozen for central-
20News under different data partition strategies. ized training and FedOpt (at 28-th round). E stands for
the embedding layer and Li means the i-th layer. The
significant lower accuracy are underlined.
This tells us that the use of AdamW as the client
optimizer may not always be a good choice, es-
pecially for a complex task such as the Seq2Seq skew partitions have a smoother curve, while the
ones, as its adaptive method for scheduling learn- variance is smaller for a larger α (e.g., 10).
ing rates might cause implicit conflicts. These ob- • Quantity skew does not introduce a great chal-
servations suggest that federated optimization al- lenge for federated learning when the label dis-
gorithms need to be tailored for various NLP tasks, tribution is closer to the uniform one.
and exploring FL-friendly model architecture or These findings suggest that it is important to
loss function can also be promising directions to design algorithms to mitigate data heterogene-
address these challenges. ity. One promising direction is personalized FL,
which enables each client to learn its personalized
Q2: How do different non-IID partitions of model via adapting its local data distribution and
the same data influence FL performance? system resources (Dinh et al., 2020; Fallah et al.,
2020; Li et al., 2021b).
The FedNLP platform supports users to inves-
tigate the performance of an FL algorithm with a Q3: How does freezing of Transformers in-
wide range of data partitioning strategies, as dis- fluence the FL performance?
cussed in §3.2. Here we look at the training curves
of the FedOPT on different partitions, as shown in Communication cost is a major concern in the
Figure 5. We reveal several findings: federated learning process. It is thus natural to
• When α is smaller (i.e., the partition is more consider freezing some Transformer layers of the
non-IID in terms of their label distribution), the client models to reduce the size of the trainable pa-
performance tends to degrade, based on the three rameters that will be transmitted between servers
curves (α = {1, 5, 10}). and clients. To study the influence of freezing lay-
• The variance is also larger when the label distri- ers on the FL performance, we conduct a series of
bution shift is larger. Both uniform and quantity- experiments that freeze the layers from the embed-
 0.6 base achieves better performance, the performance
None
0.5 E of DistilBERT is not significantly worse. Consid-
E+L0
E+L0 1
ering the communication cost (BERT-base is al-
0.4 E+L0 2 most 2x larger), we argue that using DistilBERT is
E+L0 3
0.3 E+L0 4 a more cost-effective choice for both experimental
E+L0 5 analysis and realistic applications.
0.2
0.1 5 Related Work
0.0
0 5 10 15 20 25 FL benchmarks and platforms. In the last few
years a proliferation of frameworks and bench-
Figure 6: Testing FedOPT with DistilBERT for
mark datasets have been developed to enable re-
20News under different frozen layers.
searchers to better explore and study algorithms
0.7 and modeling for federated learning, both from
bert-base academia: LEAF(Caldas et al., 2018), FedML (He
0.6 distilbert-base
et al., 2020c), Flower (Beutel et al., 2020), and
0.5 from the industry: PySyft (Ryffel et al., 2018),
0.4 TensorFlow-Federated (TFF) (Ingerman and Os-
0.3 trowski, 2019), FATE (Yang et al., 2019), Clara
(NVIDIA, 2019), PaddleFL (Ma et al., 2019),
0.2
Open FL (Intel®, 2021). However, most platforms
0.1 only focus on designing a unified framework for
0.0 federated learning methods and do not provide
0 5 10 15 20 25 a dedicated environment for studying NLP prob-
Figure 7: FedOPT for 20News with different LMs. lems with FL methods. LEAF (Caldas et al., 2018)
contains a few text datasets, however, it is limited
ding layer (E) to the top layer (L5 ) of DistilBERT to classification and next-word prediction datasets
with both centralized training and FedOPT on the and does not consider the pre-trained language
text classification task. models. We want to provide a dedicated platform
We report our results in Table 3 and Figure 6. for studying FL methods in realistic NLP applica-
We find that in centralized training, the largest tions with state-of-the-art language models.
performance gain happens when we unfreeze the
Federated learning in NLP applications.
last layer, while in FedOPT we have to unfreeze
There are a few prior works that have begun
the last three layers to enjoy a comparable per-
to apply FL methods in privacy-oriented NLP
formance with the full model. This suggests that
applications. For example, federated learning has
reducing communication costs via freezing some
been applied to many keyboard-related applica-
layers of Transformer LMs is feasible, though one
tions including (Hard et al., 2018; Stremmel and
should be aware that the experience in centralized
Singh, 2020; Leroy et al., 2019; Ramaswamy
training may not generalize to the FL experiments.
et al., 2019; Yang et al., 2018a), sentence-level
Q4: Are compact model DistilBERT ade- text intent classification using Text-CNN (Zhu
quate for FL+NLP? et al., 2020), and pretraining and fine-tuning of
BERT using medical data from multiple silos
We know that BERT has a better performance than without fetching all data to the same place (Liu
DistilBERT for its larger model size. However, and Miller, 2020). FL methods also have been
is it cost-effective to use BERT rather than Dis- proposed to train high-quality language models
tilBERT? To study this, we compare the perfor- that can outperform the models trained without
mance of both models with FedOPT on text classi- federated learning (Ji et al., 2019; Chen et al.,
fication, sharing the same setting as the above ex- 2019). Besides these applications, some work
periments. As shown in Figure 7, although BERT- has been done in medical relation extractions (Ge
et al., 2020) and medical name entity recognition trails, and recourse so that their predictions can be
(Sui et al., 2020). These methods use federated explained to and critiqued by affected parties.
learning to preserve the privacy of sensitive
Limitations. One limitation of our work is that
medical data and learn data on different platforms,
we have not analyzed the privacy leakage of FL
excluding the need for exchanging data between
methods. We argue that novel privacy-centric
different platforms.
measures are orthogonal to the development of FL
Our work aims to provide a unified platform
methods, which is beyond the scope of our work.
for studying various NLP applications in a shared
How to fairly analyze the privacy leakage is now
environment so that researchers can better design
still an open problem for both FL and NLP, and
new FL methods either for a specific NLP task or
it is only possible to study this when we have an
as a general-purpose model. The aforementioned
existing platform like FedNLP.
prior works would thus be a particular instance of
the settings supported by the FedNLP platform.
Acknowledgements
6 Conclusion and Future Directions This work is supported in part by a research
Our key contribution is providing a thorough and grant and an Amazon ML Fellowship from USC-
insightful empirical analysis of existing federated Amazon Center on Secure and Trustworthy AI
learning algorithms in the context of NLP mod- (https://trustedai.usc.edu). Xiang
els. Notably, We compare typical FL methods Ren is supported in part by the Office of the
for four NLP task formulations under multiple Director of National Intelligence (ODNI), In-
non-IID data partitions. Our findings reveal both telligence Advanced Research Projects Activity
promise and the challenges of FL for NLP. In ad- (IARPA), via Contract No. 2019-19051600007,
dition, we also provide a suite of resources to sup- the DARPA MCS program under Contract No.
port future research in FL for NLP (e.g., a unify- N660011924033, the Defense Advanced Research
ing framework for connecting Transformer mod- Projects Agency with award W911NF-19-20271,
els with popular FL methods and different non-IID NSF IIS 2048211, NSF SMA 1829268, and gift
partition strategies). Thus, we believe our well- awards from Google, Amazon, JP Morgan and
maintained open-source codebase to support fu- Sony. Mahdi Soltanolkotabi is supported by the
ture work in this area. Packard Fellowship in Science and Engineering,
Promising future directions in FL for NLP in- a Sloan Research Fellowship in Mathematics, an
clude: 1) minimizing the performance gap, 2) im- NSF-CAREER under award #1846369, DARPA
proving the system efficiency and scalability, 3) Learning with Less Labels (LwLL) and FastNICS
trustworthy and privacy-preserving NLP, 4) per- programs, and NSF-CIF awards #1813877 and
sonalized FL methods for NLP, etc. (Please see #2008443.
Appendix E for more details.)

Ethical Considerations and Limitations(*) References

2012. Proceedings of the Joint Workshop on Auto-
Ethical considerations. The key motivation of matic Knowledge Base Construction and Web-scale
FedNLP (and FL) is to protect the data privacy of Knowledge Extraction, AKBC-WEKEX@NAACL-
general users by keeping their data on their own HLT 2012, Montrèal, Canada, June 7-8, 2012.
devices while benefiting from a shared model from James Henry Bell, Kallista A Bonawitz, Adrià Gascón,
a broader community. Among the risks that need Tancrède Lepoint, and Mariana Raykova. 2020. Se-
to be considered in any deployment of NLP are cure single-server aggregation with (poly) logarith-
that responses may be wrong, or biased, in ways mic overhead. In Proceedings of the 2020 ACM
SIGSAC Conference on Computer and Communica-
that would lead to improperly justified decisions. tions Security.
Although in our view the current technology is still
relatively immature, and unlikely to be fielded in Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi
Qiu, Titouan Parcollet, and Nicholas D Lane. 2020.
applications that would cause harm of this sort, it Flower: A friendly federated learning research
is desirable that FedNLP methods provide audit framework. ArXiv preprint.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Anto- Andrew Hard, K. Rao, Rajiv Mathews, F. Beaufays,
nio Marcedone, H Brendan McMahan, Sarvar Patel, S. Augenstein, Hubert Eichner, Chloé Kiddon, and
Daniel Ramage, Aaron Segal, and Karn Seth. 2017. D. Ramage. 2018. Federated learning for mobile
Practical secure aggregation for privacy-preserving keyboard prediction. ArXiv.
machine learning. In proceedings of the 2017 ACM
SIGSAC Conference on Computer and Communica- Chaoyang He, Murali Annavaram, and Salman Aves-
tions Security. timehr. 2020a. Fednas: Federated deep learning via
neural architecture search.
Sebastian Caldas, Peter Wu, Tian Li, Jakub Konečnỳ,
H Brendan McMahan, Virginia Smith, and Ameet
Chaoyang He, Murali Annavaram, and Salman Aves-
Talwalkar. 2018. Leaf: A benchmark for federated
timehr. 2020b. Group knowledge transfer: Feder-
settings. ArXiv preprint.
ated learning of large cnns at the edge. In Advances
Mingqing Chen, Ananda Theertha Suresh, Rajiv Math- in Neural Information Processing Systems 33: An-
ews, Adeline Wong, Cyril Allauzen, Françoise Bea- nual Conference on Neural Information Processing
ufays, and Michael Riley. 2019. Federated learn- Systems 2020, NeurIPS 2020, December 6-12, 2020,
ing of n-gram language models. In Proceedings virtual.
of the 23rd Conference on Computational Natural
Language Learning (CoNLL), pages 121–130, Hong Chaoyang He, Keshav Balasubramanian, Emir Ceyani,
Kong, China. Association for Computational Lin- Yu Rong, Peilin Zhao, Junzhou Huang, M. An-
guistics. navaram, and S. Avestimehr. 2021. Fedgraphnn: A
federated learning system and benchmark for graph
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and neural networks.
Kristina Toutanova. 2019. BERT: Pre-training of
deep bidirectional transformers for language under- Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng,
standing. In Proceedings of the 2019 Conference Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth
of the North American Chapter of the Association Vepakomma, Abhishek Singh, Hang Qiu, Xinghua
for Computational Linguistics: Human Language Zhu, Jianzong Wang, Li Shen, Peilin Zhao, Yan
Technologies, Volume 1 (Long and Short Papers), Kang, Yang Liu, Ramesh Raskar, Qiang Yang, Mu-
pages 4171–4186, Minneapolis, Minnesota. Associ- rali Annavaram, and Salman Avestimehr. 2020c.
ation for Computational Linguistics. Fedml: A research library and benchmark for fed-
Canh T. Dinh, Nguyen H. Tran, and Tuan Dung erated machine learning. ArXiv preprint.
Nguyen. 2020. Personalized federated learning with
moreau envelopes. In Advances in Neural Infor- Chaoyang He, Conghui Tan, Hanlin Tang, Shuang Qiu,
mation Processing Systems 33: Annual Conference and Ji Liu. 2019. Central server free federated learn-
on Neural Information Processing Systems 2020, ing over single-sided trust social networks. ArXiv
NeurIPS 2020, December 6-12, 2020, virtual. preprint.

Matthew Dunn, Levent Sagun, Mike Higgins, V. U. Chaoyang He, Haishan Ye, Li Shen, and Tong Zhang.
Güney, Volkan Cirik, and Kyunghyun Cho. 2017. 2020d. Milenas: Efficient neural architecture search
Searchqa: A new q&a dataset augmented with con- via mixed-level reformulation. In 2020 IEEE/CVF
text from a search engine. ArXiv. Conference on Computer Vision and Pattern Recog-
nition, CVPR 2020, Seattle, WA, USA, June 13-19,
A. Elkordy and A. Avestimehr. 2020. Secure aggre- 2020, pages 11990–11999. IEEE.
gation with heterogeneous quantization in federated
learning. ArXiv.
Alex Ingerman and Krzys Ostrowski. 2019. Tensor-
P. Kairouz et al. 2019. Advances and open problems in Flow Federated.
federated learning. ArXiv.
Intel®. 2021. Intel® open federated learning.
Alireza Fallah, Aryan Mokhtari, and Asuman
Ozdaglar. 2020. Personalized federated learning: A Shaoxiong Ji, Shirui Pan, Guodong Long, Xue Li, Jing
meta-learning approach. ArXiv preprint. Jiang, and Zi Huang. 2019. Learning private neural
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, language modeling with attentive aggregation. 2019
Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 International Joint Conference on Neural Networks
shared task: Evaluating generalization in reading (IJCNN).
comprehension. In Proceedings of the 2nd Work-
shop on Machine Reading for Question Answering, Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke
pages 1–13, Hong Kong, China. Association for Zettlemoyer. 2017. TriviaQA: A large scale dis-
Computational Linguistics. tantly supervised challenge dataset for reading com-
prehension. In Proceedings of the 55th Annual
Suyu Ge, Fangzhao Wu, Chuhan Wu, Tao Qi, Meeting of the Association for Computational Lin-
Yongfeng Huang, and X. Xie. 2020. Fedner: guistics (Volume 1: Long Papers), pages 1601–
Privacy-preserving medical named entity recogni- 1611, Vancouver, Canada. Association for Compu-
tion with federated learning. ArXiv. tational Linguistics.
Peter Kairouz, H Brendan McMahan, Brendan Avent, 11th International Joint Conference on Natural Lan-
Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, guage Processing (Volume 1: Long Papers), pages
Keith Bonawitz, Zachary Charles, Graham Cor- 4582–4597, Online. Association for Computational
mode, Rachel Cummings, et al. 2019. Advances Linguistics.
and open problems in federated learning. ArXiv
preprint. D. Liu and T. Miller. 2020. Federated pretraining and
fine tuning of bert using clinical notes from multiple
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- silos. ArXiv.
field, Michael Collins, Ankur Parikh, Chris Al-
berti, Danielle Epstein, Illia Polosukhin, Jacob De- Ilya Loshchilov and Frank Hutter. 2019. Decou-
vlin, Kenton Lee, Kristina Toutanova, Llion Jones, pled weight decay regularization. In 7th Inter-
Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, national Conference on Learning Representations,
Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Natural questions: A benchmark for question an- OpenReview.net.
swering research. Transactions of the Association
Lingjuan Lyu, Han Yu, Xingjun Ma, Lichao Sun, Jun
for Computational Linguistics, 7:452–466.
Zhao, Qiang Yang, and Philip S Yu. 2020. Privacy
Ken Lang. 1995. Newsweeder: Learning to filter net- and robustness in federated learning: Attacks and
news. In Proc. of ICML. defenses. ArXiv preprint.
Yanjun Ma, Dianhai Yu, Tian Wu, and Haifeng Wang.
David Leroy, Alice Coucke, Thibaut Lavril, Thibault
2019. Paddlepaddle: An open-source deep learning
Gisselbrecht, and Joseph Dureau. 2019. Federated
platform from industrial practice. Frontiers of Data
learning for keyword spotting. In IEEE Interna-
and Domputing, (1).
tional Conference on Acoustics, Speech and Signal
Processing, ICASSP 2019, Brighton, United King- Brendan McMahan, Eider Moore, Daniel Ramage,
dom, May 12-17, 2019, pages 6341–6345. IEEE. Seth Hampson, and Blaise Agüera y Arcas. 2017a.
Communication-efficient learning of deep networks
Mike Lewis, Yinhan Liu, Naman Goyal, Mar- from decentralized data. In Proceedings of the 20th
jan Ghazvininejad, Abdelrahman Mohamed, Omer International Conference on Artificial Intelligence
Levy, Veselin Stoyanov, and Luke Zettlemoyer. and Statistics, AISTATS 2017, 20-22 April 2017,
2020. BART: Denoising sequence-to-sequence pre- Fort Lauderdale, FL, USA, volume 54 of Proceed-
training for natural language generation, translation, ings of Machine Learning Research, pages 1273–
and comprehension. In Proceedings of the 58th An- 1282. PMLR.
nual Meeting of the Association for Computational
Linguistics, pages 7871–7880, Online. Association Brendan McMahan, Eider Moore, Daniel Ramage,
for Computational Linguistics. Seth Hampson, and Blaise Agüera y Arcas. 2017b.
Communication-efficient learning of deep networks
Q. Li, Yiqun Diao, Quan Chen, and Bingsheng He. from decentralized data. In Proceedings of the 20th
2021a. Federated learning on non-iid data silos: An International Conference on Artificial Intelligence
experimental study. ArXiv. and Statistics, AISTATS 2017, 20-22 April 2017,
Fort Lauderdale, FL, USA, volume 54 of Proceed-
Tian Li, Shengyuan Hu, Ahmad Beirami, and Vir-
ings of Machine Learning Research, pages 1273–
ginia Smith. 2021b. Ditto: Fair and robust feder-
1282. PMLR.
ated learning through personalization. In Proceed-
ings of the 38th International Conference on Ma- NVIDIA. 2019. Nvidia clara.
chine Learning, ICML 2021, 18-24 July 2021, Vir-
tual Event, volume 139 of Proceedings of Machine Sameer Pradhan, Alessandro Moschitti, Nianwen Xue,
Learning Research, pages 6357–6368. PMLR. Hwee Tou Ng, Anders Björkelund, Olga Uryupina,
Yuchen Zhang, and Zhi Zhong. 2013. Towards ro-
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and bust linguistic analysis using OntoNotes. In Pro-
V. Smith. 2020a. Federated learning: Challenges, ceedings of the Seventeenth Conference on Com-
methods, and future directions. IEEE Signal Pro- putational Natural Language Learning, pages 143–
cessing Magazine. 152, Sofia, Bulgaria. Association for Computational
Linguistics.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar San-
jabi, Ameet Talwalkar, and Virginia Smith. 2020b. Saurav Prakash and Amir Salman Avestimehr. 2020.
Federated optimization in heterogeneous networks. Mitigating byzantine attacks in federated learning.
In Proceedings of Machine Learning and Systems ArXiv preprint.
2020, MLSys 2020, Austin, TX, USA, March 2-4,
2020. mlsys.org. Saurav Prakash, Sagar Dhakal, Mustafa Riza Akdeniz,
Yair Yona, Shilpa Talwar, Salman Avestimehr, and
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Nageen Himayat. 2020. Coded computing for low-
Optimizing continuous prompts for generation. In latency federated learning over wireless edge net-
Proceedings of the 59th Annual Meeting of the works. IEEE Journal on Selected Areas in Commu-
Association for Computational Linguistics and the nications, (1).
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Jinhyun So, Başak Güler, and A Salman Avestimehr.
Lee, Sharan Narang, Michael Matena, Yanqi Zhou, 2021a. Codedprivateml: A fast and privacy-
Wei Li, and Peter J Liu. 2020. Exploring the limits preserving framework for distributed machine learn-
of transfer learning with a unified text-to-text trans- ing. IEEE Journal on Selected Areas in Information
former. Journal of Machine Learning Research, Theory, (1).
(140).
Jinhyun So, Başak Güler, and A Salman Avestimehr.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and 2021b. Turbo-aggregate: Breaking the quadratic ag-
Percy Liang. 2016. SQuAD: 100,000+ questions for gregation barrier in secure federated learning. IEEE
machine comprehension of text. In Proceedings of Journal on Selected Areas in Information Theory,
the 2016 Conference on Empirical Methods in Natu- (1).
ral Language Processing, pages 2383–2392, Austin,
Texas. Association for Computational Linguistics. Joel Stremmel and Arjun Singh. 2020. Pretraining fed-
erated text models for next word prediction. ArXiv.
Swaroop Indra Ramaswamy, Rajiv Mathews, K. Rao, Dianbo Sui, Yubo Chen, Jun Zhao, Yantao Jia, Yuan-
and Franccoise Beaufays. 2019. Federated learning tao Xie, and Weijian Sun. 2020. FedED: Feder-
for emoji prediction in a mobile keyboard. ArXiv. ated learning via ensemble distillation for medical
relation extraction. In Proceedings of the 2020
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Conference on Empirical Methods in Natural Lan-
Zachary Garrett, Keith Rush, Jakub Konečný, San- guage Processing (EMNLP), pages 2118–2128, On-
jiv Kumar, and Hugh Brendan McMahan. 2021. line. Association for Computational Linguistics.
Adaptive federated optimization. In 9th Inter-
national Conference on Learning Representations, Adam Trischler, Tong Wang, Xingdi Yuan, Justin Har-
ICLR 2021, Virtual Event, Austria, May 3-7, 2021. ris, Alessandro Sordoni, Philip Bachman, and Ka-
OpenReview.net. heer Suleman. 2017. NewsQA: A machine compre-
hension dataset. In Proceedings of the 2nd Work-
General Data Protection Regulation. 2016. Regu- shop on Representation Learning for NLP, pages
lation eu 2016/679 of the european parliament 191–200, Vancouver, Canada. Association for Com-
and of the council of 27 april 2016. Offi- putational Linguistics.
cial Journal of the European Union. Avail-
able at: http://ec. europa. eu/justice/data- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
protection/reform/files/regulation_oj_en. pdf Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz
(accessed 20 September 2017). Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In Advances in Neural Information Pro-
Nils Reimers and Iryna Gurevych. 2019. Sentence- cessing Systems 30: Annual Conference on Neural
BERT: Sentence embeddings using Siamese BERT- Information Processing Systems 2017, December 4-
networks. In Proceedings of the 2019 Conference on 9, 2017, Long Beach, CA, USA, pages 5998–6008.
Empirical Methods in Natural Language Process-
ing and the 9th International Joint Conference on Hongyi Wang, Kartik Sreenivasan, Shashank Ra-
Natural Language Processing (EMNLP-IJCNLP), jput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong
pages 3982–3992, Hong Kong, China. Association Sohn, Kangwook Lee, and Dimitris S. Papailiopou-
for Computational Linguistics. los. 2020a. Attack of the tails: Yes, you really can
backdoor federated learning. In Advances in Neural
Theo Ryffel, Andrew Trask, Morten Dahl, Bobby Wag- Information Processing Systems 33: Annual Con-
ner, Jason Mancuso, Daniel Rueckert, and Jonathan ference on Neural Information Processing Systems
Passerat-Palmbach. 2018. A generic framework for 2020, NeurIPS 2020, December 6-12, 2020, virtual.
privacy preserving deep learning. ArXiv preprint. Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dim-
itris S. Papailiopoulos, and Yasaman Khazaeni.
Victor Sanh, Lysandre Debut, Julien Chaumond, and 2020b. Federated learning with matched averaging.
Thomas Wolf. 2019. Distilbert, a distilled version In 8th International Conference on Learning Repre-
of bert: smaller, faster, cheaper and lighter. ArXiv. sentations, ICLR 2020, Addis Ababa, Ethiopia, April
26-30, 2020. OpenReview.net.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and
Ameet S. Talwalkar. 2017. Federated multi-task Thomas Wolf, Lysandre Debut, Victor Sanh, Julien
learning. In Advances in Neural Information Pro- Chaumond, Clement Delangue, Anthony Moi, Pier-
cessing Systems 30: Annual Conference on Neural ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-
Information Processing Systems 2017, December 4- icz, Joe Davison, Sam Shleifer, Patrick von Platen,
9, 2017, Long Beach, CA, USA, pages 4424–4434. Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu,
Teven Le Scao, Sylvain Gugger, Mariama Drame,
Jinhyun So, Başak Güler, and A Salman Avestimehr. Quentin Lhoest, and Alexander Rush. 2020. Trans-
2020. Byzantine-resilient secure federated learn- formers: State-of-the-art natural language process-
ing. IEEE Journal on Selected Areas in Commu- ing. In Proceedings of the 2020 Conference on Em-
nications. pirical Methods in Natural Language Processing:
 System Demonstrations, pages 38–45, Online. As-
sociation for Computational Linguistics.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin
Tong. 2019. Federated machine learning: Concept
and applications. ACM Transactions on Intelligent
Systems and Technology (TIST), (2).
T. Yang, G. Andrew, Hubert Eichner, Haicheng Sun,
W. Li, Nicholas Kong, D. Ramage, and F. Beau-
fays. 2018a. Applied federated learning: Improving
google keyboard query suggestions. ArXiv.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Ben-
gio, William Cohen, Ruslan Salakhutdinov, and
Christopher D. Manning. 2018b. HotpotQA: A
dataset for diverse, explainable multi-hop question
answering. In Proceedings of the 2018 Conference
on Empirical Methods in Natural Language Pro-
cessing, pages 2369–2380, Brussels, Belgium. As-
sociation for Computational Linguistics.
Ligeng Zhu, Zhijian Liu, and Song Han. 2019. Deep
leakage from gradients. In Advances in Neural
Information Processing Systems 32: Annual Con-
ference on Neural Information Processing Systems
2019, NeurIPS 2019, December 8-14, 2019, Van-
couver, BC, Canada, pages 14747–14756.

Xinghua Zhu, Jianzong Wang, Zhenhou Hong, and

Jing Xiao. 2020. Empirical studies of institutional
federated learning for natural language processing.
In Findings of the Association for Computational
Linguistics: EMNLP 2020, pages 625–634, Online.
Association for Computational Linguistics.
Appendix formats (Section B). As the non-IID data partition
over clients is the major feature of FL problems, it
is also a challenge to simulate the realistic non-IID
A FL+NLP partition for existing NLP datasets (Section 3.2).
Finally, a platform also must integrate various FL
Many realistic NLP services heavily rely on users’
methods with the Transformer-based NLP mod-
local data (e.g., text messages, documents and
els for a variety of task types, and thus a flexible
their tags, questions and selected answers, etc.),
and extensible learning framework is needed. In
which can be located at either personal devices
particular, the conventional trainer component of
or larger data-silos for organizations. These lo-
Transformers now needs to be modified for effi-
cal data are usually regarded as highly private and
cient and safe communications towards federated
thus not directly accessible by anyone, according
learning (Section F).
to many data privacy regulations; this makes it dif-
ficult to train a high-performance model to benefit B Basic Formulations of NLP Tasks
users. Federated learning aims to solve machine
learning under such a privacy-preserving use case, There are various types of NLP applications, but
thus offering a novel and promising direction to many of them share a similar task formulation
the community: FL+NLP. (i.e., input-and-put formats). We show four com-
Apart from the goal of learning a shared global mon task formulations that can cover most of the
model for all clients, FL also provides a new per- mainstream NLP applications: text classification,
spective for many other interesting research ques- sequence tagging, question answering, sequence-
tions in NLP. One related direction is to develop to-sequence generation.
personalized models for NLP applications, which Text Classification (TC) The input is a sequence
requires both protection of data privacy and trans- of words, x = [w1 , w2 , . . . ], and the output is a la-
ferred ability on users’ own input feature distri- bel y in a fixed set of labels L. Many NLP applica-
bution caused by language styles, interested top- tions can be formulated as text classification tasks.
ics and so on. The recent concerns on adversar- For example, we can use TC models for classi-
ial attacks and safety issues of NLP models are fying the topic of a news article to be political,
also highly related to FL+NLP. We thus believe sports, entertainment, etc., or analyzing movie re-
FL+NLP is of vital importance for applying NLP views to be positive, negative or neutral.
technologies in realistic use cases and could bene-
Sequence Tagging (ST) The input is a sequence
fit many relevant research areas.
of words, x = [w1 , w2 , . . . , wN ], and the out-
A.1 Challenges of Applying FL in NLP put is a same-length sequence of tags y =
[t1 , t2 , . . . , tN ], where ti is in a fixed set of labels
Given the promising benefits of studying FL+NLP,
L. The main difference between TC and ST is that
however, this research direction is currently
ST learns to classify the label of each token in a
blocked by the lack of a standardized platform
sentence, which is particularly useful in analyzing
providing fundamental building blocks: bench-
syntactic structures (e.g., part-of-speech analysis,
mark datasets, NLP models, FL methods, evalu-
phrase chunking, and word segmentation) and ex-
ation protocols, etc. Most of the current FL plat-
tracting spans (e.g., named entity recognition).
forms either focus on unifying various FL meth-
ods and use computer vision models and datasets Question Answering (QA) Given a passage P =
for their experiments, but lack the ability to con- [w1 , w2 , . . . , wN ] and a question q as input, the
nect the study of pre-trained language models, the task is to locate a span in the passage as the an-
most popular NLP , and realistic NLP applications swer to the question. Thus, the output is a pair of
of various task formulations. token index (s, e) where s, e ∈ {1, 2, . . . , N } for
The first challenge in developing a comprehen- denoting the begin and end of the span in the pas-
sive and universal platform for FL+NLP is to deal sage. This particular formulation is also known as
with various task formulations for realistic NLP reading comprehension.
applications, which have different input and output Natural Language Generation (NLG) Both in-
 already used up. Prior works choose to stop as-
signing early and remove such clients, but it conse-
quently loses the other unused examples and also
causes the inconsistency of client numbers. Thus,
to avoid these issues, we propose a dynamic re-
Prob. Density

assigning method which complement the vacancy

of a label by filling in the examples of other la-
bels based on their current ratio of remaining unas-
signed examples.

C.1 The FedNLP Training Pipeline: Security

# examples per client
and Efficiency
Under the definition of federated learning in Algo-
Figure 8: The probability density of quantity of train- rithm 1, we design a training system to support the
ing examples in each of the 100 clients on the 20News research of NLP in the FL paradigm. We highlight
dataset with different β. When β is larger, then all its core capabilities and design as follows.
clients share more similar numbers of examples; when
β is smaller, then the range of the quantity is much Supporting diverse FL algorithms. FedNLP
wider — i.e., the larger differences between clients in aims to enable flexible customization for future
terms of their sizes of datasets. algorithmic innovations. We have supported a
number of classical federated learning algorithms,
put and output are sequence of words, x = including FedAvg (McMahan et al., 2017a), Fe-
[w1i , w2i , . . . , wN
i ] , y = [w o , w o , . . . , w o ]. It is
1 2 M
dOPT (Reddi et al., 2021), and FedProx (Li et al.,
shared by many realistic applications such as sum- 2020b). These algorithms follow the same frame-
marization, response generation in dialogue sys- work introduced in Algorithm 1. The algorithmic
tems, machine translation, etc. APIs are modularized: all data loaders follow the
Language Modeling (LM) The left-to-right lan- same format of input and output arguments, which
guage modeling task considers a sequence of are compatible with different models and algo-
words as the input x = [w1 , w2 , . . . , wn ] and a rithms and are easy to support new datasets; the
token y = wn+1 as the output. The output token method of defining the model and related trainer is
is expected to be the most plausible next word of kept the same as in centralized training to reduce
the incomplete sentence denoted as x. Although the difficulty of developing the distributed train-
the direct application of LM is limited, a high- ing framework. For new FL algorithm develop-
performance pre-trained language model can ben- ment, worker-oriented programming reduces the
efit a wide range of NLP applications (as above) difficulty of message passing and definition. More
via fine-tuning. It also serves as an excellent test details are introduced in Appendix F.3.
bed as it requires no human annotations at all.
Enabling secure benchmarking with
Others. There are some other applications that lightweight secure aggregation. In partic-
not are covered by the above four basic formu- ular, FedNLP enhances the security aspect of
lations, and our extensible platform (detailed in federated training, which is not supported by ex-
Section F) enables users to easily implement their isting non-NLP-oriented benchmarking libraries
specific tasks. For each task formulation, we show (e.g., TFF, LEAF). This is motivated by the fact
which datasets are used in FedNLP and how we that model weights from clients may still have
partition them in Section 3. the risk of privacy leakage (Zhu et al., 2019). To
break this barrier, we integrate secure aggregation
C Implementation Details
(SA) algorithms to the FedNLP system. NLP
Non-IID. Label Distribution Note that this researchers do not need to master security-
might cause a few clients not to have enough ex- related knowledge and also benefit from a secure
amples to sample for particular labels if they are distributed training environment. To be more
specific, FedNLP supports state-of-the-art SA personalization and fairness in federated training.
algorithms LightSecAgg, SecAgg (Bonawitz For trustworthiness, security and privacy are the
et al., 2017), and SecAgg+ (Bell et al., 2020). At two main research directions that are mainly con-
a high-level understanding, SA protects the client cerned with resisting data or model attacks, recon-
model by generating a single random mask and struction, and leakage during training (So et al.,
allows their cancellation when aggregated at the 2021b,a, 2020; Prakash et al., 2020; Prakash and
server. Consequently, the server can only see the Avestimehr, 2020; Elkordy and Avestimehr, 2020;
aggregated model and not the raw model from Prakash et al., 2020; Wang et al., 2020a; Lyu
each client. In this work, our main effort is to et al., 2020). Given that modern deep neural net-
design and optimize these SA algorithms in the works are over-parameterized and dominate nearly
context of the FedNLP system. We provide an all learning tasks, researchers also proposed algo-
algorithmic performance comparison in Appendix rithms or systems to improve the efficiency and
F.5. scalability of edge training (He et al., 2020b,c,
2019, 2021). We refer readers to the canonical sur-
Realistic evaluation with efficient distributed vey (Kairouz et al., 2019) for details.
system design. FedNLP aims to support dis- Although tremendous progress has been made
tributed training in multiple edge servers (e.g, in the past few years, these algorithms or systems
AWS EC2) or edge devices (e.g., IoTs and smart- have not been fully evaluated on realistic NLP
phones). To achieve this, the system is designed tasks introduced in this paper.
with three layers: the application layer, the algo-
rithm layer, and the infrastructure layer. At the ap- E Future Directions
plication layer, FedNLP provides three modules:
Minimizing the performance gap. In the FL
data management, model definition, and a single-
setting, we demonstrate that federated fine-tuning
process trainer for all task formats; at the algo-
still has a large accuracy gap in the non-IID dataset
rithm layer, FedNLP supports various FL algo-
compared to centralized fine-tuning. Develop-
rithms; at the infrastructure layer, FedNLP aims
ing algorithms for Transformer models with NLP
at integrating single-process trainers with a dis-
tasks is of the highest priority.
tributed learning system for FL. Specifically, we
make each layer and module perform its own du- Improving the system efficiency and scalabil-
ties and have a high degree of modularization. We ity. Transformer models are usually large, while
refer readers to Appendix F for a detailed descrip- resource-constrained edge devices may not be able
tion of the system architecture and design philos- to run large models. Designing efficient FL meth-
ophy. ods for NLP tasks is thus a practical problem
worth solving. How to adopt a reasonable user se-
D More Related Works lection mechanism to avoid stragglers and speed
up the convergence of training algorithms is also a
Federated Learning Methods. Federated
pressing problem to be solved.
Learning (FL) is a widely disciplinary research
area that mainly focuses on three aspects: sta- Trustworthy and privacy-preserving NLP.
tistical challenge, trustworthiness, and system We argue that it is an important future research
optimization. Numerous methods have been direction to analyze and assure the privacy-
proposed to solve statistical challenges, including preserving ability of these methods, although our
FedAvg (McMahan et al., 2017b), FedProx (Li focus in this paper is the implementation and
et al., 2020b), FedOPT (Reddi et al., 2021), performance analysis of the FL methods for NLP
FedNAS (He et al., 2020a,d), and FedMA (Wang tasks. It is now an open problem for both FL
et al., 2020b) that alleviate the non-IID issue and NLP areas, while it is an orthogonal goal
with distributed optimization, and new formu- for improving the trustworthy of decentralized
lations, MOCHA (Smith et al., 2017), pFedMe learning, and it is only possible to study privacy
(Dinh et al., 2020), perFedAvg (Fallah et al., preservation when we have an existing FL+NLP
2020), and Ditto (Li et al., 2021b), that consider platform. This is also part of our motivation in
proposing FedNLP, and we believe our framework Algorithm 2: The FedNLP Workflow
provides a set of flexible interfaces for future
# using text classification (TC) as an example
development to analyze and improve the privacy-
# initialize distributed computing environment
preserving ability of FL methods for NLP tasks process_id, ... = FedNLP_init()
and beyond. # GPU device management
device = map_process_to_gpu(process_id, ...)
Personalized FedNLP. From the perspective of # data management
the data itself, user-generated text is inherently data_manager = TCDataManager (process_id, ...)
# load the data dictionary by process_id
personalized. Designing personalized algorithms data_dict = dm.load_federated_data(process_id)

to improve model accuracy or fairness is a very # create model by specifying the task
client_model, ... = create_model(model_args,
promising direction. In addition, it is also an inter- formulation="classification")
esting problem to adapt the heterogeneous model # define a customized NLP Trainer
client_trainer = TCTrainer(device,
architecture for each client in the FL network. We client_model, ...)
show that it is feasible to only fine-tune a small # launch the federated training (e.g., FedAvg)
amount of the parameters of LMs, so it is promis- FedAvg_distributed(..., device,
client_model,
ing to adapt recent prefix-tuning methods (Li and data_dict, ...,
client_trainer)
Liang, 2021) for personalizing the parameters of
NLP models within the FedNLP framework.

F The System Design of FedNLP types of DataManager according to the task def-
inition. Users can customize their DataManager
The FedNLP platform consists of three layers: by inheriting one of the DataManager class, spec-
the application layer, the algorithm layer, and ifying data operation functions, and embedding a
the infrastructure layer. At the application layer, particular preprocessor. Note that the raw data’s
FedNLP provides three modules: data manage- H5Py file and the non-IID partition file are pre-
ment, model definition, and single-process trainer processed offline, while DataManager only loads
for all task formats; At the algorithm layer, them in runtime.
FedNLP supports various FL algorithms; At the
infrastructure layer, FedNLP aims at integrating Model Definition. We support two types of
single-process trainers with a distributed learning models: Transformer and LSTM. For Transformer
system for FL. Specifically, we make each layer models, to dock with the existing NLP ecology,
and module perform its own duties and have a high our framework is compatible with the Hugging-
degree of modularization. Face Transformers library (Wolf et al., 2020), so
that various types of Transformers can be directly
F.1 Overall Workflow reused without the need for re-implementation.
The module calling logic flow of the whole frame- Specifically, our code is compatible with the three
work is shown on the left of Figure 9. When main classes of Tokenizer, Model, and Config
we start the federated training, we first complete in HuggingFace. Users can also customize them
the launcher script, device allocation, data load- based on HuggingFace’s code. Although LSTM
ing, and model creation, and finally call the API has gradually deviated from the mainstream, we
of the federated learning algorithm. This process still support LSTM to reflect the framework’s in-
is expressed in Python-style code (see Alg. 2). tegrity, which may meet some particular use cases
in a federated setting.
F.2 The Application Layer NLP Trainer (single process perspective). As
Data Management. In data management, What for the task-specific NLP Trainer, the most
DataManager does is control the whole workflow prominent feature is that it does not require users
from loading data to returning trainable features. to have any background in distributed comput-
To be specific, DataManager is set up for read- ing. Users of FedNLP only need to complete
ing h5py data files and driving a preprocessor single-process code writing. A user should in-
to convert raw data to features. There are four herit the Trainer class in the application layer
 Figure 9: The overall workflow and system design of the proposed FedNLP platform.

to implement the four methods as shown in the learning an NLP model on a dataset.
figure: 1. the get_model_params() interface al-
lows the algorithm layer to obtain model param- FedAvg (McMahan et al., 2017a) is the de facto
eters and transmit them to the server; 2. the method for federated learning, assuming both
set_model_params() interface obtains the up-
client and server use the SGD optimizer for up-
dated model from the server’s aggregation and dating model weights.
then updates the model parameters of the local FedProx (Li et al., 2020b) can tackle statistical
model; 3. the programming of the train() and heterogeneity by restricting the local model up-
test() function only needs to consider the data dates to be closer to the initial (global) model with
of a single user, meaning that the trainer is com- L2 regularization for better stability in training.
pletely consistent with the centralized training.
FedOPT (Reddi et al., 2021) is a generalized
F.3 The Algorithm Layer version of FedAvg. There are two gradient-based
In the design of the algorithm layer, we follow optimizers in the algorithm: ClientOpt and
the principle of one-line API. The parameters of ServerOpt (please refer to the pseudo code in
the API include model, data, and single-process the original paper (Reddi et al., 2021)). While
trainer (as shown in Algorithm 2). The algorithms ClientOpt is used to update the local models,
we support include: SerevrOpt treats the negative of aggregated lo-
cal changes −∆(t) as a pseudo-gradient and ap-
Centralized Training. We concatenate all client plies it on the global model. In our FedNLP frame-
datasets and use the global data DG to train a work, by default, we set the ClientOpt to be
global model — i.e., the conventional protocol for AdamW (Loshchilov and Hutter, 2019) and the
SerevrOpt to be SGD with momentum (0.9) ??). The main idea of LightSecAgg are that
and fix server learning rate as 1.0. each user protects its local model using a locally
Each algorithm includes two core objects, generated random mask. This mask is then en-
ServerManager and ClientManager, which in- coded and shared with other users, in such a way
tegrate the communication module ComManager that the aggregate mask of any sufficiently large
from the infrastructure layer and the Trainer of set of surviving users can be directly reconstructed
the training engine to complete the distributed al- at the server. Our main effort in FedNLP is in-
gorithm protocol and edge training. Note that tegrating these algorithms, optimizing its system
users can customize the Trainer by passing a cus- performance, and designing user-friendly APIs to
tomized Trainer through the algorithm API. make them compatible with NLP models and FL
algorithms.
F.4 The Infrastructure Layer
The infrastructure layer includes three modules:
1) Users can write distributed scripts to man-
age GPU resource allocation. In particular,
FedNLP provides the GPU assignment API
(map_process_to_gpu() in Algorithm 2) to as-
sign specific GPUs to different FL Clients.
2) The algorithm layer can use a unified and ab-
stract ComManager to complete a complex al-
gorithmic communication protocol. Currently,
we support MPI (Message Passing Interface),
RPC (Remote procedure call), and MQTT (Mes-
sage Queuing Telemetry Transport) communica-
tion backend. MPI meets the distributed training
needs in a single cluster; RPC meets the communi-
cation needs of cross-data centers (e.g., cross-silo
federated learning); MQTT can meet the commu-
nication needs of smartphones or IoT devices.
3) The third part is the training engine, which
reuses the existing deep learning training engines
by presenting as the Trainer class. Our cur-
rent version of this module is built on PyTorch,
but it can easily support frameworks such as
TensorFlow. In the future, we may consider sup-
porting the lightweight edge training engine opti-
mized by the compiler technology at this level.

F.5 Enhancing Security with Secure

Aggregation (SA)
FedNLP supports state-of-the-art SA algorithms
LightSecAgg, SecAgg (Bonawitz et al.,
2017), and SecAgg+ (Bell et al., 2020). Here, we
provide a short performance comparison of these
three algorithms. In general, LightSecAgg
provides the same model privacy guarantees as
SecAgg (Bonawitz et al., 2017) and SecAgg+
(Bell et al., 2020)) while substantially reducing the
aggregation (hence run-time) complexity (Figure