LANGUAGE MODEL INTEGRATION BASED ON MEMORY CONTROL FOR

SEQUENCE TO SEQUENCE SPEECH RECOGNITION

Jaejin Cho1 , Shinji Watanabe1 , Takaaki Hori2 , Murali Karthick Baskar3 ,

Hirofumi Inaguma4 , Jesus Villalba1 , Najim Dehak1
1
Johns Hopkins University, 2 Mitsubishi Electric Research Laboratories (MERL),
3
Brno University of Technology, 4 Kyoto University
{jcho52,shinjiw,jvillal7,ndehak3}@jhu.edu, [email protected], [email protected],
[email protected]

ABSTRACT However, compared to the conventional ASR, there have been only
a few studies on ways to integrate an LM into seq2seq ASR [13, 14,
In this paper, we explore several new schemes to train a seq2seq 15]. In this direction, the authors in [5] introduce two methods inte-
model to integrate a pre-trained language model (LM). Our proposed grating an LM into a decoder of the end-to-end neural machine trans-
fusion methods focus on the memory cell state and the hidden state lation (NMT) system. The first method was shallow fusion where the
in the seq2seq decoder long short-term memory (LSTM), and the model decodes based on a simple weighted sum of NMT model and
memory cell state is updated by the LM unlike the prior studies. recurrent neural network LM [16] (RNNLM) scores. The next one
This means the memory retained by the main seq2seq would be ad- was called deep fusion where they combine a mono-lingual RNNLM
justed by the external LM. These fusion methods have several vari- with an NMT model by learning parameters that connect hidden
ants depending on the architecture of this memory cell update and states of a separately trained NMT model and RNNLM. While the
the use of memory cell and hidden states which directly affects the parameters connecting the hidden states are trained, parameters in
final label inference. We performed the experiments to show the ef- NMT and RNNLM are frozen. Recently in ASR research, a scheme
fectiveness of the proposed methods in a mono-lingual ASR setup on called cold fusion was introduced, which trains a seq2seq model
the Librispeech corpus and in a transfer learning setup from a multi- from scratch in assistance with a pre-trained RNNLM [17]. In con-
lingual ASR (MLASR) base model to a low-resourced language. In trast to the previous methods, the parameters of the seq2seq model
Librispeech, our best model improved WER by 3.7%, 2.4% for test are not frozen during training although pre-trained RNNLM parame-
clean, test other relatively to the shallow fusion baseline, with multi- ters are still kept being frozen. The results showed the model trained
level decoding. In transfer learning from an MLASR base model this way outperforms deep fusion in decoding as well as reducing the
to the IARPA Babel Swahili model, the best scheme improved the amount of data in domain adaptation. Later, more experiments were
transferred model on eval set by 9.9%, 9.8% in CER, WER relatively done comparing all three methods [18]. In the paper, they observe
to the 2-stage transfer baseline. that cold fusion works best among all three methods in the second
Index Terms: Automatic speech recognition (ASR), sequence to pass re-scoring with a large and production-scale LM. The previ-
sequence, language model, shallow fusion, deep fusion, cold fusion ous research has shown the potential of training a seq2seq model
utilizing a pre-trained LM. However, it seems only effective in lim-
1. INTRODUCTION ited scenarios such as domain adaptation and second-pass re-scoring.
Thus, studies on better ways of integrating both models need to be
As deep learning prospers in most research fields, systems based on explored.
it keep improving and become the state-of-the-art in most of the sce- In this paper, we explored several new fusion schemes to train
narios. The sequence to sequence (seq2seq) model is one of the kind a seq2seq model jointly with a pre-trained LM. Among them, we
that heavily depends on deep learning techniques, and it is used in found one method that works consistently better than other fusion
many sequence mapping problems such as automatic speech recog- methods over more general scenarios. The proposed methods focus
nition (ASR) [1, 2, 3, 4] and machine translation [5, 6, 7]. In [4], on updating the memory cell state as well as the hidden state of the
a seq2seq model with attention mechanism is introduced in ASR. seq2seq decoder long short-term memory (LSTM) [19], given the
Though the performance lagged behind highly-optimized conven- LM logit or hidden state. This means that the memory retained by
tional systems, e.g. CLDNN HMM system [8], it enabled to map a the main seq2seq model will be adjusted by the external LM for bet-
sequence of feature vectors to a sequence of characters, with a sin- ter prediction. The fusion methods have several variants according
gle neural network, in an end-to-end manner. In [9], the authors to the architecture of this memory cell update and the use of memory
apply a multi-task learning scheme to train an attentional seq2seq cell and hidden states, which directly affects the final label inference.
model with connectionist temporal classification (CTC) objective Note that we used LSTM networks for RNNs throughout whole ex-
function [1, 10] as auxiliary loss. Adding the CTC loss to train the planations and experiments. The proposed methods, however, can
model reduces the burden of the attention model to learn monotonic be applied to different variant RNNs such as gated recurrent unit
attention. (GRU) [20] only with minimal modification.
In the seq2seq ASR setup, the language model (LM) takes an im- The organization of this paper is as follows. First, we describe
portant role as it is already shown in hybrid ASR systems [11, 12]. previous fusion methods as a background in Section 2. Then, in Sec-

978-1-5386-4658-8/18/$31.00 ©2019 IEEE 6191 ICASSP 2019

 $ 𝒚𝑷$ 𝒕𝒚𝒕𝒚𝒚'𝒕
𝑷 , 𝒙)
tion 3, we explain our proposed methods in detail. Experiments with '𝒕, 𝒙)

⊙⊙
previous and proposed methods are presented in Section 4. Lastly,
we conclude the paper in Section 5. 𝜎𝜎

PrePre trained
trained
RNNLM
RNNLM
Decoder
Decoder
2. BACKGROUND: SHALLOW FUSION, DEEP FUSION,
AND COLD FUSION IN ASR Seq2seq
Seq2seq
model
model

2.1. Shallow fusion

(a) cold fusion
In this paper, we denotes as shallow fusion a decoding method based
on the following convex score combination of a seq2seq model and $ 𝒚𝒕 𝒚'𝒕 , 𝒙)
𝑷
LM during beam search, ⊙
𝜎 2
ŷ = argmax (log p(y|x) + γ log p(y)) (1) 1 3
y Pre trained
RNNLM
Decoder

where x is an input acoustic frame sequence while ŷ is the predicted

Seq2seq
label sequence selected among all possible y. The predicted label model
sequence can be a sequence of characters, sub-words, or words, and
this paper deals with character-level sequences. log p(y|x) is calcu- (b) cell update
lated from the seq2seq model and log p(y) is calculated from the
RNNLM. Both models are separately trained but their scores are $ 𝒚𝒕 𝒚'𝒕 , 𝒙)
𝑷
combined in a decoding phase. γ is a scaling factor between 0 and 1 ⊙
that needs to be tuned manually. 𝜎 2
1
3
Pre trained
Decoder
2.2. Deep fusion RNNLM

In deep fusion, the seq2seq model and RNNLM are combined with Seq2seq
learnable parameters. Two models are first trained separately as in model

shallow fusion, and then both are frozen while the connecting linear
(c) cell and state update + cold fusion
transformation parameters, i.e. v, b, W , and b below, are trained.

gt = σ(v T sLM Fig. 1. High-level illustration of fusion methods in training. (b) and
t + b) (2a)
(c) correspond to cell control fusion 1 and cell control fusion 3
respectively among our proposed methods. Output layer here is a
sDF LM
t = [st ; gt st ] (2b)
linear transformation, with non-linearity only when having it bene-
p̂(yt |y<t , x) = softmax(W sDF
t + b) (2c) fits empirically
sLM
t and st are hidden states at time t from an RNNLM and a de-
coder of the seq2seq model, respectively. σ(·) in Eq. (2a) is the sig-
moid function to generate gt , which acts as a scalar gating function 3. PROPOSED METHODS
in Eq. (2b) to control the contribution of sLM
t to the final inference.
In this paper, we propose several fusion schemes to train a seq2seq
model well integrated with a pre-trained RNNLM. We mainly focus
2.3. Cold fusion on updating hidden/memory cell states in the seq2seq LSTM de-
coder given the LM logit/hidden state. The first proposed method
In contrast to the two previous methods, cold fusion uses the
uses the LM information to adjust the memory cell state of the
RNNLM in a seq2seq model training phase. The seq2seq model
seq2seq decoder. Then, the updated cell state replaces the origi-
is trained from scratch with the pre-trained RNNLM model whose
nal input cell state of the LSTM decoder to calculate states at the
parameters are frozen and learnable parameters Wk and bk , where
next time step. This fusion scheme is inspired by cold fusion, but
k = 1, 2, 3. The equation follows
they differ in that the new method directly affects the memory cell
maintaining mechanism in the LSTM decoder to consider the lin-
hLM
t = W1 ltLM + b1 (3a)
guistic context obtained from the LM. Then, we further extend this
gt = σ(W2 [st ; hLM idea with multiple schemes that use the LM information not only for
t ] + b2 ) (3b)
updating the memory cell states but also hidden states in the seq2seq
sCF
t = [st ; gt hLM
t ] (3c) LSTM decoder, which further affect the final inference and attention
calculation directly. Figure 1 visualizes the schemes in high-level to
p̂(yt |y<t , x) = softmax(ReLU(W3 sCF
t + b3 )) (3d) help understanding.
where ltLM is the logit at time step t from the RNNLM. As opposed
to a scalar gt in Eq. (2b) of deep fusion, it is now a vector gt in cold 3.1. LM fusion by controlling memory cell state
fusion, meaning the gating function is applied element-wise, where
means element-wise multiplication. ReLU(·) is a rectified linear In Eq. (3c) at cold fusion, the gated RNNLM information, gt hLM t ,
function applied element-wise. Applying it before softmax(·) was is concatenated with the decoder hidden state, st . Then, the fused
shown to be helpful empirically in [17]. state, sCF
t is used to predict the distribution of the next character.

6192
However, the gated RNNLM information can be used in a different sCCF3
t = W4 [st ; gtstate hLM
t ] + b4 (6d)
way to update the cell state in the seq2seq decoder, as in Eq. (4c).
cCCF3
t = CellUpdate(ct , gtcell hLM
t ) (6e)
hLM
t = tanh(W1 ltLM + b1 ) (4a) st+1 , ct+1 = LSTM(input, sCCF3
t , cCCF3
t ) (6f)
gt = σ(W2 [ct ; hLM
t ] + b2 ) (4b) p̂(yt |x, y<t ) = softmax(ReLU(W5 sCCF3
t + b5 )) (6g)
cCCF
t = ct + gt hLM
t (4c) For CellUpdate function in Eq. (6e), we compared two different
calculations:
st+1 , ct+1 = LSTM(input, st , cCCF
t ) (4d) ct + gtcell hLM (7)
t
p̂(yt |y<t , x) = softmax(W3 st + b3 ) . (4e) W0 [ct ; gtcell hLM
t ] + b0 (8)
In this method, we add the gated RNNLM information to the original [ct ; gtcell hLM
In the case of Eq. (8), the affine transformation of t ]
cell state, ct . LSTM(·) in Eq. (4d) is the standard LSTM function, would cause gradient vanishing problem in theory. However, we
which takes the previous cell and hidden states and an input came found that in practice, the method works best among all proposed
from an attention context vector, and updates the cell and hidden methods.
states for the next time step. In our case, when the LSTM decoder
updates its cell state, it uses cCCF
t instead of ct , which contains ad-
ditional linguistic context obtained from an external LM. We call 4. EXPERIMENTS
this fusion as cell control fusion 1 throughout the paper. Here, this
method does not include ReLU(·) before softmax(·) in Eq. (4e) since We first compared all our proposed methods described in Sec-
it did not show any benefit empirically unlike in other methods. tion 3 with shallow fusion, deep fusion, and cold fusion on the
100hrs subset of the Librispeech corpus [21] as a preliminary ex-
periment. Then, we further investigate some selected methods with
3.2. LM fusion by updating both hidden and memory cell states
two other experiments: mono-lingual ASR setup on the Librispeech
In cold fusion, the update of the hidden state output from the LSTM 960hrs corpus and a transfer learning setup from a multilingual
decoder in Eq. (3c) directly affects the final inference unlike cell ASR (MLASR) base model to a low-resourced language, Swahili
control fusion 1. Therefore, this section combines the concepts of in IARPA Babel [22].
the cold fusion and cell control fusion 1 and further proposes variants We used shallow fusion all the time in decoding phase for every
of novel fusion methods by extending the cell control fusion 1 with trained model. For example, we can additionally use shallow fusion
the above hidden state consideration. decoding for the seq2seq model trained with a cold/cell-control fu-
First, we simply combine cell control fusion 1 in Section 3.1 sion scheme. We refer to [18] to justify the comparison between
with cold fusion. We call this scheme as cell control fusion 2. The the baseline seq2seq model with shallow/deep fusion decoding and
detailed equations are the seq2seq model trained using cold/cell-control fusion with shal-
low fusion decoding. The baseline seq2seq model above means a
hLM
t = W1 ltLM + b1 (5a) seq2seq model trained without any fusion method.
All models are trained with joint CTC-attention objective as pro-
gtcell = σ(W2 [ct ; hLM
t ] + b2 ) (5b) posed in [9],
cCCF2
t = ct + gtcell hLM
t (5c)
LMTL = αLCTC + (1 − α)LAttention (9)
st+1 , ct+1 = LSTM(input, st , cCCF2
t ) (5d)
gtstate = σ(W3 [st ; hLM where LCTC and LAttention are the losses for CTC and attention repec-
t ] + b3 ) (5e)
tively, and α ranges between 0 and 1 inclusively. In decoding, we did
sCCF2
t = [st ; gtstate hLM
t ] (5f) attention/CTC joint decoding with RNNLM [23]. In all the experi-
p̂(yt |y<t , x) = softmax(ReLU(W4 sCCF2 + b4 )) . (5g) ments, we represented each frame of 25ms windowed audio having
t
10ms shift by a vector of 83 dimensions, which consists of 80 Mel-
Note the calculations of Eq. (5a), (5e)-(5g) are exactly the same as filter bank coefficients and 3 pitch features. The features were nor-
of Eq. (3), and calculations of Eq. (5a)-(5d) are same as of Eq. (4a)- malized by the mean and the variance of the whole training set. All
(4d) other than tanh(·) used in Eq. (4a), which shows some effective- experiments were done based on ESPnet toolkit [24].
ness on our preliminary investigation. However, this straightforward For Librispeech, training and decoding configurations of the
extension does not outperform both cell control fusion 1 and cold seq2seq model are shown in Table 1. We trained both character-
fusion. level and word-level RNNLMs on 10% of the text publicly available
As a next variant, we apply a similar cell control update mecha- for Librispeech1 , which is roughly 10 times the 960 hours of the
nism (Eq. (5d)) to the seq2seq decoder hidden state st as well. That transcriptions in terms of the data size.
is, the original hidden state, st , is replaced by sCCF3
t in the LSTM In the MLASR transfer learning scenario, the base MLASR
update Eq. (6f), which is transformed from the fused state in cold model was trained exactly the same as in [25]. The base model
fusion to match dimension. sCCF3
t is expected to have more informa- was trained using 10 selected Babel languages, which are roughly
tion since it contains additional linguistic context obtained from an 640 hours of data: Cantonese, Bengali, Pashto, Turkish, Vietnamese,
external LM. The Eq. (6) explains this fusion method more in detail. Haitian, Tamil, Kurmanji, Tokpisin, and Georgian. The model pa-
We call this type of fusion cell control fusion 3 in this paper. rameters were then fine-tuned using all Swahili corpus in Babel,
which is about 40 hours. During the transfer process, we used the
hLM
t = tanh(W1 ltLM + b1 ) (6a) same MLASR base model with three different ways: 2-stage trans-
gtstate = σ(W2 [st ; hLM
t ] + b2 ) (6b) fer (see [25] for more details), cold fusion, and cell control fusion
gtcell = σ(W3 [ct ; hLM
t ] + b3 ) (6c) 1 http://www.openslr.org/11/

6193
3 (affine). We included cold fusion in this comparison since cold by shallow fusion, and cold fusion. Also, we observed that the gap
fusion showed its effectiveness in domain adaptation in [17]. The in the WER between cell control fusion 3 (affine) and shallow fusion
character-level RNNLM was trained on all transcriptions available is larger when we use multi-level decoding. This suggests that with
for Swahili in IARPA Babel. the advanced decoding algorithm, the cell control fusion 3 benefits
more in performance. Note that γ was set to 0.3 for character-level
Table 1. Experiment details decoding and 0.5 for multi-level decoding.
Model Configuration
Table 3. LibriSpeech 960 hours: Character-level decoding (%WER)
Encoder Bi-LSTM Fusion dev dev test test
# encoder layers 8 method clean other clean other
# encoder units 320 Shallow fusion 6.1 17.6 6.1 18.1
# projection units 320 Cold fusion 6.1 18.1 6.3 18.7
Decoder Bi-LSTM Cell control fusion 3 (affine) 6.0 17.1 6.1 17.9
# decoder layers 1
# decoder units 300 Table 4. LibriSpeech 960 hours: Multi-level decoding (%WER)
Attention Location-aware Fusion dev dev test test
Training Configuration method clean other clean other
Shallow fusion 5.4 15.8 5.4 16.6
Optimizer AdaDelta [26]
Initial learning rate 1.0
Cold fusion 5.4 16.2 5.6 17.1
AdaDelta  1e−8
Cell control fusion 3 (affine) 5.2 15.5 5.2 16.2
AdaDelta  decay 1e−2
Batch size 36 4.3. Transfer learning to low-resourced language
ctc-loss weight (α) 0.5
Finally, Table 5 shows the result of the transfer learning to a low-
Decoding Configuration resourced language (Swahili). The cold fusion transfer improved the
Beam size 20 performance from simple 2-stage, showing its effectiveness in this
ctc-weight (λ [23]) 0.3 scenario but cell control fusion 3 (affine) improved the performance
further. cell control fusion 3 (affine) outperforms cold fusion not
only in this MLASR transfer learning setup but also in the previous
4.1. Preliminary experiments: Librispeech 100hrs mono-lingual ASR setup. In decoding, γ was set to 0.4.
Table 2 compares the proposed cell control fusion methods to the Table 5. Transfer learning from an MLASR base model to Swahili:
conventional fusion methods. Both cold fusion and cell control fu- Character-level decoding (%CER, %WER)
sion 1 show similar performance, but the performance of cell control Fusion eval set eval set
fusion 2 was degraded. This implies that simply combining cold fu- method %CER %WER
sion and cell control fusion 1 does not have any benefit. Then, The Shallow fusion (2-stage transfer) [25] 27.2 56.2
cell control fusion 3 methods, which extended cell control fusion 2 Cold fusion 25.8 52.9
by applying cell control update mechanism (Eq. (5d)) to the seq2seq cell control fusion 3 (affine) 24.5 50.7
decoder hidden state (Eq. (6f)), outperformed all the previous fusion
methods in most cases. This suggests that applying the cell control
update mechanism for both cell and hidden states consistently, fur- 5. CONCLUSION
ther improves the performance. Among the two cell control fusion
3 methods, cell control fusion 3 (affine) outperformed all the other In this paper, several methods were proposed to integrate a pre-
methods in every case. During the decoding, the shallow fusion pa- trained RNNLM during a seq2seq model training. First, we used
rameter γ in Eq. (1) was set to 0.3. information from the LM output to update the memory cell in the
seq2seq model decoder, which performed similarly to cold fusion.
Table 2. Comparison of previous and cell control fusion methods on
Then, we extended this model to additionally update the seq2seq
Librispeech 100 hours: Character-level decoding (%WER)
Fusion dev test dev test model hidden state given the LM output. For the scheme, Several
method clean clean other other formulas were compared. Among the proposed methods, cell con-
Shallow fusion 16.9 16.7 45.6 47.9 trol fusion 3 (affine) showed the best performance consistently on
Deep fusion 17.1 17.0 45.9 48.3 all experiments. In Librispeech, our best model improved WER by
Cold fusion 16.7 16.4 45.5 47.8 3.7%, 2.4% for test clean, test other relatively to the shallow fusion
Cell control fusion 1 16.4 16.5 45.4 47.7 baseline, with multi-level decoding. For the transfer learning setup
Cell control fusion 2 17.4 16.8 45.9 48.4 from an MLASR base model to the IARPA Babel Swahili model, the
Cell control fusion 3 (sum) 16.7 16.3 45.3 47.2 best scheme improved the transferred model performance on eval set
Cell control fusion 3 (affine) 16.0 16.0 44.7 46.6
by 9.9%, 9.8% in CER, WER relatively to the 2-stage transfer base-
line. In the future, we will explore how the best method performs by
4.2. Librispeech 960 hrs the amount of additional text data used for RNNLM training.
In this setting, we decoded in two ways: character-level decoding
and multi-level (character and word) decoding [23]. The word-level 6. ACKNOWLEDGMENTS
RNNLM used for the multi-level decoding has 20,000 as its vocab- The work reported here was started during JSALT 2018, and sup-
ulary size. The results are shown in Table 3 and 4 respectively. For ported by JHU with gifts from Amazon, Facebook, Google, Mi-
both cases, cell control fusion 3 (affine), performed the best followed crosoft and Mitsubishi Electric.

6194
 7. REFERENCES [14] Alex Graves, “Sequence transduction with recurrent neural
networks,” arXiv preprint arXiv:1211.3711, 2012.
[1] Alex Graves and Navdeep Jaitly, “Towards end-to-end speech [15] Yajie Miao, Mohammad Gowayyed, and Florian Metze,
recognition with recurrent neural networks,” in International “Eesen: End-to-end speech recognition using deep rnn models
Conference on Machine Learning, 2014, pp. 1764–1772. and wfst-based decoding,” in Automatic Speech Recognition
[2] Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and and Understanding (ASRU), 2015 IEEE Workshop on. IEEE,
Yoshua Bengio, “End-to-end continuous speech recognition 2015, pp. 167–174.
using attention-based recurrent nn: first results,” arXiv preprint [16] Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ,
arXiv:1412.1602, 2014. and Sanjeev Khudanpur, “Recurrent neural network based lan-
[3] Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Phile- guage model,” in Eleventh Annual Conference of the Interna-
mon Brakel, and Yoshua Bengio, “End-to-end attention-based tional Speech Communication Association, 2010.
large vocabulary speech recognition,” in Acoustics, Speech and [17] Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam
Signal Processing (ICASSP), 2016 IEEE International Confer- Coates, “Cold fusion: Training seq2seq models together with
ence on. IEEE, 2016, pp. 4945–4949. language models,” arXiv preprint arXiv:1708.06426, 2017.
[4] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals, [18] Shubham Toshniwal, Anjuli Kannan, Chung-Cheng Chiu,
“Listen, attend and spell: A neural network for large vocab- Yonghui Wu, Tara N Sainath, and Karen Livescu, “A
ulary conversational speech recognition,” in IEEE Interna- comparison of techniques for language model integration
tional Conference on Acoustics, Speech and Signal Processing in encoder-decoder speech recognition,” arXiv preprint
(ICASSP). IEEE, 2016, pp. 4960–4964. arXiv:1807.10857, 2018.
[5] Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, [19] Sepp Hochreiter and Jürgen Schmidhuber, “Long short-term
Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780,
and Yoshua Bengio, “On using monolingual corpora in neural 1997.
machine translation,” arXiv preprint arXiv:1503.03535, 2015.
[20] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and
[6] Minh-Thang Luong, Hieu Pham, and Christopher D Manning, Yoshua Bengio, “Empirical evaluation of gated recurrent
“Effective approaches to attention-based neural machine trans- neural networks on sequence modeling,” arXiv preprint
lation,” arXiv preprint arXiv:1508.04025, 2015. arXiv:1412.3555, 2014.
[7] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, [21] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev
“Neural machine translation by jointly learning to align and Khudanpur, “Librispeech: an asr corpus based on public do-
translate,” arXiv preprint arXiv:1409.0473, 2014. main audio books,” in Acoustics, Speech and Signal Process-
ing (ICASSP), 2015 IEEE International Conference on. IEEE,
[8] Tara N Sainath, Oriol Vinyals, Andrew Senior, and Haşim Sak,
2015, pp. 5206–5210.
“Convolutional, long short-term memory, fully connected deep
neural networks,” in Acoustics, Speech and Signal Process- [22] Martin Karafiát, Murali Karthick Baskar, Pavel Matějka, Karel
ing (ICASSP), 2015 IEEE International Conference on. IEEE, Veselỳ, František Grézl, and Jan Černocky, “Multilingual
2015, pp. 4580–4584. blstm and speaker-specific vector adaptation in 2016 but ba-
bel system,” in Spoken Language Technology Workshop (SLT),
[9] Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey,
2016 IEEE. IEEE, 2016, pp. 637–643.
and Tomoki Hayashi, “Hybrid CTC/attention architecture for
end-to-end speech recognition,” IEEE Journal of Selected Top- [23] Takaaki Hori, Shinji Watanabe, and John R Hershey, “Multi-
ics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017. level language modeling and decoding for open vocabulary
end-to-end speech recognition,” in Automatic Speech Recogni-
[10] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen tion and Understanding Workshop (ASRU), 2017 IEEE. IEEE,
Schmidhuber, “Connectionist temporal classification: la- 2017, pp. 287–293.
belling unsegmented sequence data with recurrent neural net-
works,” in Proceedings of the 23rd international conference [24] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki
on Machine learning. ACM, 2006, pp. 369–376. Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta So-
plin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al.,
[11] George E Dahl, Dong Yu, Li Deng, and Alex Acero, “Espnet: End-to-end speech processing toolkit,” arXiv preprint
“Context-dependent pre-trained deep neural networks for arXiv:1804.00015, 2018.
large-vocabulary speech recognition,” IEEE Transactions on
audio, speech, and language processing, vol. 20, no. 1, pp. 30– [25] Jaejin Cho, Murali Karthick Baskar, Ruizhi Li, Matthew Wies-
42, 2012. ner, Sri Harish Mallidi, Nelson Yalta, Martin Karafiat, Shinji
Watanabe, and Takaaki Hori, “Multilingual sequence-to-
[12] Lalit R Bahl, Peter F Brown, Peter V de Souza, and Robert L sequence speech recognition: architecture, transfer learning,
Mercer, “A tree-based statistical language model for natural and language modeling,” arXiv preprint arXiv:1810.03459,
language speech recognition,” IEEE Transactions on Acous- 2018.
tics, Speech, and Signal Processing, vol. 37, no. 7, pp. 1001–
1008, 1989. [26] Matthew D Zeiler, “Adadelta: an adaptive learning rate
method,” arXiv preprint arXiv:1212.5701, 2012.
[13] Takaaki Hori, Shinji Watanabe, Yu Zhang, and William Chan,
“Advances in joint CTC-attention based end-to-end speech
recognition with a deep CNN encoder and RNN-LM,” in IN-
TERSPEECH, 2017.

6195