COVID_MTNet: COVID-19 Detection with Multi-Task Deep Learning

Approaches
Md Zahangir Alom, M M Shaifur Rahman, Mst Shamima Nasrin, Tarek M. Taha, and Vijayan K. Asari
Department of Electrical and Computer Engineering, University of Dayton, Dayton, OH, USA
Emails: {alomm1, rahmanm24, nasrinm1, ttaha1, asari1}@udayton.edu

Abstract

COVID-19 is currently one the most life-threatening

problems around the world. The fast and accurate
detection of the COVID-19 infection is essential to identify,
take better decisions and ensure treatment for the patients
which will help save their lives. In this paper, we propose a
fast and efficient way to identify COVID-19 patients with Normal COVID-19 Input Output
multi-task deep learning (DL) methods. Both X-ray and CT Figure 1: Example X-ray and CT images for normal
scan images are considered to evaluate the proposed versus COVID-19 detection and outputs for infected
technique. We employ our Inception Residual Recurrent region localization with NABLA-3 network.
Convolutional Neural Network with Transfer Learning
(TL) approach for COVID-19 detection and our NABLA-N patients through home quarantining is the best way to
network model for segmenting the regions infected by protect people not already infected with this deadly virus
COVID-19. The detection model shows around 84.67% [2]. Thus, patients with COVID-19 symptoms must remain
testing accuracy from X-ray images and 98.78% accuracy isolated and wear masks when near others. For primary
in CT-images. A novel quantitative analysis strategy is also examination, a sample of nasopharyngeal exudate is taken
proposed in this paper to determine the percentage of to test reverse transcription polymerase chain reaction
infected regions in X-ray and CT images. The qualitative (RT-PCR) followed by a chest X-ray. If the chest X-ray
and quantitative results demonstrate promising results for film is normal, then the patient can go home and take rest. If
COVID-19 detection and infected region localization. COVID-19 is confirmed, the patient must be admitted to a
hospital. At the initial stages, COVID-19 affects both
lungs, particularly the lower lobes, and especially the
1. Introduction posterior segments, with a fundamentally peripheral and
subpleural distribution. The chest CT is used for detecting
COVID-19 is a highly contagious and unsafe COVID-19 in practically 50% of patients in the first two
coronavirus that has not been previously identified. The days. The researcher from Tongji Hospital in Wuhan,
coronavirus disease 2019 (COVID-19) family is causing China, has concluded that the CT should be used as the
thousands of deaths per day around the world. Even though primary screening or diagnostic method to identify
this virus first started from Wuhan, China in Dec. 2019, it is COVID-19.
spreading around the world so quickly that presently more
than one million people from around 188 countries have Even though the RT-PCR has been considered as the gold
been infected with this dangerous virus as of this writing. standard for SARS-CoV-2 diagnosis, due to the limited
Due to coronavirus, more than 70 thousand people have supply and strict requirements for the laboratory
already died worldwide, and the number of new deaths has environment, it delays the accurate diagnosis of suspected
been increasing rapidly day by day. There is no treatment patients. Hence, this is very difficult to pose the unexpected
specifically approved for this virus. Thus, the World Health spreading of infection of COVID-19 diseases. By
Organization (WHO) has declared COVID-19 as a combining the clinical symptoms and signs, the chest
pandemic disease. The mutation of this virus moving faster computed tomography (CT) is a faster and easier approach
than align and more deadly too. This is one of the fastest to identify COVID-19 in clinical practices. According to
progressive disease ever seen before [2]. Since it is a new doctors and researchers in China, CT image analysis is the
type of virus and it changes formation quickly, there is no key method to examine suspected patients for COVID-19
specific guideline for the assessment or examination confirmation. Recent reports also support this phenomenon
process, and proper treatment. According to findings from [2,3]. It is crucial to detect COVID-19 infections at an early
Wuhan, Hubei province in China, isolation of infected stage for suspected patients. It is also important for patient
prognosis, control of this epidemic, and public health
1
security. As the rate of newly affected global regions is (AI) and image analysis-based methods for COVID-19
increasing every day, and the number of new patients added detection. In most of the cases, the Deep Learning based
is increasing geometrically in those areas, it has become a approaches are applied and have achieved very promising
burden or sometimes impossible to provide the necessary detection accuracy for COVID-19.
treatment for them [3,4,5]. Hence, an Artificial Intelligence A study has been conducted to illustrate the importance of
(AI) system can be helpful to identify the COVID-19 early detection and management of COVID-19 patients in
patients quickly and accurately using X-ray and CT images [8]. A literature review was published recently and claimed
as shown in Figure 1. that the ground-glass and consolidative opacities on CT are
There are several motivations behind this research. First, sometimes undetectable on chest radiography. The study
accurate and faster detection of COVID-19 patients is very has also suggested that CT is a more sensitive modality of
important for the prevention and control of this pandemic. medical imaging. The COVID-19 infection pulmonary
Most importantly, it can protect human lives around the manifestation is predominantly characterized by
world. Second, deep learning-based radiology image ground-glass opacification with occasional consolidation
analysis methods have been providing state-of-the-art on CT [9]. The pros and cons of using X-ray and CT image
testing accuracy by far compared to existing methods [6]. analysis and its effectiveness for the screening of
Recently, several Deep Learning (DL) based methods have COVID-19 patients were explained in recent studies. In
been proposed and shown to be highly accurate for most of the cases, chest X-ray is considered as the primary
COVID-19 detection from CT images [7,12,17]. Third, screening method. In cases of patients without predominant
around 70,000 people around the world have already died disease, COVID-19 can be defined by chest X-ray.
due to the COVID-19. Thus, there is a strong demand for Otherwise the CT was recommended for determining
faster and accurate detection of patients infected with COVID-19 infection. Another study demonstrates that
COVID-19. Chest CT has a pivotal role for the diagnosis and
In this study, we apply advanced deep learning methods to assessment of lung involvement in COVID-19 pneumonia
investigate both X-ray and CT manifestations of [10]. The CT plays a central role in the diagnosis of
coronavirus (COVID-19) pneumonia using multiple COVID-19 pneumonia and the decision has been taken by
models. These include classification and segmentation for conducting an evaluation with 366 CT scans which were
analysis of samples from two different modalities for reviewed by two groups of radiologists. The objective of
detection and localization. For chest X-ray image analysis, the study was to define disease progression and recovery of
due to the scarcity of publicly available COVID-19 X-ray the illness and found that the peak period during illness
samples, we have trained our model with a pneumonia were days 6-11 [11]. The performance of the Radiologists
dataset, and then utilized a Transfer Learning (TL) method has been analysis and observed the radiologist has high
for training with samples for COVID-19. The contribution specificity but moderate sensitivity in differentiating
of this paper can be summarized as follows: COVID-19 from viral pneumonia on CT scan [12].For
• Proposed an end-to-end COVID-19 detection and COVID-19 detection, a CT scan image analysis method
infected region detection method using deep found that in 20 (56%) patients out of 36 patients about 2
learning approaches days after symptom onset had normal [13]. Furthermore,
for critical cases, images from both modalities are
• The proposed methods are evaluated for both X-ray recommended for decision making.
and CT images and achieved promising detection
and infected region localization tasks.
Meanwhile, there are several DL based systems that have
• Training and validation are performed on publicly been proposed for X-ray and CT images in the last few
available samples which are collected from months for COVID-19 detection. A DL based method has
different sources around the world. been proposed for COVID-19 detection from chest X-ray
The paper is summarized as: section 2 discusses related which was named COVID-19 Detection Neural Network
works. Section 3 represents the method and the model used (COVNet). The ReseNet50 was used as the backbone for
in the implementation. The experimental setup is discussed COVNet. This model is tested on 4356 chest CTs from
in Section 5. Sections 6, 7, 8 explain about detail discussion 3,322 patients and showed 0.96 and 0.95 score for Area
on results, limitations, and conclusion respectively. Under Curve and community acquired pneumonia (CAP)
score respectively [14]. The study also demonstrates CT
2. Related works findings were compared for asymptomatic and
There are several AI systems that have been proposed for symptomatic patients for COVID-19 detection and the
COVID-19 detection from X-ray and CT images. Some results showed that there were no significant differences in
studies demonstrate the efficiency of X-ray and CT scans age, sex distribution, or comorbidities for symptomatic and
image analysis for detecting the COVID-19. On the other asymptomatic cases [14]. One of the most relevant recent
hand, other papers have demonstrated artificial Intelligence models, COVID-Net [15]], uses a deep neural network with

2
 Figure 2: COVID-Det_Xray: End-to-end system for pneumonia detection in the top row and COVID-19 detection with TL learning
from pneumonia detection at the bottom. Bottom row shows the training phase for COVID-19 which include inputs, data
augmentation, training model with TL and outputs (N: normal and C: COVID-19).

high architectural diversity and selective long-range deep learning-based method shows around 86.7% testing
connectivity. The model experimented on two open access accuracy from CT images [17].
data repositories, one for COVID-19 detection dataset and
another for pneumonia detection dataset. The experimental According to the literature studies, the X-ray and CT scans
results show around 83.5% testing accuracy. In this case, have been used very often to recognize the patients with
both databases were merged for performing training and COVID-19. Hence, we present here an end-to-end deep
testing of the proposed method [15]. However, in our learning-based system for COVID-19 detection and
proposed approach, we have trained a model with a infected regions detection from both X-ray and CT images.
pneumonia dataset and used Transfer Learning (TL) for
training the same model with a COVID-19 dataset. 3. Methodology
In this implementation, we have used multiple models
A deep learning-based detection for COVID-19 from Chest for different tasks where the classification model is for
3D CT volumes with weak labeled samples has been COVID-19 detection and the segmentation model is used
proposed recently. For each patient, a pretrained UNet for Region of Interest (ROI) detection for COVID-19. The
model was applied for 3D lung region segmentation and Inception Recurrent Residual Neural Network (IRRCNN)
then a 3D-CNN model was applied for predicting the model is used for the COVID-19 detection task [18]. The
probability of COVID-19 infections. The model was NABLA-N network is applied for infected region
trained on 499 CT volumes and tested on 131 CT volumes segmentation from X-ray and CT images.
and obtained 0.959 ROC AUC with 0.907 and 0.911
sensitivity and specificity respectively. Overall, this model COVID-19 Detection from X-ray-Image
shows around 90% accuracy [16]. In addition, the recent (COVID-Det_Xray): first, we have used the IRRCNN
study shows very promising accuracy for pneumonia model to classify the normal and pneumonia detection from
detection tasks where the model developed grouped three chest X-ray images. Then, the Transfer Learning (TL)
different categories: COVID-19, Influenza-A viral method is used to train with samples to determine
pneumonia, and healthy cases from CT images. The VNet COVID-19 and normal images from Chest X-ray images.
based VNET-IR-RPN is used for region of interest The end-to-end training method is shown in Figure 2. The
segmentation for pulmonary tuberculosis. The proposed IRRCNN model is used for this implementation where five
3
 Figure 3: Chest segmentation model for X-ray (Chest-Seg_Xray): the top row of the figure shows the training method and bottom
row demonstrates the different steps for testing phase.

Inception Recurrent Residual Unit (IRRUs) are used [18]. China. For COVID-Det_CT, we have used the IRRCNN
The IRRUs are shown for unit 3 in the top of Figure classification model, which is trained on our own datasets.
2. After successfully training the model for COVID-19 The database samples are collected from normal CT and
detection, the system has been tested with completely new CT scan with COVID-19. From this implementation, the
samples collected from new patients. same methods are followed as described for
COVID-19 segmentation from X-ray-Image COVID-Det_Xray.
(COVID-Seg_Xray): another system is developed for
detecting infected regions for COVID-19 in X-ray images. COVID-19 segmentation from CT scans
In this case, the NABLA-N model is applied for only chest (COVID-Seg_CT): for mining the specific infected region
segmentation [19]. The end to end training and testing of COVID-19 virus in lung region, we have developed an
method for chest segmentation method (Chest-Seg_Xray) end to end system for segmentation of lung region from 2D
is shown in Figure 3. Mathematical morphological
approaches are applied for selecting the refinement and
selecting appropriate contours for chest region extraction,
which is shown in the last column of the second row in
Figure 3. In the testing phase, precise chest regions are
extracted. After generating segmentation masks for the
chest regions, the mask is used to extract only the chest (a) (b) (c)
regions from the input images, which is shown in Figure Figure 4: The pipeline processing for COVID-19 infected
4(a). A classical image processing method and adaptive region detection: (a) chest regions, (d) mask for COVID-19
thresholding approach are applied for extracting the infected regions and (c) final heatmap image.
features to identify the infected regions of COVID-19 from
the segmented chest region in Figure 4(b). Finally, the images where the same NABLA-N model is used as
output heatmap image with COVID-19 infected regions are mentioned in COVID-Seg_Xray. First, we have trained the
shown in Figure (c). model on publicly available 2D lung images, which is
COVID-19 detection from CT scans (COVID-Det_CT): called Lung_Seg_CT. Then the trained Lung_Seg_CT
CT scans are also used for detecting coronavirus which is model is directly applied to segment the lung region from
very effective to detect the infection of COVID-19 within a the entire set of CT images. Classical image processing and
couple of days after infection. This is stated according to morphological analysis are then applied to extract infected
the assessment system followed by the expert in Wuhan, regions with COVID-19. The entire system is named

4
“COVID-Seg_CT”. For the testing phase, the same number of samples, 80% of samples are used for training
processing pipeline CT is used, which is demonstrated for and the remaining 20% are used for validation and testing
X-ray, as demonstrated in Figure 3 and 4. of the COVID-Seg_Xray model. This is the first step of
infected regions extraction from input chest X-ray
Architecture details: The IRRCNN model consists of an images. The randomly selected X-ray images and
input layer, five IRRUs, a Global Average Pooling (GAP) corresponding masks are shown in Figure 6.
layer, and a Softmax output layer. For this model, we have
applied 1×1, and 3×3 kernels for IRRU. The entire model
utilizes around 34M network parameters. On the other
hand, the NABLA-N network consists of an encoding and
three decoding units. The model architecture is as followed:
3→16×(3×3) →32×(3×3) →64×(3×3) →128×(3×3)
→256×(3×3) →512×(3×3) →256×(3×3) →128×(3×3)
→64×(3×3) →32×(3×3) →16×(3×3) →1. The general
notation of FN× (M×N) where FN represents the number of Figure 5: Randomly selected samples without pneumonia in first
feature maps and (M×N) represents the kernel size which is row and second row shows the X-ray image with pneumonia.
(3×3) kernels used in each layer except the last layer. The
Rectified Linear Unit (ReLU) activation function is used all In addition, for COVID-19 detection from CT images, there
through the network in this implementation. At the end, a is no labeled dataset available for this specific task. Hence,
1×1 convolutional layer is used to reduce the dimension of we have collected samples from different sources for
the feature maps to the single channel outputs with a normal CT scans. The COVID-19 CT images are
sigmoid activation function. In addition, we have utilized considered from different CT scans from confirmed
the up-sampled feature maps from 3 different encoding patients. A total of 420 samples are collected where 247
layers including the bottleneck layer. Therefore, we named samples are for normal and 178 samples are for COVID-19.
the model NABLA-3 network. The NABLA-3 model We have observed the variant sizes of the samples, the
utilizes totally 18.98 Million (M) network parameters. The lowest dimension of the images 450×338 pixels and highest
network is initialized with the He initialization method 630×630 pixels.
[21].

4. Experiments

4.1. Experimental setup

The We have collected several datasets for implementing
this multi-modality learning method. The pneumonia
detection samples are collected from publicly available
datasets in [22]. The total number of samples are 5,216, Figure 6: Example image for Chest X-ray segmentation.
where only 1,341 samples are for normal and 3,875
samples for pneumonia. The average size of the images is Thus, all samples are resized to 192×192 pixels. From the
around 1168×984 pixels. Due to the limitation of our total samples, we randomly selected 375 samples for
computing system, we have resized the images to 128×128 training and validation, and the remaining 45 samples are
pixels. To resolve the class imbalance problem, we have used for testing. To increase the number of samples, a data
applied class specific data augmentation. The example augmentation method is applied during the training.
images are shown in Figure 5. The COVID-19 dataset has Randomly selected samples are shown in Figure 7.
been collected from different sources around the world and
a publicly available dataset [23]. Due to the scarcity of
training samples, we have applied data augmentation
techniques for increasing the number of samples.

For segmentation of the chest regions, we collected 704

chest X-ray images and corresponding masks, which are
shown in Figure 5[24]. The original average size of the
sample is 2437×2806×3. We have resized the image to Figure 7: Randomly selected for normal and COVID-19 CT
192×192×3 pixels images. As a result, input samples images.
significantly lose essential information. From the total For lung segmentation from CT images, we have used a
publicly available 2D CT scans from Kaggle competition
5
[25]. This dataset contains 267 samples with corresponding
masks in total with corresponding labels images. Some
example samples and mask are shown in Figure 8. The
actual size of the images is 512×512. We resized them to
256×256-pixel single channel images. The total number of
samples was 267, where 80 percent of the images were used
for training and the remaining 20 percent used for
validation and testing.

Figure 10: Receiver operating characteristic (ROC)-AUC curve

for COVID-19 detection.

The classification model (COVID-Seg_CT) is trained with

same parameters as COVID-Seg_Xray, however, due to
the less number of training samples, the model is trained
150 epoch with batch size of 16. For COVID-Seg_CT is
Figure 8: Example image for lung segmentation from 2D images. trained with Adam optimizer with learning rate 3 ,
First row shows the input image and second row show DC loss, As the number of images is less, therefore, we
corresponding masks.
have used the batch size of 8. In this implementation, we
have used max min normalized single channel images for
4.2. Training methods training and testing.
The IRRCNN model is trained and tested on pneumonia
dataset. For COVID-19 detection method for X-ray 5. Results
(COVID-Det_Xray), the model is trained with the COVID-Det_Xray outputs: After successfully training
following hyperparameters: Adam optimization method the model, we have tested the pneumonia detection system
with learning rate , and Batch size 32. The system with completely new 624 which includes 234 normal and
is trained for 75 epochs in total where the learning rate is 390 pneumonia samples. The quantitative results show
decrease with respect to the factor of 10 after each 25 around 87.26% testing accuracy for pneumonia detection.
epochs. The training and validation accuracy for Then, the same IRRCNN model is used for training and
pneumonia detection method is shown in 9. From the testing for the COVID-19 detection where the pretrained
figure, it can be seen that the proposed pneumonia detection weights from pneumonia are used as initial weights for
method shown around 98.2% validation accuracy. For training the COVID-19 model. We have achieved around
COVID-Seg_Xray is trained with Adam optimizer with 84.67% testing accuracy on the completely new 67
learning rate 3 , Dice Coefficient (DC) loss. COVID-19 testing samples. The Receiver operating
characteristic (ROC)-AUC for pneumonia is shown in
Figure 10. This shows an AUC of 0.93.

COVID-Seg_Xray outputs: For further evaluation and to

define the specific infected regions with COVID-19, we
have implemented an end to end segmentation system for
extracting the chest regions from the entire X-ray image.
The NABLA-N network is used for chest segmentation
from X-ray images. This model has been trained on the
chest-x-ray segmentation challenge dataset. After
successfully training the model, the performance is
evaluated on the testing samples provided by the organizer.
We have tested on completely new 57 chest X-ray images.
The model shows 0.9452 and 0.9466 in terms of global
Figure 9: Training and validation accuracy for pneumonia accuracy and F1-score respectively. In addition, it shows
detection task for 75 epochs. around 0.8650 and 0.8846 for Intersection over Union
(IoU) and Dice Similarity Coefficient score (DC).

6
 (a) (b) (c) (d) (e)
Figure 11: Chest segmentation (Chest-Seg_Xray) results: (a) input chest images (b) Ground Truth (c) Model outputs (d) Refined
outputs and (e) Final outputs with contours where the green and red contour represent the ground truth and detection respectively.

The experimental results for the segmentation model for column, the percentage of infection is around 51.23%,
COVID-19 X-ray samples are shown in Figure 12. The first 27.66%, and 47.89% respectively.
column shows the inputs images, the second column shows
the outputs from COVID-Seg_Xray, the third column COVID-Seg_Xray outputs for COVID-19 from
shows the polished segmentation masks, the fourth column Abdominal CT: We have also observed that for some
represents outputs for only chest regions, and the fifth cases, patients have been confirmed with COVID-19 after
column shows the final outputs with COVID-19 infected analysis of the abdominal CT images. Thus, we have
regions. The qualitative results clearly demonstrate that the evaluated the same model for X-ray as has been tested on
proposed model is able to segment and detect contaminated abdominal CT images. The outputs for abdominal CT
regions of COVID-19 accurately from the chest X-ray images are shown in Figure 13. The first column shows the
images. input images, the second column represents the refined
segmented mask, where only large regions have been
In addition, for quantitative justification, we have selected among a set of segmented regions, and the fourth
calculated the total number of pixels of the lung regions and column shows the extracted lung region with respect to the
the total number of pixels for infected regions with refined mask in Figure 13(b). The classical image
COVID-19. The percentage is calculated with respect to the processing method and adaptive thresholding method are
total areas of the lung, which can be used for measuring the applied to extract the pixels to represent COVID-19, and
contingency and severeness of corona-virus patients. For the sixth column demonstrates the heatmaps for the
the first row and third column in Figure 12, the number of infected regions for COVID-19. For the first row, the total
total pixels for lung is 6696 and total number of infected number of pixels for the lung regions is 5184 and the total
pixels with COVID-19 is 2245. Thus, the percentage of number of infected pixels with COVID-19 is 1599. Hence,
infection is 33.52% with respect to the total pixels of lung the percentage of infection is 30.84%. For the second and
regions. In the last row of Figure 12, the total number of third row, the percentage of infection is 80.39% and
pixels for lung regions is 9601 and the total number of 23.18% respectively. The quantitative and qualitative
infected pixels with COVID-19 is 3609.Thus the results clearly show that the proposed classification and
percentage of infection is 37.58%. This quantitative segmentation for X-ray images demonstrate promising
analysis can be applied to define the severeness of the detection and infected region extraction.
COVID-19 disease. For the second, third and fourth

7
 Figure 12: Experimental results of COVID-Seg_Xray model: first column represents the input images, second column shows the
predicted segmentation masks, third column shows the refined outputs with chest regions, fourth column represent only chest
regions, and fifth column represent the heatmap in the infected regions.

COVID-Det_CT outputs: From the literature survey, we which is named COVID-Det_CT. The system is trained and
have observed that CT scans are used to identify the tested on our own dataset. The COVID-Det_CT shows
COVID-19 detection directly. In some critical cases, around 98.78% testing accuracy for 45 samples.
Doctors fail to take a decision directly from the X-ray
analysis. Instead, the CT scans are used to take final COVID-Seg_CT outputs: For extracting infected regions
decisions on the patients. Different studies have claimed with COVID-19, the NABLA-N network-based
that CT is most efficient to confirm the COVID-19 patients. segmentation model is trained and tested on a publicly
Thus, we have included a detection model for CT images available 2D lung segmentation dataset. This model is

8
 (a) (b) (c) (d) (e)
Figure 13: Experimental results of COVID-Seg_Xray model for abdominal CT: (a)inputs, (b) segmented masks, (c) lung regions,
(d) extracted pixels for COVID-19, and (d)final heat with heatmap.

named COVID-Seg_CT. After training the

COVID-Seg_CT model, the testing is done on completely
new samples. The quantitative results demonstrate 0.9885
and 0.9956 for F1-score and global accuracy respectively.
The qualitative results are shown in Figure 14. The first
column shows the input images, the second column shows
the ground truth, and the third column shows the outputs
from the COVID-Seg_CT model. The results clearly
demonstrate that the proposed model produces very
accurate segmentation results compared to ground truth

Since, we did not have any label samples for COVID-19 for
segmentation tasks, the COVID-Seg_CT model is directly (a) (b) (c)
applied to segment the lung regions from the samples with Figure 14: COVID-Seg_CT outputs for testing samples:
COVID-19. After segmenting the lung region from CT (a) inputs (b) Ground Truth (GT) and (c) model outputs.
images, mathematical morphological operations are
performed to refine the segmentation masks. Lung regions cases, we have observed false detection, as shown in the
are then extracted with respect to the mask. The second row in Figure 15(d).
segmentation results of COVID-Seg_CT model for
COVID-19 samples are shown in Figure 15. The first 6. Discussions
column shows the input images, the second column shows
The proposed pneumonia detection method shows
the lung segmentation results, the third column represents
around 87.26% testing accuracy whereas the recently
lung regions, and the fourth column represents the result
published paper shows 84.67% testing accuracy in [15].
with heatmap. In most of the cases, the proposed model
Thus, our IRRCNN based detection model shows around
provides good accuracy of detection. However, in some
3.76% better testing accuracy for pneumonia detection
9
 (a) (b) (c) (d)
Figure 15: The COVID-19 infected region detection results from lung CT images: (a) inputs images, (b) segmented and refined masks
with TL learning approach, and (d) infected region with heatmaps.

tasks. In addition, most of the COVID-19 infected region detection shown in the second row in Figure 15 (d). This
detection methods proposed are based on patch-based needs to be improved.
detection methods for infected region extraction, where
there is a big possibility to have false positive and false 8. Conclusion
negative detection as the decision is taken based on the In this study, we propose an end-to-end system for
class [17]. On the other hand, we proposed an infected COVID-19 detection and infected region localization from
region with different level of evaluation for pixel level two different modalities of medical imaging. For
analysis. Thus, the qualitative results clearly demonstrated classification, and segmentation tasks, our improved
the infected region from the lung part and the proposed Inception Recurrent Residual Neural Network (IRRCNN)
COVID-19 detection significantly reduces the possibility and NABLA-3 network models are applied. The models are
of false positive and false negative detection for tested on X-ray, abdominal CT, and full body CT images
COVID-19. on publicly available datasets. The observed results show
very promising detection results with 84.67% and 98.78%
7. Limitations testing accuracy for COVID-19 from X-ray and CT images
Both detection and segmentation methods for X-ray respectively. In addition, the qualitative results clearly
images provide very promising accuracy. However, there demonstrate high accuracy in the segmentation and
are some limitations of this study which need to be detection of infected regions by COVID-19 in both X-ray
addressed in the near future. First, the COVID-Det_Xray and CT images. In the near future, we would like to collect
model needs to be trained and tested with more COVID-19 more samples of COVID-19 affected subjects to develop a
samples. Second, as the COVID-Det_CT model is trained robust and more accurate system.
and tested on only 300 samples in the initial
implementation, the model provides very good detection References
accuracy. From our point of view, the COVID-Det_CT [1] The Global Initiative on Sharing All Influenza Data
model needs to be trained and tested with more samples to (GISAID). Coronavirus COVID-19 Global Cases by Johns
generalize and make the model more robust and accurate. Hopkins CSSE.
Third, due to the scarcity of the label samples for lung https://www.gisaid.org/epiflu-applications/global-casescovi
segmentation from CT for COVID-19, the d-19
COVID-Seg_CT provides outputs with false positive [2] Wei-jie Guan, Zheng-yi Ni, Yu Hu, Wen-hua Liang,
Chun-quan Ou, Jian-xing He, Lei Liu, Hong Shan,

10
 Chun-liang Lei, David SC Hui, et al. Clinical characteristics Distinguishes COVID-19 from Community Acquired
of 2019 novel coronavirus infection in china. MedRxiv, Pneumonia on Chest CT.” Radiology, 2020, 200905.
2020.
https://doi.org/10.1148/radiol.2020200905.
[3] Junqiang Lei, Junfeng Li, Xun Li, and Xiaolong Qi. Ct
imaging of the 2019 novel coronavirus (2019-ncov) [15] Wang, Linda, and Alexander Wong. "COVID-Net: A
pneumonia. Radiology, page 200236, 2020. Tailored Deep Convolutional Neural Network Design
[4] Fengxiang Song, Nannan Shi, Fei Shan, Zhiyong Zhang, Jie for Detection of COVID-19 Cases from Chest
Shen, Hongzhou Lu, Yun Ling, Yebin Jiang, and Yuxin Shi. Radiography Images." arXiv preprint
Emerging coronavirus 2019-ncov pneumonia. Radiology,
arXiv:2003.09871 (2020).
page 200274, 2020.
[5] Michael Chung, Adam Bernheim, Xueyan Mei, Ning Zhang, [16] Zheng, Chuansheng, Xianbo Deng, Qing Fu, Qiang
Mingqian Huang, Xianjian Zeng, Zhou, Jiapei Feng, Hui Ma, Wenyu Liu, and Xinggang
JiufaCui,WenjianXu,YangYang,ZahiFayad,etal. CT Wang. "Deep Learning-based Detection for
imaging features of 2019 novel corona virus (2019-ncov). COVID-19 from Chest CT using Weak
Radiology, page 200230, 2020.
Label." medRxiv (2020).
[6] Litjens, Geert, Thijs Kooi, Babak Ehteshami Bejnordi,
Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen [17] Xu, Xiaowei, Xiangao Jiang, Chunlian Ma, Peng Du,
Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Xukun Li, Shuangzhi Lv, Liang Yu et al. "Deep
Ginneken, and Clara I. Sánchez. "A survey on deep learning Learning System to Screen Coronavirus Disease 2019
in medical image analysis." Medical image analysis 42 Pneumonia." arXiv preprint arXiv:2002.09334 (2020).
(2017): 60-88. [18] Alom, Md Zahangir, Mahmudul Hasan, Chris
[7] Gozes, Ophir, Maayan Frid-Adar, Hayit Greenspan, Patrick Yakopcic, Tarek M. Taha, and Vijayan K. Asari.
D. Browning, Huangqi Zhang, Wenbin Ji, Adam Bernheim, "Improved inception-residual convolutional neural
and Eliot Siegel. "Rapid ai development cycle for the
network for object recognition." Neural Computing
coronavirus (covid-19) pandemic: Initial results for
and Applications (2018): 1-15.
automated detection & patient monitoring using deep
learning ct image analysis." arXiv preprint [19] Md Zahangir Alom, Theus Aspiras, Tarek M. Taha,
arXiv:2003.05037 (2020). and Vijayan K. Asari. Skin Cancer Segmentation and
[8] Yang, Wenjie, and Fuhua Yan. "Patients with RT-PCR Classification with NABLA-N and Inception
confirmed COVID-19 and normal chest CT." Radiology Recurrent Residual Convolutional Networks.
(2020): 200702. Published in SPIE Medical Imaging Conference, 15-20
[9] Ng, Ming-Yen, Elaine YP Lee, Jin Yang, Fangfang Yang, February 2020, Houston, Texas, USA.
Xia Li, Hongxia Wang, Macy Mei-sze Lui et al. "Imaging [20] Alom, Md Zahangir, Chris Yakopcic, Mahmudul
profile of the COVID-19 infection: radiologic findings and
Hasan, Tarek M. Taha, and Vijayan K. Asari.
literature review." Radiology: Cardiothoracic Imaging 2, no.
"Recurrent residual U-Net for medical image
1 (2020): e200034.
[10] Poggiali, Erika, Alessandro Dacrema, Davide Bastoni, segmentation." Journal of Medical Imaging 6, no. 1
Valentina Tinelli, Elena Demichele, Pau Mateo Ramos, (2019): 014006.
Teodoro Marcianò, Matteo Silva, Andrea Vercelli, and [21] Alom, Md Zahangir, Tarek M. Taha, Chris Yakopcic,
Andrea Magnacavallo. "Can Lung US Help Critical Care Stefan Westberg, Paheding Sidike, Mst Shamima
Clinicians in the Early Diagnosis of Novel Coronavirus Nasrin, Mahmudul Hasan, Brian C. Van Essen, Abdul
(COVID-19) Pneumonia?." Radiology (2020): 200847. AS Awwal, and Vijayan K. Asari. "A state-of-the-art
[11] Wang, Yuhui, Chengjun Dong, Yue Hu, Chungao Li, survey on deep learning theory and
Qianqian Ren, Xin Zhang, Heshui Shi, and Min Zhou.
architectures." Electronics 8, no. 3 (2019): 292.
"Temporal Changes of CT Findings in 90 Patients with
[22] Mooney. Kaggle chest x-ray images (pneumonia)
COVID-19 Pneumonia: A Longitudinal Study." Radiology
(2020): 200843. dataset.
[12] Bai, Harrison X., Ben Hsieh, Zeng Xiong, Kasey Halsey, Ji https://github.com/ieee8023/covid-chestxray-dataset,
Whae Choi, Thi My Linh Tran, Ian Pan et al. "Performance 2020. 2, 3
of radiologists in differentiating COVID-19 from viral [23] Cohen. Covid chest x-ray dataset.
pneumonia on chest CT." Radiology (2020): 200823. https://github.com/ieee8023/covid-chestxray-dataset,
[13] Bernheim, Adam, Xueyan Mei, Mingqian Huang, Yang 2020. 2, 3
Yang, Zahi A. Fayad, Ning Zhang, Kaiyue Diao et al. "Chest [24] https://www.kaggle.com/arturscussel/lung-segmentati
CT findings in coronavirus disease-19 (COVID-19):
on-and-candidate-points-generation
relationship to duration of infection." Radiology (2020):
200463.
[14] Li, Lin, Lixin Qin, Zeguo Xu, Youbing Yin, Xin Wang, Bin
Kong, Junjie Bai, et al. “Artificial Intelligence

11