Object Tracking using CSRT Tracker and RCNN

Khurshedjon Farkhodov1, Suk-Hwan Lee2 and Ki-Ryong Kwon1

1Dept. of IT Convergence and Applications Engineering, Pukyong National University, South Korea
2Dept. of Information Security, Tongmyong University, South Korea

[email protected], [email protected], [email protected]

Keywords: Object Tracking, Object Detection, CSRT, Faster RCNN, CSR-DCF, CNN, Opencv, Deep Learning, DNN
Module.

Abstract: Nowadays, Object tracking is one of the trendy and under investigation topic of Computer Vision that
challenges with several issues that should be considered while creating tracking systems, such as, visual
appearance, occlusions, camera motion, and so on. In several tracking algorithms Convolutional Neural
Network (CNN) has been applied to take advantage of its powerfulness in feature extraction that convolutional
layers can characterize the object from different perspectives and treat tracking process from misclassification.
To overcome these problems, we integrated the Region based CNN (Faster RCNN) pre-trained object
detection model that the OpenCV based CSRT (Channel and Spatial Reliability Tracking) tracker has a high
chance to identifying objects features, classes and locations as well. Basically, CSRT tracker is C++
implementation of the CSR-DCF (Channel and Spatial Reliability of Discriminative Correlation Filter)
tracking algorithm in OpenCV library. Experimental results demonstrated that CSRT tracker presents better
tracking outcomes with integration of object detection model, rather than using tracking algorithm or filter
itself.

1 INTRODUCTION sharing their convolutional features using the recently

popular terminology of neural networks with
The main goal in this proposed tracking method is to “attention” mechanisms, the RPN component tells the
focus on exact object to track, furthermore because of unified network where to look (R. Girshick, 2016). In
the real-time object tracking environment there are this paper, we propose an algorithm that to apply
some parameters should be considered, such as OpenCV-based CSRT tracker into person detection
camera movement, distance between object and and tracking, which we combined with Faster R-CNN
camera, and so on. As an object detector Faster R- based object detector with the support of OpenCV’s
CNN have been used after comparing all object DNN module and obtained trained object detector
detection methods, for instance Viola-Jones model. With the great afford of deep learning-based
algorithm: the first efficient face detector (P. Viola object detection technique that we can easily avoid
and M. Jones, 2001), much more efficient detection target misclassification and lost. Results of proposed
technique: Histograms of Oriented Gradients (N. tracking method has gained remarkable outcome, and
Dalal and B. Triggs, 2005), and from 2012 the deep tested in a different dataset with single and overlap-
learning methods, which is called “Convolutional ping images, also the tracking system experimented
Neural Networks” became the gold standard for with video and real-time sequence and got much
image classification after Kriszhevsky's CNN's better performance that rather than using tracking
performance during ImageNet (A. Krizhevsky, et al., filter without any additional supporter like detector.
2012), While these results are impressive, image This article consists of following sections: our
classification is far simpler than the complexity and suggested tracking system structure explained in
diversity of true human visual understanding. A better system overview section comes with related work. In
approach, R-CNN has been proposed (R. Girshick, et section 3, the detailed results will be presented which
al., 2013) after CNN realized. R-CNN creates obtained during the experiments and we also
bounding boxes, or region proposals, using a process compared visual value with the conventional methods
called Selective Search. Later, where further merge of performance, conclusion and reference follows as
R-CNN and Fast R-CNN into a single network by well.
2 RELATED WORKS do not use your computer for other high memory or
performance required processes while training, that
can affect to obtain training outcomes as well as it
2.1 Overview of the Object Detection could lead your training process to be time consuming.
Task Training classifier should train until the loss is
consistently below 0.05 or so that the law starts to
Object detection has applications in many areas of plateau out. A total loss graph estimates that while
computer vision, including image retrieval and video learning training dataset images it can loss or
surveillance. Variations in position, color, lighting misidentify objects by their features, shapes and other
conditions, size, etc., greatly affect the performance parameters in one average graph performance. The
of the model. Object detection and tracking processes total loss of training process performance together
should be fine-tuned, that depending on what kind of with objectiveness loss, which can show us the
problem is going to be solved. This is usually objectiveness score (4e-3≈0.004) of the dataset’s
determined by the characteristics of the target. For images, to indicate if this box contains an object or
instance, detecting vehicles require different not while training process is given in Figure 1 below:
parameters tuning than detecting pedestrians, animals,
or faces, and so on. This feature-based technique
exploits a notable differentiation character of the
objects that taken from 2D pixel information in an
image (Gonzalez, R. C., and Woods, R. E., 2017).
While using feature points of 2D images, such as
color, intensity, background information it is easy
way to identify object from frames if it will not
change the appearance, position, and size as well. Figure 1: Performance of the total losses along with
objectiveness loss.
2.2 Faster RCNN Object Detection
Architecture 2.4 Proposed Object Tracking Method
Faster RCNN is becoming one of the most used and The basis of our tracking method taken from the
popular an object detection architecture presented by Discriminative Correlation Filter with Channel and
R. Girshick, Sh. Ren, K. He and J. Sun in 2015 that Spatial Reliability (CSR-DCF) (A. Lukezic, et al.,
uses Convolutional Neural Network like other 2018) tracking algorithm. Moreover, this algorithm has
famous detectors, such as YOLO (You Look Only been implemented in a C++ and integrated into Open
Once), SSD (Single Shot Detector) and so on. In CV library as a DNN (Deep Neural Networks) module
general, Faster RCNN composed from three main (OpenCV dnn module, n.d.). We proposed a tracking
part at all that can be managed building object system that integration of Faster RCNN object
detection model process. They are: a) convolution detection as an object detector and OpenCV_CSRT_
layers; b) region proposal network; c) classes and tracker as a tracking algorithm for tracking method.
bounding boxes prediction (Faster RCNN, n.d.): Implemented object tracking process contains two
parts: first is already explained above, training object
2.3 Training Dataset classifier and generating object detection model from
training outcome file. From coming frame algorithm
We have used our own dataset (600 images) for takes blobs and gives it to object detection model to
training (400) and testing (200) process, which has predict location of object and classify as an object
been collected throughout internet sources, such as class. Output predictions of the detection passes to
blogs, posts, and so on. The difference between our tracking algorithm to track predicted box of object
dataset apart from other datasets is that our dataset class and so on, any other circumstances of object
images taken by drone camera and environment. prediction changes tracker will invoke the object
Training process speed and time depends on what classifier and restarts process from the beginning.
kind of CPU and GPU system we have, if our OS has
last version of GPU hardware system that means we
can get training results faster than using CPU system. 3 EXPERIMENTAL RESULTS
If your computer has no supporting hardware
platform or latest GPU system, we recommend you After finishing 200K times training our dataset we got
our object classifier model and we have tested our Initially, conventional tracking method
object detection (classifier) model work by applying performance was good, it tracked a target properly,
images from different open source resources that but after some movement of target frame by frame
accomplishment of our detector demonstrated perfect and changing a foreground/background color of
results, such as shown in Figure 2. We can see the frame the tracker could not predict properly (given in
remarkable results of object detection model from 1611, 1696, 1854 frames) a new location of target and
different sources; however, the object classifier is not position of the maximum in correlation between
detecting all objects in frame at once in a section (c) backgrounds and image patch features extraction on
and accuracy rate is under 100%. While tracking we position and weight by the channel reliability scores.
may face thousands of positions, location, shape as Our tracker performance precision (accuracy) has
well that means it requires more quantity of images been calculated by the ability of our tracking model
on dataset with different position and environment that to identify only the relevant objects. It is the
object located images to get perfect results. percentage of correct positive predictions. Precision
Nevertheless, with 600 labeled images dataset gave calculation equation given below:
unusual result after training by Faster RCNN object
classifier model.
= =
+

Here, TP is a True Positive – a correct detection,

detection with intersection over union (IOU ≥
threshold); FP is a False Positive – a wrong detection,
detection with IOU < threshold.
a) b) c) d)
Furthermore, a proposed tracking method has
Figure 2: Object detection results of the pre-trained Faster been tested with several well-known open source
RCNN object classifier: (a) image from internet sources
datasets, like OTB (Object Tracking Benchmark),
taken by drone camera; (b, c) picture taken from drone
video sequence; (d) image from VOT2018 dataset.
VOT (Visual Object Tracking), and others. However,
we have compared results in Table 1 with some open
source datasets which conventional method has been
We have tested our tracking method with different
open source datasets and got good results and tested and posted as a final approach performance in
several internet blogs. We can see the comparison
compered with conventional tracker itself. The
results of conventional and proposed tracking
experiments show that CSR-DCF tracker algorithm
methods in Table 1, where has been tested with three
cannot re-establish object once it gone from current
different open source datasets that the main difference
frame or tracks object not properly and if the shape or
between two methods is deep learning-based
appearance of the tracking object changes in a
approach and real time frames per second as well. As
noticeable drape tracker can’t track object properly
we can see below in the first column of TABLE 1
and consequently it will lose a target. We have tested
results of CVRP2013 dataset while conventional
this situation on video sequence taken by drone, here
method presents 80 percent accuracy, proposed
is some result of tested video shown in Figure 3 below:
method gained 89 percent tracking accuracy
respectively. Furthermore, outcomes of the tested
technique performance can be seen in next two
column with some of systems feature.

Table 1: Comparsion Results.

0 485 981 1242
Deep
Precision- Precision- Precision Real
Tracker Learning
CVPR2013 OTB100 -OTB50 Time
(Yes/No)

CSR-DCF 0.8 0.733 N Y(13)

Proposed 0.89 0.829 0.81 Y Y(30)

1611 1696 1854 2106
Figure 3: Qualitative results of conventional (CSR-DCF,
blue) and proposed (FRCNN_CSRT, green) trackers (0,
485, 981,… - frames).
4 CONCLUSIONS Information Processing Systems 25 (NIPS),
DOI: 10.1145/3065386.
R. Girshick, J. Donahue, T. Darrell, and J. Malik,
In conclusion, we can say that our proposed tracking November 2013. “Rich feature hierarchies for accurate
method has gained good performance as well as object detection and semantic segmentation”,
comparable results. Our trained object classifier DOI: 10.1109/CVPR.2014.81.
model has showed excellent work by training 200K R. Girshick, 6 Jan. 2016. “Faster R-CNN: Towards Real-
times only 400 images for training and 200 images for Time Object Detection with Region Proposal
testing. In this method we have integrated deep Networks,” - Computer Vision and Pattern Recognition
learning-based object classifier model with CSRT (cs.CV), arXiv:1506.01497 [cs.CV].
Gonzalez, R.C., and Woods, R.E., 2017: Digital Image
tracker OpenCV implementation version of CSR-
Processing. Pearson, 4th edition.
DCF algorithm with the support of OpenCV DNN Faster RCNN, https://towardsdatascience.com/faster-rcnn-
module. We have compared performance of CSR- object-detection- f865e5ed7fc4
DCF tracking algorithm with our integrated tracking A. Lukezic, T. Vojır, L. C. Zajc, J. Matas, and M. Kristan,
method. Comparison results showed a much better July 2018. “Discriminative Correlation Filter Tracker
performance in proposed tracking method that in with Channel and Spatial Reliability,” International
tracking process an object classifier gives exact Journal of Computer Vision: Volume 126, Issue 7, pp
location of the target in frame with the benefit of pre- 671–688.
trained object classifier. In the future work we will try OpenCV dnn module, cv::dnn::Net Class Reference,
https://docs.opencv.org/master/db/d30/classcv_1_1dnn
to create CNN tracker that can be work without pre-
_1_1Net.html
trained model.

ACKNOWLEDGEMENTS
This research was supported by the MSIT (Ministry
of Science and ICT), Korea, under the ICT
Consilience Creative program (IITP-2020-2016-0-
00318) supervised by the IITP (Institute for
Information & communications Technology Planning
& Evaluation), and Basic Science Research Program
through the National Research Foundation of Korea
(NRF) funded by the Ministry of Science, ICT &
Future Planning (2016R1D1A3B03931003, No.
2017R1A2B2012456), and Ministry of Trade,
Industry and Energy for its financial support of the
project titled “the establishment of advanced marine
industry open laboratory and development of realistic
convergence content”

REFERENCES
P. Viola and M. Jones, July 13, 2001. “Robust Real-time
Object Detection,” Second international workshop on
statistical and computational theories of vision –
modeling, learning, computing, and sampling,
Vancouver, Canada, IJCV 2001 See pp 1-3.
N. Dalal and B. Triggs, Jun, 2005. “Histograms of Oriented
Gradients for Human Detection,” International
Conference on Computer Vision & Pattern Recognition
(CVPR '05), San Diego, United States. pp.886-893.
A. Krizhevsky, I. Sutskever, Geoffrey E. Hinton, 2012.
“ImageNet Classification with Deep Convolutional
Neural Networks” - Part of: Advances in Neural