OPNET Modeler and Ns-2: Comparing the Accuracy Of Network

Simulators for Packet-Level Analysis using a Network Testbed

Gilberto Flores Lucio, Marcos Paredes-Farrera, Emmanuel Jammeh, Martin Fleury, Martin J. Reed
Electronic Systems Engineering Department
University of Essex
Colchester, Essex C04 3SQ
United Kingdom
[email protected], http://privatewww.essex.ac.uk/~gflore/

Abstract: - This paper presents a comparative study of two well-known network simulators: OPNET Modeler and Ns-
2. Other studies in this area have generally been confined to just one simulator. The motivation of this paper is to
provide a guide to researchers undertaking packet-level network simulations. The simulator outputs were compared to
the output from a live network testbed. The experimental comparison consisted of deploying both CBR data traffic,
and an FTP session, both on the network testbed and the simulators. CBR data traffic was used due to its simplicity of
modeling. A custom analysis tool was employed to examine the behavior of the network in different background
traffic scenarios. The same scenarios were then recreated in the simulators in order to ascertain their realism. The
results show the necessity of fine-tuning the parameters within a simulator so that it closely tracks the behavior of a
real network.

Keywords: network simulator, OPNET Modeler, Ns-2, simulation methodology

results. Therefore, this paper presents real and

1 Introduction simulated results (from the two simulator tools), where
Network simulators have grown in maturity since CBR (Constant Bit Rate) data traffic, and an FTP (File
they first appeared as performance, management and Transfer Protocol) session are generated in order to
prediction tools. Simulators are normally used as test the network performance with different types of
network management tools, for which packet level traffic. CBR traffic was selected due to the simplicity
analysis is not commonly employed. However, more of its nature and FTP due to its popular use and
studies are needed to establish guidelines for dynamic behavior. CBR in particular is a useful
researchers so that they may select and customise a analysis as it provides “ground truth” i.e. there are few
simulator to suite fine-grained packet level analysis external factors that influence the model. Whereas
[1][2]. Reference [3] reports `the breach of credibility' FTP was chosen at this stage as any more complex
that studies based on simulation tools need to tackle; protocol (e.g. HTTP) would have too many degrees of
one motivation behind this paper is to address this freedom. The analysis is made at a packet-by-packet
need. The ease and facility that simulators provide in level concentrating on the bandwidth utilisation
evaluating “radical” changes to a network environment analysis.
cannot be discarded. There are a considerable number The initial tests used the default configuration
of simulations tools in the market. The main profiles provided with the simulators, however, several
characteristics that divide them are: accuracy, speed, manual modifications were made at different stages of
ease of use, and monetary expense. the experiments to investigate possible improvements.
This paper concentrates on the accuracy of the An in-house analysis tool called tcpflw was used to
simulation in comparison to a real network for packet examine the behavior of the real network. tcpflw
level analysis. Two of the currently popular network has successfully been used to analyze the performance
simulators that can perform this type of analysis are of a video streaming session [7]. The results reported
Modeler from OPNET1 [4] and Ns-2 from the Virtual here were directly compared with the results from the
Internetwork Testbed project VINT [5]. These are simulators.
selected because of their popularity within academia, This paper does not aim to define which network
commercial and industrial communities [6]. simulator is best as there are too many different
An ideal way to test simulator accuracy is to measure parameter variations and different possible network
real network traffic and compare it with the simulators scenarios to adequately determine this in a single
paper. Instead, this paper demonstrates how the
suitability of simulators can be validated for the
1
OPNET Modeler was provided under the OPNET particular case of packet level forwarding in an IP
University Programs
transport network supporting both real-time CBR and research community. It includes numerous models of
non-real time data transport. It is hoped that it will act common Internet protocols including several newer
as a guideline to other researchers of the validation protocols, such as reliable multicast and TCP selective
steps required when choosing and optimising a acknowledgement [11]. A network animator, Nam
network simulator for a different scenario. [12], provides packet-level animation and protocol
The rest of the paper is organized as follows: specific graph for design and debugging of network
Section 2 presents the network simulators, the protocols. Additionally, different levels of
experimental testbed and experiments used for the configuration are present in Ns-2 due to its open
tests. Section 3 presents the comparative results source nature, including the capability of creating
obtained. Section 4 gives a constructive discussion of custom applications and protocols as well as
the difficulties encountered with each simulator. modifying several parameters at different layers.
Finally, in Section 5, conclusions are drawn, and
further work is proposed.
2.2 The network testbed
Fig. 1 shows the network testbed used for these
2 Overview of the experimental setup experiments. It is formed by five PC's, each with
Network simulators can be divided into several processor rating of 1.7 GHz, running the Linux
types (by protocol, technology or processing method), operating system with IntelPro 400 network cards.
but the most general categorization is their method of Two of them operate as a client-server pair, and the
simulation. There are two typical methods of other two as a traffic -generator/traffic -sink pair
simulation: discrete event or analytical simulation [8]. (Traffic G. and Traffic S. respectively), and the fifth
The former produce predictions in the network at a acts as a router. In addition, a 10 Mbit hub is used to
low level (packet-by-packet), which makes them connect two PC's to the Router. Links of 10 and 100
accurate but slow to generate results. The latter, use Mbit/s were employed. The Linux systems had a
mathematical models to produce their results at a kernel patch applied (Kansas University Real-Time
much faster speed, but may sacrifice accuracy. The Linux, KURT [13]) to reduce the scheduling
usual approach is to combine both methodologies with granularity. This was required so that the scheduling of
the aim to provide a reasonable performance in terms high-rate CBR traffic (of the order 5 Mbit/s used in the
of speed but maintaining accuracy in the critical areas. experiments) could be maintained with reasonably low
The two simulators used in these experiments are delay variation (jitter). Without this patch the
hybrid simulators. performance was severely degraded due to scheduler
problems.

2.1 The simulation tools

This Section gives a brief description of the
capabilities of each simulator.
Traffic G.
Desktop System

OPNET Modeler The Modeler is just one of the

10Mbit/Link
many tools from the OPNET Technologies suite. The Client
Desktop System

version used for these experiments was 9.0.A. The Hub

Hub
100Mbit/Link
engine of the OPNET Modeler is a finite state machine 10Mbit/Link
Router
Desktop System

model in combination with an analytical model. 10Mbit/Link

Modeler can model protocols, devices and behaviors
100Mbit/Link
with about 400 special-purpose modelling functions
[9]. The GUI (Graphical User Interface) along with the
considerable amount of documentation and study cases Server
Desktop System

that come along with the license are an attractive Traffic S

Desktop System

feature of the Modeler. A number of editors are

provided to simplify the different levels of modelling Fig. 1 Network testbed.
that the network operator requires. Modeler is not open
source software though model parameters can be
altered, and this can have a significant effect on the 2.3 Network representation in the simulators
simulation accuracy. OPNET Modeler provides different levels of
modeling depending on the necessities and
Ns-2 is the second version of a network simulator requirements of the simulation. The GUI establishes
tool developed by the Virtual InterNetwork Testbed an overall environment called a Project [4]. From that
(VINT) project [10]. It is an event-driven network Project, the operator can develop several network
simulator, which is popular with the networking scenarios in order to evaluate and analyze the
performance of that network in different “what-if”
circumstances. Fig. 2 shows the network
representation used in OPNET for these experiments.
Two separate scenarios were used. The first is used for
the CBR experiment, and the second for the FTP
session. Both representations contained the same
elements; with the difference that the former was
created from custom process levels to simulate more
accurately the real characteristics of the network
testbed and the latter uses the standard and a custom
tuned version of FTP.

Fig. 3 Network Representation in Ns-2.

CBR traffic is characterized by a fixed bandwidth

across the network and is typically used by
applications such as video and audio. Furthermore,
these types of applications usually require strict delay
and jitter bounds on the CBR packets used for
transport. A CBR traffic stream can be generated by
fixing the packet size and using the same inter-arrival
time between these packets.

This type of traffic was produced by introducing

Fig. 2 Network representation used in OPNET trace files into the simulators tools. The traffic
Modeler. generators, stg and rtg, which come with the NCTUns
[15] simulator tool were used in the network testbed.
The Ns-2 architecture takes an object-oriented The stg program can read from a trace file and
approach using C++ and otcl [14]. Otcl is an replicate the UDP (User Datagram Protocol) packets.
object-oriented variant of the well-known scripting
language tcl. C++ is utilized for per-packet processing FTP session. FTP is intended to share, transfer
and forms the core of the simulator. Otcl is used for and transmit information between two computers.
simulation scenario generation, periodic or triggered Bhushan [16], in the MIT Project MAC, proposed the
action generation and for manipulating existing C++ first file transfer mechanism in 1971. Nearly all of the
objects. Fig. 3 shows the network representation for FTP servers and clients follow the more recent RFC
Ns-2 for these experiments. no. 959 [17], in which is explained: the usage model
for FTP; the set of commands employed; and how the
protocol handles the control commands and the data
connection.
2.4 The experiments For the FTP experiments, two types of tests were
Two sets of experiments were conducted. The first
performed:
one was to test the simulators accuracy with raw CBR
1. -FTP sessions using the default settings of the
data traffic and the second with an FTP session to
simulators.
simulate best-effort data traffic:
2. -Finely tuned FTP parameters at different
levels for each simulator capability. The actual
parameters changed are discussed next.

Several tests were performed to evaluate the tuned

parameters and the values that better “mimic” the
characteristics of the networks testbed for each
simulator. The most significant parameters tuned in
each simulator with the default and “tuned” values are
given in Table 1. For OPNET Modeler, three
parameters were changed, first the version of TCP was
changed from Reno to New Reno; this improves the In total, 24 experiments were conducted with 8
fast recovery capability of the protocol. When multiple different scenarios. 12 experiments were performed for
packets are lost, New Reno can recover without a CBR traffic and 12 for the FTP session. Table 2 shows
retransmission timeout [18]. Window Scaling was the different scenarios, some with cross-traffic and
activated, this allows the advertisement of window some without. Each scenario was repeated in the two
sizes bigger than 65 kB [19], and, the Time-Stamp [19] simulators and in the network testbed.
option was activated to imitate the echoing capability
of the testbed in both directions. For Ns-2 [11], the Scenario # Client-Server Traffic G. - Traffic S.
link latency was activated and this affects the Round Load load
Trip Time (RTT) of the packets. Max segment size was CBR1 2Mb/s 0 Mb/s
increase for bigger Ethernet packets, and Window size CBR2 2Mb/s 2 Mb/s
to have more range for sliding the window value. With CBR3 5Mb/s 0 Mb/s
Ns-2 there was no discernable change between Reno CBR4 5Mb/s 6Mb/s
and New Reno. FTP1 10MB file 0Mb/s (CBR)
FTP2 10MB file 6Mb/s (CBR)
Simulator Parameters Default/tuned FTP3 10MB file 0Mb/s (CBR)
value FTP4 10MB file 6Mb/s (CBR)
Table 2. The CBR and FTP traffic scenarios.
New Reno Disabled/Enabled
OPNET Modeler Window Scaling Disabled/Enabled
Timestamp Disabled/Enabled
Link Latency Disabled/Enabled 3 Results
Ns-2 Max segment size 1000/1500 Figs.4–8 compare the most rele vant results
Window size 20/100 obtained by the two simulators and the network
Table 1. Parameters tuned for both simulators. testbed for the first four CBR scenarios. Fig. 4 shows
the results obtained in scenario CBR1, from the
router's perspective. As can be seen, Ns-2 displays a
2.5 The methodology more realistic CBR behavior than Modele r. We
To evaluate the accuracy of the simulators, actual surmise that the reason for this could the method used
network data was measured. Two sets of experiments to import CBR traffic pattern into Modeler. There are
were conducted with different scenarios. First, CBR three techniques to import this traffic into Modeler:
data traffic was introduced to the network testbed and 1. - Import traffic measurements as trace files
second, a FTP session was performed from a client to using the OPNET ACE flow analysis tool.
a server to transfer a 10 MB file. Under all the 2. - Create a custom process model that imports
scenarios, the measurement technique consisted in trace files.
monitoring the IP traffic using tcpdump [20], which is 3. - Import trace files as a background traffic
a tool widely used for packet capture in the research source within OPNET.
community [21]. The information provided for every We have used the latter as OPNET ACE flow
packet was: measurement timestamp, analysis was not available to us and generating a
source/destination IP address and port source, Ethernet custom process model was not required for Ns-2.
packet size, and IP packet size. Modeler used the trace file as a source to generate
Afterwards, the data were analyzed with the in- background traffic at the given rate during this part of
house program tcpflw, obtaining the Ethernet the experiment, which no doubt explains the “bursty
throughput in every node in the testbed. tcpflw is behavior” [22] displayed.
software developed in the University of Essex by
Marcos Paredes-Farrera that groups and analyzes IP Figs. 5 and 6 (scenarios CBR2 and CBR3) show
data obtained from tcpdump. The data consists of very similar simulator behavior compared to the CBR1
packets that are grouped in different categories called behavior.
flows. Every flow can be characterized by its IP
source, IP destination, port source/port destination, and
protocol. The program can provide graphs and trace
files of every flow by performing additional post-
processing of the raw data. The granularity can be
changed to analyze the traffic from an application,
service, and computer, at subnet and network level.
The program can analyze packet size, bandwidth, and
inter-arrival time versus time. For this study,
bandwidth analysis was used.
 provides a more accurate behavior in comparison to
the network testbed than the other simulator.
It was found that in the real network testbed the
traffic from different flows was apportioned unequally
by the router packet scheduler, whereas both
simulators split the flows equally in the router.
Clearly, the simulators have made an assumption
about the router scheduling, whereas, in practice, a
number of different algorithms may be used in the
router kernel. This shows one weakness in packet level
simulation, in that, actual systems have a great number
of variables that control the forwarding of packets and
not all of these processes can be accurately replicated.
Fig. 4 CBR1 Router perspective.

Figs. 5 and 6 show lower throughput for the

network testbed. This is in fact an artifact of the
scheduling problems mentioned in Section 2.2; even
with the applied KURT kernel patch a target CBR rate
of 5 Mbit/s was not quite achieved, and without the
patch this rate was reduced much
further.

Fig. 7 CBR4: Client perspective.

Fig. 5 CBR2 Router perspective.

Fig. 8 CBR4: Router perspective.

Fig. 6 CBR3 Router perspective

Figs.7–9 show results from scenario CBR4. Again,

the testbed traffic discrepancy when generating CBR
traffic appears in Figs. 7 and 9. Turning to the
simulators, Fig. 7 shows a sudden drop in bandwidth Fig. 9 CBR4: Server perspective.
in Ns-2 from the client perspective. The sudden drop
in bandwidth came as a surprise, so this result was Figs.10–13 compare the most relevant results
repeated a number of times to confirm the anomalous obtained by the two simulators and the network
problem. In scenario CBR4, the Modeler from OPNET testbed for the two FTP scenarios with and without the
fine-tuning. In both scenarios, a 10 MB file was
transferred from the Server to the Client. Traffic going
from client to server consists of session level packets
and TCP data transfer acknowledgements. Figs. 10 and
11 show the results for scenarios FTP1 and FTP3 from
the router perspective with no CBR traffic the former
is the “standard” version of FTP and the later with the
tuning parameters active. For scenarios FTP2 and
FTP4, CBR traffic of 6 Mbit/s was also sent from G to
S (see Fig. 1 for node definition). The results displayed
only show the file transfer traffic; they do not consider
the preliminary traffic to establish the FTP session and
the subsequent traffic to tear the connection down. The
FTP used in the network testbed was vsFTPd version Fig. 11 FTP1 and FTP3: Router perspective (from
1.1.0. 2 Client to Server).
Though vsFTPd has more security features than
some other versions of FTP, its traffic behavior
appears to be typical. From the results, we can
conclude that to model a FTP server is not an easy
task, an observation echoed by others [23]. While Ns-2
took more time to transfer the file due to its limited
and almost constant bandwidth (as shown in Figs. 10
and 11), Modeler had a tendency to more closely
model the network testbed over time. A simple FTP
session, due to its dynamic behavior and number of
interrelated factors, is a difficult application to
simulate. Nevertheless, after “fine-tuning” simulator
parameters, the results from the Modeler were quite
accurate in comparison to those from Ns-2. In Fig. 12 FTP2 and FTP4: Router perspective (from
particular, note the behavior of the FTP3 session of Server to Client).
Modeler shown in Figs. 10 and 11 matches the actual
network testbed quite closely.
Figs. 12 and 13 show the results for scenario FTP2
and FTP4 from the router perspective. Again, the
former represents the standard FTP version of the
simulators and the second the “fine-tuned” one. The
results show significantly different tendencies between
the Modeler and Ns-2. The OPNET Modeler is a more
accurate replication of the testbed results.

Fig. 13 FTP2 and FTP4: Router perspective (from

Client to Server).

4 Discussion
In terms of accuracy of bandwidth estimation for
the pure CBR-type traffic, Ns-2 performed better than
OPNET Modeler using the default Modeler package.
Fig. 10 FTP1 and FTP3: Router perspective (from This may be improved using additional OPNET
Server to Client). packages; the authors cannot verify this. Nevertheless,
in scenario CBR4, Ns-2 behaved differently to the
testbed, and instead Modeler gave more accurate
results.
2
Turning to the FTP experiments. The Ns-2 FTP
vsFTPd is available at http://vsftpd.beasts.org/ (current v 3 simulation model only indicated a general transfer rate
03).
rather than replicating the actual network flow.
Modeler performed closely to the testbed results, and standard form. FTP (through TCP flow control) adapts
in the case of the FTP2 and FTP4 scenarios the results its output to prevailing network conditions, whereas
were remarkably similar. the response of Ns-2 and OPNET Modeler did not
In terms of simulation speed, both Ns-2 and the always mimic this performance. However, when “fine-
Modeler proved to be fast, requiring less than a minute tuning” of parameters was performed, it was found
to obtain the results. that Modeler was a more accurate simulator for this
Ns-2 has a “small suite”, but for large-scale particular case.
networks several modifications and extra care has to We did not find creating these types of
be taken to manage memory allocation and CPU time comparisons between different network simulation
(abstraction techniques)[24]. OPNET Modeler has a tools to be an easy task. Further work to be performed
“heavy suite” (large software overhead) but provides includes establishing a scheme to model HTTP and
diverse statistics modules at different levels [25]. The other more complex protocols in the simulators and
freeware nature of Ns-2 is attractive to researchers the testbed.
compared to the need to enter into an OPNET Modeler
license agreement and associated direct costs. References:
Additionally, the constraint of not having access to [1] R. Currier. Test-drive your network designs.
other modules outside the standard version of OPNET, Network World, May 1999.
such as the Application Characterization Environment [2] J. Heidemann and K. Mills. Expanding confidence
(ACE) and the SPGuru package, limits the possibility in network simulations. IEEE Network Magazine,
of evaluating other performance measures such as 15(5): 58-63, 2001.
flow-analysis. [3] K. Pawlikowski, H-D. J. Jeong, and J-S. R. Lee.
At first glance, the evaluation of the different On the credibility of simulation studies of
network simulators by comparison with the testbed telecommunication networks. IEEE Communications
appears to be a straightforward procedure. However, Magazine, 40(1): 132-139, 2002.
this was not the case. The learning curve for each of [4] C. Zhu, O. W. W. Yang, J. Aweya, M. Oullette,
the simulators was different, and sometimes steeper and D. Y. Montuno. A comparison of active queue
than expected. To create an Ns-2 model involves management algorithms using the OPNET Modeler.
writing a script in an extension of tcl, which will be IEEE Communications Magazine, 40(6): 158-167,
unfamiliar to most of those using the simulator for the 2002.
same time. On the other hand, as benefits a [5] V. Paxson and S. Floyd. Why we don't know how
commercial product, OPNET has a well-engineered to simulate the Internet. In Winter Simulation
user-interface using mainstream software and Conference, pages 1037-1044, 1997.
operating system. [6] L. Breaslu, K. Estrin, D. Fall, S. Floyd, J.
Heidemann, A. Helmy, P. Huang, S. McCanne, K.
5 Conclusions and further work Varadhan, X. Ya, and H. Yu. Advances in network
In this paper, two network simulators were simulation. IEEE Computer, 33(5): 59-67, 2000.
compared with a network testbed. The accuracy of Ns- [7] E. Jammeh, M. Paredes, and M. Ghanbari.
2 and Modeler from OPNET was compared using Transporting real time transcoded video over the
CBR data traffic and an FTP session. Several scenarios Internet using end-to-end control. In International
were evaluated and regenerated in the simulation tools Packet Video Workshop, 2002.
and the network testbed. The results provide [8] R. Currier. Test-drive your network designs.
interesting guidelines to network researchers in the Network World, May 1999.
selection of network simulation tools. [9] OPNET Users' Manual, OPNET Architecture,
From the researchers point of view, Ns-2 provides OV.415.http://forums.opnet.com.
very similar results compared to OPNET Modeler, but [10] S. Bajaj, L. Breslau, D. Estrin, K. Fall, S. Floyd,
the “freeware” version of Ns-2 makes it more M. Haldar, P. Handley, A. Helmy, J. Heidemann, P.
attractive to a researcher. However, the complete set of Huang, S. Kumar, S. McCanne, R. Rejajie, S. Punnet,
OPNET Modeler modules provides more features than K. Varadhan, X. Ya, H. Yu, and D. Zappala.
Ns-2, and it therefore will be more attractive to Improving simulation for network research. Technical
network operators. Report 99-702b, USC Computer Science Department,
One specific observation can be made about these 1999.
network simulators as a result of these experiments. [11] K. Fall. Network emulation in the VINT/ns
For a simple CBR data traffic, it appears that the simulator. In Fourth IEEE Symposium on Computers
simulators had no significant problem in terms of and Communications (ISCC'99), pages 59-67,1999.
accurately modeling the testbed behavior. However, in [12] D. Estrin, M. Handley, J. Heidemann, S.
the case of an FTP session, the simulators using McCanne, Y. Xu, and H. Yu. Network visualization
default simulation settings did not adequately model with the VINT Network Animator Nam. Technical
the dynamic behavior of FTP when is used in its basic
Report 99-703b, USC Computer Science Department,
1999.
[13] W. Dinkel, D. Niehaus, M. Frisbie, and J.
Woltersdorf. KURT Linux Users Manual, 2002.
http://www.ittc.ku.edu/kurt/papers/user-manual-
DRAFT.pdf.
[14] The Otcl Tutorial, 1995.
http://bmrc.berkeley.edu/research/cmt/cmtdoc/otcl/tuto
rial.html (Current v 3 03).
[15] S.Y. Wang and H. T. Kung. A new methodology
for easily constructing extensible and high fidelity
TCP/IP network simulators. Computer Networks,
40(2): 257-278, 2002.
[16] A. Bhushan. A file transfer protocol, Internet
Engineering Task Force RFC 114. Technical report,
1971.
[17] J. Postel. File transfer protocol (ftp), Internet
Engineering Task Force RFC 959. Technical report,
1985.
[18] The New Reno Modification to TCP's Fast
Recovery Algorithm, Internet Engineering Task Force
RFC 2582.
[19] TCP Extensions for High Performance, Internet
Engineering Task Force RFC 1323. pages 9-11.
[20] tcpdump manual pages.
http://www.tcpdump.org/tcpdump man.html.
[21] V. Paxson, "Measurements and Analysis of End-
to-End Internet Dynamics," Ph.D. dissertation, U.C.
Berkeley, 1997
[22] Organizing server assets: Methodology for
organizing server assets, 2003. From the OPNET
Methodology and Case Studies Website, part of the
Technical Resources papers.
[23] M. Masuda and J. Ichimura. Evaluation of
simulation accuracy and network planning case study.
OPNET website white paper.
[24] The Network Simulator ns-2: Tips and Statistical
Data for Running Large Simulations in NS;
http://www.isi.edu/nsnam/ns/ns-largesim.html.
[25] W. G. Bragg. Which network design tool is right
for you? IEEE IT Pro Magazine, 2(5): 23-31,2000.