See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/316175587

A comparison of IoT application layer protocols through a smart parking

implementation

Conference Paper · January 2017

DOI: 10.1109/ICIN.2017.7899436

CITATIONS READS
20 1,061

2 authors, including:

Paridhika Kayal
University of Toronto
4 PUBLICATIONS   23 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

IoT Protocols View project

Fog computing View project

All content following this page was uploaded by Paridhika Kayal on 09 November 2017.

The user has requested enhancement of the downloaded file.

 A Comparison of IoT application layer protocols
through a smart parking implementation
Paridhika Kayal and Harry Perros
{pkayal,hp}@ncsu.edu
Computer Science Department
North Carolina State University

Abstract—Several IoT protocols have been introduced in order to high performance, real-time data sharing or real-time device
provide an efficient communication for resource-constrained control. In many cases data is collected for subsequent
applications. However, their performance is not as yet well “offline” processing. The WebSocket (WS) standard provides
understood. To address this issue, we evaluated and compared bi-directional Web communication and connection
four communication protocols, namely, CoAP, MQTT, XMPP,
and WebSocket. For this, we implemented a smart parking
management. WebSocket is a good IoT solution if the devices
application using open source software for these protocols and can afford the WebSocket payload. Other protocols, such as,
measured their response time by varying the traffic load. SMQ and CoSIP are also gaining traction. All these protocols
Keywords—CoAP, MQTT, XMPP, WebSocket, smart parking, are positioned as real-time publish-subscribe IoT protocols,
response time. with support for millions of devices. Depending on how you
define “real time” (seconds, milliseconds or microseconds)
I. INTRODUCTION and “things” (WSN node, multimedia device, personal
An IoT application typically involves a large number of wearable device, medical scanner, engine control, etc.), the
deployed and interconnected sensors and gateways. The protocol selection for an application is critical.
sensors measure the physical environment and send the data to
II. RELATED WORK
a gateway. The gateway aggregates the data from various
sensors and then sends it to a server/broker. Meanwhile, Several surveys have provided description and
clients that are interested to receive sensor data connect to the comparative analysis of IoT application layer protocols
server to obtain the data. The integration of sensor devices into without providing real data measurements, see [1], [2], [3],
the Internet requires an IP-compatible protocol stack which is [4]. All these surveys identified CoAP, MQTT, XMPP,
bandwidth-efficient, energy-efficient and capable of working AMQP and REST services as the most representative
with limited hardware resources. The lack of optimized protocols for internet of things. The article in [2] provides a
application protocols for sensors can cause performance description of these key protocols their architectures and
degradation in terms of bandwidth usage and battery lifetime strengths and weaknesses. The authors argued about suitability
for wireless sensors. of these protocols for the IoT by considering reliability,
IoT is a big place, with room for many application security, and energy consumption aspects without any
protocols suitable for sensors. The fundamental goals of all statistical comparisons between the protocols.
protocols differ, the architectures differ, and the capabilities Several works experimentally tested the most popular IoT
differ. It is important to understand the class of use that each application layer protocols, typically comparing two selected
of these protocols addresses and choose the one for an protocols. In [5] the authors compared MQTT and CoAP by
application carefully, especially when key system creating a middleware component in order to perform testing.
requirements such as performance, QoS, interoperability, fault They found that MQTT has a lower latency for smaller packet
tolerance and security are taken into account. loss than CoAP, and in contrast, higher latency than CoAP for
In this paper, we report on the response time of the higher packet loss. Two IoT protocols CoAP and MQTT have
following communication protocols: CoAP, MQTT, XMPP, and been assessed in terms of energy consumption, bandwidth
WebSocket, measured for different traffic loads. utilization and reliability [6]. According to the result, CoAP is
The MQTT protocol is used for collecting device data and the most efficient in terms of energy consumption and
communicating it to servers, XMPP is used for connecting bandwidth usage while MQTT provides high reliability. [7]
devices to people. It can support distributed message provide a qualitative and quantitative comparison between
exchanges between processes on a single node (Intra Device). MQTT and CoAP when used as smartphone application
However XMPP was not designed for high performance protocols. In [8] performance tests are conducted by
message exchanges within the same mode and is more implementing the experimental design for three different
appropriate when used to communicate between nodes or with techniques in a LAN for supporting real-time communication
internet based applications. CoAP is a specialized web transfer with XMPP on the Web.
protocol for use in constrained nodes and networks. It can be In paper [9] CoAP, WebSocket and MQTT are evaluated
used for data collection in systems that do not require very for their performance in terms of protocol efficiency, strictly
related to the overhead, and average Round Trip Time (RTT) publish and subscribe messages sent from the clients. Below is
with no consideration of increased load. Authors in [10] an example.
evaluated the performance, energy consumption, and resource ./client/mosquitto_pub -t “topic“ -m “Message to be sent”
usage characteristics of IoT protocols including CoAP, ./client/mosquitto_sub -C 1 -t “topic“
MQTT, MQTT-SN, WebSocket, and TCP for varying packet
size. [11] compared web performance of MQTT, AMQP, C. XMPP (eXtensible Messaging and Presence Protocol)
XMPP, and DDS by measuring the latency of sensor data XMPP [15] is a distributed client/server architecture and
message delivery and the message throughput rate for different TCP communications protocol based on XML that enables
message size and format. near-real-time exchange of structured data between two or
There are only a few studies available on protocol more connected entities. XMPP provides a wide range of
efficiency, strictly related to overhead, RTT, packet applications, such as, instant messaging, multi-party chat,
fragmentation, QoS and retransmission. To the best of our voice and video calls, collaboration, and lightweight
knowledge there is no performance study of the response time middleware. It provides SASL authentication and has built-in
with varying traffic loads. In this work we take an TLS encryption.
experimental approach by considering a real IoT scenario. We We used the Openfire server licensed under the Open
implemented a smart parking application using the following Source Apache License. It uses the only widely adopted open
application layer protocols: CoAP, MQTT, XMPP and protocol for instant messaging, XMPP (also called Jabber).
WebSocket. Being implemented on the same platform, the D. MQTT over WebSocket
comparison between theses protocols is fair and realistic. The
measured performance metrics is the average response time WebSocket [18] operate over TCP as an upgrade to a
and its confidence interval and the 99th percentile for varying standard HTTP connection allowing for full-duplex, low-
server utilizations. latency communication between a server and a client. In
MQTT over WebSocket, an MQTT message is encapsulated
III. PROTOCOLS OVERVIEW within a WebSocket packet. Messages over WebSocket are
In this section, we review briefly the four protocols under sent in frames, which have only 2 bytes overhead. WebSocket
study. are suitable as transport for MQTT because the
communication is bidirectional, ordered and lossless.
A. CoAP (Constrained Application Protocol) We used the following software: a) HiveMQ Broker, a
CoAP [12] is a one-to-one protocol for transferring state simple and secure, highly scalable enterprise MQTT broker
information between client and server over the Internet using designed for lowest latency and high throughput; b) the
UDP, and it is primarily designed for constrained devices. HiveMQ Plugin, which allows plugin development on top of
Clients may send GET, PUT, POST and DELETE resource the broker to achieve required functionality, and c) the Paho
requests to the server. CoAP messages are encoded in a simple python client library, which provides a client class with
binary format. Packets are simple to generate and can be support for both MQTT v3.1 and v3.1.1 on Python 2.7 or 3.x.
parsed in place without consuming extra RAM in constrained
IV. SMART PARKING SYSTEM
devices.
We used the CoAP libcoap library that provides a coap- In this paper, we use a smart parking testbed to evaluate
client which is a wget-like tool to generate the resource and compare the above four communication protocols. Smart
requests. Below is an example of the commands used to send a parking makes it easier for drivers to find a spot in a car
coap-uri request for a DELETE message: parking to park their car and avoids the overhead of moving
./coap-client -m delete coap://[::1]/location -e "Message to be around in the parking looking for an empty slot.
sent" We considered a car parking with a sensor attached to
coap-URI = "coap:" "//" host [ ":" port ] path-abempty [ "?" every parking position. These sensors sense whether a parking
query ] spot is occupied or not. The architecture consists of a
server/client model. Every sensor has a client that sends status
B. MQTT (Message Queuing Telemetry Transport) information (occupied/empty) as a PUT or a DELETE
MQTT [16] is a client/server publish/subscribe messaging message to update their status at the server.
protocol designed for lightweight M2M communications. The Also, there is a sensor attached to every car and as a car
protocol runs over TCP/IP to provide ordered, lossless, arrives at the parking lot, the sensor sends a GET request to
bidirectional connections. MQTT supports three QoS levels reserve an empty space in the parking. If there is an available
and messages can be encrypted with SSL/TLS. MQTT brokers spot, the server reserves it and sends the exact location of the
may require username and password authentication from reserved spot back to the car sensor. Cars can also come in and
clients to connect. park at an empty position without requesting the server to
We used the Eclipse Mosquitto (EPL/EDL licensed) reserve a spot. In this case, the sensor at the parking spot sends
a PUT request to the server to indicate that it has become
message broker that implements the MQTT protocol versions occupied. The server stores the information of all the occupied
3.1 and 3.1.1. It provides an MQTT server which can handle and empty spots in the parking, so that it can serve requests
from GET clients.
 The following assumptions were made: the server. CoAP and MQTT maintain a single server, and
● The car arrival rate is assumed to follow a Poisson therefore U is the server utilization. However, this is not the
distribution. Let λ1 and λ2 be the arrival rates of case in XMPP and WebSocket because they spawn multiple
reservations and the arrival rate of cars entering the threads depending on the arrival rate. For presentation
parking lot without reservation respectively. purposes we will refer to U as the server utilization.
● The time a car stays at a parking spot follows an We picked a value for 1/ξ for each protocol, and then used
exponential distribution with a mean 1/ξ. the above M/M/100/100 queue to calculate all possible
● The length of the communication protocol server queue combinations of λ1 and λ2 so that pb = 0. In this case η = λ1 + λ2.
is assumed to be large enough to accommodate all Subsequently, we chose a pair of λ1 and λ2 that corresponds to
messages pending execution. 20%, 40%, 60% and 80% utilization of the server. We could
● The communication between clients and server is have used any combination of λ1 and λ2, since we can calculate
assumed to be fully reliable with zero packet loss. η, and thus the total arrival rate to the server. Then, vary λ1 and
λ2 so that to obtain a given server utilization. However, this
In order to implement the testbed, we considered a parking does not make any difference since we run the experiments
space of 100 cars depicted as a 10x10 array with a sensor using the server utilization.
attached at every location. The server stores the status of every The experiments are run as follows. The PUT and GET
spot as either occupied or empty. The simulation runs three threads issue messages with an exponential inter-arrival time
client threads namely PUT, DELETE, and GET for each of the of 1/λ1 and 1/λ2 respectively. Once a car occupies a position, its
three types of request described above. The GET thread parking time is generated from an exponential distribution
generates client requests for reservation according to a Poisson with a mean 1/ξ. The departure time is placed into an event list
distribution with rate λ1. The PUT thread generates messages which is kept sorted in an ascending manner. The next
due to cars occupying a parking spot without going through departure to occur is the one at the top of the event list. We
the reservation process, according to a Poisson distribution note that the simulation clock is the real clock. For a given
with rate λ2. The DELETE thread generates messages due to protocol, each experiment corresponds to a given utilization.
cars departing from the parking lot after staying for an The chosen values of λ1 and λ2 were such that they were as
exponentially distributed time with mean 1/ξ. The server close as possible, i.e., λ1/λ2 ~ 1. This can be varied, but we do
synchronizes the parking map according to the requests it not expect any difference in the results for a given utilization.
receives. We ran each experiment for 1000 GETs and 1000 PUTs,
The three client threads and the server run on the same which creates 2000 DELETE messages. We measured the
machine, which is an Ubuntu 14.04 64 bit system with Intel amount of time it takes to process each message including
Core i5-5200U CPU @ 2.20GHz * 4. As a result the waiting time in the protocol server queue. (Recall that the
propagation delay is zero. A programmable propagation delay propagation delay is zero.) These measurements were done at
can be easily introduced, but we were only interested in the thread level by starting the clock as soon as a request is sent to
response time of the protocols without including two-way the server, i.e., the time when a process thread starts, and
propagation delays. noting the time when it arrives back after being processed at
We also note that the exponential assumption could be the server. The difference in time is stored as the response
easily replaced by a non-bursty general distribution. The time for that message. Consequently, we obtained three
service time for the GET, PUT and DELETE message was separate sets of samples of response times, 1000 GET
first calculated by issuing a single request to the server which samples, 1000 PUT samples and 2000 DELETE samples. For
was idle. Let 1/μ1, 1/μ2, and 1/μ3 be the time required by the each sample we calculated the mean, standard deviation and
idle server to process a single GET, PUT and DELETE the 99th of the response time. The three samples were also
request respectively. combined to calculate similar overall statistics. The order in
The testbed can be seen as consisting of 2 queueing which they were combined is immaterial as far as these
systems. The first one is an M/M/s/K queue, where s=K=100 statistics are concerned. The results of the interval estimation
and it depicts the parking lot consisting of the 100 parking of the mean response time are presented in the next section.
spots with an arrival rate of λ1 + λ2 and a mean service time of The unit of time is seconds.
each server equal to the mean waiting time of a car in parking
V. PERFORMANCE EVALUATION
lot, i.e., 1/ξ. Let η be the rate of departure of cars from the
parking lot, then η can be calculated as (λ1 + λ2)(1- pb), where pb A. CoAP:- libcoap library
is the blocking probability that a car will be blocked due to the We implemented the three types of messages using the
parking lot being full. pb can be obtained using the Erlang-B GET, PUT and DELETE commands provided by CoAP. The
formula. service time for a PUT and a DELETE request is found to be
The second queueing model is an M/M/1 queue and it the same, but the service time of a GET request is significantly
depicts the server as implemented with the given protocol less. The three commands belong to three different classes and
where GET, PUT and DELETE messages queue up to get the libcoap server library handles them differently. Because of
processed. In order for this queue to be stable, U< 1, where U= this, the commands have different priorities and different
λ1/μ1 + λ2/μ2 + η/μ3. U can be seen as the workload offered to service times.
 time of the PUT and DELETE requests were equal but that of
the GET request was higher.

Fig. 2 Mean Response Time vs Mosquitto Broker Utilization

From Fig. 2, we can see that the mean response time of

Fig. 1 Mean Response Time vs libcoap Server Utilization every individual process increases (or lies within the
confidence interval of each other) as the server utilization is
As can be seen in Fig. 1, as the server utilization is increased.
increased, the mean response time of DELETE and PUT also
goes up as expected. In the case of GET, the response time C. XMPP:- Openfire Server & Smack Client
decreases as the utilization increases from 20% to 40%. This We used XMPP’s Instant Messaging feature to implement
can be explained by the fact that CoAP endpoints cache the three different requests and hence the service time for all
responses in order to reduce the response time and network the three messages was found to be equal. From Fig. 3, we see
bandwidth consumption on future, equivalent requests. A that as the load increases the mean response time increases for
validation and freshness mechanism is used for this purpose each individual DELETE, PUT and GET client requests but
[12]. The PUT and DELETE commands of CoAP have non- the change is not significant. This is because it opens a new
cacheable response codes while the GET command’s response TCP connection for each client.
codes are cached. Hence the response time for each individual
GET process includes the time to process the request and
return the response code plus the time to write the response in
cache. In view of this, when the arrival rate for GET messages
increases, the server skips some of the cache writes for new
similar codes. Thus the mean response time for GET messages
reduces as the arrival rate increases, while PUT and DELETE
process remain unaffected as their is no concept of caching.
The response time again increases as the server utilization
increases to 60%, this can be explained by the fact that
initially the CoAP server provides a buffer of size k bytes to
handle client requests and as the number of clients connecting
to the server increases, it increases the buffer size. This
requires buffer reallocation, which is added up to the response
time. Further as the utilization increases to 80% we observe a
similar decrease in the mean response time because of making Fig. 3 Mean Response Time vs Openfire Server Utilization
use of cached responses with a larger buffer this time.
B. MQTT:- Mosquitto Broker From Fig. 3, we can see that the mean response time for
the GET and PUT processes are similar as they have the same
With MQTT’s publisher/subscriber model, we used the arrival rate while that of the DELETE is lower. This is because
publish message to implement PUT and DELETE requests, the server gives priority to messages with the highest presence
and the subscribe message followed by a publish message to priority as "most available"(M) resource [13]. It maintains a
implement a GET request. Hence, we found that the service stack of users connected to the server with their presence
status as "available" and sends the response back to the "most
available" (i.e., the last arrived client first). This results in a E. Comparisons of the Protocols
lower mean response time for DELETE messages because of Fig. 5 shows the overall mean response time comparison
their higher arrival rate. for the protocols for different server utilizations. It can be
D. WebSocket:- HiveMQ Broker and Paho Python Client concluded from the results that MQTT and CoAP have a much
higher response time, since they use a message queue to
We used the publish request to implement PUT and
process client requests, as compared to XMPP and MQTTWS
DELETE messages while a subscribe request to implement the
which are multi-threaded.
GET message. The service time of the PUT and DELETE
messages was found to be equal while that of the GET
message was greater.

Fig. 4 Mean Response Time vs HiveMQ Server Utilization Fig. 6 Mean Response Time for CoAP & MQTT vs Server Utilization

From Fig. 4, we see that the mean response times of each Fig. 6 shows the comparison of the two message queue
individual process goes down on increasing server utilization. based protocols. We see that CoAP takes advantage of using
This is because the WebSocket protocol also has the ability to UDP for communication and hence performs better at lower
multiplex several streams simultaneously over the same utilization. MQTT performs better at higher server utilizations
connection. Hence, as the arrival rate increases, it skips few making use of some extra optimization features of the
handshakes and the new client connects over the same protocol.
websocket, resulting in a decrease in the mean response time.

Fig. 7 Mean Response Time for XMPP & WebSocket vs Server Utilization

Fig. 7 shows the comparison of the two multi-threaded

protocols XMPP and WebSocket. We can see that XMPP
Fig. 5 Mean Response Time of the Four Protocols vs Server Utilization
performs better at lower server utilization as it transfers
 View publication stats

messages directly over a TCP connection while WebSocket [9] S. Mijovic, E. Shehu, and C. Buratti. Comparing application
ues an extra WebSocket handshake. As load increases layer protocols for the Internet of things via experimentation. In
WebSocket takes advantage of multiplexing and hence the Research and Technologies for Society and Industry Leveraging
mean response time goes down. a Better Tomorrow (RTSI), pages 1–5, 2016.
[10] D.-H. Mun, M. Le Dinh, and Y.-W. Kwon. An assessment of
VI. CONCLUSIONS AND FUTURE WORK internet of things protocols for resource-constrained
We implemented a smart parking application using CoAP, applications. In COMPSAC, vol. 1, pages 555–560, 2016.
MQTT, XMPP and WebSocket as application layer protocols [11] Z. B. Babovic, J. Protic, and V. Milutinovic. Web performance
in order to measure and compare their response time for evaluation for Internet of things applications. IEEE Access, vol
different loads. The results showed that at lower server 4, pages 6974–6992, 2016.
utilization, CoAP performs best between the two message [12] Z. Shelby, K. Hartke, and C. Bormann. The constrained
queue based protocols. When the application can support application protocol (CoAP). Technical report, 2014.
multi-threading, then XMPP performs the best for lower [13] P. Saint-Andre. Extensible messaging and presence protocol
server utilization. As we increase the server utilization, the (XMPP): Instant messaging and presence. Tech. report, 2004.
mean response time of the protocols increases except that of [14] S. Cirani, M. Picone, and L. Veltri. CoSIP a constrained session
WebSocket which follows an opposite trend. This is because it initiation protocol for the Internet of things. In European
Conference on Service-Oriented and Cloud Computing, pages
multiplexes several streams onto the same connection due to
13–24. Springer, 2013.
higher arrival rates, resulting in less connection terminations
[15] P. Saint-Andre. Extensible messaging and presence protocol
and consequently fewer handshakes. So, at higher server
(XMPP): Core. [Online] Available: xmpp.org/rfcs/rfc3920.html.
utilization WebSocket outperforms the other three protocols
[16] A. Banks and R. Gupta. MQTT 3.1. 1. OASIS standard, 2014.
given that the application has enough CPU to allow multi-
threading. Also, both XMPP and WebSocket provide [17] SMQ protocol specification. [Online] Available: https://realtime.
horizontal scalability while this feature is missing in CoAP com/downloads/docs/SMQ-specification.pdf.
and MQTT being prone to single point of failure. We also [18] I Fette and A Melnikov. RFC 6455: The WebSocket protocol.
found that XMPP serves the processes in the order of most IETF, Dec, 2011.
available first while others are based on FIFO scheduling. [19] P. Kayal, A Comparison of IoT application layer protocols
The 99th percentiles are not reported due to lack of space, through a smart parking implementation. MS thesis, North
Carolina State University, Dec. 2016.
but they follow the same pattern as the mean response times.
They can be found along with the interval estimates of the
mean response times and the input values to the experiments
in [19].
REFERENCES
[1] T. Jaffey. MQTT and CoAP, IoT protocols. Feb-2014.[Online].
Available:http://www.eclipse.org/community/eclipse_newsletter/201
4/february/article2.php, 2014.
[2] V. Karagiannis, P. Chatzimisios, F. Vazquez-Gallego, and J.
Alonso-Zarate. A survey on application layer protocols for the
internet of things. Transaction on IoT and Cloud Computing, vol
3, pages 11–17, 2015.
[3] M. B. Yassein, M. Shatnawi, et al. Application layer protocols
for the Internet of things: A survey. In Engineering & MIS
(ICEMIS), pages 1–4, 2016.
[4] Z. B. Babovic, J. Protic, and V. Milutinovic. Web performance
evaluation for Internet of things applications. IEEE Access, vol
4, pages 6974–6992, 2016.
[5] D. Thangavel, X. Ma, A. Valera, H.-X. Tan, and C. K.-Y. Tan.
Performance evaluation of MQTT and CoAP via a common
middleware. In ISSNIP, 2014, pages 1–6, 2014.
[6] S. Bandyopadhyay and A. Bhattacharyya. Lightweight Internet
protocols for web enablement of sensors using constrained
gateway devices. In ICNC, pages 334–340, 2013.
[7] N. De Caro, W. Colitti, K. Steenhaut, G. Mangino, and G. Reali.
Comparison of two lightweight protocols for smartphone-based
sensing. In 2013 IEEE 20th Symp. on Comm. and Vehicular
Technology in the Benelux (SCVT), pages 1–6, 2013.
[8] M. Laine and K. Säilä. Performance evaluation of XMPP on the
web. Technical report, Citeseer, 2012.