1

A Survey on Edge Computing Systems and Tools

Fang Liu, Guoming Tang, Youhuizi Li, Zhiping Cai, Xingzhou Zhang, Tongqing Zhou

Abstract—Driven by the visions of Internet of Things and 5G Architecture Applications

communications, the edge computing systems integrate comput-
ing, storage and network resources at the edge of the network
Cloudlet Cachier ……
Push from
to provide computing infrastructure, enabling developers to CloudPath FocusStack
cloud
quickly develop and deploy edge applications. Nowadays the edge Some other systems, e.g.,
PCloud Pull from IoT
computing systems have received widespread attention in both Vigilia, HomePad, LAVEA,
industry and academia. To explore new research opportunities MUVR, OpenVDAP, Hybrid cloud-
ParaDrop SafeShareRide, VideoEdge edge analytics
and assist users in selecting suitable edge computing systems for
arXiv:1911.02794v1 [cs.DC] 7 Nov 2019

specific applications, this survey paper provides a comprehensive SpanEdge Programming models
overview of the existing edge computing systems and introduces Cloud-sea AirBox Firework
representative projects. A comparison of open source tools is computing
presented according to their applicability. Finally, we highlight Supporting Supporting
energy efficiency and deep learning optimization of edge com-
Open-source systems
puting systems. Open issues for analyzing and designing an edge
Akraino
computing system are also studied in this survey. CORD
Edge Stack
EdgeX Apache Azure IoT
Foundry Edgent Edge
I. I NTRODUCTION
In the post-Cloud era, the proliferation of Internet of Things Fig. 1. Categorization of edge computing systems.
(IoT) and the popularization of 4G/5G, gradually changes the
public’s habit of accessing and processing data, and challenges
the linearly increasing capability of cloud computing. Edge AzureStack in 2017, which allows cloud computing ca-
computing is a new computing paradigm with data processed pabilities to be integrated into the terminal, and data can
at the edge of the network. Promoted by the fast growing be processed and analyzed on the terminal device.
demand and interest in this area, the edge computing systems • Pull from IoT. Internet of Things (IoT) applications
and tools are blooming, even though some of them may not pull services and computation from the faraway cloud
be popularly used right now. to the near edge to handle the huge amount of data
There are many classification perspectives to distinguish generated by IoT devices. Representative systems include
different edge computing system. To figure out why edge PCloud, ParaDrop, FocusStack and SpanEdge. Advances
computing appears as well as its necessity, we pay more in embedded Systems-on-a-Chip (SoCs) have given rise
attention to the basic motivations. Specifically, based on dif- to many IoT devices that are powerful enough to run
ferent design demands, existing edge computing systems can embedded operating systems and complex algorithms.
roughly be classified into three categories, together yielding Many manufacturers integrate machine learning and even
innovations on system architecture, programming models and deep learning capabilities into IoT devices. Utilizing edge
various applications, as shown in Fig. 1. computing systems and tools, IoT devices can effectively
• Push from cloud. In this category, cloud providers push share computing, storage, and network resources while
services and computation to the edge in order to leverage maintaining a certain degree of independence.
locality, reduce response time and improve user experi- • Hybrid cloud-edge analytics. The integration of advan-
ence. Representative systems include Cloudlet, Cachier, tages of cloud and edge provides a solution to facilitate
AirBox, and CloudPath. Many traditional cloud comput- both global optimal results and minimum response time
ing service providers are actively pushing cloud services in modern advanced services and applications. Represen-
closer to users, shortening the distance between customers tative systems include Firework and Cloud-Sea Comput-
and cloud computing, so as not to lose market to mo- ing Systems. Such edge computing systems utilize the
bile edge computing. For example, Microsoft launched processing power of IoT devices to filter, pre-process,
and aggregate IoT data, while employing the power and
F. Liu is with the School of Data and Computer Science, Sun Yat-sen Uni-
versity, Guangzhou, Guangdong, China (e-mail: [email protected]). flexibility of cloud services to run complex analytics
G. Tang is with the Key Laboratory of Science and Technology on Information on those data. For example, Alibaba Cloud launched its
System Engineering, National University of Defense Technology, Changsha, first IoT edge computing product, LinkEdge, in 2018,
Hunan, China (e-mail: [email protected]). Y. Li is with the School of
Compute Science and Technology, Hangzhou Dianzi University, China. Z. which expands its advantages in cloud computing, big
Cai and T. Zhou are with the College of Computer, National University of data and artificial intelligence to the edge to build a
Defense Technology, Changsha, Hunan, China (e-mail: [email protected], cloud/edge integrated collaborative computing system;
[email protected]). X. Zhang is with State Key Laboratory of
Computer Architecture, Institute of Computing Technology, Chinese Academy Amazon released AWS Greengrass in 2017, which can
of Sciences, China. extend AWS seamlessly to devices so that devices can
 2

System View

Energy Efficiency
Edge Computing Systems & Tools
(Sec. II)

(Sec. IV)
Open Source Edge Computing Projects
(Sec. III)

Application View

Deep Learning Optimization at the Edge

(Sec. V)

Fig. 3. Cloudlet Component Overview and Functions that support application

Fig. 2. Major building blocks and organization of this survey paper. mobility. A: Cloudlet Discovery, B: VM Provisioning, C: VM Handoff.

perform local operations on the data they generate, while

data is transferred to the cloud for management, analysis, proposed to support the deep learning models at the edge (in
and storage. Sec. V).
From a research point of view, this paper gives a detailed Our main contributions in this work are as follows:
introduction to the distinctive ideas and model abstractions of • Reviewing existing systems and open source projects for
the aforementioned edge computing systems. Noted that the edge computing by categorizing them from their design
three categories are presented to clearly explain the necessity demands and innovations. We study the targets, architec-
of edge computing, and the classification is not the main line of ture, characteristics, and limitations of the systems in a
this paper. Specifically, we review systems designed for archi- comparative way.
tecture innovation first, then introduce those for programming • Investigating the energy efficiency enhancing mechanism
models and applications (in Sec. II). Besides, some recently for edge computing from the view of the cloud, the edge
efforts for specific application scenarios are also studied. servers, and the battery-powered devices.
While we can find a lot of systems using edge computing • Studying the technological innovations dedicated to de-
as the building block, there still lacks standardization to such ploying deep learning models on the edge, including
a paradigm. Therefore, a comprehensive and coordinated set systems and toolkits, packages, and hardware.
of foundational open-source systems/tools are also needed • Identifying challenges and open research issues of edge
to accelerate the deployment of IoT and edge computing computing systems, such as mobility support, multi-user
solutions. Some open source edge computing projects have fairness, and privacy protection.
been launched recently (e.g., CORD). As shown in Fig. 1, We hope this effort will inspire further research on edge
these systems can support the design of both architecture and computing systems. The contents and their organization in this
programming models with useful APIs. We review these open paper are shown in Fig. 2. Besides the major four building
source systems with a comparative study on their characteris- blocks (Sec. II∼Sec. V), we also give a list of open issues for
tics (in Sec. III). analyzing and designing an edge computing system in Sec. VI.
When designing the edge computing system described The paper is concluded in Sec. VII.
above, energy efficiency is always considered as one of the
major concerns as the edge hardware is energy-restriction. II. E DGE C OMPUTING S YSTEMS AND T OOLS
Meanwhile, the increasing number of IoT devices is bringing
In this section, we review edge computing systems and tools
the growth of energy-hungry services. Therefore, we also
presenting architecture innovations, programming models, and
review the energy efficiency enhancing mechanisms adopted
applications, respectively. For each part, we introduce work
by the state-of-the-art edge computing systems from the three-
under the “push”, “pull”, and “hybrid” demand in turn.
layer paradigm of edge computing (in Sec. IV).
In addition to investigations from the system view, we
also look into the emerging techniques in edge computing A. Cloudlet
system from the application view. Recently, deep learning In 2009, Carnegie Mellon University proposed the concept
based Artificial Intelligence (AI) applications are widely used of Cloudlet [1], and the Open Edge computing initiative was
and offloading the AI functions from the cloud to the edge is also evolved from the Cloudlet project [2]. Cloudlet is a
becoming a trend. However, deep learning models are known trusted, resource-rich computer or cluster of computers that are
for being large and computationally expensive. Traditionally, well-connected to the Internet and available to nearby mobile
many systems and tools are designed to run deep learning devices. It upgrades the original two-tier architecture “Mobile
models efficiently on the cloud. As the multi-layer structure of Device-Cloud” of the mobile cloud computing to a three-
deep learning, it is appropriate for edge computing paradigm, tier architecture “Mobile Device-Cloudlet-Cloud”. Meanwhile,
and more of its functions can be offloaded to the edge. Ac- Cloudlet can also serve users like an independent cloud,
cordingly, this paper also studies the new techniques recently making it a “small cloud” or “data center in a box”. Although
 3

the Cloudlet project is not proposed and launched in the

name of edge computing, its architecture and ideas fit those
of the edge computing and thus can be regarded as an edge
computing system.
The Cloudlet is in the middle layer of the three-tier edge
computing architecture and can be implemented on a personal
computer, low-cost server, or small cluster. It can be composed
of a single machine or small clusters consisting of multiple
machines. Like WiFi service access points, a Cloudlet can
be deployed at a convenient location (such as a restaurant, a
cafe, or a library). Multiple Cloudlets may form a distributed
computing platform, which can further extend the available
resources for mobile devices [3]. As the Cloudlet is just one
hop away from the users’ mobile devices, it improves the QoS
with low communication delay and high bandwidth utilization.
In detail, Cloudlet has three main features as follows.
1) Soft State: Cloudlet can be regarded as a small cloud
computing center located at the edge of the network. There-
fore, as the server end of the application, the Cloudlet
generally needs to maintain state information for interacting Fig. 4. CloudPath Architecture [9].
with the client. However, unlike Cloud, Cloudlet does not
maintain long-term state information for interactions, but only
temporarily caches some state information. This reduces much 3) VM Handoff: Migrating the VM running the application
of the burden of Cloudlet as a lightweight cloud. to another cloudlet.
2) Rich Resources: Cloudlet has sufficient computing re-
sources to enable multiple mobile users to offload computing
tasks to it. Besides, Cloudlet also has stable power supply so B. CloudPath
it doesn’t need to worry about energy exhaustion. CloudPath [9] is an edge computing system proposed by
3) Close to Users: Cloudlets are deployed at those places the University of Toronto. In such a system, diverse resources
where both network distance and physical distance are short to like computing and storage are provided along the path from
the end user, making it easy to control the network bandwidth, the user device to the cloud data center. It supports on-demand
delay and jitter. Besides, the physical proximity ensures that allocation and dynamic deployment of the multi-level archi-
the Cloudlet and the user are in the same context (e.g., the tecture. The main idea of CloudPath is to implement the so-
same location), based on which customized services (e.g., the called “path computing”, such that it can reduce the response
location-based service) could be provided. time and improve the bandwidth utilization, compared with
To further promote Cloudlet, CMU built up an open edge the conventional cloud computing.
computing alliance, with Intel, Huawei and other compa- As illustrated in Fig. 4, the bottom layer of CloudPath is
nies [2], to develop standardized APIs for Cloudlet-based edge user devices, and the top layer is the cloud computing data
computing platforms. Currently, the alliance has transplanted center. The system reassigns those tasks of the data centers
OpenStack to the edge computing platform, which enables along the path (for path computing) to support different types
distributed Cloudlet control and management via the standard of applications, such as IoT data aggregation, data caching
OpenStack APIs [4]. With recent development of the edge services, and data processing services. Developers can select
computing, the Cloudlet paradigm has been widely adopted an optimal hierarchical deployment plan for their services by
in various applications, e.g., cognitive assistance system [5], considering the factors such as cost, delay, resource availability
[6], Internet of Things data analysis [7], and hostile environ- and geographic coverage. Path computing builds a multi-tiered
ments [8]. architecture, and from the top (traditional data center) to
Unlike the cloud, cloudlets are deployed on the edge of the the bottom (user terminal equipment), the device capability
network and serve only nearby users. Cloudlet supports appli- becomes weaker, while the number of devices gets larger.
cation mobility, allowing devices to switch service requests to On the premise of clear separation of computing and states,
the nearest cloudlet during the mobile process. As shown in CloudPath extends the abstract shared storage layer to all data
Fig. 3, Cloudlet supports for application mobility relying on center nodes along the path, which reduces the complexity
three key steps. of third-party application development and deployment, and
1) Cloudlet Discovery: Mobile devices can quickly discover meanwhile keeps the RESTful development style.
the available Cloudlets around them, and choose the most The CloudPath application consists of a set of short-cycle
suitable one to offload tasks. and stateless functions that can be quickly instantiated at
2) VM Provisioning: Configuring and deploying the service any level of the CloudPath framework. Developers either tag
VM that contains the server code on the cloudlet so that it is functions to specify where (such as edges, cores, clouds, etc.)
ready to be used by the client. their codes run, or tag performance requirements (such as
 4

APP1 APP2 APP3

Intent-initiate Application-
service created services

Instance1 Instance2 Instance3

Personal Cloud Runtime

Intent Composition

Virtual Platform (STRATUS)

Networked Resources

Cloud
Local

Fig. 6. ParaDrop System [12].

Edge

Fig. 5. PCloud Architecture [10].

the computing resources of the newly generated instance
may come from multiple devices, which is equivalent to one
integrated computing device for the external applications.)
response latency) to estimate the running location. CloudPath
Thus, an application program is actually a combination of
does not migrate a running function/module, but supports
services running on the PCloud instance. For example, a
service mobility by stopping the current instance and re-
media player application can be a combination of the storage,
starting a new one at the expected location. Each CloudPath
decoding and playing services. These services could be local
node usually consists of the following six modules. 1) PathEx-
or remote but are transparent to the applications. Furthermore,
ecute: implements a serverless cloud container architecture
the PCloud system also provides basic system services, such as
that supports lightweight stateless application functions/mod-
permission management and user data aggregation, to control
ules. 2) PathStore: provides a distributed eventual consistent
the resources access of other users.
storage system that transparently manages application data
In the actual operation, the mobile application describes
across nodes. 3) PathRoute: transmits the request to the
the required resources to the PCloud through interfaces. The
most appropriate CloudPath node, according to the information
PCloud will find out the optimal resource configuration by
such as user’s location in the network, application preferences,
analyzing the description and the currently available resources,
or system status. 4) PathDeploy: dynamically deploys and
and then generates an instance to provide corresponding
removes applications on CloudPath nodes based on application
services for the application. PCloud integrates edge resources
preferences and system policies. 5) PathMonitor: provides
with cloud resources so that they can complement each other.
real-time monitoring and historical data analysis function to
The abundant resources from the cloud can make up for
applications and CloudPath nodes. It collects the metrics of
the lack of computing and storage capabilities at the edge;
other CloudPath modules on each node through the PathStore
meanwhile, due to the physical proximity, edge devices can
and presents the data to users using web pages. 6) PathInit: is
provide low-latency services to the user that cloud cannot offer.
an initialization module in the top-level data center node and
In addition, PCloud also enhances the availability of the entire
can be used to upload applications/services to the CloudPath.
system and can choose alternate resources when encountering
network and equipment failures.
C. PCloud
PCloud [10] integrates the edge computing and storage
D. ParaDrop
resources with those at the cloud to support seamless mobile
services. The architecture of PCloud is shown in Fig. 5. ParaDrop [12] is developed by the WiNGS Laboratory at
Specifically, these resources are virtualized through a spe- the University of Wisconsin-Madison. It is an edge computing
cial virtualization layer named STRATUS [11], and form a framework that makes the computing/storage resources close
distributed resource pool that can discover new resources to mobile devices and data sources available to the third party
and monitor resource changes. With the resource pool, the developers. Its goal is to bring intelligence to the network edge
runtime mechanism is responsible for resource application and in a friendly way.
allocation. Through a resource description interface, the run- ParaDrop upgrades the existing access point to an edge
time mechanism selects and combines appropriate resources computing system, which supports applications and services
based on the requirements of specified applications. After like a normal server. To isolate applications under the multi-
the resources are combined, it generates a new instance tenancy scenario, ParaDrop leverages the lightweight container
to provide corresponding services for external applications, virtualization technique. As Fig. 6 shows, the ParaDrop server
according to the resource access control policy. (Note that, (in the cloud) controls the deployment, starting and deletion
 5

processing systems, such as Apache Spark and Flink, are

2nd tier
also designed and optimized for one centralized data center.
Spoke-Worker
However, this approach cannot effectively handle the huge
data generated by a lot of devices at the edge of the network,
Manager
and the situation is even worse when the applications require
1st tier
low latency and predictability. SpanEdge [13] is a research
Hub-Worker
project of the Royal Institute of Technology in Sweden. It
unifies the cloud central node and the near-edge central node,
reduces network latency in WAN connections, and provides a
1st tier 1st tier
Hub-Worker Hub-Worker programming environment that allows the program to run near
the data source. Developers can focus on developing streaming
applications without considering where the data sources are
2nd tier 2nd tier 2nd tier 2nd tier located and distributed.
Spoke-Worker Spoke-Worker Spoke-Worker Spoke-Worker The data center in SpanEdge is composed of two levels:
the cloud data center is the first level and the edge data center
(such as the operator’s cloud, Cloudlet, or Fog) is the second
level. Partial streaming processing tasks run on the edge cen-
tral nodes to reduce latency and boost performance. SpanEdge
Fig. 7. SpanEdge Architecture [13].
uses the master-worker architecture (as shown in Fig. 7) with
one manager and multiple workers. The manager collects the
streaming processing requests and assigns tasks to the workers.
of the applications. It provides a group of APIs, via which Workers mainly consist of cluster nodes whose primary re-
the developer can monitor and manage the system resources sponsibility is to execute tasks. There are two types of workers:
and configure the running environment. The web UI is also hub-worker (first level) and spoke-worker (second level). The
provided, through which the user can directly interact with the network transmission overhead and latency are related to the
applications. geographical location and network connection status between
The design goals of ParaDrop include three aspects: multi- workers. The communication in SpanEdge is also divided into
tenancy, efficient resource utilization and dynamic application a system management communication (worker-manager) and
management. To achieve these goals, the container technology a data transmission communication (worker-worker). System
is applied to manage the multi-tenancy resources separately. management communication aims to schedule tasks between
As the resources of the edge devices are very limited, com- managers and workers, and data transmission communication
pared with the virtual machine, the container consumes less takes care of the data flow in each task. Each worker has
resources and would be more suitable for delay sensitive and an agent that handles system management operations, such as
high I/O applications. Moreover, as applications running in sending and receiving management information, monitoring
the container, ParaDrop can easily control their startup and compute nodes to ensure that they are running normally, and
revocation. periodically sending heartbeat messages to the manager to
ParaDrop is mainly used for IoT applications, especially ensure immediate recovery when the task fails.
IoT data analysis. Its advantages over traditional cloud system SpanEdge allows developers to divide the tasks into local
can be summarized as follows: a) since sensitive data can be ones and global ones. Local tasks should run on the node
processed locally, it protects the users privacy; b) WiFi access near the data source and provide only part of the required
point is only one hop away from the data source, leading to low data; global tasks are responsible for further processing the
network delay and stable connection; c) only user requested results of local tasks and aggregating all results. SpanEdge
data are transmitted to the equipment through the Internet, creates a copy of the local task on each spoke-worker which
thus cutting down the total traffic amount and saving the has all corresponding data sources. If the data is insufficient,
bandwidth of the backbone network; d) the gateway can obtain the task is dispatched to the hub-worker. The global task runs
the location information of the edge devices through radio on a separate hub-worker, and the scheduler selects the optimal
signals (e.g., the distance between devices, and the location hub-worker based on the network delay (i.e. the distance from
of the specific device), which facilitates the location-aware the spoke-worker in the network topology).
services; e) when edge devices cannot be connected to the
Internet, the edge service can still work.
F. Cloud-Sea Computing Systems
Cloud-Sea Computing Systems project [14] is a main re-
E. SpanEdge search thrust of the Next Generation Information and Com-
Streaming processing is one important type of applications munication Technology initiative (the NICT initiative) and a
in edge computing, where data is generated by various data 10-year strategic priority research initiative, launched by the
sources in different geographical locations and continuously Chinese Academy of Science in 2012. The NICT initiative
transmitted in the form of streams. Traditionally, all raw data is aims to address the three major technology challenges in
transmitted over the WAN to the data center server, and stream the coming Zettabyte era: i) improving the performance per
 6

Sea-Side Functions Cloud-Side Functions Cachier

Sensing, Interaction, Local Processing Aggregation, Request-Response, Big Data
Mobile Users Could
Sea Zone
Responses
Optimizer
Trained
HTTP 2.0+ Profiler
SeaHTTP Data
EB-Scale
Seaport Billion-Thread 1000s Structure
Severs Requests Distribution Offline
Recognizer Estimator Analysis
PB-Scale
Severs 100Ks
Sea Zone

TP~PB-Scale
Millions
CDN/CGN
SeaHTTP Fig. 9. Cachier System [16].
Seaport HTTP 1.1
Billions, TB HTTP 2.0

Trillions, KB~GB

local processing while cloud-side functions include aggrega-

Fig. 8. Cloud-Sea Computing Model. tions, request-response, and big data processing.
3) Scalability to ZB and trillion devices: This future Net
will collectively need to support trillions of sea devices and
watt by 1000 times, ii) supporting more applications from
to handle ZBs of data.
the human-cyber-physical ternary computing, and iii) enabling
4) Minimal extension to existing ecosystems: The REST 2.0
transformative innovations in devices, systems and applica-
cloud-sea computing architecture attempts to utilize existing
tions, while without polluting beneficial IT ecosystems.
Web computing ecosystems as much as possible.
In the cloud-sea computing system, “cloud” refers to the
Overall, the cloud-sea computing model is proposed to
datacenters and “sea” refers to the terminal side (the client
migrate the cloud computing function to the sea side, and it
devices, e.g., human-facing and physical world facing subsys-
focuses more on the devices at the “sea” side and the data at
tems). The design of the project can be depicted from three
the “cloud” side. Typical edge computing is more generalized
levels: the overall systems architecture level, the datacenter
and may care about any intermediate computing resources
server and storage system level, and the processor chip level.
and network resources between the “sea” and the “cloud”.
The project contains four research components: a computing
The researches in cloud-sea computing (e.g., energy efficient
model called REST 2.0 which extends the representational
computing and elastic processor designing) are consultative
state transfer (REST) [15] architectural style of Web com-
for the edge computing.
puting to cloud-sea computing, a three-tier storage system
architecture capable of managing ZBs of data, a billion-thread
datacenter server with high energy efficiency, and an elastic G. Cachier and Precog
processor aiming at energy efficiency of one trillion operations Cachier [16] and Precog [17] are two edge caching systems
per second per watt. that proposed by the researchers from Carnegie Mellon Uni-
As shown in Fig. 8, the cloud-sea computing model includes versity for image recognition. Recognition applications have
sea-side functions and cloud-side functions. The sea zone is strict response time requirements, while the computation is
expanded from the traditional cloud client, e.g., a home, an huge due to the model complexity and dataset size. With edge
office, a factory manufacturing pipeline. There can be multiple computing, we can leverage the computing resource of the
client devices inside a sea zone, and each device can be human edge nodes to process the matching, so the network delay
facing or physical world facing. In a sea zone, there is a special can be reduced. Considering that edge nodes mainly provide
device (like a home datacenter or a smart TV set) designated services to nearby users, the spatio-temporal characteristics
as the seaport for three purposes: i) a gateway interfacing the of service requests can be leveraged. From the perspective of
sea zone to the cloud, ii) a gathering point of information and caching, Cachier system proposes that edge nodes can be used
functionalities inside a sea zone, and iii) a shield protecting as “computable cache devices”, which can cache recognition
security and privacy of the sea zone. A device inside a sea results, reduce matching dataset size and improve response
zone does not communicate to the cloud directly, but through time. By analyzing the response delay model and requests’
the seaport, either implicitly or explicitly. The SeaHTTP, a characteristics, the system can dynamically adjust the size of
variant based on HTTP 2.0, is the widely-used protocol in sea the dataset on the edge nodes according to the environment
zones of the cloud-sea system to connect with the cloud. factors, thus ensuring optimal response time.
The cloud-sea computing model has four distinct features. As Fig. 9 shows, Cachier consists of the Recognizer Mod-
1) Ternary computing via sea devices: Human and physical ule, the Optimizer Module, and the Offline Analysis Module.
world entities interface and collaborate with the cyberspace The Recognizer Module is responsible for analyzing and
through the sea side. For example, users can leverage a smart matching the received figures according to the cached training
phone application to read and control a sensor device in a data and model. If there is a match, the result will be directly
home through an Internet application service. returned to the user, otherwise the figure will be transmitted
2) Cooperation with locality: A specific network computing to the cloud for recognition. The distribution estimator in
system will partition its functions between the sea side and the the Optimizer Module can use the maximum a posteriori
cloud side. Sea-side functions include sensing, interaction, and estimation to predict the request distribution. Secondly, given
 7

Containers
Geocast query

Geocast Cloud based nova-

Gclib nova compute node
Georouter
Edge Devices VMs
Conductor App Server
FocusStack API SAMonitor

OpenStack nova

Containers
Apps
LSA Subsystem
OSE Subsystem
Gclib nova
Edge Devices
LSA awareness messages
Edge Devices
OSE Messages when
in focus of attention
OSE Always-on messages

Fig. 11. FocusStack System [18].

Fig. 10. Precog System [17].

the end device.

the classification algorithm and training dataset, the cache
searching delay and the cache accuracy can also be calculated
by the Offline Analysis Module. At last, the Profiler sub- H. FocusStack
module is responsible for estimating the network delay and FocusStack [18] is developed by AT&T Labs, which sup-
cloud latency incurred by cache misses. It measures and ports the deployment of complex applications on a variety of
records the delay under corresponding distance in real time, potential IoT edge devices. Although the resources, such as
and uses the moving average filter to remove the noise data. computing, power consumption, connectivity, are limited on
By taking such information into the delay expectation time edge devices, they have the nature of mobility. Hence, it is
model, Cachier is able to calculate the optimal cache size, very important to have a system that can discover and organize
and then adjusts the cache on the edge nodes accordingly. available edge resources. FocusStack is such a system that can
Precog is an extension of Cachier. The cached data is not discover a set of edge devices with sufficient resources, deploy
only on the edge nodes, but also on the end devices which applications and run them accordingly. Thus, the developers
are used for selection calculation in order to reduce the image can focus more on the design of applications than on how to
migration between the edge nodes and the cloud. Based on find and track edge resources.
the prediction model, Precog prefetches some of the trained FocusStack consists of two parts (as shown in Fig. 11):
classifiers, uses them to recognize the images, and cooperates i) Geocast system, which provides location-based situational
with the edge nodes to efficiently complete the tasks. Precog awareness (LSA) information, and ii) OpenStack extension
pushes computing capability to the end device and leverages (OSE), which is responsible for deploying, executing, and
the locality and selectivity of the user requests to reduce the managing the containers on the edge devices. FocusStack
last mile delay. For the end device, if the cache system uses the builds a hybrid cloud of edge devices (containers) and data
same cache replacement policy as the edge node applies, it will center servers (virtual machines). When a user initiates a cloud
lead to a large number of forced misses. For a single user, the operation through the FocusStack API (such as instantiating
device only has the request information of the user, and usually a container), the LSA subsystem analyzes the scope of the
the user will not identify the same object multiple times request based on the Geocast route and sends a resource
in the near future. Therefore, Precog constructs a Markov list of geographic locations to the target area, and waits
model using the request information from the edge node to for the response from (online) edge devices that can satisfy
describe the relationship among the identified objects, and the requirements. Then, the selected edge device runs the
then predicts the potential future requests. Precog improves corresponding OpenStack operations with the help of the
the delay expectation time model by considering the network conductor module.
state, device capabilities and the prediction information from The situational aware subsystem enables one group of
edge nodes. The system architecture of Precog is illustrated devices to monitor the survival status of each other, and each
in Fig. 10. We can see that it is mainly composed of the device can update the sensing information of other devices.
feature extraction module, the offline analysis module, the The geo-processing projection service is primarily intended
optimizer module and the profiler module. The edge node to pass requests and reply messages between areas of interest,
is mainly responsible for the construction of Markov model. send out device control information (like drones) and broadcast
Based on the offline analysis results, the Markov model location-based information. The service is based on a two-
prediction results, and the network information provided by the layer network, and data can be transmitted over a dedicated
profiler module, the optimizer module determines the number WiFi network or the Internet. The sender transmits the data
of feature to be extracted as well as the data to be cached on packet to the geo-router server which tracks the location and
 8

To ensure the security of the AirBox, the EF consists of

a trusted part and an untrusted part. The untrusted part is
responsible for all network and storage interactions. Based
on OpenSGX APIs, AirBox provides four extensible APIs
to implement secure communication and storage: Remote
Attestation, Remote Authentication, Sealed Storage, and EF
Defined Interface. AirBox supports a variety of edge features
such as aggregation, buffering, and caching.

Fig. 12. AirBox Architecture [19].

J. Firework
Wayne State University’s MIST Lab proposes a program-
ming model for edge computing—Firework [20]. In the model
metadata of each edge device, and then the data packet is sent of Firework, all services/functions are represented in a data
to each device (including edge devices and a cloud device view of datasets and functions which can be the results of
running a SAMonitor instance) in the specified area based on processing with their own data or secondary processing with
the address. Location and connection statuses are maintained other services/datasets. A system consisting of nodes that
by the geo-routing database (GRDB). The GCLib is a software apply the Firework model is called the Firework system. In the
framework that provides data acquisition services, and it runs Firework system, instead of dividing nodes into edge nodes
on edge devices and cloud applications that use SAMonitor. and cloud ones, they are all considered as Firework nodes.
Any application or service in need of the state of the device in Through the mutual invocation of the Firework nodes, the
the area needs a SAMonitor component to communicate with data can be distributed and processed on each node. Along
the LSA subsystem. The application server makes a request to the path of data transmission, the edge nodes perform a series
SAMonitor through the FocusStack API, and then SAMonitor of calculations upon the data, thus forming a “computational
builds a real-time graph of the current area and returns a flow”. The beginning of the computational flow could be user
list of available edge devices. The regional graph is sent to terminals, edge servers close to the user, edge servers close to
the conductor in the OSE, which is responsible for checking the cloud, or cloud nodes.
whether the devices are capable of running the tasks, whether In the Firework system, the data processing service is split
the pre-defined policy rules are met, and so on. After that, the into multiple sub-services, and the scheduling of the data
available edge device list is submitted to the application server processing is performed at the following two layers.
and the server selects devices to be used. OSE manages and 1) Same sub-service layer scheduling: A Firework node
deploys the program through the OpenStack Nova API. The can cooperate with another for the same sub-services in the
edge devices run a custom version of Nova Compute to interact surrounding area to achieve optimal response time. For idle
with the local Docker to manage the container. The container Firework nodes, the system can schedule sub-service programs
on the edge devices supports all OpenStack services, including onto them, such that a cluster providing specific sub-service
access to virtual networks and application-based granularity is formed dynamically and complete the service faster.
configuration. 2) Computational flow layer scheduling: Firework nodes
along the computational flow can cooperate with each other
and dynamically schedule execution nodes to achieve an
I. AirBox optimal solution. For example, depending on the states of the
AirBox [19] is a secure, lightweight and flexible edge network, the system can choose Firework nodes for service
function system developed by Georgia Institute of Technology. providing based on the nodes’ locations (e.g., selecting those
It supports fast and flexible edge function loading and provides closest to the users).
service security and user privacy protection. The Edge Func- As shown in Fig. 13, Firework divides the nodes into com-
tion (EF) in AirBox is defined as a service that can be loaded puting nodes and managers, depending on the type of service
on an edge node, and the software stack that supports the edge those nodes provide. In general, each node with the Firework
function is named Edge Function Platform (EFP). model has three modules: the job management module, the
As shown in Fig. 12, the AirBox consists of two parts: the actuator management module, and the service management
AB console and the AB provisioner. The back-end service module.
manager deploys and manages the EF on the edge nodes 1) Service management module: This type of module is
through the AB console. The AB provisioner which runs at designed for the management of data views. It provides
the edge is responsible for providing dynamic EF seamlessly. interfaces to update the data view, as well as relevant programs
The edge functions are implemented through system-level for data processing.
containers with minimal constraints on developers. Security is 2) Job management module: This type of module is re-
enhanced by using the hardware security mechanisms like Intel sponsible for the scheduling, monitoring, and evaluation of
SGX. AirBox provides centrally controlled backend services in the task executions. When the local computing resources are
discovering edge nodes and registering them. The AB console insufficient, the module can look into the node list and the
is a web-based management system, which activates the docker data view and make resource re-scheduling at the same sub-
start-up process on the edge node with AB provisioner. service layer. When the sub-service is running, the module
 9

TABLE I
S UMMARY OF EDGE COMPUTING SYSTEMS

Edge
Application End Computation
Computing Edge Nodes Features/Targets
Scenarios Devices Architecture
Systems
Hybrid Lightweight
Cloudlet Mobile devices Cloudlet
(3-tier) VM migration
Mobile devices, Hybrid Resource integration,
PCloud Mobile devices
local server, PC (3-tier) dynamic allocation
General Hybrid Hardware,
ParaDrop IoT devices Home gateway
Usage (3-tier) developer support
Scenario Mobile devices, Hybrid Figure recognition,
Cachier & Precog Mobile devices
local server, PC (3-tier) identification
Hybrid Location-based info,
FocusStack IoT devices Router, server
(3-tier) OpenStack extension
Local cluster, Hybrid Streaming processing,
SpanEdge IoT devices
Cloudlet, Fog (2-tier) local/global task
Mobile devices, Hybrid
AirBox IoT devices Security
local server, PC (3-tier)
Multi-level Hybrid
CloudPath Mobile devices Path computing
data centers (multi-tier)
Two-layer
Firework Firework.Node Firework.Node Programming model
scheduling
Hybrid Minimal extension,
Cloud-Sea Sea Seaport
(3-tier) transparency
Hybrid General
Vehicular OpenVDAP CAVs XEdge
(2-tier) platform
Data
Smartphones Hybrid In-vehicle
Analytics SafeShareRide Smartphones
and vehicles (2-tier) security
Smart Edge Smart
Smart Vigilia Hubs
home devices only Home Security
Home
Smart Edge Smart
HomePad Routers
home devices only Home Security
Edge Low
Video LAVEA ∼ ∼
or cloud latency response
Stream
Cameras and
Analytics Hybrid Resource-accuracy
VideoEdge Cameras private
(3-tier) tradeoff
clusters
Video Autonomous Portable Edge Bandwidth
on drones drones edge computers only saving
Resource utilization
Virtual Individual Edge
MUVR Smartphones efficiency
Reality households only
optimization

upper layer functions could be decoupled, so that the nodes of

Firework system are not limited to a certain type of devices,
and the data processing environment is not limited to a certain
type of computing platform.

K. Other Edge Computing Systems

The edge systems introduced above depict some typical
and basic innovations on the exploitation of edge analytics
for highly-responsive services. The previous systems are used
in general cases, which lay the foundation of further devel-
opment. We highlight that, in addition to these efforts, there
Fig. 13. An example of a Firework instance that consists of heterogeneous are many other edge systems tuned for a series of different
computing platforms [20].
application scenarios.In Table I, we briefly summarize the
general and application-specific edge systems, which leverage
different kinds of edge nodes to serve diverse end devices
can also provide necessary monitoring information and give using either a hybrid or edge only computation architecture.
feedback to other upstream and downstream nodes for flow Compared to both on-board computation and cloud-based
layer scheduling. computation, edge computing can provide more effective data
3) Actuator management module: This type of module is analytics with lower latency for the moving vehicles [21].
mainly responsible for managing all hardware resources and In [21], an open full-stack edge computing-based platform
hosting the execution processes of different tasks. With the OpenVDAP is proposed for the data analytics of connected
help of this module, the device, running environment and the and autonomous vehicles (CAVs). OpenVDAP proposes sys-
 10

tematic mechanisms, including varied wireless interfaces, to Portable edge computers are required here to support dynamic
utilize the heterogeneous computation resources of nearby transportation during a mission. Totally four different video
CAVs, edge nodes, and the cloud. For optimal utilization of the transmission strategies are presented to build an adaptive and
resources, a dynamic scheduling interface is also provided to efficient computer vision pipeline. In addition to the analytics
sense the status of available resources and to offload divided work (e.g., object recognition), the edge nodes also train filters
tasks in a distributed way for computation efficiency. Safe- for the drones to avoid the uploading of the uninteresting video
ShareRide is an edge-based attack detection system addressing frames.
in-vehicle security for ridesharing services [22]. Its three In order to provide flexible virtual reality (VR) on unteth-
detection stages leverage both the smartphones of drivers and ered smartphones, edge computing can be useful to transport
passengers as edge computing platform to collect multime- the heavy workload from smartphones to their nearby edge
dia information in vehicles. Specifically, speech recognition cloud [29]. However, the rendering task of the panoramic VR
and driving behavior detection stages are first carried out frames (i.e., 2GB per second) will also saturate the individual
independently to capture in-vehicle danger, and video capture households as common edge in the house. In [29], the MUVR
and uploading stage is activated when abnormal keywords or system is designed to support multi-user VR with efficient
dangerous behaviors are detected to collect videos for cloud- bandwidth and computation resources utilization. MUVR is
based analysis. By using such an edge-cloud collaborative built on a basic observation that the VR frames being rendered
architecture, SafeShareRide can accurately detect attacks in- and transmitted to different users are highly redundant. For
vehicle with low bandwidth demand. computation efficiency, MUVR maintains a two-level hierar-
Another scenario that edge computing can play an impor- chical cache for invariant background at the edge and the user
tant role is the IoT devices management in the smart home end to reuse frames whenever necessary. Meanwhile, MUVR
environment. Wherein, the privacy issue of the wide range of transmits a part of all frames in full and delivers the distinct
home-devices is a popular topic. In [23], the Vigilia system portion for the rest frames to further reduce the transmission
is proposed to harden smart home systems by restricting the costs.
network access of devices. A default access deny policy and
an API-granularity device access mechanism for applications III. O PEN S OURCE E DGE C OMPUTING P ROJECTS
are adopted to enforce access at the network level. Run time Besides the designed edge computing systems for specific
checking implemented in the routers only permits those de- purposes, some open source edge computing projects have
clared communications, thus helping users secure their home- also been launched recently. The Linux Foundation pub-
devices. Similarly, the HomePad system in [24] also proposes lished two projects, EdgeX Foundry in 2017 and Akraino
to execute IoT applications at the edge and introduces a Edge Statck [30] in 2018. The Open Network Founda-
privacy-aware hub to mitigate security concerns. Homepad tion(ONF) launched a project namely CORD (Central Office
allows users to specify privacy policy to regulate how ap- Re-architected as a Datacenter) [31]. The Apache Software
plications access and process their data. Through enforcing Foundation published Apache Edgent. Microsoft published
applications to use explicit information flow, Homepad can use Azure IoT Edge in 2017 and announced it as an open source
Prolog rules to verify whether applications have the ability to in 2018.
violate the defined privacy policy at install time. Among them, CORD and Akraino Edge Stack focus on
Edge computing has also been widely used in the analysis providing edge cloud services; EdgeX Foundry and Apache
of video stream. LAVEA is an edge-based system built for Edgent focus on IoT and aim to solve problems which bring
latency-aware video analytics nearby the end users [25]. In difficulties to practical applications of edge computing in IoT;
order to minimize the response time, LAVEA formulates an Azure IoT Edge provides hybrid cloud-edge analytics, which
optimization problem to determine which part of tasks to be helps to migrate cloud solutions to IoT devices.
offloaded to the edge computer and uses a task queue priori-
tizer to minimize the makespan. It also proposes several task
placement schemes to enable the collaboration of nearby edge A. CORD
nodes, which can further reduce the overall task completion CORD is an open source project of ONF initiated by AT&T
time. VideoEdge is a system that provides the most promising and is designed for network operators. Current network in-
video analytics implementation across a hierarchy of clusters frastructure is built with closed proprietary integrated systems
in the city environment [26]. A 3-tier computation architecture provided by network equipment providers. Due to the closed
is considered with deployed cameras and private clusters as property, the network capability cannot scale up and down
the edge and remote server as the cloud. The hierarchical dynamically. And the lack of flexibility results in inefficient
edge architecture is also adopted in [27] and is believed utilization of the computing and networking resources. CORD
to be promising in processing live video stream at scale. plans to reconstruct the edge network infrastructure to build
Technically, VideoEdge searches thousands of combinations datacenters with SDN [32], NFV [33] and Cloud technolo-
of computer vision components implementation, knobs, and gies. It attempts to slice the computing, storage and network
placement and finds a configuration to balance the accuracy resources so that these datacenters can act as clouds at the
and resource demands using an efficient heuristic. In [28], a edge, providing agile services for end users.
video analytics system for autonomous drones is proposed, CORD is an integrated system built from commodity hard-
where edge computing is introduced to save the bandwidth. ware and open source software. Fig. 14 shows the hardware
 11

network operating system which is used to manage network

CORD Virtualized Central Office components like the switching fabric and provide communi-
cation services to end-users. XOS provides a control plane
Mobile Enterprise Residential to assemble and compose services. Other software projects
Cord Controller (XOS) provide component capabilities, for example, vRouter(Virtual
Subscribers
Router) provides with virtual routing functionality.
Mobile The edge of the operator network is a sweet spot for edge
Mobile Access
computing because it connects customers with operators and is
Enterprise
Enterprise
Access WAN close to customers’ applications as data sources. CORD takes
Residential edge computing into consideration and moves to support edge
Access Shared Server Resources ROADM(Core)
Residential computing as a platform to provide edge cloud services (from
the released version 4.1). CORD can be deployed into three
solution, M-CORD (Mobile CORD), R-CORD (Residential
CORD) and E-CORD (Enterprise CORD) for different use
Fig. 14. Hardware Architecture of CORD. cases. M-CORD focuses on mobile network, especially 5G
network, and it plans to disaggregate and virtualize cellular
network functions to enable services be created and scaled
ACCESS-AS-A- SUBSCRIBER-AS-A- INTERNET-AS-A- MONITORING-AS-
dynamically. This agility helps to provide multi-access edge
CDN
SERVICE SERVICE SERVICE A-SERVICE
services for mobile applications. For those use cases like
XOS
driverless cars or drones, users can rent the edge service to
run their edge applications. Similarly, R-CORD and E-CORD
are designed to be agile service delivery platforms but for
CEILOMETER vSG vCDN VTN
FABRIC MULTICAST
vOLT vROUTER
different users, residential and enterprise users relatively.
CONTROL CONTROL
So far, the deployment of CORD is still under test among
OPENSTACK/DOCKER ONOS
network operators, and more researches are needed to combine
CORD with various edge applications.

Fig. 15. Software Architecture of CORD. B. Akraino Edge Stack

Akraino Edge Stack, initiated by AT&T and now hosted by
Linux Foundation, is a project to develop a holistic solution
architecture of CORD [31]. It uses commodity servers that for edge infrastructure so as to support high-availability edge
are interconnected by a Fabric of White-box switches. White- cloud services [30]. An open source software stack, as the
box switch [34] is a component of SDN switch, which software part of this solution, is developed for network carrier
is responsible to regulate the flow of data according to to facilitate optimal networking and workload orchestration
SDN controller. These commodity servers provide computing, for underlying infrastructure in order to meet the need of
storage resources, and the fabric of switches are used to edge computing such as low latency, high performance, high
build the network. This switching fabric is organized to a availability, scalability and so on.
Spine-Leaf topology [35], a kind of flat network topology To provide a holistic solution, Akraino Edge Stack has
structure which adds a horizontal network structure parallel a wide scope from infrastructure layer to application layer.
to the trunk longitudinal network structure, and then adds Fig. 16 [30] shows the scope with three layers. In the ap-
corresponding switching network on the horizontal structure. plication layer, Akraino Edge Stack wants to create a virutal
Comparing to traditional three-tier network topology, it can network function (VNF) ecosystem and calls for edge applica-
provide scalable throughput for greater East-to-West network tions. The second layer consists of middleware which supports
traffic, that is, traffic coming from network diagram drawings applications in the top layer. In this layer, Akraino Edge Stack
that usually depict local area network (LAN) traffic hori- plans to develop Edge API and framework for interoperability
zontally. In addition, specialized access hardware is required with 3rd party Edge projects such as EdgeX Foundry. At
to connect subscribers. The subscribers can be divided into the bottom layer, Akraino Edge Stack intends to develop
three categories for different use cases, mobile subscribers, an open source software stack for the edge infrastructure in
enterprise subscribers and residential subscribers. Each cat- collaboration with upstream communities. It interfaces with
egory demands different access hardware due to different and maximizes the use of existing open source projects such
access technologies. In terms of software, Fig. 15 shows the as Kubernetes, OpenStack and so on. Akraino Edge Stack
software architecture of CORD [31]. Based on the servers provides different edge use cases with blueprints, which are
and the fabric of switches, OpenStack provides with IaaS declarative configurations of entire stack including hardware,
capability for CORD, it manages the compute, storage and software, point of delivery, etc [30]. The application domains
networking resources as well as creating virtual machines and of these blueprints start from Telco industry and are expected
virtual networks. Docker is used to run services in containers to applied in more domains like Enterprise and industrial IoT.
for isolation. ONOS(Open Network Operating System) is a Now Akraino Edge Stack has put forward several blueprints
 12

Services, so that it can support for any combination of device

interfaces and protocols by programming.
EdgeX consists of a collection of microservices, which
allows services to scale up and down based on device ca-
pability. These microservices can be grouped into four service
layers and two underlying augmenting system services, as
depicted in Fig. 17. The four service layers include Device
Services Layer, Core Services Layer, Supporting Services
Layer and Export Services Layer, respectively; the two un-
derlying augmenting system services are System Management
and Security, respectively. Each of the six layers consists of
several components and all components use a common Restful
Fig. 16. Akraino Edge Stack’s Scope.
API for configuration.
1) Device Services Layer: This layer consists of Device
Services. According to the Device Profiles, Device Service
such as Micro-MEC and Edge Media Processing. Micro-MEC Layer converts the format of the data, sends them to Core
intends to develop a new service infrastructure for smart cities, Services Layer, and translates the command requests from the
which enables developing services for smart city and has high Core Services Layer.
data capacity for citizens. Edge Media Processing intends to 2) Core Services Layer: This layer consists of four com-
develop a network cloud to enable real-time media processing ponents: Core Data, Command, Metadata, and Registry &
and edge media AI analytics with low latency. Configuration. Core Data is a persistence repository as well as
As an emerging project, Akraino Edge Stack has been taken a management service. It stores and manages the data collected
to execution since August 2018. Thus more researches need from the south side objects. Command is a service to offer
to be done with the development of this project. the API for command requests from the north side to Device
Services. Metadata is a repository and management service for
C. EdgeX Foundry metadata about IoT objects. For example, the Device Profiles
are uploaded and stored in Metadata. Registry & Configu-
EdgeX Foundry is a standardized interoperability frame- ration provides centralized management of configuration and
work for IoT edge computing, whose sweet spots are edge operating parameters for other microservices.
nodes such as gateways, hubs, and routers [36]. It can con-
3) Supporting Services Layer: This layer is designed to
nect with various sensors and devices via different protocols,
provide edge analytics and intelligence [33]. Now the Rules
manage them and collect data from them, and export the data
Engine, Alerting and Notification, Scheduling and Logging
to a local application at the edge or the cloud for further
microservices are implemented. A target range of data can be
processing. EdgeX is designed to be agnostic to hardware,
set to trigger a specific device actuation as a rule and Rules
CPU, operating system, and application environment. It can
Engine helps realize the rule by monitoring the incoming data.
run natively or run in docker containers.
Alerting and Notifications can send notifications or alerts to
Fig. 17 [36] shows the architecture of EdgeX Foundry.
another system or person by email, REST callback or other
“South side” at the bottom of the figure includes all IoT
methods when an urgent actuation or a service malfunction
objects, and the edge of the network that communicates
happens. The scheduling module can set up a timer to regularly
directly with those devices, sensors, actuators and other IoT
clean up the stale data. Logging is used to record the running
objects to collect the data from them. Relatively, “north side”
information of EdgeX.
at the top of the figure includes the Cloud (or Enterprise
system) where data are collected, stored, aggregated, analyzed, 4) Export Services Layer: This layer connects EdgeX with
and turned into information, and the part of the network that North Side and consists of Client Registration and Export
communicates with the Cloud. EdgeX Foundry connects these Distribution. Client Registration enables clients like a specific
two sides regardless of the differences of hardware, software cloud or a local application to register as recipients of data
and network. EdgeX tries to unify the manipulation method from Core Data. Export Distribution distributes the data to
of the IoT objects from the south side to a common API, so the Clients registered in Client Registration.
that those objects can be manipulated in the same way by the 5) System Management and Security: System Management
applications of north side. provides management operations including installation, up-
EdgeX uses a Device Profile to describe a south side object. grade, starting, stopping and monitoring, as EdgeX is scalable
A Device Profile defines the type of the object, the format of and can be deployed dynamically. Security is designed to
data that the object provides, the format of data to be stored protect the data and command of IoT objects connected with
in EdgeX and the commands used to manipulate this object. EdgeX Foundry.
Each Device Profile involves a Device Service, which is a EdgeX is designed for the user cases dealing with multitudes
service that converts the format of the data, and translates of sensors or devices, such as automated factories, machinery
the commands into instructions that IoT objects know how to systems and lots of other cases in IoT. Now EdgeX Foundry
execute. EdgeX provides SDK for developers to create Device is in the rapid upgrading phase, and more features will be
 13

Fig. 17. Architecture of EdgeX Foundry.

is responsible for data analysis. The streams of data can be

Provider
filtered, split, transformed or processed by other operations
Topology A
Tstream in a topology. Edgent uses a provider to act as a factory to
Filter create and execute topologies. To build an Edgent application,
users should firstly get a provider, then create a topology and
split
data
Connector data add the processing flow to deal with the streams of data, and
Connector
control
Transform
finally submit the topology. The deployment environments of
The Cloud
Edgent are Java 8, Java 7 and Android.
...
Edgent provides APIs for sending data to back-end systems
Sensors or devices and now supports MQTT, IBM Watson IoT Platform, Apache
Topology B Kafka and custom message hubs. Edgent applications analyze
... the data from sensors and devices, and send the essential data
Edgent application on edge devices to the back-end system for further analysis. For IoT use cases,
Edgent helps reduce the cost of transmitting data and provide
local feedback.
Fig. 18. Model of the Edgent Applications.
Edgent is suitable for use cases in IoT such as intelligent
transportation, automated factories and so on. In addition, the
data in Edgent applications are not limited to sensor readings,
added in future releases. An EdgeX UI is in development as they can also be files or logs. Therefore, Edgent can be applied
a web-based interface to add and manage the devices. to other use cases. For example, it can perform local data
analysis when embedded in application servers, where it can
D. Apache Edgent analyze error logs without impacting network traffic [37].
Apache Edgent, which was known as Apache Quarks pre-
viously, is an Apache Incubator project at present. It is an
E. Azure IoT Edge
open source programming model for lightweight runtime data
analytics, used in small devices such as routers and gateways Azure IoT Edge, provided by Microsoft Azure as a cloud
at the edge. Apache Edgent focuses on data analytics at the service provider, tries to move cloud analytics to edge devices.
edge, aiming to accelerate the development of data analysis. These edge devices can be routers, gateways or other devices
As a programming model, Edgent provides API to build which can provide computing resources. The programming
edge applications. Fig. 18 illustrates the model of the Edgent model of Azure IoT Edge is the same as that of other Azure
applications. Edgent uses a topology as a graph to represent IoT services [38] in the cloud, which enables user to move
the processing transformation of streams of data which are their existing applications from Azure to the edge devices for
abstracted to a Tstream class. A connector is used to get lower latency. The convenience simplifies the development of
streams of data from external entities such as sensors and edge applications. In addition, Azure services like Azure Func-
devices in physical world, or to send streams of data to tions, Azure Machine Learning and Azure Stream Analytics
back-end systems like a cloud. The primary API of Edgent can be used to deploy complex tasks on the edge devices such
 14

this module image to the edge device with the help of the IoT
instantiate deploy module Edge interface. Then the IoT Edge receives the deployment
module IoT Edge
ensure running agent information, pulls the module image, and instantiates the
report module
IoT Edge
status module instance.
Data
hub
Analystics Azure IoT Edge has wide application areas. Now it has ap-
Module IoT Hub
data
data plication cases on intelligent manufacturing, irrigation system,
Meachine
Learning drone management system and so on. It is worth noting that
Module data
Azure IoT Edge is open-source but the Azure services like
Azure
Fuctions Azure Functions, Azure Machine Learning and Azure Stream
Module IoT Edge cloud
interface are charged.
IoT Edge runtime
sensors
and
Azure IoT edge device
devices
F. Comparative Study
We summarize the features of the above open source edge
Fig. 19. Diagram of Azure IoT Edge.
computing systems in Table II. Then we compare them from
different aspects in Table II, including the main purpose of the
systems,application area, deployment, target user, virtualiza-
as machine learning, image recognition and other tasks about tion technology, system characteristic, limitations, scalability
artificial intelligence. and mobility. We believe such comparisons give better under-
Azure IoT Edge consists of three components: IoT Edge standings of current open source edge computing systems.
modules, IoT Edge runtime and a cloud-based interface as 1) Main purpose: The main purpose shows the target
depicted in Fig. 19. The first two components run on edge problem that a system tries to fix, and it is a key factor
devices, and the last one is an interface in the cloud. IoT Edge for us to choose a suitable system to run edge applications.
modules are containerized instances running the customer code As an interoperability framework, EdgeX Foundry aims to
or Azure services. IoT Edge runtime manages these modules. communicate with any sensor or device in IoT. This ability is
The cloud-based interface is used to monitor and manage the necessary for edge applications with data from various sensors
former two components, in other words, monitor and manage and devices. Azure IoT Edge offers an efficient solution to
the edge devices. move the existing applications from cloud to edge, and to
IoT Edge modules are the places that run specific applica- develop edge applications in the same way with the cloud
tions as the units of execution. A module image is a docker applications. Apache Edgent helps to accelerate the develop-
image containing the user code. A module instance, as a ment process of data analysis in IoT use cases. CORD aims
docker container, is a unit of computation running the module to reconstruct current edge network infrastructure to build
image. If the resources at the edge devices are sufficient, these datacenters so as to provide agile network services for end-user
modules can run the same Azure services or custom applica- customers. From the view of edge computing, CORD provides
tions as in the cloud because of the same programming model. with multi-access edge services. Akraino Edge Stack provides
In addition, these modules can be deployed dynamically as an open source software stack to support high-availability edge
Azure IoT Edge is scalable. clouds.
IoT Edge runtime acts as a manager on the edge devices. It 2) Application area: EdgeX Foundry and Apache Edgent
consists of two modules: IoT Edge hub and IoT Edge agent. both focus on IoT edge, and EdgeX Foundry is geared toward
IoT Edge hub acts as a local proxy for IoT Hub which is a communication with various sensors and devices, while Edgent
managed service and a central message hub in the cloud. As a is geared toward data analysis. They are suitable for intelligent
message broker, IoT Edge hub helps modules communicate manufacturing, intelligent transportation and smart city where
with each other, and transport data to IoT Hub. IoT Edge various sensors and devices generate data all the time. Azure
agent is used to deploy and monitor the IoT Edge modules. IoT Edge can be thought as the expansion of Azure Cloud. It
It receives the deployment information about modules from has an extensive application area but depends on the compute
IoT Hub, instantiates these modules, and ensures they are resources of edge devices. Besides, it is very convenient to
running, for example, restarts the crashed modules. In addition, deploy edge applications about artificial intelligence such as
it reports the status of the modules to the IoT hub. machine learning and image recognition to Azure IoT Edge
IoT Edge cloud interface is provided for device manage- with the help of Azure services. CORD and Akraino Edge
ment. By this interface, users can create edge applications, Stack support edge cloud services, which have no restriction
then send these applications to the device and monitor the on application area. If the edge devices of users don’t have
running status of the device. This monitoring function is useful sufficient computing capability, these two systems are suitable
for use cases with massive devices, where users can deploy for users to run resource-intensive and interactive applications
applications to devices on a large scale and monitor these in connection with operator network.
devices. 3) Deployment: As for the deployment requirements,
A simple deployment procedure for applications is that: EdgeX Foundry, Apache Edgent and Azure IoT Edge are
users choose a Azure service or write their own code as an deployed in edge devices such as routers, gateways, switchers
application, build it as an IoT Edge module image, and deploy and so on. Users can deploy EdgeX Foundry by themselves,
 15

TABLE II
C OMPARISON OF O PEN E DGE S YSTEM C HARACTERISTICS

Azure IoT Akraino

Aspect EdgeX Foundry Apache Edgent CORD
Edge Edge Stack
User access Restful API Web service, API or
API N/A
interface or EdgeX UI Command-line XOS-GUI
OS support Various OS Various OS Various OS Ubuntu Linux
Programming Java, .NET, C, Shell script,
Not provided Java N/A
framework Python, etc. Python
Accelerate the Transform edge of Support edge
Provide with Support hybrid
development the operator network clouds with an
Main purpose Interoperability cloud-edge
process of into agile service open source
for IoT edge analytics
data analysis delivery platforms software stack
Application area IoT Unrestricted IoT Unrestricted Unrestricted
Deployment Dynamic Dynamic Static Dynamic Dynamic
Network
Target user General users General users General users Network operators
operators
Virtualization Virtual Machine Virtual Machine
Container Container JVM
technology and Container and Container
System A common API for Powerful APIs for Widespread Widespread
characteristics device management Azure services data analytics edge clouds edge clouds
Lack of
Azure Services Limited to Unable to Unable to
Limitation programable
is chargeable data analytics be offline be offline
interface
Scalability Scalable Scalable Not scalable Scalable Scalable
Mobility Not support Not support Not support Support Support

add or reduce microservices dynamically, and run their own 6) System characteristic: System characteristics show the
edge applications. Differently, users need the help of cloud- unique features of the system, which may help users to
based interface to deploy Azure IoT Edge and develop their develop, deploy or monitor their edge applications. It will
edge applications. CORD and Akraino Edge Stack are de- save lots of workload and time if making good use of these
signed for network operators, who need fabric switches, access characteristics. EdgeX Foundry provides a common API to
devices, network cards and other related hardware apart from manage the devices, and this brings great convenience to de-
compute machines. Customers have no need to think about ploying and monitoring edge applications in large scale. Azure
the hardware requirements and management process of the IoT Edge provides powerful Azure services to accelerate the
hardware, but to rent the services provided by the network development of edge applications. Apache Edgent provides a
operators like renting a cloud service instead of managing a series of functional APIs for data analytics, which lowers the
physical server. difficulty and reduces the time on developing edge analytic
applications. CORD and Akraino Edge Stack provide with
4) Target user: Though these open source systems focus multi-access edge services on edge cloud. We only need to
on edge computing, their target users are not the same. keep connection with operator network, and we can apply for
EdgeX Foundry, Azure IoT Edge and Apache Edgent have these services without the need to deploy an edge computing
no restriction on target users. Therefore, every developer can system on edge devices by ourselves.
deploy them into local edge devices like gateways, routers
7) Limitation: This subsection discusses the limitation of
and hubs. Differently, CORD and Akraino Edge Stack are
the latest version of them to deploy edge applications. The
created for network operators because they focus on edge
lastest version of EdgeX Foundry has not provided a pro-
infrastructure.
grammable interface in its architecture for developers to write
5) Virtualization technology: Virtualization technologies their own applications. Although EdgeX allows us to add
are widely used nowadays. Virtual machine technology can custom implementations, it demands more workload and time.
provide better management and higher utilization of resources, As for Azure IoT Edge, though it is open-source and free,
stability, scalability and other advantages. Container technol- Azure services are chargeable as commercial software. For
ogy can provide services with isolation and agility but with Apache Edgent, it is lightweight and it focuses on only data
negligible overhead, which can be used in edge devices [39]. analytics. As for CORD and Akraino Edge Stack, these two
Using OpenStack and Docker as software components, CORD systems demand stable network between data sources and the
and Akraino Edge Stack use both of these two technologies to operators because the edge applications are running on the
support edge cloud. Different edge devices may have different edge of operator network rather than local devices.
hardware and software environment. For those edge systems 8) Scalability: Increasing applications at edge make the
which are deployed on edge devices, container is a good network architecture more complex and the application man-
technology for services to keep independence in different agement more difficult. Scalability is one major concern in
environment. Therefore, EdgeX Foundry and Azure IoT Edge edge computing. Among these edge computing systems, Azure
choose to run as docker containers. As for Edgent, Edgent IoT Edge, CORD and Akraino Edge Stack apply docker
applications run on JVM. technology or virtual machine technology to support users to
 16

Device EdgeX
management Foundry Top Cloud Layer
capability
Local solutions while
supporting
third-party clouds

Accelerate the Apache

development of Edgent
Deployment by analytics
developers

Need for Azure IoT Edge,

Choice of edge third-party AWS IoT Greengrass Middle Edge Server Layer
computing clouds or Link IoT Edge
systems

Third-party CORD or Akraino

services Edge Stack

Bottom (Battery-powered) Device Layer

Fig. 20. The choice of open-source tools in different scenarios.
Fig. 21. The Three-layer Edge Computing Paradigm from a Power Source
View.
scale up or down their applications efficiently by adding or
deleting module images. EdgeX Foundry is also a scalable
platform that enables users to dynamically add or reduce optimized in the cloud. Link IoT Edge provides Function
microservices to adapt to the actual needs. But Apache Edgent Compute and other functions. Based on the application re-
is not scalable enough because every Edgent application is a quirements, a suitable system could be chosen among them
single Java application and performance cannot be changed by taking account the functions they provide.
dynamically. In the third scenario, suppose the users expect to directly
9) Mobility: For EdgeX Foundry, Apache Edgent and leverage third-party services to deploy edge applications with-
Azure IoT Edge, once the applications are executed on some out any hardware or software system locally. In this case,
edge devices, they cannot be dynamically migrated to other edge systems such as CORD and Akraino Edge Stack are
devices. CORD and Akraino Edge Stack, deployed in the suitable. The users could choose one of them dependent on
telecom infrastructure, support mobile edge services through their application requirements. These systems are deployed
mobile access network like 4G/5G. The mobility of these by telecom operators. And telecom operators are also the
systems meet the need of cases such as unmanned cars and providers of network services. Thus when there exist special
drones. network requirements of edge applications, these systems
10) Scenarios: We discuss the choice of open-source tools could satisfy the requirements. For example, edge computing
from the perspective of the following three scenarios, as shown applications on unmanned vehicles or drones need the support
in Fig. 20. of wireless telecom networks (such as 4G/5G), in this case, a
mobile edge computing service provided by these systems is
In the first scenario, suppose IoT edge applications are
a good choice.
running on local area network (LAN), and the local enterprise
system is regarded as back-end system with no need for third- In addition to the systems described above, there are other
party clouds. In this case, EdgeX Foundry or Apache Edgent emerging open source projects. Device Management Edge, as
are favorable because they enable users to build and control part of Mbed IoT Platform published by ARM, is responsible
their own back-end system without being bound to any specific for edge computing and provides the ability to access, manage
cloud platform. Furthermore, for the sake of managing and and control Edge devices. KubeEdge, released by Huawei,
controlling edge devices on a large scale, EdgeX Foundry provides edge nodes with native containerized application
is a better choice for good device management capability. orchestration capabilities.
If considering data analysis, Apache Edgent is preferable
to EdgeX Foundry. It provides a programming model to IV. E NHANCING E NERGY E FFICIENCY OF E DGE
accelerate the development of edge analytic applications and C OMPUTING S YSTEMS
a set of powerful APIs for data processing. In the design of an edge computing system, energy con-
In the second scenario, suppose the cloud services push to sumption is always considered as one of the major concerns
the edge of network to improve the quality. In this case, Auzre and evaluated as one of the key performance metrics. In this
IoT Edge provides a convenient way for developers to migrate section, we review the energy efficiency enhancing mecha-
the applications from the cloud to the edge devices, and to nisms adopted by the state-of-the-art edge computing systems,
leverage third-party high value services or functions when from the view of top cloud layer, middle edge server layer and
developing edge systems. Besides, AWS IoT Greengrass and bottom device layer, respectively (the three-layer paradigm is
Link IoT Edge, which are published by Amazon and Alibaba illustrated by Fig. 21).
Cloud, are good choices as the competitive projects. More
specifically, Azure IoT Edge provides Azure services such as
Azure Functions, Azure Stream Analytics, and Azure Machine A. At the Top Cloud Layer
Learning. AWS IoT Greengrass can run AWS Lamda functions For cloud computing, a centralized DC can comprise thou-
and machine learning models that are created, trained, and sands of servers and thus consume enormous energy [40].
 17

As an alternative to the cloud computing, does the edge/fog that the obtained results with DockerCap is comparable to that
computing paradigm consume more or less energy? Different from the power capping solution provided by the hardware
points of view have been given: some claim that decentralized (Intel RAPL).
data storage and processing supported by the edge computing An energy aware edge computer architecture is designed
architecture are more energy efficient [41], [42], while some to be portable and usable in the fieldwork scenarios in [51].
others show that such distributed content delivery may con- Based on the architecture, a high-density cluster prototype is
sume more energy than that of the centralized way [43]. built using the compact general-purpose commodity hardware.
The authors in [44] give a thorough energy analysis for Power management policies are implemented in the prototype
applications running over the centralized DC (i.e., under cloud to enable the real-time energy-awareness. Through various
mode) and decentralized nano DCs (i.e., under fog mode), experiments, it shows that both the load balance strategies and
respectively. The results indicate that the fog mode may cluster configurations have big impacts on the system energy
be with a higher energy efficiency, depending on several consumption and responsiveness.
system design factors (e.g., type of application, type of access 2) Green-energy powered sustainable computing: Dual en-
network, and ratio of active time), and those applications that ergy sources are employed to support the running of a fog
generate and distribute a large amount of data in end-user computing based system in [52], where solar power is utilized
premises result in best energy saving under the fog mode. as the primary energy supply of the fog nodes. A comprehen-
Note that the decentralized nano DCs or fog nodes in sive analytic framework is presented to minimize the long-term
our context are different from the traditional CDN datacen- cost on energy consumption. Meanwhile, the framework also
ters [45], [46], [47]. They are also designed in a decentralized enables an energy-efficient data offloading (from fog nodes to
way but usually with much more powerful computing/com- the cloud) mechanism to help provide a high quality of service.
municating/storage capacities. In [53], a rack-scale green-energy powered edge infrastruc-
ture, InSURE (in-situ server system using renewable energy)
is implemented for data pre-processing at the edge. InSURE
B. At the Middle Edge Server Layer can be powered by standalone (solar/wind) power and with
At the middle layer of the edge computing paradigm, energy batteries as the energy backup. Meanwhile, an energy buffering
is also regarded as an important aspect, as the edge servers mechanism and a joint spatio-temporal power management
can be deployed in a domestic environment or powered by scheme are applied, to enable efficient energy flow control
the battery (e.g., a desktop or a portable WiFi router, as from the power supply to the edge server.
shown in Fig. 21). Thus, to provide a higher availability, many
power management techniques have been applied to limit the C. At the Bottom Device Layer
energy consumption of edge servers, while still ensuring their As a well-recognized fact, the IoT devices in edge comput-
performances. We give a review of two major strategies used ing usually have strict energy constraints, e.g., limited battery
at the edge server layer in recent edge computing systems. life and energy storage. Thus, it remains a key challenge to
1) Low-power system design and power management: power a great number (can up to tens of billions) of IoT
In [48], the tactical cloudlet is presented and its energy devices at the edge, especially for those resource-intensive
consumption when performing VM synthesis is evaluated applications or services [54]. We review the energy saving
particularly, under different cloudlet provisioning mechanisms. strategies adopted at the device layer of the edge computing
The results show that the largest amount of energy is consumed diagram. Specifically, we go through three major approaches
by i) VM synthesis due to the large payload size, and ii) to achieving high energy efficiency in different edge/fog
on-demand VM provisioning due to the long application- computing systems.
ready time. Such results lead to the high energy efficiency 1) Computation offloading to edge servers or cloud: As a
policy: combining cached VM with cloudlet push for cloudlet natural idea to solve the energy poverty problem, computa-
provision. tion offloading from the IoT devices to the edge servers or
A service-oriented architecture for fog/edge computing, Fog cloud has been long investigated [55], [56], [57]. It was also
Data, is proposed and evaluated in [49]. It is implemented with demonstrated that, for some particular applications or services,
an embedded computer system and performs data mining and offloading tasks from IoT devices to more powerful ends can
data analytics on the raw data collection from the wearable reduce the total energy consumption of the system, since the
sensors (in telehealth applications). With Fog Data, orders task execution time on powerful servers or cloud can be much
of magnitude data are reduced for transmission, thus leading shortened [58]. Although it increases the energy consumption
to enormous energy saving. Furthermore, Fog Data is with a of (wireless) data transmission, the tradeoff favors the offload-
low power architecture design, and even consumes much less ing option as the computational demand increases [59].
energy than that of a Raspberry Pi. Having realized that the battery life is the primary bottle-
In [50], a performance-aware orchestrator for Docker con- neck of handheld mobile devices, the authors in [57] present
tainers, named DockerCap, is developed to meet the power MAUI, an architecture for mobile code offloading and re-
consumption constraints of the edge server (fog node). Follow- mote execution. To reduce the energy consumption of the
ing the observe-decide-act loop structure, DockerCap is able smartphone program, MAUI adopts a fine-grained program
to manage container resources at run-time and provide soft- partitioning mechanism and minimizes the code changes re-
level power capping strategies. The experiments demonstrate quired at the remote server or cloud. The ability of MAUI
 18

in energy reduction is validated by various experiments upon cation performance through an LTE-optimized protocol, where
macro-benchmarks and micro-benchmarks. The results show Device-to-Device (D2D) communication is applied as an im-
that MAUI’s energy saving for a resource-intensive mobile portant supporting technology (to pull memory replicas from
application is up to one order of magnitude, also with a IoT devices). The total energy consumption of REPLISON
significant performance improvement. is generally worse than the conventional LTE scenario, as it
Like MAUI [57] , the authors in [59] design and implement needs more active devices. However, the energy consumed
CloneCloud, a system that helps partition mobile application per device during a single replicate transmission is much less.
programs and performs strategically offloading for fast and With further evaluation results, it shows that REPLISOM has
elastic execution at the cloud end. As the major difference an energy advantage over the conventional LTE scenarios as
to MAUI, CloneCloud involves less programmer help during long as the size of replica is sufficiently small.
the whole process and only offloads particular program parti- For IoT devices distributed at the edge, the authors in [64]
tions on demand of execution, which further speeds up the leverage the software agents running on the IoT devices to
program execution. Evaluation shows that CloneCloud can establish an integrated multi-agent system (MAS). By sharing
improve the energy efficiency of mobile applications (along data and information among the mobile agents, edge devices
with their execution efficiency) by 20 times. Similarly in [60], are able to collaborate with each other and improve the
by continuous updates of software clones in the cloud with a system energy efficiency in executing distributed opportunistic
reasonable overhead, the offloading service can lead to energy applications. Upon the experimental platform with 100 sensor
reduction at the mobile end by a factor. For computation nodes and 20 smartphones as edge devices, the authors show
intensive applications on resource constrained edge devices, the great potential of data transmission reduction with MAS.
their executions usually need to be offloaded to the cloud. This leads to a significant energy saving, from 15% to 66%,
To reduce the response latency of the image recognition under different edge computing scenarios. As another work
application, Precog is presented which has been introduced in applying data reduction for energy saving, CAROMM [65]
Sec. II-G. With the on-device recognition caches, Precog much employs a change detect technique (LWC algorithm) to control
reduces the amount of images offloading to the edge server or the data transmission of IoT devices, while maintaining the
cloud, by predicting and prefetching the future images to be data accuracy.
recognized.
2) Collaborated devices control and resource management: V. D EEP L EARNING O PTIMIZATION AT THE E DGE
For energy saving of the massive devices at the edge, besides In the past decades, we have witnessed the burgeoning of
offloading their computational tasks to more powerful ends, machine learning, especially deep learning based applications
there is also a great potential via sophisticated collaboration which have changed human being’s life. With complex struc-
and cooperation among the devices themselves. Particularly, tures of hierarchical layers to capture features from raw data,
when the remote resources from the edge server or cloud are deep learning models have shown outstanding performances in
unavailable, it is critical and non-trivial to complete the edge those novel applications, such as machine translation, object
tasks while without violating the energy constraint. detection, and smart question and answer systems.
PCloud is presented in [10] to enhance the capability of Traditionally, most deep learning based applications are
individual mobile devices at the edge. By seamless using deployed on a remote cloud center, and many systems and
available resources from the nearby devices, PCloud forms tools are designed to run deep learning models efficiently
a personal cloud to serve end users whenever the cloud on the cloud. Recently, with the rapid development of edge
resources are difficult to access, where device participation computing, the deep learning functions are being offloaded
is guided in a privacy-preserving manner. The authors show to the edge. Thus it calls for new techniques to support the
that, by leveraging multiple nearby device resources, PCloud deep learning models at the edge. This section classifies these
can much reduce the task execution time as well as energy technologies into three categories: systems and toolkits, deep
consumption. For example, in the case study of neighborhood learning packages, and hardware.
watch with face recognition, the results show a 74% reduction
in energy consumption on a PCloud vs. on a single edge
device. Similar to PCloud, the concept of mobile device cloud A. Systems and Toolkits
(MDC) is proposed in [61], where computational offloading Building systems to support deep learning at the edge
is also adopted among the mobile devices. It shows that the is currently a hot topic for both industry and academy.
energy efficiency (gain) is increased by 26% via offloading There are several challenges when offloading state-of-the-
in MDC. The authors of [62] propose an adaptive method art AI techniques on the edge directly, including computing
to dynamically discovery available nearby resource in het- power limitation, data sharing and collaborating, and mismatch
erogeneous networks, and perform automatic transformation between edge platform and AI algorithms. To address these
between centralized and flooding strategies to save energy. challenges, OpenEI is proposed as an Open Framework for
As current LTE standard is not optimized to support a large Edge Intelligence [66]. OpenEI is a lightweight software
simultaneous access of IoT devices, the authors in [63] propose platform to equip edges with intelligent processing and data
an improved memory replication architecture and protocol, sharing capability. OpenEI consists of three components: a
REPLISON, for computation offloading of the massive IoT package manager to execute the real-time deep learning task
devices at the edge. REPLISON improves the memory repli- and train the model locally, a model selector to select the
 19

most suitable model for different edge hardware, and a library pre-fused activations, and quantized kernels that allow smaller
including a RESTFul API for data sharing. The goal of OpenEI and faster (fixed-point math) models.
is that any edge hardware will has the intelligent capability Facebook published Caffe2 [76] as a lightweight, modular,
after deploying it. and scalable framework for deep learning in 2017. Caffe2
In the industry, some top-leading tech-giants have published is a new version of Caffe which is first developed by UC
several projects to move the deep learning functions from the Berkeley AI Research (BAIR) and community contributors.
cloud to the edge. Except Microsoft published Azure IoT Edge Caffe2 provides an easy and straightforward way to play with
which have been introduced in Sec. III-E, Amazon and Google the deep learning and leverage community contributions of
also build their services to support deep learning on the edge. new models and algorithms. Comparing with the original Caffe
Table III summarizes the features of the systems which will framework, Caffe2 merges many new computation patterns,
be discussed below. including distributed computation, mobile, reduced precision
Amazon Web Services (AWS) has published IoT Greengrass computation, and more non-vision use cases. Caffe2 supports
ML Inference [67] after IoT Greengrass. AWS IoT Greengrass multiple platforms which enable developers to use the power
ML Inference is a software to support machine learning of GPUs in the cloud or at the edge with cross-platform
inferences on local devices. With AWS IoT Greengrass ML libraries.
Inference, connected IoT devices can run AWS Lambda func- PyTorch [71] is published by Facebook. It is a python pack-
tions and have the flexibility to execute predictions based on age that provides two high-level features: tensor computation
those deep learning models created, trained, and optimized with strong GPU acceleration and deep Neural Networks built
in the cloud. AWS IoT Greengrass consists of three software on a tape-based auto-grad system. Maintained by the same
distributions: AWS IoT Greengrass Core, AWS IoT Device company (Facebook), PyTorch and Caffe2 have their own ad-
SDK, and AWS IoT Greengrass SDK. Greengrass is flexible vantages. PyTorch is geared toward research, experimentation
for users as it includes a pre-built TensorFlow, Apache MXNet, and trying out exotic neural networks, while caffe2 supports
and Chainer package, and it can also work with Caffe2 and more industrial-strength applications with a heavy focus on
Microsoft Cognitive Toolkit. the mobile. In 2018, Caffe2 and PyTorch projects merged into
Cloud IoT Edge [68] extends Google Cloud’s data process- a new one named PyTorch 1.0, which would combine the user
ing and machine learning to edge devices by taking advantages experience of the PyTorch frontend with scaling, deployment
of Google AI products, such TensorFlow Lite and Edge TPU. and embedding capabilities of the Caffe2 backend.
Cloud IoT Edge can either run on Android or Linux-based MXNet [72] is a flexible and efficient library for deep
operating systems. It is made up of three components: Edge learning. It was initially developed by the University of Wash-
Connect ensures the connection to the cloud and the updates ington and Carnegie Mellon University, to support CNN and
of software and firmware, Edge ML runs ML inference by long short-term memory networks (LSTM). In 2017, Amazon
TensorFlow Lite, and Edge TPU specific designed to run announced MXNet as its choice of deep learning framework.
TensorFlow Lite ML models. Cloud IoT Edge can satisfy the MXNet places a special emphasis on speeding up the devel-
real-time requirement for the mission-critical IoT applications, opment and deployment of large-scale deep neural networks.
as it can take advantages of Google AI products (such as It is designed to support multiple different platforms (either
TensorFlow Lite and Edge TPU) and optimize the performance cloud platforms or the edge ones) and can execute training and
collaboratively. inference tasks. Furthermore, other than the Windows, Linux,
and OSX operating systems based devices, it also supports the
Ubuntu Arch64 and Raspbian ARM based operating systems.
B. Deep Learning Packages CoreML [73] is a deep learning framework optimized
Many deep learning packages have been widely used to for on-device performance at memory footprint and power
deliver the deep learning algorithms and deployed on the consumption. Published by Apple, users can integrate the
cloud data centers, including TensorFlow [69], Caffe [70], trained machine learning model into Apple products, such as
PyTorch [71], and MXNet [72]. Due to the limitations of Siri, Camera, and QuickType. CoreML supports not only deep
computing resources at the edge, the packages designed for learning models, but also some standard models such as tree
the cloud are not suitable for edge devices. Thus, to support ensembles, SVMs, and generalized linear models. Built on top
data processing with deep learning models at the edge, several of low level technologies, CoreML aims to make full use of
edge-based deep learning frameworks and tools have been re- the CPU and GPU capability and ensure the performance and
leased. In this section, we introduce TensorFlow Lite, Caffe2, efficiency of data processing.
PyTorch, MXNet, CoreML [73], and TensorRT [74], whose The platform of TensorRT [74] acts as a deep learning infer-
features are summarized in Tables IV. ence to run the models trained by TensorFlow, Caffe, and other
TensorFlow Lite [75] is TensorFlow’s lightweight solution frameworks. Developed by NVIDIA company, it is designed to
which is designed for mobile and edge devices. TensorFlow reduce the latency and increase the throughput when executing
is developed by Google in 2016 and becomes one of the the inference task on NVIDIA GPU. To achieve computing
most widely used deep learning frameworks in cloud data acceleration, TensorRT leverages several techniques, including
centers. To enable low-latency inference of on-device deep weight and activation precision calibration, layer and tensor
learning models, TensorFlow Lite leverages many optimization fusion, kernel auto-tuning, dynamic tensor memory, and multi-
techniques, including optimizing the kernels for mobile apps, stream execution.
 20

TABLE III
C OMPARISON OF D EEP LEARNING S YSTEMS ON E DGE

Features AWS IoT Greengrass Azure IoT Edge Cloud IoT Edge
Developer Amazon Microsoft Google
IoT Greengrass Core, IoT Device SDK, IoT Edge modules, IoT Edge runtime,
Components Edge Connect, Edge ML, Edge TPU
IoT Greengrass SDK Cloud-based interface
OS Linux, macOS, Windows Windows, Linux, macOS Linux, macOS, Windows, Android
Target device Multiple platforms (GPU-based, Raspberry Pi) Multiple platforms TPU
Characteristic Flexible Windows friendly Real-time

Considering the different performance of the packages and Following the above work, Qiu et al. [82] propose a CNN
the diversity of the edge hardware, it is challenging to choose a accelerator designed upon the embedded FPGA, Xilinx Zynq
suitable package to build edge computing systems. To evaluate ZC706, for large-scale image classification. It presents an
the deep learning frameworks at the edge and provide a in-depth analysis of state-of-the-art CNN models and shows
reference to select appropriate combinations of package and that Convolutional layers are computational-centric and Fully-
edge hardware, pCAMP [77] is proposed. It compares the Connected layers are memory-centric. The average perfor-
packages’ performances (w.r.t. the latency, memory footprint, mances of the CNN accelerator at convolutional layers and the
and energy) resulting from five edge devices and observes that full CNN are 187.8 GOPS and 137.0 GOPS under a 150M Hz
no framework could win over all the others at all aspects. It working frequency, respectively, which outperform previous
indicates that there is much room to improve the frameworks approaches significantly.
at the edge. Currently, developing a lightweight, efficient and An efficient speech recognition engine (ESE) is designed
high-scalability framework to support diverse deep learning to speed up the predictions and save energy when applying
modes at the edge cannot be more important and urgent. the deep learning model of LSTM. ESE is implemented in a
In addition to these single-device based frameworks, more Xilinx XCKU060 FPGA opearting at 200M Hz. For the sparse
researchers focus on distributed deep learning models over LSTM network, it can achieve 282GOPS, corresponding to
the cloud and edge. DDNN [78] is a distributed deep neural a 2.52 TOPS on the dense LSTM network. Besides, energy
network architecture across cloud, edge, and edge devices. efficiency improvements of 40x and 11.5x are achieved,
DDNN maps the sections of a deep neural network onto respectively, compared with the CPU and GPU based solution.
different computing devices, to minimize communication and 2) GPU-based hardware: GPU can execute parallel pro-
resource usage for devices and maximize usefulness of features grams at a much higher speed than CPU, which makes it fit
extracted from the cloud. for the computational paradigm of deep learning algorithms.
Neurosurgeon [79] is a lightweight scheduler which can Thus, to run deep learning models at the edge, building the
automatically partition DNN computation between mobile hardware platform with GPU is a must choice. Specifically,
devices and datacenters at the granularity of neural network NVIDIA Jetson TX2 and DRIVE PX2 are two representative
layers. By effectively leveraging the resources in the cloud and GPU-based hardware platforms for deep learning.
at the edge, Neurosurgeon achieves low computing latency, NVIDIA Jetson TX2 [83] is an embedded AI computing
low energy consumption, and high traffic throughput. device which is designed to achieve low latency and high
power-efficient. It is built upon an NVIDIA Pascal GPU with
256 CUDA cores, an HMP Dual Denver CPU and a Qualcomm
C. Hardware System ARM CPU. It is loaded with 8GB of memory and 59.7GB/s of
The hardware designed specifically for deep learning can memory bandwidth and the power is about 7.5 watts. The GPU
strongly support edge computing. Thus, we further review rele- is used to execute the deep learning task and CPUs are used to
vant hardware systems and classify them into three categories: maintain general tasks. It also supports the NVIDIA Jetpack
FPGA-based hardware, GPU-based hardware, and ASIC. SDK which includes libraries for deep learning, computer
1) FPGA-based Hardware: A field-programmable gate ar- vision, GPU computing, and multimedia processing.
ray (FPGA) is an integrated circuit and can be configured by NVIDIA DRIVE PX [84] is designed as the AI supercom-
the customer or designer after manufacturing. FPGA based puter for autonomous driving. The architecture is available in a
accelerators can achieve high performance computing with variety of configurations, from the mobile processor operating
low energy, high parallelism, high flexibility, and high secu- at 10 watts to a multi-chip AI processors delivering 320 TOPS.
rity [80]. It can fuse data from multiple cameras, as well as lidar, radar,
[81] implements a CNN accelerator on a VC707 FPGA and ultrasonic sensors.
board. The accelerator focuses on solving the problem that 3) ASIC: Application-Specific Integrated Circuit (ASIC) is
the computation throughput does not match the memory the integrated circuit which supports customized design for
bandwidth well. By quantitatively analyzing the two factors a particular application, rather than the general-purpose use.
using various optimization techniques, the authors provide a ASIC is suitable for the edge scenario as it usually has a
solution with better performance and lower FPGA resource smaller size, lower power consumption, higher performance,
requirement, and their solution achieves a peak performance and higher security than many other circuits. Researchers and
of 61.62 GOPS under a 100M Hz working frequency. developers design ASIC to meet the computing pattern of deep
 21

TABLE IV
C OMPARISON OF D EEP L EARNING PACKAGES ON E DGE

Features TensorFlow Lite Caffe2 PyTorch MXNet CoreML TensorRT

Developer Google Facebook Facebook DMLC, Amazon Apple NVIDIA
Open Source
Apache-2.0 Apache-2.0 BSD Apache-2.0 Not open source Not open source
License
Task Inference Training, Inference Training, Inference Training, Inference Inference Inference
Mobile and
Target Device Multiple platform Multiple platform Multiple platform Apple devices NVIDIA GPU
embedded device
Lightweight, modular, Large-scale Memory footprint and Latency and
Characteristic Latency Research
and scalable deep neural networks power consumption. throughput

learning. 2) Multi-user Fairness: For edge devices with limited

DianNao family [85] is a series of hardware accelera- resources, how to ensure the fairness of multi-user usage,
tors designed for deep learning models, including DianNao, especially for the shared resources and rare resources. For
DaDianNao, ShiDianNao, and PuDianNao. They investigate example, a smartphone made up of various sensors and com-
the accelerator architecture to minimize memory transfers puting resources can act as an edge node to serve multiple
as efficiently as possible. Different from other accelerators users. However, as the smartphone has limited battery life,
in DianNao family, ShiDianNao [86] focuses on image ap- it is a problem to fairly allocate resources when receiving
plications in embedded systems which are widely used in many requests. The resource competition is more intense,
edge computing scenarios. For the CNN-based deep learning the requests are coming from different irrelevant tasks, it is
models, it provides a computing speed 30x faster than that of hard to decide who should use the resource with only local
NVIDIA K20M GPU, with an body area of 4.86mm2 and a information. Besides, the unfairness can be used as an attack
power of 320mW . strategy to make the critical task resource hungry, which may
Edge TPU [87] is Google’s purpose-built ASIC for edge lead to main task failure. The existing edge computing systems
computing. It augments Google’s Cloud TPU and Cloud IoT do not pay much attention to the multi-user fairness, the basic
to provide an end-to-end infrastructure and facilitates the solution (bottom-line) is to provide resource isolation, users
deployment of customers’ AI-based solutions. In addition, only get what the edge node promised when it accepts the
Edge TPU can combine the custom hardware, open software, request. More fairness strategies require system support, like
and state-of-the-art AI algorithms to achieve high performance related task status updates, etc.
with a small physical area and low power consumption. 3) Privacy Protection: Unlike cloud computing, edge de-
vices can be privately owned, such as gateway devices for
smart home systems. When other users use such edge devices,
VI. K EY D ESIGN I SSUES
obtain their data, and even take control of them, how to
The edge computing system manages various resources ensure the owner’s privacy and guest users’ data privacy
along the path from the cloud center to end devices, shielding is important. AirBox is a lightweight flexible edge function
the complexity and diversity of hardware and helping devel- system, which leverages hardware security mechanism (e.g.
opers quickly design and deploy novel applications. To fully Intel SGX) to enhance system security. Other system have
leverages the advantages, we discuss the following key issues not pay much attention to privacy and security. Existing cloud
which need to be paid attention when analyzing and designing security approaches can be applied with the consideration of
a new edge computing system. the resource limitation. Enhancing resource isolation, setting
1) Mobility Support: Mobility support has two aspects: user up privilege management and access control policies can be
potential directions to solve the privacy problem.
mobility and resource mobility. User mobility refers to how to
automatically migrate the current program state and necessary 4) Developer Friendliness: The system ultimately provides
data when the user moves from one edge node coverage hardware interaction and basic services for upper-level appli-
domain to another so that the service handover is seamlessly. cations. How to design interactive APIs, program deployment
Currently, Cloudlet and CloudPath provide service migration module, resource application and revocation, etc., are the
through terminating/finishing existing task and starting a new key factors for the system to be widely used. Therefore, to
VM/instance in the target edge node. Fine-grain migration design an edge computing system, we should think from an
and target edge node prediction are not supported. Resource application developer’s perspective. Specifically, to provide
mobility refers to i) how to dynamically discover and manage effective development and deployment services is a good
available resources, including both the long-term resources and idea to help improve the ecosystem of the edge computing
short-term ones, and ii) when some edge devices are damaged system. For instances, EdgeX Foundry and Apache Edgent
how the system can be resumed as soon as possible with provide APIs to manage devices and analyze data respectively.
replacements. For example, PCloud and FocusStack supports ParaDrop supports monitoring devices and application status
edge devices dynamic join and leave with no task running. through developer APIs, and provides a web UI for users to
Intelligent dynamic resource management is still required for manage/configure gateway as well as deploy applications to
edge computing systems. devices.
 22

5) Multi-domain Management and Cooperation: Edge R EFERENCES

computing involves multiple types of resources, each of which
[1] M. Satyanarayanan, V. Bahl, R. Caceres, and N. Davies, “The case for
may belong to a different owner. For examples, the smart home vm-based cloudlets in mobile computing,” IEEE pervasive Computing,
gateways and sensors of the house owner, networks resources 2009.
and base stations from the Internet service providers (ISP), [2] (2018) Open edge computing. [Online]. Available: http:
//openedgecomputing.org/
traffic cameras owned by the government. How to access these [3] Z. Li, C. Wang, and R. Xu, “Computation offloading to save energy
resources and organize them according to the requirements of on handheld devices: a partition scheme,” in Proceedings of the 2001
application and services, especially in emergency situations, is international conference on Compilers, architecture, and synthesis for
embedded systems. ACM, 2001, pp. 238–246.
a problem that the edge computing system needs to consider.
[4] K. Ha and M. Satyanarayanan, “Openstack++ for cloudlet deployment,”
Existing researches assume we have the permission to use School of Computer Science Carnegie Mellon University Pittsburgh,
various devices belongs to different owners. In the initial stage, 2015.
systems focus on the functionality perspective rather than [5] K. Ha, P. Pillai, W. Richter, Y. Abe, and M. Satyanarayanan, “Just-in-
time provisioning for cyber foraging,” in Proceeding of the 11th annual
implementation issues like price model. With the developing of international conference on Mobile systems, applications, and services.
edge computing, the real deployment issues should be solved ACM, 2013, pp. 153–166.
before we enjoy all the benefits. [6] K. Ha, Z. Chen, W. Hu, W. Richter, P. Pillai, and M. Satyanarayanan,
“Towards wearable cognitive assistance,” in Proceedings of the 12th
6) Cost Model: In cloud computing, the corresponding vir- annual international conference on Mobile systems, applications, and
tual machine can be allocated based on the resource requested services. ACM, 2014, pp. 68–81.
by the user, and the cost model can be given according to [7] M. Satyanarayanan, P. Simoens, Y. Xiao, P. Pillai, Z. Chen, K. Ha,
W. Hu, and B. Amos, “Edge analytics in the internet of things,” IEEE
the resource usage. In edge computing, an application may Pervasive Computing, no. 2, pp. 24–31, 2015.
use resources from different owners. Thus, how to measure [8] M. Satyanarayanan, G. Lewis, E. Morris, S. Simanta, J. Boleng, and
resource usage, calculate overall overhead, and give an appro- K. Ha, “The role of cloudlets in hostile environments,” IEEE Pervasive
Computing, vol. 12, no. 4, pp. 40–49, 2013.
priate pricing model are crucial problems when deploying an [9] S. H. Mortazavi, M. Salehe, C. S. Gomes, C. Phillips, and E. de Lara,
edge computing system. Generally, edge computing systems “Cloudpath: a multi-tier cloud computing framework,” in Proceedings of
focus on how to satisfy the resource requirements of various the Second ACM/IEEE Symposium on Edge Computing. ACM, 2017,
p. 20.
services and applications, some of them pay more attention
[10] M. Jang, K. Schwan, K. Bhardwaj, A. Gavrilovska, and A. Avasthi, “Per-
to the energy consumption due to the nature of mobile nodes, sonal clouds: Sharing and integrating networked resources to enhance
more complex overhead are not considered. end user experiences,” in Proceedings of IEEE INFOCOM. IEEE, 2014,
7) Compatibility: Currently, specialized edge computing pp. 2220–2228.
[11] M. Jang and K. Schwan, “Stratus: Assembling virtual platforms from
applications are still quite few. For examples, ParaDrop appli- device clouds,” in Proceedings of the IEEE International Conference on
cations require additional XML configuration file to specify Cloud Computing. IEEE, 2011, pp. 476–483.
the resource usage requirements, SpanEdge needs developers [12] P. Liu, D. Willis, and S. Banerjee, “Paradrop: Enabling lightweight
multi-tenancy at the networks extreme edge,” in 2016 IEEE/ACM
to divide/configure tasks into local tasks and global tasks. Symposium on Edge Computing (SEC), Oct 2016, pp. 1–13.
Common applications are not directly supported to run in edge [13] H. P. Sajjad, K. Danniswara, A. Al-Shishtawy, and V. Vlassov,
systems. How to automatically and transparently convert ex- “Spanedge: Towards unifying stream processing over central and near-
the-edge data centers,” in IEEE/ACM Symposium on Edge Computing
isting programs to the edge version and conveniently leverages (SEC). IEEE, 2016, pp. 168–178.
the advantages of edge computing are still open problems. The [14] Z.-W. Xu, “Cloud-sea computing systems: Towards thousand-fold im-
compatibility should be considered in the edge system design. provement in performance per watt for the coming zettabyte era,”
Journal of Computer Science and Technology, vol. 29, no. 2, pp. 177–
Specifically, how to adapt traditional applications to the new 181, 2014.
architecture and realize the basic functions so that they can run [15] R. T. Fielding and R. N. Taylor, Architectural styles and the design of
successfully, deserving further exploration and development. network-based software architectures. University of California, Irvine
Irvine, USA, 2000, vol. 7.
[16] U. Drolia, K. Guo, J. Tan, R. Gandhi, and P. Narasimhan, “Cachier:
VII. C ONCLUSIONS Edge-caching for recognition applications,” in 2017 IEEE 37th Interna-
Edge computing is a new paradigm that jointly migrates tional Conference on Distributed Computing Systems (ICDCS). IEEE,
2017, pp. 276–286.
the capability of networking, computation, and storage from [17] U. Drolia, K. Guo, and P. Narasimhan, “Precog: prefetching for image
the remote cloud to the user sides. Under the context of recognition applications at the edge,” in Proceedings of the Second
IoT and 5G, the vision of edge computing is promising ACM/IEEE Symposium on Edge Computing. ACM, 2017, p. 17.
[18] B. Amento, B. Balasubramanian, R. J. Hall, K. Joshi, G. Jung, and
in providing smarter services and applications with better K. H. Purdy, “Focusstack: Orchestrating edge clouds using location-
user experiences. The recently proposed systems and tools based focus of attention,” in IEEE/ACM Symposium on Edge Computing
on edge computing generally reduce the data processing and (SEC). IEEE, 2016, pp. 179–191.
transmission overhead, and improve the efficiency and efficacy [19] K. Bhardwaj, M.-W. Shih, P. Agarwal, A. Gavrilovska, T. Kim, and
K. Schwan, “Fast, scalable and secure onloading of edge functions using
of mobile data analytics. Additionally, the integration of edge airbox,” in IEEE/ACM Symposium on Edge Computing (SEC). IEEE,
computing and deep learning techniques further fosters the 2016, pp. 14–27.
research on edge-based intelligence services. This article intro- [20] Q. Zhang, X. Zhang, Q. Zhang, W. Shi, and H. Zhong, “Firework:
Big data sharing and processing in collaborative edge environment,”
duced the representative systems and open source projects for in Proceedings of the IEEE Workshop on Hot Topics in Web Systems
edge computing, presented several energy efficiency enhancing and Technologies (HotWeb). IEEE, 2016, pp. 20–25.
strategies for performance consideration and technologies for [21] Q. Zhang, Y. Wang, X. Zhang, L. Liu, X. Wu, W. Shi, and H. Zhong,
“Openvdap: an open vehicular data analytics platform for cavs,” in
deploying deep learning models at the edge, and suggested a Proceedings of the IEEE 38th International Conference on Distributed
few research directions during system design and analysis. Computing Systems (ICDCS), 2018.
 23

[22] L. Liu, X. Zhang, M. Qiao, and W. Shi, “Safeshareride: edge-based at- [48] G. Lewis, S. Echeverrı́a, S. Simanta, B. Bradshaw, and J. Root, “Tactical
tack detection in ridesharing services,” in Proceedings of the IEEE/ACM cloudlets: Moving cloud computing to the edge,” in Military Commu-
Symposium on Edge Computing (SEC). IEEE, 2018. nications Conference (MILCOM), 2014 IEEE. IEEE, 2014, pp. 1440–
[23] R. Trimananda, A. Younis, B. Wang, B. Xu, B. Demsky, and G. Xu, 1446.
“Vigilia: securing smart home edge computing,” in Proceedings of the [49] H. Dubey, J. Yang, N. Constant, A. M. Amiri, Q. Yang, and
IEEE/ACM Symposium on Edge Computing (SEC). IEEE, 2018, pp. K. Makodiya, “Fog data: Enhancing telehealth big data through fog
74–89. computing,” in Proceedings of the ASE BigData & SocialInformatics
[24] I. Zavalyshyn, N. O. Duarte, and N. Santos, “Homepad: a privacy-aware 2015. ACM, 2015, p. 14.
smart hub for home environments,” in Proceedings of the IEEE/ACM [50] A. Asnaghi, M. Ferroni, and M. D. Santambrogio, “Dockercap: A
Symposium on Edge Computing (SEC). IEEE, 2018, pp. 58–73. software-level power capping orchestrator for docker containers,” in
[25] S. Yi, Z. Hao, Q. Zhang, Q. Zhang, W. Shi, and Q. Li, “Lavea: latency- Computational Science and Engineering (CSE) and IEEE Intl Confer-
aware video analytics on edge computing platform,” in Proceedings of ence on Embedded and Ubiquitous Computing (EUC) and 15th Intl
the Second ACM/IEEE Symposium on Edge Computing (SEC). ACM, Symposium on Distributed Computing and Applications for Business
2017, p. 15. Engineering (DCABES), 2016 IEEE Intl Conference on. IEEE, 2016,
[26] C.-C. Hung, G. Ananthanarayanan, P. Bodik, L. Golubchik, M. Yu, pp. 90–97.
P. Bahl, and M. Philipose, “Videoedge: processing camera streams using [51] T. Rausch, C. Avasalcai, and S. Dustdar, “Portable energy-aware cluster-
hierarchical clusters,” in Proceedings of the IEEE/ACM Symposium on based edge computers,” in 2018 IEEE/ACM Symposium on Edge Com-
Edge Computing (SEC). IEEE, 2018, pp. 115–131. puting (SEC). IEEE, 2018, pp. 260–272.
[27] L. Tong, Y. Li, and W. Gao, “A hierarchical edge cloud architecture for [52] Y. Nan, W. Li, W. Bao, F. C. Delicato, P. F. Pires, Y. Dou, and A. Y.
mobile computing,” in Proceedings of the 35th Annual IEEE Interna- Zomaya, “Adaptive energy-aware computation offloading for cloud of
tional Conference on Computer Communications (INFOCOM). IEEE, things systems,” IEEE Access, vol. 5, pp. 23 947–23 957, 2017.
2016, pp. 1–9. [53] C. Li, Y. Hu, L. Liu, J. Gu, M. Song, X. Liang, J. Yuan, and T. Li,
[28] J. Wang, Z. Feng, Z. Chen, S. George, M. Bala, P. Pillai, S.-W. Yang, and “Towards sustainable in-situ server systems in the big data era,” in ACM
M. Satyanarayanan, “Bandwidth-efficient live video analytics for drones SIGARCH Computer Architecture News, vol. 43, no. 3. ACM, 2015,
via edge computing,” in Proceedings of the IEEE/ACM Symposium on pp. 14–26.
Edge Computing (SEC). IEEE, 2018, pp. 159–173. [54] Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, “A survey
[29] Y. Li and W. Gao, “Muvr: supporting multi-user mobile virtual reality on mobile edge computing: The communication perspective,” IEEE
with resource constrained edge cloud,” in Proceedings of the IEEE/ACM Communications Surveys & Tutorials, vol. 19, no. 4, pp. 2322–2358,
Symposium on Edge Computing (SEC). IEEE, 2018, pp. 1–16. 2017.
[30] “Akraino edge statck,” 2018, https://www.akraino.org. [55] A. Rudenko, P. Reiher, G. J. Popek, and G. H. Kuenning, “Saving
[31] “Cord,” 2018, https://www.opennetworking.org/cord. portable computer battery power through remote process execution,”
[32] B. A. A. Nunes, M. Mendonca, X.-N. Nguyen, K. Obraczka, and ACM SIGMOBILE Mobile Computing and Communications Review,
T. Turletti, “A survey of software-defined networking: Past, present, and vol. 2, no. 1, pp. 19–26, 1998.
future of programmable networks,” IEEE Communications Surveys & [56] R. Kemp, N. Palmer, T. Kielmann, F. Seinstra, N. Drost, J. Maassen, and
Tutorials, vol. 16, no. 3, pp. 1617–1634, 2014. H. Bal, “eyedentify: Multimedia cyber foraging from a smartphone,” in
[33] H. Hawilo, A. Shami, M. Mirahmadi, and R. Asal, “Nfv: state of the 2009 11th IEEE International Symposium on Multimedia. IEEE, 2009,
art, challenges, and implementation in next generation mobile networks pp. 392–399.
(vepc),” IEEE Network, vol. 28, no. 6, pp. 18–26, 2014. [57] E. Cuervo, A. Balasubramanian, D.-k. Cho, A. Wolman, S. Saroiu,
[34] A. W. Manggala, Hendrawan, and A. Tanwidjaja, “Performance anal- R. Chandra, and P. Bahl, “Maui: making smartphones last longer with
ysis of white box switch on software defined networking using open code offload,” in Proceedings of the 8th international conference on
vswitch,” in International Conference on Wireless and Telematics, 2016. Mobile systems, applications, and services. ACM, 2010, pp. 49–62.
[35] K. C. Okafor, I. E. Achumba, G. A. Chukwudebe, and G. C. Onon- [58] K. Ha, “System infrastructure for mobile-cloud convergence,” Ph.D.
iwu, “Leveraging fog computing for scalable iot datacenter using dissertation, Carnegie Mellon University, 2016.
spine-leaf network topology,” Journal of Electrical and Computer [59] B.-G. Chun, S. Ihm, P. Maniatis, M. Naik, and A. Patti, “Clonecloud:
Engineering,2017,(2017-4-24), vol. 2017, no. 2363240, pp. 1–11, 2017. elastic execution between mobile device and cloud,” in Proceedings of
[36] “Edgex foundry,” 2018, https://www.edgexfoundry.org. the sixth conference on Computer systems. ACM, 2011, pp. 301–314.
[37] “Apache edgent,” 2018, http://edgent.apache.org. [60] M. V. Barbera, S. Kosta, A. Mei, and J. Stefa, “To offload or not to
[38] “Azure iot,” 2018, https://azure.microsoft.com/en-us/overview/iot/. offload? the bandwidth and energy costs of mobile cloud computing,”
[39] R. Morabito, “Virtualization on internet of things edge devices with in INFOCOM, 2013 Proceedings IEEE. IEEE, 2013, pp. 1285–1293.
container technologies: a performance evaluation,” IEEE Access, vol. 5, [61] A. Fahim, A. Mtibaa, and K. A. Harras, “Making the case for compu-
pp. 8835–8850, 2017. tational offloading in mobile device clouds,” in Proceedings of the 19th
[40] T. Mastelic, A. Oleksiak, H. Claussen, I. Brandic, J.-M. Pierson, and annual international conference on Mobile computing & networking.
A. V. Vasilakos, “Cloud computing: Survey on energy efficiency,” Acm ACM, 2013, pp. 203–205.
computing surveys (csur), vol. 47, no. 2, p. 33, 2015. [62] W. Liu, T. Nishio, R. Shinkuma, and T. Takahashi, “Adaptive resource
[41] V. Valancius, N. Laoutaris, L. Massoulié, C. Diot, and P. Rodriguez, discovery in mobile cloud computing,” Computer Communications,
“Greening the internet with nano data centers,” in Proceedings of the vol. 50, pp. 119–129, 2014.
5th international conference on Emerging networking experiments and [63] S. Abdelwahab, B. Hamdaoui, M. Guizani, and T. Znati, “Replisom:
technologies. ACM, 2009, pp. 37–48. Disciplined tiny memory replication for massive iot devices in lte edge
[42] S. Sarkar and S. Misra, “Theoretical modelling of fog computing: A cloud,” IEEE Internet of Things Journal, vol. 3, no. 3, pp. 327–338,
green computing paradigm to support iot applications,” Iet Networks, 2016.
vol. 5, no. 2, pp. 23–29, 2016. [64] S. Abdelwahab, B. Hamdaoui, M. Guizani, T. Znati, Taieb Leppnen,
[43] A. Feldmann, A. Gladisch, M. Kind, C. Lange, G. Smaragdakis, and F.- and J. Riekki, “Energy efficient opportunistic edge computing for the
J. Westphal, “Energy trade-offs among content delivery architectures,” internet of things,” Web Intelligence and Agent Systems, p. In Press,
in Telecommunications Internet and Media Techno Economics (CTTE), 2018.
2010 9th Conference on. IEEE, 2010, pp. 1–6. [65] P. P. Jayaraman, J. B. Gomes, H. L. Nguyen, Z. S. Abdallah, S. Kr-
[44] F. Jalali, K. Hinton, R. Ayre, T. Alpcan, and R. S. Tucker, “Fog ishnaswamy, and A. Zaslavsky, “Cardap: A scalable energy-efficient
computing may help to save energy in cloud computing,” IEEE Journal context aware distributed mobile data analytics platform for the fog,” in
on Selected Areas in Communications, vol. 34, no. 5, pp. 1728–1739, East European Conference on Advances in Databases and Information
2016. Systems. Springer, 2014, pp. 192–206.
[45] G. Tang, H. Wang, K. Wu, and D. Guo, “Tapping the knowledge of dy- [66] X. Zhang, Y. Wang, S. Lu, L. Liu, L. Xu, and W. Shi, “OpenEI: An Open
namic traffic demands for optimal cdn design,” IEEE/ACM Transactions Framework for Edge Intelligence,” in 2019 IEEE 39th International
on Networking (TON), vol. 27, no. 1, pp. 98–111, 2019. Conference on Distributed Computing Systems (ICDCS), July 2019.
[46] G. Tang, K. Wu, and R. Brunner, “Rethinking cdn design with distributee [67] (2019) Aws iot greengrass. https://aws.amazon.com/greengrass/”.
time-varying traffic demands,” in INFOCOM. IEEE, 2017, pp. 1–9. [Online]. Available: https://aws.amazon.com/greengrass/
[47] G. Tang, H. Wang, K. Wu, D. Guo, and C. Zhang, “When more may not [68] (2019) Cloud iot edge: Deliver google ai capabilities at the
be better: Toward cost-efficient cdn selection,” in INFOCOM WKSHPS. edge. https://cloud.google.com/iot-edge/. [Online]. Available: https:
IEEE, 2018, pp. 1–2. //cloud.google.com/iot-edge/
 24

[69] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin,

S. Ghemawat, G. Irving, M. Isard et al., “Tensorflow: A system for
large-scale machine learning.” in OSDI, vol. 16, 2016, pp. 265–283.
[70] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,
S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for
fast feature embedding,” in Proceedings of the 22nd ACM international
conference on Multimedia. ACM, 2014, pp. 675–678.
[71] (2019) From research to production. https://pytorch.org. [Online].
Available: https://pytorch.org
[72] T. Chen, M. Li, Y. Li, M. Lin, N. Wang, M. Wang, T. Xiao, B. Xu,
C. Zhang, and Z. Zhang, “Mxnet: A flexible and efficient machine
learning library for heterogeneous distributed systems,” in LearningSys
at NIPS. ACM, 2015.
[73] (2019) Core ml: Integrate machine learning models into your app.
https://developer.apple.com/documentation/coreml. [Online]. Available:
https://developer.apple.com/documentation/coreml
[74] (2019) Nvidia tensorrt: Programmable inference accelerator.
https://developer.nvidia.com/tensorrt. [Online]. Available: https:
//developer.nvidia.com/tensorrt
[75] (2019) Introduction to tensorflow lite.
https://www.tensorflow.org/mobile/tflite/. [Online]. Available:
https://www.tensorflow.org/mobile/tflite/
[76] (2019) Caffe2: A new lightweight, modular, and scalable deep learning
framework. https://caffe2.ai/. [Online]. Available: https://caffe2.ai/
[77] X. Zhang, Y. Wang, and W. Shi, “pcamp: Performance comparison
of machine learning packages on the edges,” in USENIX Workshop
on Hot Topics in Edge Computing (HotEdge 18). Boston, MA:
USENIX Association, 2018. [Online]. Available: https://www.usenix.
org/conference/hotedge18/presentation/zhang
[78] S. Teerapittayanon, B. McDanel, and H. Kung, “Distributed deep neural
networks over the cloud, the edge and end devices,” in Distributed
Computing Systems (ICDCS), 2017 IEEE 37th International Conference
on. IEEE, 2017, pp. 328–339.
[79] Y. Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, and
L. Tang, “Neurosurgeon: Collaborative intelligence between the cloud
and mobile edge,” ACM SIGPLAN Notices, vol. 52, no. 4, pp. 615–629,
2017.
[80] T. Wang, C. Wang, X. Zhou, and H. Chen, “A survey of fpga based
deep learning accelerators: Challenges and opportunities,” arXiv preprint
arXiv:1901.04988, 2018.
[81] C. Zhang, P. Li, G. Sun, Y. Guan, B. Xiao, and J. Cong, “Optimizing
fpga-based accelerator design for deep convolutional neural networks,”
in Proceedings of the 2015 ACM/SIGDA International Symposium on
Field-Programmable Gate Arrays. ACM, 2015, pp. 161–170.
[82] J. Qiu, J. Wang, S. Yao, K. Guo, B. Li, E. Zhou, J. Yu, T. Tang,
N. Xu, S. Song et al., “Going deeper with embedded fpga platform for
convolutional neural network,” in Proceedings of the 2016 ACM/SIGDA
International Symposium on Field-Programmable Gate Arrays. ACM,
2016, pp. 26–35.
[83] (2019) Nvidia jetson tx2 module.
https://developer.nvidia.com/embedded/buy/jetson-tx2. [Online].
Available: https://developer.nvidia.com/embedded/buy/jetson-tx2
[84] (2019) Nvidia drive px: Scalable ai supercomputer for autonomous driv-
ing. https://www.nvidia.com/en-au/self-driving-cars/drive-px/. [Online].
Available: https://www.nvidia.com/en-au/self-driving-cars/drive-px/
[85] Y. Chen, T. Chen, Z. Xu, N. Sun, and O. Temam, “Diannao family:
energy-efficient hardware accelerators for machine learning,” Communi-
cations of the ACM, vol. 59, no. 11, pp. 105–112, 2016.
[86] Z. Du, R. Fasthuber, T. Chen, P. Ienne, L. Li, T. Luo, X. Feng, Y. Chen,
and O. Temam, “Shidiannao: Shifting vision processing closer to the
sensor,” in ACM SIGARCH Computer Architecture News, vol. 43, no. 3.
ACM, 2015, pp. 92–104.
[87] (2019) Edge tpu: Googles purpose-built asic designed to run inference
at the edge. https://cloud.google.com/edge-tpu/. [Online]. Available:
https://cloud.google.com/edge-tpu/