EDX Merged
EDX Merged
EDX Merged
A report submitted in the partial fulfillment of the requirements for the skill oriented
course
BACHELOR OF TECHNOLOGY
IN
“ELECTRONICS & COMMUNICATION ENGINEERING”
Submitted by
NAME:
ROLL NO:
2023– 2024
AVANTHI INSTITUTE OF ENGINEERING & TECHNOLOGY
(Accredited By NAAC A+, Approved by AICTE and Permanently Affiliated to JNTUG
VIZIANAGARAM, AP)
TAMARAM (V), MAKAVARAPALEM (M), ANAKAPALLE DISTRICT-531113
CERTIFICATE
This is to certify that the course entitled “IOT101 BUILD YOUR FIRST IOT
APPLICATION WITH ARM ” is being submitted for the partial fulfillment of requirements for
the completion of the skill oriented course of done by under
guidance during year 2023 – 2024 and it has been found suitable for acceptance according to the
requirements of the University.
EXTERNAL EXAMINER
Build Your First IoT Application with Arm
CONTENTS
MODULE 1- INTERNET OF THINGS ................................................................................................. 2
1.1 The Key Elements of an IoT Device ............................................................................................. 3
1.2 The Evolution of the Internet of Things........................................................................................ 4
1.3 Key Technologies ......................................................................................................................... 5
1.4 Major Challenges for the Internet of Things ................................................................................. 7
1.5 Risks and Opportunities ................................................................................................................ 9
MODULE 2- SYSTEM ARCHITECTURE ......................................................................................... 11
2.1 Key Considerations ..................................................................................................................... 11
2.2 Cloud, Fog and Edge Computing................................................................................................ 12
2.3 Gateways in Fog Architectures ................................................................................................... 14
2.4 Internet of Things standards........................................................................................................ 16
MODULE 3- DEVICE ARCHITECTURE .......................................................................................... 17
3.1 Hardware ..................................................................................................................................... 17
3.2 Memory ....................................................................................................................................... 18
3.3 Sensors ........................................................................................................................................ 19
MODULE 4- CONNECTIVITY OF THINGS ..................................................................................... 21
4.1 Bluetooth Architecture ................................................................................................................ 21
4.2 Bluetooth Protocols ..................................................................................................................... 22
4.3 Bluetooth Security ...................................................................................................................... 24
4.4 Zigbee Architecture .................................................................................................................... 25
MODULE 5- CONNECTIVITY ON THE INTERNET ...................................................................... 27
5.1 WLAN......................................................................................................................................... 27
5.2 IEEE 802.11n .............................................................................................................................. 29
5.3 LPWAN ...................................................................................................................................... 29
5.4 LoRaWAN .................................................................................................................................. 30
5.5 NB-IoT ........................................................................................................................................ 32
MODULE 6- THE CLOUD.................................................................................................................. 33
6.1 What is the Cloud? ...................................................................................................................... 33
6.2 Virtualisation............................................................................................................................... 34
6.3 MQ Telemetry Transport ............................................................................................................ 35
6.4 Constrained Application Protocol (CoAP) ................................................................................. 37
6.5 Data Processing Pipelines ........................................................................................................... 38
DEPT OF ECE 1
Build Your First IoT Application with Arm
These things usually contain sensors that generate data, actuators that perform
physical actions, and technologies for communicating with other devices or systems.
The Cloud plays a big role in the IoT, hosting the programs that consume and process
this data.
Observe how this fitness tracker has a sensor. Heart rate data is passed to the Cloud,
which processes it. In turn, the Cloud sends a command back to the fitness tracker to
instruct it to indicate that the wearer is working at 80% of their maximum heart rate.
There are four major parts to an IoT system which includes the device, the
communication method, the Cloud, and the app. This is a typical IoT style
architecture. The device is the physical object that is the connected thing, and usually
contains an embedded system.
The Cloud is basically a large collection of servers spread across data centers. Cloud
applications process the incoming data from the IoT devices and in turn instruct
devices to perform physical actions based on that processing. The Cloud can also
provide visualisations of data, which something like a mobile app might want to look
at. The app is user-facing software that is used to interact with the device via the
Cloud. This interaction could be looking at visualisations of the data, or it could be
actually triggering outputs on the device.
There are plenty of examples of IoT devices. Home appliances, including washing
machines, fridges, kettles and vacuum cleaners are all starting to become more
intelligent and internet connected. Civil engineering structures such as bridges,
railways, and traffic lights, can all be IoT controlled. There are also wearable devices
such as smartwatches and fitness trackers, entertainment devices such as TVs and
games consoles, and even in the medical industry, biomedical devices such as
pacemakers, blood pressure monitors and hearing aids, are now IoT devices.
DEPT OF ECE 2
Build Your First IoT Application with Arm
Let's look at what an IoT system might look like using the example of a smart light
bulb. The device in this context is the light bulb itself. That device contains an LED, a
dimmer circuit to dim the LED, a Wi-Fi module for communications, and a
microprocessor to handle the communications and alter the inputs and outputs.
The communication method here is Wi-Fi because a Wi-Fi module is built-in. This
means the smart light bulb can directly communicate with the Cloud over a Wi-Fi
connection through your home router.
The Cloud is the management and control system for the smart light bulb. The Cloud
is what tells the smart device to set its brightness and this is based on how the
controller is configured. The Cloud application might work autonomously, so it might
turn the light on in the evening after a certain time, or if it detects that it has gone
dark.
The app that is on the smartphone can talk to the Cloud and say turn the light on or
turn the light off, and the Cloud can report back to the app to see if the light is
currently on or off. The app can also be used to configure how the controller works if
you want to set a lighting schedule, for example.
The Internet of Things is growing rapidly. IoT systems can be characterised by their
DEPT OF ECE 3
Build Your First IoT Application with Arm
large-scale and the billions of devices that are already in existence. The devices
themselves are characterised by the sensors and actuators that are employed. A sensor
is something for sensing the environment and an actuator is something for making
changes in the environment. Devices can employ a large range of sensors to collect
data, and then they can in turn perform actions in the real-world.
The initial conception of the IoT can be traced back to the 1980s, when students from
Carnegie Mellon University installed monitoring equipment in a vending machine to
discover the state of its contents. The students installed stock level monitoring
hardware on the vending machine, which in turn fed data into something called
ARPANET. ARPANET is the precursor to the Internet. Workstations connected to
ARPANET could then interrogate the stock level monitoring hardware and find out
what the vending machine had. This is a very early conception of the Internet of
Things.
A couple of decades later, the term IoT was coined when a marketing team was trying
to excite high-level executives about supply chain optimisation. The term “supply
chain optimisation” was thought to sound too boring and so engineers came up with
the phrase “Internet of Things”. The idea was to employ sensors throughout the supply
chain, which could track the state of products and feed this into a central algorithm
that would process all the data to optimise that supply chain and give useful
information at the end.
From these early beginnings, mainstream consumer use of IoT devices exploded.
There was the emergence of smart thermostats where you can control your home's
heating equipment remotely, a dramatic rise in the popularity of wearable fitness
trackers, and home systems and smart home devices are now prevalent.
The number of IoT devices has already surpassed the human population, and huge
growth is predicted for the future. According to various researchers, by 2030 the total
number of connected devices may reach more than 125 billion. Arm hesitates that by
2035, there could be a total of 1 trillion connected devices.
DEPT OF ECE 4
Build Your First IoT Application with Arm
You’ll also find off-the-shelf commodity hardware. Simulation and rapid prototyping
is now available so you can visualise a design virtually before you manufacture it.
There are varied sensors available for hardware, which can accurately sense the
physical environment. Power management is a key concern for these devices.
IoT devices are very small and suitable for harsh environments. Ensuring you extract
every ounce of power from the power source available is essential. We need to take
advantage of sleep states in modern microcontrollers, efficient battery systems, and
where feasible additional power sources such as solar or wind when designing an IoT
device.
Communications technology for IoT systems can be complex. Typically, IoT devices
employ wireless technology which can be divided into short-range, medium-range,
and long-range.
Short-range technologies such as Bluetooth and ZigBee can be used for device-to-
device communication or device-to-gateway communication. When a device talks to
the Internet, it talks to the gateway first, which forwards the traffic to the Internet.
Medium-range technologies such as Wi-Fi, 4G, or LTE, can be used when the device
wants to talk directly to the internet or when the gateway itself is talking to the
Internet.
DEPT OF ECE 5
Build Your First IoT Application with Arm
IoT devices can be deployed in the harshest and most distant environments.
Communication protocols are therefore important on many levels. Due to the current
explosion in the number of IoT devices, we need a way to identify each individual
one. How do we address a trillion IoT devices?
The IPv4 protocol is outdated, unscalable, and we have already run out of IPv4
addresses for desktop hardware. IPv6 which is rich in IoT features provides an
excellent solution to this problem, but comes with a learning curve for those used to
IPv4.
One key consideration for any protocol is to ensure low overheads, so embedded
systems can adequately handle them. A complex protocol requires complex
processing, increasing complexity of the embedded system and impacting on cost,
size, and energy performance.
The Cloud can seem like a daunting black-box of computing resources, and in fact
that is the idea. Cloud computing enables IoT technology because it provides
reliability and scalability of systems completely transparent to the application
developer. The reliability of Cloud computing resources enables fault tolerance for
safety critical systems to be upheld and also provides uptime guarantees for users. On
the scalability side, the Cloud provides this elastic, scalable infrastructure. When
thousands of devices suddenly turn on, the computing resources can cope.
As Cloud computing keeps everything central in this magical box, edge computing
keeps the data closer to the device and performs operations on it more locally. With
edge computing, data processing happens on the device itself rather than sending it to
the Cloud for analysis. Any results are then sent back to the device. Fog computing is
related to edge computing since some computations can be done at gateway or base
station level. The data still remains close to the device, and does not have to be
shipped from the Cloud for further analysis.
In this scenario, although processing is local, typically summaries and interesting bits
of information will still be sent to the Cloud. The serverless paradigm or function-as-
a-service provides a low-cost way to get code running in the Cloud without worrying
about computer resources. It enables rapid development of technologies because you
DEPT OF ECE 6
Build Your First IoT Application with Arm
just write a function and then it executes it. You don’t need to worry about scalability,
reliability, or even managing your own virtual machines in the Cloud.
Using data descriptions such as JSON or XML can be valuable to ensure device-
agnostic communication. There are plenty of online communities that support IoT
device development for both hobbyists and businesses. The Arm Mbed system and
even YouTube are great sources of information for IoT development, and there are
plenty of open standards to explore to guide system design.
DEPT OF ECE 7
Build Your First IoT Application with Arm
• We’ve examined the addressability of devices and the IPv6 protocol solution,
but we need to ensure this will work on resource-constrained systems. How can
we champion this? Networks must be designed with communication
characteristics in mind, so we need to consider; how much data is transmitted?
How reliable is that transmission? Where is that data going? What is the
shortest path to get data from the device to the data centers and the server for
processing? How can we ensure compatibility and interoperability with other
systems?
• Protocols and data formats need to be designed for compatibility to ensure that
legacy products still work with future versions of the Cloud.
• Once we get our data into the Cloud, we need to deal with it efficiently, and
support the sheer volume of data transmitted through networks and stored on
disk. The process needs to be efficient, resilient and accessible. If large
amounts of data require analysis, processing power must increase. We must
also optimise applications to minimise delays, reduce cost and decrease energy
consumption. Cloud computing still has costs, typically billed per second.
• How can we reduce execution times? If we shave off one millisecond across a
million devices, we gain around 60 minutes of execution time, with significant
scaled-up savings. In a data center itself, how do we maximise resource
utilisation? You don’t want the majority of services to be sitting idle, you want
to efficiently scale resources up and down on demand, remaining ready for
surges in requests.
• Security and privacy are also key issues for IoT systems: how do we ensure
devices are secure? Once they leave our possession, we can lose control and be
at the mercy of end-users. How can we verify the data generation sources?
How can we establish that it is authentic and came from a real device? How
can we confirm that devices are operating correctly and have not been
reprogrammed with malicious firmware? How can we ensure the integrity of
data generated and transmitted by these devices? Can hardware security
techniques help? There are many security and privacy issues to consider.
• One option is to employ end-to-end encryption data transfer. Device data is
encrypted and can only be decrypted at the endpoint server. Although those
packets pass through gateways and routers, they have no knowledge of packet
contents and cannot decrypt it. An additional question is: Can we anonymise
data to avoid unnecessarily tracking users?
• There are certain regulations that we may need to comply with including
banking, medical, and radio-frequency licensing laws to consider when
designing an IoT application in these domains.
• Finally, maintaining a user's trust is absolutely key for user retention. If a user
does not trust a company, they most likely will not buy products from them.
DEPT OF ECE 8
Build Your First IoT Application with Arm
Efficiency is a key driver for IoT adoption because IoT can create more efficient
business processes. In turn, this can lead to massive cost savings meaning a huge
return on investment. If you spend money on developing an IoT system for your
business that optimises the way that you operate, you can save much more money than
you put into the development of the IoT system.
If you have more information about all of the systems in one place, this helps you as a
business owner to make more intelligent decisions and automate some of these
processes as well. However, you need to be careful because too much data can
complicate systems or dilute the intended outcome. Striking a balance here is vital.
Another opportunity is in the integration of IoT systems. Two distinct systems that
never used to talk to each other can now communicate. This means you can make
more intelligent decisions about the system as a whole when you are considering
sources of data. However, it can be very challenging to support the diversity of
systems.
Old legacy systems need to be supported, which means you need to build hardware to
support them or incorporate the right software that can interface with these old
DEPT OF ECE 9
Build Your First IoT Application with Arm
systems. Heterogeneity in the designs also needs to be considered if you have different
styles of processes or different networking paradigms. Bad integrations can actually
make existing situations worse. Incomplete, untimely or invalid data that is being
brought in from legacy systems can lead to poor decision making or automation not
working at all.
DEPT OF ECE 10
Build Your First IoT Application with Arm
In this module, we will be exploring IoT systems architecture. By the end of this
module, you will be able to modify and debug a simple program using the tools and
hardware platform that will be used throughout the course.
To begin the module, we will examine the key considerations that underpin IoT
architectures such as application domains, the location of the intelligence system and
networking structures, as well as their cost and scalability implications.
We will then move on to explore Cloud computing, Fog computing and Edge
computing, which will enable you to identify and differentiate between the three
paradigms. You will be presented with examples of each to secure your understanding
of how they work and how they differ. We will then focus on Fog architectures
looking specifically at gateways, where another example will be provided.
It is important to be able to choose the right architecture for any given application, so
we will look at the type of questions you need to be asking to make this decision,
before we move on to discuss the various IoT standards that will also inform your
choices.
DEPT OF ECE 11
Build Your First IoT Application with Arm
Another consideration is where to place the intelligence in the system. IoT systems
generally have intelligence somewhere and typically you might think this lives in the
Cloud. The IoT devices send the data to the Cloud which processes it and stores it and
returns it back, but could the IoT devices themselves be smart? If so, where should the
data processing be done? Can it be done locally on the IoT device or is the IoT device
just a simple data relay?
In the first example, we can see an IoT device that has a couple of sensors and an
output. In the simple sensor data relay model, this IoT device simply forwards
information from sensors to the Cloud and the Cloud returns commands to instruct the
IoT device to change its output. Alternatively, we could bring the processor to the IoT
device itself so that the output could be changed, based on the sensors that are local to
the IoT device, and only summary information sent up to the Cloud for data storage.
A big question is what networking structure should be employed. There are latency
and throughput requirements that need to be taken into account. The latency is how
long it takes for a package to reach its destination and the throughput is the average
rate that data packets are transmitted. Do we require any bandwidth and delivery
guarantees? Is any packet loss acceptable at all?
A typical approach is to take a layered system design where you can logically separate
all of the different components of an IoT system, keeping the physical world away
from the networking fabric and away from the actual platform and the applications
that run on it. This approach enables us to break up complexity, share resources more
easily and promote interoperability.
The cost and scalability implications of the application are very important practical
aspects. A system might be feature rich but expensive to deploy and run. Cost can
impact scalability, but so can technological aspects, such as the capacity of a wireless
link. Considering the anticipated number of devices, and how we could reactively
scale to it, will help us design the system so it can cope.
DEPT OF ECE 12
Build Your First IoT Application with Arm
In the Cloud computing paradigm, IoT devices simply ship their sense of data straight
into the Cloud. There is little, if any, local processing performed on the device itself.
All of the data processing is done on powerful servers that exist in some big black box
- the Cloud. Data storage, control functions, analytics, and visualisation all live in the
Cloud. The networking infrastructure forwards the information from the IoT device to
the Cloud service. This particular model has some scalability issues as the data grows.
The particular scalability challenges are growing costs associated with routing and
storing big data and also the control signaling overheads incurred by billions of
devices that might be competing for resources with all the data traffic. As all data
processing is performed in the Cloud, this introduces latencies that might not be
appropriate for applications with real time constraints.
In the fog computing paradigm, some of the data processing functionality is offloaded
to the devices or the gateways themselves. Data processing has now moved closer to
where the information is generated. The sort of processing that you can expect to see
in fog computing is data aggregation or compression and even some processing such
as statistical generation or noise removal. There is also the ability to act locally on the
data without having to ship it to the Cloud for analysis for a result to be returned. IoT
devices are still embedded systems, so Cloud systems are still used for the main data
processing and storage.
In this example of fog computing, consider a smart home that has a number of
different devices. Taking a look at data flow one, you can see that the mobile phone is
connected to the gateway and has a direct line of communication with the smart bulb.
This means that the mobile phone can instantly tell the smart bulb to turn on or off
without having to send its command up through to the Cloud. In data flow two, we
have a smart doorbell and a bell unit. In this case, the smart doorbell can trigger the
bell unit by talking directly to it through the gateway.
There is no need for the smart doorbell to talk to the Cloud to trigger the bell unit.
However, in data flow three, you can see that the smart doorbell still sends its
information up to the Cloud for additional processing. The benefit of this is that the
Cloud can then send information to a mobile phone that might not be connected to the
gateway. It could be connected to a cell tower if it is outside of the home.
DEPT OF ECE 13
Build Your First IoT Application with Arm
In the edge computing paradigm, the computation and intelligence are executed at the
device level. Processing is performed where the data is collected and only key
information or summaries are transmitted up to the Cloud.
In the example here, we can consider a smart vehicle and IoT device where the
majority of the processing is done on the vehicle itself, by taking a range of input such
as cameras, radar, lidar, the human computer interface and GPS modules into a central
processor. This can then operate the engine, display outputs on the driver's heads up
display, operate the braking system or operate the comfort system.
All of this is done locally on the device without having to offload to the Cloud.
However, the smart vehicle itself can still generate summaries and key information
and send this up to the Cloud for processing, storage and analysis. For example, if a
fault developed with the car, this information could be transmitted to the Cloud while
still being acted upon locally.
We have looked at the fog computing paradigm and seen where and how data
processing can occur on gateways but what does the gateway actually do? A gateway
provides a means for an IoT device to communicate with other devices or systems. An
IoT device might only have a limited range of connectivity protocols, like Bluetooth
or Wi-Fi, which are not necessarily compatible with all of the other devices that want
to speak with it. A gateway can therefore be used to transform messages from one
protocol to another. In the simple data passing scenario, a gateway acts as a bridge
between different communication methods, for example, Bluetooth and Ethernet.
DEPT OF ECE 14
Build Your First IoT Application with Arm
Gateways may need to combine multiple data sources onto the same IP connection
towards a server in the Cloud. Gateways might also apply security technologies such
as encryption, to preserve confidentiality and firewalls to prevent unauthorised traffic
from entering the IoT network. More functional gateways can even act upon the data
they receive, for example triggering an output based on IoT devices sensor readings.
If we take the example fog architecture of a smart doorbell again, we can take a look
at the protocol being used. The smart doorbell itself may contain a push button and a
camera. The push button may use a Zigbee connection and the camera may use a Wi-
Fi connection. In this case, when somebody presses the push button, a Zigbee message
is sent to the gateway, the gateway can then locally relay that message to the bell
ringing unit to say “sound the doorbell”.
However, the camera data might be transmitted all the way up into the Cloud so that a
remote mobile phone can access the images. In this case, the camera is connected by
Wi-Fi to the gateway and the gateway transmits information over the home Internet
connection, which might be DSL for example, into the Internet and then from the
Internet, through fiber optic backhauls into the Cloud and to cell towers. The gateway
itself maintains a firewall which means that unauthorised access cannot be made to the
local Zigbee network or the Wi-Fi network.
DEPT OF ECE 15
Build Your First IoT Application with Arm
processors so they can do local data processing. They can run complex networking
stacks and they can employ really good security algorithms. They can maintain
multiple connections to devices in their proximity, for example, a smartphone might
have Wi-Fi, LTE, Bluetooth and NFC connections, all of which can talk to devices
and talk to the Internet.
Standardisation is also driven by industry alliances and consortia. Such is the case of
the widely used Bluetooth specification of which the 5.2 revision has been recently
released. Bluetooth is a wireless personal area networking technology. Zigbee is
similar to Bluetooth. You could think of it as a competitor but with different
characteristics. The LoRaWAN technology is specified by the Lora Alliance that
targets long range bidirectional communications for battery powered IoT devices. The
National Institute of Standards and Technologies defines encryption standards such as
the Advanced Encryption Standard - more commonly known as AES, which can be
used for security and privacy purposes, and ISO produces an IoT reference
architecture. The ITU or International Telecommunications Union provides
recommendations and reference models for IoT communication.
DEPT OF ECE 16
Build Your First IoT Application with Arm
Initially, we will discuss the hardware that can be used and the factors that influence
the device design, and then progress to focus specifically on memory, sensors,
conversion techniques, and energy consumption. We will explore the different types
of memory, from EEPROM to SRAM, equipping you with the tools to differentiate
between them.
We will then discuss the role of sensors in IoT systems and look at some examples
before exploring input and output. There are two main conversion techniques,
analogue-to-digital and digital-to-analogue, which we will consider in turn.
Finally, we approach a key issue: energy consumption. It is important that IoT devices
operate efficiently, so we will discuss some of the main ways to save energy.
3.1 Hardware
There are several factors that influence the design of a hardware platform. Depending
on the application type, constraints, and budget, different types of microcontrollers or
CPUs are adopted. For more specialised applications, graphical processing units
(GPUs), and digital signal processors (DSPs) may also be added. Whether the
platform is expected to be line-powered or battery-powered will further influence the
choice of computing modules.
Another factor considered in the design is the type of connectivity intended to be used
with the target application. Choices vary between wired and different types of wireless
solutions that not only depend on the desired throughput, but also on the energy
constraints imposed. How the system will interact with peripherals influences the
DEPT OF ECE 17
Build Your First IoT Application with Arm
choice of I/O. Options include general-purpose I/O, or GPIO, and different types of
serial interfaces. For example, I2C or SPI interfaces. Finally, the method of debugging
should also be factored into a hardware platform’s design because software will need
to be developed and tested for this platform.
3.2 Memory
Memory can be classified into two main types; volatile and non-volatile. Volatile
memory is when the contents of the memory are lost when power is removed and non-
volatile memory is when the contents remain when power is removed. You can then
sub-categorise these types into different memory technologies. In the volatile memory
arena these are Dynamic Random Access Memory, or DRAM, and Static Random
Access Memory, or SRAM. In the non-volatile arena, there's EEPROM, which is
electronically erasable, programmable, read-only memory, or flash. All of these
different types of memory have different characteristics and different use cases. Quite
often, a hardware platform might include all of them.
Typically, non-volatile memory will be used for storing persistent data such as the
program code that is running on the microcontroller, any configuration data, and any
long-term or sensitive program data that might have been generated as part of the
application's runtime. Volatile memory will be used for storing transient data such as
short-term program data or the variables that are in a program.
EEPROM and flash are types of non-volatile memory. The differences are EEPROM
is byte-erasable and flash is block-erasable. This means individual bytes of EEPROM
memory can be addressed and erased and rewritten, whereas for flash, it has to be
done in blocks of bytes. Program data is typically stored in EEPROM, whereas
program code is typically stored in flash. This is because program code does not
change at runtime, and so you do not need the ability to modify flash memory at a
granular level. Conversely, program data might need to be changed at runtime. Being
DEPT OF ECE 18
Build Your First IoT Application with Arm
able to address it a byte at a time allows fine grained access to that data.
In the non-volatile domain, choosing between DRAM and SRAM can be a cost
exercise. DRAM is cheap and SRAM is expensive. There are also additional
technological constraints on DRAM. DRAM memory must be refreshed periodically,
but SRAM does not. DRAM is much slower compared to SRAM but SRAM is
expensive compared to DRAM. The trade-off here is the cost versus speed.
3.3 Sensors
Sensors are input devices as they convert a particular physical property of the
environment into an electrical signal. Typically, sensors have a linear relationship
between the physical property and the electrical signal. For example, a light sensor
will take in photons from the physical world and convert that into a voltage that
represents the amount of light falling on the sensor. That voltage will vary between
some minimum and some maximum depending on the number of photons hitting the
light sensor. The sensitivity of a sensor indicates the minimum value of the measured
input that can produce a certain output signal. An analog-to-digital converter converts
the electrical signals produced by sensors into a form that can be processed by
microcontrollers. Different physical phenomena underpin the operation of sensors.
There are many different types of sensors, such as push buttons for receiving user
input, temperature for measuring the environment, acceleration sensors for detecting
direction and movement, pressure sensors, optical sensors, humidity, force, and even
microphones are considered to be sensors.
Input and outputs, or I/O, defines how a microcontroller can interact with the
environment. Different I/O protocols are available. The universal asynchronous
receiver transmitter, or UART, is a simple two-wire serial communication protocol
that allows data to be sent between two devices. The serial peripheral interface is a
serial bus that allows many devices to communicate with each other. Inter-integrated
circuits (Inter-IC or I2C), is similar to SPI, but uses a different physical layer and in
general combines the features of UART and SPI. General purpose I/O is the most
straightforward kind of input output.
DEPT OF ECE 19
Build Your First IoT Application with Arm
a low-power or sleep mode. GPIO pins can be grouped into GPIO ports and controlled
as such. Pulse-width modulation allows a GPIO port to be activated at a certain
frequency. This is employed when linear processes must be controlled, for example,
fans or motors. GPIO is typically limited to low current applications. If you need to
drive something that is higher current, transistors and relays are used to facilitate this.
DEPT OF ECE 20
Build Your First IoT Application with Arm
By the end of this module, you will be able to design a simple Android app to display
the data received from a Bluetooth Low Energy device.
We will begin by exploring the origins of Bluetooth and then examine Bluetooth
architecture, in addition to the associated protocols that will influence any decision
making. This will help broaden your understanding of Bluetooth’s functionality, its
key features, and Bluetooth Low Energy protocols.
We will also concentrate on Bluetooth security, which is very important for any
Bluetooth device. We will then spend some time focusing on Bluetooth 5,
concentrating on its specifications, and new features.
Bluetooth started as a late 1980s project where the initial goal of streaming music to
headphones wirelessly was pursued. Eventually, it materialised as a technology for
serial communication without cables. The first consumer product that had Bluetooth
capabilities was a Hand Street mobile headset which launched in 1999. Bluetooth is
wireless technology that works in the 2.4GHz band. The core functionality is to enable
a range of embedded devices to connect and pair with each other and exchange data
securely.
DEPT OF ECE 21
Build Your First IoT Application with Arm
This is what the Bluetooth architecture looks like. Bluetooth networks are effectively
Wireless Personal Area Networks. They comprise a Bluetooth controller and multiple
agents that connect to the controller. Here we have the Bluetooth controller with
multiple Bluetooth agents connected to it. These agents may or may not be active. The
controller can have up to seven active agents at a time and up to 255 parked or
inactive agents.
We can see here that we have a few different Bluetooth agents like fitness trackers,
audio headsets, or printers, all connected to a mobile phone acting as a controller. That
mobile phone itself could also be an agent if it is connected to a desktop PC which is
acting as a controller. That desktop PC may be connected to multiple agents too, such
as another wireless printer.
The Bluetooth Protocol is highly complex. The Bluetooth 5.2 core specification,
which exceeds 3,000 papers, is an iterative development, in that later specifications
encompass features from all the previous versions. Many of the characteristics of the
specification remain the same across the different versions, including the protocol
stack but a new one is added with each release, for example, to improve energy
DEPT OF ECE 22
Build Your First IoT Application with Arm
The Bluetooth Protocol stack is not fully aligned with models used in other
networking technology such as OSI or TCP/IP models. A certain degree of layering
does exist but some functions can cut across multiple layers as illustrated in the
diagram. What is unique to the Bluetooth specification is the definition of profiles that
essentially define the scope of a particular application. 36 distinct profiles are defined
in Bluetooth Classic whilst Bluetooth low energy adds even more. Some of these
profiles have a fundamental role as they allow for the building of other profiles on top,
such as the case of the generic access profile that handles connection establishment.
On the physical layer, or the radio layer, Bluetooth operates in the 2.4 GHz,
Industrial, Scientific, and Medical or ISM band. This band is license exempt, but it
also means that Bluetooth needs to share the radio frequency spectrum with other
technologies. To minimise the interference in such technologies whilst mitigating
interference towards Bluetooth, frequency hopping spread spectrum is adopted. This
means that the communication channel is not fixed but changes periodically in an
DEPT OF ECE 23
Build Your First IoT Application with Arm
order agreed between the controller and the agent. This also offers some degree of
confidentiality.
In the link layer, Bluetooth devices are uniquely identified by 6 byte addresses that
can be logically divided into two parts; an organisation part and a device part. The
most significant 3 bytes is the organsation part which is assigned to the manufacturer
by the Tripoli Registration Authority, and the least significant 3 bytes is assigned to
the device by the manufacturer. Data transmission and Bluetooth is preceded by an
Access Code and a Header. Bluetooth frames do not necessarily contain a payload, for
instance, when they are used for control purposes. The Access Code itself is further
divided into a preamble that allows a receiver to detect the start of a frame; a sink
word that identifies the piconet and optionally a trailer; and a Header that follows
containing various control information.
Pairing is the technique that is used to secure a link through data encryption. If the
communication is meant to be secure, it needs to have data confidentiality guarantees.
The connection setup is followed by a pairing procedure to establish link encryption.
There are three phases involved in pairing I/O capabilities: exchange, association, and
key distribution.
In the first phase, devices exchange information about their input output or I/O
capabilities. I/O capabilities are things like “is there any input available?” “Are there
at least two buttons available?” Or “is it a full keyboard?” Similarly, “is there any
output on the device?” Based on these capabilities, three different association modes
are possible. Just Works, PassKey entry, and out of band. Out of band is the most
secure, but it is also the most expensive as it requires additional hardware. Just Works
is the least secure because it uses default passcodes. PassKey entry falls between these
two.
Once the device is associated, a temporary key is generated from which a short term
DEPT OF ECE 24
Build Your First IoT Application with Arm
key is then derived. Knowing the numbers used during pairing does not help an
attacker decrypt the data exchange layer, but intercepting the initial communication
may allow an attacker to brute force a temporary key. The short term key may be
computed as the other data is sent in clear and the methodology is known. Once the
link is secured using the short term key, a key distribution protocol is used to transmit
a long-term key (or LTK) that will be used to encrypt all communication along with
an identity resolving key to support private device addresses and a connection
signature resolving key to support data signing. The LTK can therefore be
compromised and to address this problem, the Diffie-Hellman key exchange
technology was introduced into later revisions of the Bluetooth Standard.
The Bluetooth 5 specification was released in August 2018 with two new subversions.
Since then, there are four physical layer modes, all of which are Gaussian Frequency
Shift Keying (or GFSK) based. The four modes can be divided into uncoded and
coded. Coded packets can increase the range, as parts of the frames use forward error
correction to combat bit error due to signal fading, but this also diminishes the
effective data rate.
There are three types of device in a Zigbee network; a coordinator which sends
beacons that manage network-specific addresses; a router which runs some
applications and can also forward packets between other Zigbee nodes; and a Zigbee
end device, which has reduced functionality and communicates only with the Zigbee
coordinator or Zigbee router. Zigbee is more suitable for a different set of applications
where messages are less frequent and shorter.
DEPT OF ECE 25
Build Your First IoT Application with Arm
Zigbee has a well structured protocol stack that follows the OSI reference model to a
large degree. The PHY and MAC layers are effectively specified by the IEEE802.15.4
standard, whereas the other layers are specified by the Zigbee alliance. Applications
running on Zigbee devices are also referred to as Zigbee device objects or ZDOs.
DEPT OF ECE 26
Build Your First IoT Application with Arm
In this module we will explore the connectivity of the internet. From wireless LANs to
Narrow-band IoT, you will end the module with a deeper knowledge and
understanding of what connects the Internet of Things.
By the end of the module, you will be able to program an embedded device to
communicate with a service in the cloud, by transmitting sensor data through the WiFi
interface to a hosted publish/subscribe system.We begin by considering the features of
wireless local area networks in contrast to wireless personal area networks such as
Bluetooth networks. You’ll learn about the advantages of wireless LAN networks, and
their importance for applications such as smart home automation devices.
We’ll then examine definitions of wireless LANs and the protocols that implement
them, with a review of more recent performance enhancements and their benefits for
IoTsystems.
We then turn to consider long range networking for IoT systems, and introduce low
power wide area network technologies. We will consider the most popular of these,
enabling you to identify the key differences between them and their distinctive
applications.
Finally, we will investigate the Long Range Wide Area Networking (or LoRaWAN)
architecture—its protocol stacks, device types and security measures—before
considering the characteristics and architecture of Narrow-band IoT.
5.1 WLAN
In contrast to a wireless personal area network, like a Bluetooth network, a wireless
local area network (or WLAN) is a bigger deployment whose range extends to
hundreds of meters and can support higher data rates. Wireless LANs are usually
encountered in home and office buildings and can underpin smart home automation
devices. Wireless LANs date back to the 1970s when researchers sought to
interconnect the Hawaiian Islands without telephone lines. The solution was a wireless
packet data networking protocol called ALOHAnet.
The way in which devices work with ALOHA is to transmit a data packet as soon as
they arrive through the transmission queue without checking if another transmission is
already ongoing. If that is the case, and the packet delivery proves unsuccessful due to
collision, the sender would not receive an acknowledgment, and that would indicate
DEPT OF ECE 27
Build Your First IoT Application with Arm
that a retransmission is required. Retransmission involves backoff, but how much time
a node experiencing a failed transmission should wait was not specified. This clearly
should depend on the number of contenders and the amount of traffic they need to
send, but optimisation was not considered. Limitations aside, ALOHA became the
foundation for subsequent wireless LAN technology that became mainstream.
The de facto wireless LAN technology in use today follows the IEEE 802.11 family of
standards that are frequently marketed under the Wi-Fi trademark. 802.11 networks
can operate in two distinct modes: infrastructure and ad hoc. The former is the default
and the most used mode of operation, whereby an access point device acts as an
arbiter. That is, all communication takes place through it. The access point can support
contention-free operation, whereby it periodically polls client stations for
transmission.
In contrast, in ad hoc mode, the role of an access point is shared; a station advertises
an independent service set for others to join. This role however can be swapped
among participants.
IEEE 802.11 wireless LANs operate in the license-exempt bands, namely the 2.4-2.5
gigahertz and 4.915-5.85 gigahertz. These are the ISM bands or the industrial,
scientific, and medical bands. The default channel width is 20 megahertz. Twelve
logical channels are used in the 2.4 gigahertz range, but only three of these are non-
overlapping. Many more non-overlapping channels are in the five gigahertz band, but
availability depends on the regional regulation and deployment scenario, whether or
not it is indoor or outdoor.
This table summarises the 802.11 protocols as they have been advanced over time.
DEPT OF ECE 28
Build Your First IoT Application with Arm
A number of protocol enhancements have been developed over the years, primarily to
improve the throughput performance of 802.11 networks. One such enhancement is
802.11n that makes use of twice as much bandwidth if needed, by bonding together
two channels. It is also possible to employ a signal processing technique called
MIMO, M-I-M-O to allow the simultaneous transmission of multiple data streams
between two pairs and encode more bits of information per symbol to increase data
rates. The overhead associated with contention and hedges is reduced through
different frame aggregation techniques. With some, it is also possible to retransmit
only a particular part of a message if it is affected by errors. This improves channel
efficiency.
802.11ac adds new features to further enhance throughput performance. For instance,
channels up to 160 megahertz can be used. Given the spectral width, only the five
gigahertz band becomes suitable, and as such, 802.11ac are reserved for this band. Up
to eight MIMO streams are supported, effectively multiplying the data rate eight
times, and the new modulation schemes enable transmitting up to 16 bits per symbol.
In addition, a new multi-user MIMO technique is adopted that allows an access point
to transmit to multiple stations at the same time. This is called spatial multiplexing.
5.3 LPWAN
To support long-range networking for IoT, the radio frequencies employed need to be
lowered. This inherently expands the collision domains that limit the data rates that
are supported. This is to some degree an advantage for IoT because infrequent
messaging brings lower power consumption. There are a number of competing
DEPT OF ECE 29
Build Your First IoT Application with Arm
technologies in the low-power wide area network (or LPWAN) space, including:
• IEEE 802.11ah, operating in the 900 megahertz band, typically on two
megahertz channels.
• Weightless-P, deployed in sub gigahertz bands on 12.5 kilohertz channels.
• LoRa and LoRaWan, a chirp spread spectrum technique with typical data rates
less than six kilobits per second.
• Extended coverage GSM, which is based on cellular GPRS.
• Narrowband IoT, which is meant to co-exist with LTE
• and LTE Cat-M, similar to NB-IoT, but with higher data rates and mobility
support, requiring more bandwidth and must be in the same band.
It is therefore clear that there are several factors that drive the choice for a particular
technology, including the coverage-throughput trade-off (a low frequency narrow
bandwidth versus a higher frequency and larger bandwidth). Other factors include the
deployment cost and interference, licensed versus unlicensed spectrum, any
complexity, for example, centralised versus decentralised access and compatibility
with existing wireless or cellular systems, for example, 3G or LTE. Support for
mobility, for example, automotive IoT, and latency guaranteed scheduled airtime
versus contention based multiplexing are other considerations and security is now a
key factor in all LPWAN solutions.
5.4 LoRaWAN
DEPT OF ECE 30
Build Your First IoT Application with Arm
LoRaWAN networks consist of end node IoT devices that connect wirelessly to
gateways. In turn, these relay information from end nodes to the network servers
where different applications consume the data received and vice versa. The
communication over the wireless link uses LoRa modulation, whereas the backhaul-
links transport TCP/IP traffic over virtually any networking technology. The
communication between end nodes and applications is secured end to end.
LoRaWAN is effectively the protocol stack laid on top of the LoRa physical layer.
Gateways have the primary role of forwarding packets between end nodes and
network servers, but they are also in charge of end node activation, authorising access,
and securing the links. The channel, which is slotted and multiplexed, follows an
ALOHA like protocol. The data transmitted by end nodes, for example sensor
readings, is encrypted as well, and only the intended application can decrypt this. In
the same way, potential commands issued by an application running on a Cloud-based
network server to control an actuator are encrypted.
Three classes of devices are defined by the LoRaWAN specification, all of which
mandate bidirectional communication between end nodes and applications. Any
device must implement Class A functionality whereby a node can transmit within
duty-cycle limits and each transmission is followed by two short windows for
receiving down link traffic. Class B devices can work with additional receive
windows that are precisely scheduled, hence ensuring controlled latency. Lastly, Class
C devices listen continuously, unless they are engaged in a transmission. This limits
the possibility of running on batteries.
Two layers of security exist in LoRaWAN. Links are secured with AES-128
encryption using a network key known to the end device and the gateway. The
payload is AES-128 encrypted with an application key that is known to the end node
and the service logic consuming data from controlling the end device. The advantage
is that the network entity along the path cannot inspect the content of the messages.
DEPT OF ECE 31
Build Your First IoT Application with Arm
5.5 NB-IoT
Some of the characteristics of NB-IoT systems are the bandwidth operating at 180
kilohertz. Data rates of 25 kilobits per second down, and 64 kilobits per second up are
achievable, and latency less than 10 milliseconds are also achievable, because
transmissions are aligned with LTE frames. NB-IoT implements a hybrid Automatic
Repeat Request scheme for reliability. Energy efficiency is maintained as the base
station controls the transmit power and sleep modes via signaling.
DEPT OF ECE 32
Build Your First IoT Application with Arm
Up to this point, you have explored both systems and device architecture, whilst
examining both the connectivity of things and the connectivity of the internet. We
hope this has helped you to develop a thorough understanding of these topics, and that
your understanding of IoT will now be much deeper than when you began the
course.In this final module, we will conclude the course with the consideration of the
Cloud; What is the Cloud? How does it work? And how does it help the Internet of
Things?
By the end of this final module, you will be able to create a wearable activity tracker
that uses a simple neural network model in the cloud to predict different classes of
activities using sensors in real time.We will begin simply by defining the Cloud before
moving on to consider different approaches to virtualisation—the process by which a
virtual version of a physical platform is created. We will also look at hardware virtual
machines as well as containers, and consider the benefits and limitations of each.
You will discover the inner workings of MQ Telemetry Transport with its privacy and
scalability advantages. We’ll compare these with Constrained Application Protocols,
which is designed for IoT environments with constrained devices and low-power
networks. Finally you will consider the role of data processing pipelines to perform
computation on data collected from IoT device sensors.
Cloud computing is a paradigm that allows computer system resources, such as data
storage or compute power, to be accessed on-demand without the user having to
concern themselves with system administration. The term Cloud is widely used to
refer to collections of high availability servers that store and process large volumes of
data, allowing the developers to focus on software development and data analytics
without worrying about system administration.
The Cloud offers three main service models; Infrastructure as a Service, Platform as a
Service, and Software as a Service. Infrastructure as a Service is where the user has
access to raw virtual machines to install and control their own operating systems or
access the networking fabric that connects them together, as well as raw storage
platforms. The end-user does not manage the hardware themselves, but can request
virtual resources from the provider.
DEPT OF ECE 33
Build Your First IoT Application with Arm
In the Platform as a Service offering, the user can access web servers, application
runtimes, databases and development tools, all without having to manage the
underlying operating system. This is a level above Infrastructure as a Service.
The next level is software as a service. Here, actual applications are running in a
Cloud environment such as office productivity suites, email systems, customer
relationship management systems, and financial systems. Here, the user does not
know anything about the underlying architecture of the hardware that is running it,
other than the fact that they have access to these applications.
The Cloud can be used for lots of different aspects of IoT and can support a number of
key tasks, such as the remote management and configuration of devices. This means
managing devices’ connectivity or sending firmware updates. Aggregating,
processing, storing, analysing, and visualising data, are key applications of compute
resources in the Cloud. Another useful aspect of the Cloud is the sharing of computing
infrastructure. Multiple tenants can share the same physical compute resources
without interfering with each of those processes. Providers can offer a customised
version of the same service to different users while still being based on the same
operating system or software libraries.
6.2 Virtualisation
A key enabler for Cloud computing is that of virtualisation. This refers to the process
of creating logical partitions or copies of a physical resource to enable sharing and
customisation between different users. In general, virtualisation is the process by
which a virtual version of a Physical Platform is created. Virtualisation can be applied
to hardware, networks, storage, or even operating systems. In theory, any physical
thing can be virtualised right down from the hardware through to the networking
infrastructure, to the storage layer, or the operating systems themselves.
DEPT OF ECE 34
Build Your First IoT Application with Arm
and monitored by a hypervisor. There are two types of virtual machine hypervisor: a
type 1 or bare-metal hypervisor, runs directly on the physical host machine, for
example, Microsoft Hyper-V. A type 2, or a hosted hypervisor runs on top of the
operating system (or OS) of a host, for example, VMware or virtual box.
Containers are much faster to instantiate than a virtual machine and can be created and
destroyed as needed. There is no need for a hypervisor, just a Container Runtime.
Sharing operating system Libraries among Containers is quite common. Containers
are also really easy to scale.
There are many similarities between serverless computing and containerisation, but
containers are running all the time and are required to configure libraries upon
deployment. Instead, functions that make up applications are stored in the Cloud and
can be instantiated much faster, with the tenant only paying for the amount of time the
function is running.
DEPT OF ECE 35
Build Your First IoT Application with Arm
Like any other protocol that relies on TCP for message transport, communication can
be secured through Transport Layer Security (or TLS) encryption. Given the short size
of the packets exchanged with MQTT, the protocol is particularly suitable for low
bandwidth network deployments, so is perfect for the Internet of Things. Two
mainstream versions of the protocol exist, namely version 3.1 and 5. The latter
introduces new features such as session and message expiry, shared subscriptions, the
ability to specify the maximum packet size supported, enhanced authentication and
OAuth.
MQTT nodes are interconnected in a star topology whereby a central entity, called the
broker, mediates between publishers and subscribers. Despite the risk of network
collapse due to broker failure, this approach has two main advantages: privacy—
clients do not need to know about each other's addresses, — and scalability—more
clients can always be added.
Publishers are able to choose different quality of service levels for each message
transmitted to the broker based on network reliability. Three quality of service (or
QoS) levels are defined with a lower number indicating fewer delivery guarantees, but
faster delivery in the absence of packet loss. The highest QoS level, level 2 employs a
four-way handshake to guarantee a message is delivered exactly once.
Packages with QoS1 are faster to deliver and equivalent reliability can be guaranteed
if the service can handle duplicates. With QoS0 each message is transmitted at most
once with the best effort delivery. The message is not acknowledged by the broker,the
message is not stored, and delivery is not re-attempted if the message is lost in transit.
DEPT OF ECE 36
Build Your First IoT Application with Arm
The quality of service feature allows publishers to obtain certain guarantees about the
delivery of their messages to a broker, but offers no guarantee about delivery to
subscribers. Similarly, a subscriber to a topic where, for example, messages are
published once a day would have a hard time distinguishing between a publisher
failure and a late join. Retained messages seek to address these problems. Specifically,
brokers will store messages that have the retained message set. New subscribers will
just immediately receive the latest message published on a topic, even if this has been
long before they subscribed.
To further reduce the message size, CoAP packets only have a 4-byte header, and
some constraints to the message body apply as well. For instance, when the
communication relies on IPV6 over low-power wireless personal area networks,
6LoWPAN, the message should fit into a single IEEE 802.15.4 frame. Being expected
to operate all the lossy links, CoAP nodes have the ability to cache request and
response messages. Unlike HTTP, CoAP also has multicast support that can be useful
in machine-to-machine communication scenarios and the mechanism for resource
DEPT OF ECE 37
Build Your First IoT Application with Arm
Unlike in MQTT with a star topology, where a clear separation between clients and
broker exists, with CoAP, a device can act both as a client and the server. To be able
to match a request to a response in the context of lossy links, CoAP messages embed a
token that helps with the said matching. A number of options can be enclosed in a
packet as well. For instance, the observe option is particularly useful to monitor a
specific resource, for example, when new firmware was published, or the state of a
device through remote management purposes. A value of zero in the observed option
is used to register interest in receiving updates about a target resource. This effectively
sets up a notification mechanism by which the volume of network traffic can be
drastically reduced.
Sometimes it may be required to transfer larger content that does not fit inside the
small CoAP messages. Instead of relying on IP fragmentation, CoAP defines a block-
wise transfer mode that introduces a pair of block options for transferring multiple
blocks of information comprising a resource representation through multiple request
response pairs. Datagram TLS can be optionally added to guarantee high-level
communication security.
A CoAP packet contains the following layout. A version flag indicates the protocol
version. A type flag indicates whether the request is confirmable or non confirmable,
or the response is an acknowledgment or a reset. The TKL field contains the length of
the token. The code field contains a status code similar to HTTP status codes. The
message ID is used to detect duplicate messages, and the token is an optional 0-8
bytes request response matching identifier. Following this header information are
additional options, and finally, the message payload.
The device may act on the raw data or some level of pre-processing would be
performed close to the device at the edge, for instance, on an IoT Hub. Such
preprocessing includes the removal of noise or converting the raw data into higher
level representation summaries. For instance, a wearable device might not transmit
DEPT OF ECE 38
Build Your First IoT Application with Arm
accelerometer readings, but step counts that are being computed on the device. Data
streams arriving to the Cloud from different IoT devices may be subject to real-time
computation, such as online machine learning or real-time analytics facilitated by
platforms such as Apache Storm. Several systems may need to be fed with data
streams, each of which will act on this independently. Platforms such as Apache
Kafka facilitate such stream processing and topic based publishing. When real-time
analytics are not desired, the challenge is how to store big data for later processing.
One solution consists of non-relational distributed databases that allows for rapid
query and selection among huge numbers of records. Apache HBase offers this
capability, and it is open-source. Bigtable's go hand in hand with distributed storage
solutions, such as the Hadoop Distributed File System, where clusters of nodes are
used to store large amounts of data and split-apply-combine strategy programming
models for data analysis act on the store data, for example, MapReduce.
Data is also stored longer-term and combined with different sources to serve more
complex query and analytics engines. One relevant example is Apache Hive, which
offers an SQL-like interface to interrogate data stored in different databases and file
systems to integrate with Hadoop. When multiple processes or pipelines are expected
to run in parallel, workload scheduling tools are beneficial. The core idea of workflow
scheduling is to combine multiple complex jobs and specify their order of execution
so as to achieve a bigger task.
Distributed file systems (or DFS) have been developed to handle the storage of large
data volumes and fast retrieval of the information of interest. Unlike conventional
storage, data in DFS resides on different inter-networked machines and/or sites. The
DFS offers a unified transparent view to the client.
A namespace augmented with metadata, enables locating the content of interest fast
without disclosing to the client where it is stored. All of this is possible without
requiring an application and database to manage the metadata. Given that data is
physically stored into blocks that may be replicated on multiple machines or sites,
concurrency is very important to ensure all clients have the same view of data.
Workflows can be seen as directed acyclic graphs or DAGs comprising control flow
and action nodes. Control flow nodes specify the start and the end of workflows and
the mechanism to control execution paths. Action nodes trigger the execution of
processing and computation tasks. It is possible to run multiple jobs in parallel.
Apache Oozie is one example of a workflow scheduler that is tightly integrated with
Hadoop and can incorporate different types of Apache jobs.
DEPT OF ECE 39