Ookla Speedtest Methodology 2020
Ookla Speedtest Methodology 2020
Ookla Speedtest Methodology 2020
Contents
Introduction 1
How Ookla® collects data 2
Speedtest measures the full capacity of an internet connection 3
Quantity + quality of test data 3
Benefits of a dedicated server network 4
Testing to a CDN alone does not provide an accurate picture of network performance 5
Understanding the last mile of a network connection 5
Testing 5G and super-fast fiber connections 6
Accurate 5G connection detection and identifcation 6
Background testing does not provide a reliable measurement of speed 7
The hidden downsides of background testing from other providers 7
Ookla provides a reliable, consistent test experience across devices 8
Testing coverage, availability and quality of service 8
How Ookla aggregates and analyzes data 9
Ookla metrics 10
Standard internet speed metrics measured by Speedtest 10
Ookla performance metrics for mobile and fixed broadband 11
Coverage metrics for mobile broadband 12
Operational definitions 15
The Ookla product suite 16
Introduction
When a consumer decides to take a test using Speedtest, it is Numbers matter in data collection, and it’s
important to look deeply at what the numbers
run as a dedicated foreground service. This is the most accurate
represent. If a testing provider were to run 100
measure of internet performance capability because it is able to background tests per day on 100 phones for 100
use enough data to flood the internet connection and measure days that record 100 values each, they’d have
the full capability of both the network connection and the 1 million measurements. That overall number
device. The connection must be fully utilized to accurately assess might sound impressive — until you realize that
network performance and quality metrics, such as download 100 phones do not give you a very wide (or
speed, upload speed, latency, packet loss, jitter and other interesting) distribution of phone types, locations
or even experiences. The numbers are even less
indicators of network conditions.
impressive when you realize that the 100 data
points they’re collecting include less significant
Companies that rely on background tests hidden inside of other details like the device’s screen width and battery
apps to measure speed only send small amounts of data back type. Those might be interesting data points
and forth. This is not a sound method for accurately measuring for tiny cross sections of our industry, but they
performance, particularly at high speeds. While useful for contribute little valuable information about
assessing metrics like signal and coverage, background tests internet speed or performance at a global scale.
are not able to flood an internet connection with a sufficient When test providers bury their SDK in third-party
apps, the number of “measurements” can be
volume of data to measure the full throughput capability. Many
smoke and mirrors designed to distract from a
companies claiming to test speed do so via background testing, low-quality user base.
which means that their measurements are incomplete and do not
offer a true picture of performance. Each and every day, over 10 million tests are
actively initiated by pressing the “Go” button
Because Speedtest operates in the foreground and measures on Speedtest, and we receive additional data
the full throughput capacity of a connection, we can properly from over 300 million coverage scans. We see
daily results on almost every mobile and fixed
measure the performance capability of each connection. This is
broadband network in the world, which provides
why speeds measured with Speedtest are often higher than those us with a real-time view of how the internet
measured with other methods. This difference can be substantial is performing at a global scale. This constant
when testing the performance of newer technologies like 5G and flow of immense amounts of data allows us to
multi-gigabit fiber. Furthermore, background tests that run on a precisely track how networks respond to events
predefined schedule that suits the data providers’ needs don’t like large crowds, the capabilities of new devices,
always reflect accurate information about internet speeds at the the impact of network upgrades and the rollout
of new technologies like 5G.
times and locations that are important to consumers.
The ability to measure a network’s full capacity is especially critical with modern network
speeds increasing rapidly across connection types like 4G, 5G and fiber broadband. Speedtest
is uniquely capable of measuring the throughput of a 5G connection and other gigabit-and-
beyond speeds. Speedtest uses a client and server testing engine that dynamically scales
the number of connections to the server in order to saturate and accurately measure client-
side connections up to 10 Gbps. This allows Ookla to measure the full extent of real-world
performance and overcome the effects of network bottlenecks such as TCP slow start (a
transmission control protocol that avoids sending more data than the network is capable
of forwarding).
Not all 5G-capable devices natively identify the 5G connection type when reporting to
applications. That’s why Ookla has directly partnered with device manufacturers worldwide
to implement accurate in-app 5G detection in Speedtest. Consumers can reliably see when
they’re testing a 5G connection in the Speedtest app, which provides a valuable feedback
loop of 5G network service.
Other testing providers often report much slower speeds than Speedtest. That’s because their
tests either run in the background of other apps without consumers’ knowledge or simply
download a very small file from a content delivery network (CDN) or server. These quick tests
cannot scale up to assess the full capacity of a connection or accurately determine upload
speed — and the results do not indicate how a connection might actually perform while using
large amounts of data over a 5G connection. Because a significant amount of demand on the
network must be generated for 5G carrier aggregation to even take effect, shorter-duration
tests are incapable of ensuring the network assigns the maximum number of component
carriers to the user equipment (UE) to trigger a 5G connection.
Other internet testing solutions run in the background of third-party applications that users
are often unaware of, such as messengers, call recorders, dating apps and media converters.
Speedtest is used and trusted by hundreds of millions of consumers worldwide to do exactly
what they expect: to provide an accurate, reliable measure of the performance and quality of
their networks.
Due to protections Apple has in place for user privacy, iOS doesn’t surface information like
connection type to apps that run in the background. It is vital that a testing solution run in
the foreground on iOS to accurately test the network and collect information about the active
SIM and access technology. Aggregate speeds you might see from other network testing
providers do not adequately represent the results of iOS users, which make up a large and
important segment of the market.
As Google continues to update their data privacy policies for Android, having a dedicated
app with a transparent consumer experience and clear user permissions around location data
collection is increasingly important.
Speedtest provides an accurate, consistent test experience across devices. We have mobile apps for both Android and
iOS that are optimized to measure the specific intricacies of a mobile connection. There are dedicated Speedtest apps for
computers (Mac and Windows), a browser extension for Chrome, an AppleTV app and embedded solutions that integrate
with routers, hardware and other connected devices. For developers who want to use the trusted technology and global
server network behind Speedtest to power their own solutions, we also offer a command-line interface tool (Speedtest CLI)
and an SDK. Our original web test at Speedtest.net covers most other platforms, and we continue to explore new platforms
for Speedtest.
Fast speeds only matter when you actually have mobile coverage. Consumer-initiated testing is the best measure for speed
and several other metrics, but to measure signal and coverage Speedtest also collects over 300 million daily scans of
coverage data in the background, submitted by Speedtest Android app users. These coverage scans capture where service is
offered and what the quality of service is at each location.
android
Ookla metrics
Here at Ookla, we are continuously researching new and better metrics to describe real-world internet performance and
coverage. Our metrics help consumers to understand their network performance and service providers to understand their
customer experience, gain competitive insights and identify how to improve their networks.
Ookla determines an operator’s geographic coverage using a sample of scans received from devices on that operator’s
network in each 100 m2 area. Because coverage is a spatially-focused metric, only scans with precise and legitimate location
information are included as we build samples that normalize data by user, operator, location and timeframe.
Android Speedtest users can enable or disable coverage scan collection at any time. When enabled, Android devices running
the Speedtest app periodically capture key characteristics of each user’s experience with the surrounding mobile networks
throughout the day. Each coverage scan captures information about the mobile network that device is connected to, including
information about the connection to that network.
As with our consumer-initiated tests, each coverage scan provides a wealth of information, including:
Information about the Information about signal Data connection type Tracking area Latitude, longitude,
subscriber network and strength and quality (e.g., LTE, WiFi) and (LAC/TAC), base station altitude, accuracy, and
the network to which the (e.g., RSRP, RSRQ, cellular connection type ID, cell ID, physical cell ID age of location (when
device is connected RSSI), frequency band (PCI) accurate GPS locations
(EARFCN, UARFCN, are not available, IP
ARFCN) and other address geolocation
properties of the cellular is used)
connection
Coverage Score™
We created Coverage Score to account for both the quality Coverage Score and Average Tile Quality
and coverage of service for mobile operators. Coverage Score
multiplies the proportion of locations in which an operator
was seen with service (its footprint) against the average
tile quality score (based on availability metrics) among all
locations in which that operator is present. Coverage Score
has a range of 0 to 1000 to avoid any potential for confusion
that te Coverage Score represents a percentage of an area or
population with coverage.
Availability
Whether or not service is available is closely tied to coverage.
We divide our data on availability into three separate
views: General Availability, 4G Availability, and On-Network
Availability. These metrics indicate how likely a user, on
average, is to have service available in the places they go. They
are individually defined as:
• General Availability
The percentage of an operator’s known locations where
a device has access to any kind of service (including
roaming).
• 4G Availability
The percentage of an operator’s known locations where a
device has access to 4G LTE service (including roaming).
• On-Network Availability
The percentage of an operator’s known locations where a
device has access to service from that operator.
Time Spent
While it might be easy to assume that a user connects to a 4G
Mean Percent Time Spent
signal 100% of the time, actual experience varies. We use Time
Spent to give mobile operators insight into the percent of time
that an average user spends on a given cellular technology —
both on and off of their subscriber (SIM) network. For example,
we could see that users in one area are using 74% 4G, 6%
Roaming 4G and 20% 3G. This can help pinpoint gaps in LTE
coverage, among other things. Time Spent uses coverage scans
to construct daily timelines for each device. Then, daily per-
device durations on subscriber network, active network and
cellular technology are calculated for a given area of interest.
Next, the percent time spent on cellular technology per-device
based on the area and time period of interest are determined.
Finally, we aggregate the average percent time over devices for
a subscriber network to give the final metrics seen in Speedtest
Intelligence™.
Operational definitions
It’s also important to define four of the key phrases we use in our articles and reports.
Top providers
To be designated as a top provider, an operator or ISPs must account for 3% or more of total test samples in the market for
the period. We use this designation to ensure that most consumers in the area actually have access to the provider to qualify
it as the fastest, most consistent or having the best coverage.
Modern chipsets
Ookla maintains a list of device models built on chipsets capable of utilizing the latest network technologies. When calculating
Speed Score and determining the winners of Speedtest Awards, we use results from devices built on modern chipsets so that
an operator’s score is not negatively impacted if a portion of its subscriber base continues to use older technology.
Footprint
Footprint is the fraction of locations within a given market, across all operators in that market, where a device has access to
service. We use Footprint in calculating Coverage Score. Operators with a relatively small footprint will have a lower Coverage
Score than competitors with equal availability and a larger footprint. We require that an operator’s footprint in their market is at
least 30% before we will calculate a Coverage Score, however, to indicate that the operator is generally available to the public.
Speedtest Intelligence provides current Powered by billions of measurements Leveraging over 22 million monthly problem
insights into global fixed broadband and from Speedtest, Cell Analytics provides reports, Downdetector Enterprise provides
mobile network performance, availability insights about wireless service quality, RF real-time outage detection, early alerting,
and quality. Leveraging billions of consumer- measurements, data usage, user density, historical data around service outages and
initiated tests, Speedtest Intelligence is an indoor vs. outdoor performance, cell site insight into third-party outages, so your team
essential tool to help organizations make locations and much more — surfaced in a can reduce downtime and increase customer
business and technical decisions with a full powerful, yet easy-to-navigate portal. satisfaction.
understanding of both the speed and quality
of networks in their markets.