INTERNATIONAL TELECOMMUNICATION UNION

ITU-T E.804.1
TELECOMMUNICATION (10/2020)
STANDARDIZATION SECTOR
OF ITU

SERIES E: OVERALL NETWORK OPERATION,

TELEPHONE SERVICE, SERVICE OPERATION AND
HUMAN FACTORS
Quality of telecommunication services: concepts, models,
objectives and dependability planning – Terms and
definitions related to the quality of telecommunication
services
Application guide for Recommendation
ITU-T E.804: Quality of service aspects for
popular services in mobile networks

CAUTION !
PREPUBLISHED RECOMMENDATION
This prepublication is an unedited version of a recently approved Recommendation.
It will be replaced by the published version after editing. Therefore, there will be
differences between this prepublication and the published version.
 FOREWORD
The International Telecommunication Union (ITU) is the United Nations specialized agency in the field of
telecommunications, information and communication technologies (ICTs). The ITU Telecommunication
Standardization Sector (ITU-T) is a permanent organ of ITU. ITU-T is responsible for studying technical,
operating and tariff questions and issuing Recommendations on them with a view to standardizing
telecommunications on a worldwide basis.
The World Telecommunication Standardization Assembly (WTSA), which meets every four years, establishes
the topics for study by the ITU-T study groups which, in turn, produce Recommendations on these topics.
The approval of ITU-T Recommendations is covered by the procedure laid down in WTSA Resolution 1.
In some areas of information technology which fall within ITU-T's purview, the necessary standards are
prepared on a collaborative basis with ISO and IEC.

NOTE
In this Recommendation, the expression "Administration" is used for conciseness to indicate both a
telecommunication administration and a recognized operating agency.
Compliance with this Recommendation is voluntary. However, the Recommendation may contain certain
mandatory provisions (to ensure, e.g., interoperability or applicability) and compliance with the
Recommendation is achieved when all of these mandatory provisions are met. The words "shall" or some other
obligatory language such as "must" and the negative equivalents are used to express requirements. The use of
such words does not suggest that compliance with the Recommendation is required of any party.

INTELLECTUAL PROPERTY RIGHTS

ITU draws attention to the possibility that the practice or implementation of this Recommendation may involve
the use of a claimed Intellectual Property Right. ITU takes no position concerning the evidence, validity or
applicability of claimed Intellectual Property Rights, whether asserted by ITU members or others outside of
the Recommendation development process.
As of the date of approval of this Recommendation, ITU [had/had not] received notice of intellectual property,
protected by patents, which may be required to implement this Recommendation. However, implementers are
cautioned that this may not represent the latest information and are therefore strongly urged to consult the TSB
patent database at http://www.itu.int/ITU-T/ipr/.

© ITU 2020
All rights reserved. No part of this publication may be reproduced, by any means whatsoever, without the
prior written permission of ITU.
Recommendation ITU-T E.804.1

Application guide for Recommendation ITU-T E.804: Quality of service aspects

for popular services in mobile networks
Summary
This Recommendation provides detailed guidance on the application of QoS metrics defined in
Recommendation [ITU-T E.804] section 7.
Keywords
Guidance; QoS; mobile services; metrics; trigger points; user perception

Introduction
The current list of services presented in section 7 of [ITU-T E.804], and the associated list of metrics
defined, is rather large:
- 11 service independent QoS parameters (section 7.2)
- 14 direct services, with a total of 175 QoS parameters (section 7.3)
- 3 store-and-forward services, with a total of 70 QoS parameters (section 7.4)
When readers and potential users (mostly service providers and national regulation authorities, for
monitoring purposes) consider this material, and before considering the provision in the following
sections of [ITU-T E.804], the first principal question arising is how to practically use it.
The solution to this question can be divided into two separate threads:
1) The prioritization of services and metrics1. Indeed, the supervision of a network cannot be
based on a too large amount of criteria. A subset of relevant services and metrics must be
identified. To do so, recent evolutions in the usage of mobile applications, as well as new
philosophies for monitoring, which are not yet incorporated in the current version of [ITU-T
E.804], must also be taken into account.
2) The application of the selected metrics. Even if [ITU-T E.804] provides trigger points as well
as instantiation across various access technologies or guidance for the usage of measurement
tools, in some cases the elements provided need to be completed to achieve an applicable
measurement procedure.
This Recommendation answers both parts of the question, taking into account the relevance of the
service or metric for end-users. To do so, the following information is provided in the following
sections of this Recommendation:
- In section 6, service independent QoS parameters from section 7.2 of [ITU-T E.804] are
reviewed, a subset of meaningful KPIs is proposed, and when needed detailed information
provided to better apply them practically.

1
Since the adoption of E.804 (in 2014), the usage of vocabulary around mobile services has slightly
evolved. The term “service” itself in E.804 corresponds to what end users experience, while
nowadays it is often understood with a meaning restricted to the access to network interface, and
replaced by the term “application” taken from the vocabulary of Google and Apple. The usage of
mobile services or applications has also become less monolithic than before, what could result in
some difficulty to apply the categories (both in terms of services and parameters) defined in
E.804. In the current version of this document however, most of the vocabulary and assumptions
of E.804 are kept.

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 1

 - In sections 7 and 8, services and applications from sections 7.3 and 7.4 of [ITU-T E.804]
respectively are reviewed and a subset of services is proposed; then for each selected service,
corresponding parameters are reviewed, a subset of meaningful QoS parameters is proposed,
and when needed detailed information provided to better apply them practically.
The proposal made in this Recommendation for services is based on assumptions mostly related to
an average business model of a mobile operator. This selection may differ from the appropriate one
(potentially operator-specific) at local level. On the other hand, commercial considerations may
suggest a set of services which cover the largest possible ground, in the sense that a particular service
may be not the most popular service but its KPI could be good proxies for a large set of actual use
cases. In that perspective, [ETSI TR 103 559] contains an agreed collection of test scenarios that can
be considered relevant. A more generic Recommendation can also be found in [ITU-T E.830].
Detailed information is provided on metrics only in case of incompleteness of the definition and
trigger points in [ITU-T E.804]. Otherwise, users will refer to information already present in [ITU-T
E.804]. A summary of all recommended services and metrics and the respective placeholders for
relevant information is provided in section 9.
The material provided in this Recommendation is a complement to [ITU-T E.804]. It does not replace
provision contained in sections 6 and 8 to 12 of [ITU-T E.804].
The lists of services and metrics in this Recommendation, and associated information, can be
expanded in the future based on new contributions.

1 Scope
This Recommendation provides detailed guidance on the application of QoS metrics defined in
Recommendation ITU-T E.804 section 7 on the definition of quality of service parameters and their
computation.

2 References
The following ITU-T Recommendations and other references contain provisions which, through
reference in this text, constitute provisions of this Recommendation. At the time of publication, the
editions indicated were valid. All Recommendations and other references are subject to revision;
users of this Recommendation are therefore encouraged to investigate the possibility of applying the
most recent edition of the Recommendations and other references listed below. A list of the currently
valid ITU-T Recommendations is regularly published. The reference to a document within this
Recommendation does not give it, as a stand-alone document, the status of a Recommendation.
[ITU-T E.804] Recommendation ITU-T E.804 (02/2014), Quality of service aspects for
popular services in mobile networks.
[ITU-T E.807] Recommendation ITU-T E.807 (02/2014), Definitions, associated
measurement methods and guidance targets of user-centric parameters for
call handling in cellular mobile voice service.
[ITU-T E.830] Recommendation ITU-T E.830 (10/1992), Models for the specification,
evaluation and allocation of serveability and service integrity.
[ITU-T G.1028] Recommendation ITU-T G.1028 (06/2019), End-to-end quality of service for
voice over 4G mobile networks.
[ITU-T G.1028.1] Recommendation ITU-T G.1028.1 (02/2019), End-to-end quality of service
for video telephony over 4G mobile networks.

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 2

[ITU-T G.1028.2] Recommendation ITU-T G.1028.2 (06/2019), Assessment of the LTE circuit
switched fall back - Impact on voice quality of service.
[ITU-T J.343] Recommendation ITU-T J.343 (11/2014), Hybrid perceptual bitstream
models for objective video quality measurements
[ITU-T P.501] Recommendation ITU-T P.501 (03/2017), Test signals for use in
telephonometry.
[ITU-T P.862] Recommendation ITU-T P.862 (02/2001), Perceptual evaluation of speech
quality (PESQ): An objective method for end-to-end speech quality
assessment of narrow-band telephone networks and speech codecs.
[ITU-T P.862.1] Recommendation ITU-T P.862.1 (11/2003), Mapping function for
transforming P.862 raw result scores to MOS-LQO.
[ITU-T P.862.2] Recommendation ITU-T P.862.2 (11/2007), Wideband extension to
Recommendation P.862 for the assessment of wideband telephone networks
and speech codecs.
[ITU-T P.862.3] Recommendation ITU-T P.862.3 (11/2007), Application guide for objective
quality measurement based on Recommendations P.862, P.862.1 and
P.862.2.
[ITU-T P.863] Recommendation ITU-T P.863 (03/2018), Perceptual objective listening
quality prediction.
[ITU-T P.863.1] Recommendation ITU-T P.863.1 (06/2019), Application guide for
Recommendation ITU-T P.863.
[ITU-T P.1201.1] Recommendation ITU-T P.1201.1 (10/2012), Parametric non-intrusive
assessment of audio-visual media streaming quality – Lower resolution
application area
[ITU-T P.1202.1] Recommendation ITU-T P.1202.1 (10/2012), Parametric non-intrusive
bitstream assessment of video media streaming quality – Lower resolution
application area
[ITU-T P.1203.1] Recommendation ITU-T P.1203.1 (01/2019), Parametric bitstream-based
quality assessment of progressive download and adaptive audio-visual
streaming services over reliable transport – Video quality estimation module
[ITU-T P.1203.2] Recommendation ITU-T P.1203.2 (10/2017), Parametric bitstream-based
quality assessment of progressive download and adaptive audio-visual
streaming services over reliable transport – Audio quality estimation module
[ITU-T P.1203.3] Recommendation ITU-T P.1203.3 (01/2019), Parametric bitstream-based
quality assessment of progressive download and adaptive audio-visual
streaming services over reliable transport – Quality integration module
[ITU-T P.1204.3] Recommendation ITU-T P.1204.3 (01/2020), Video quality assessment of
streaming services over reliable transport for resolutions up to 4K with
access to full bitstream information
[ITU-T P.1204.4] Recommendation ITU-T P.1204.4 (01/2020), Video quality assessment of
streaming services over reliable transport for resolutions up to 4K with
access to full and reduced reference pixel information

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 3

[ITU-T P.1204.5] Recommendation ITU-T P.1204.5 (01/2020), Video quality assessment of
streaming services over reliable transport for resolutions up to 4K with
access to transport and received pixel information
[ETSI TR 101 578] ETSI Technical Report 101 578 v 1.3.1 (10/2018), QoS aspects of TCP-based
video services like YouTube™
[ETSI TR 103 138] ETSI Technical Report 103 138 v 1.5.1 (08/2018), Speech samples and their
use for QoS testing
[ETSI TR 103 559] ETSI Technical Report 103 559 v 1.1.1 (08/2019), Best practices for robust
network QoS benchmark testing and scoring

3 Definitions

3.1 Terms defined elsewhere

This Recommendation uses the terms defined in [ITU-T E.804].

3.2 Terms defined in this Recommendation

None

4 Abbreviations and acronyms

This Recommendation uses the following abbreviations and acronyms:
2G Second generation of mobile networks
3G Third generation of mobile networks
4G Fourth generation of mobile networks
CLIP Calling Line Identification Presentation
CSFB Circuit Switched FallBack
DNS Domain Name System
FB Full Band (telephony)
GSM Global System for Mobile Communications
HD High Definition (for voice)
http Hypertext Transfer Protocol
IMS IP multimedia system
KPI Key Performance Indicator
MMS Multimedia Messaging Service
MOS Mean Opinion Score
MOS-LQO MOS – Voice Listening Quality (Objective assessment)
MTSI Multimedia Telephony Service for IMS
NB Narrow Band (telephony)
PDP Packet Data Protocol
QoS Quality of service

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 4

RTP Real time Transport Protocol
RTSP Real Time Streaming Protocol
SMS Short Messages services
SWB Super Wide Band (telephony)
TCP Transmission Control Protocol
UDP User Datagram Protocol
UMTS Universal Mobile Telecommunications System
VoLTE Voice over Long Term Evolution of mobile networks
WAP Wireless Application Protocol
WB Wide Band (telephony)
WLAN Wireless Local Access Network

5 Conventions
None.

6 Service independent QoS parameters

The parameters exposed in this section are taken from section 7.2 of [ITU-T E.804].

6.1 Radio network unavailability

See section 7.2.1 of [ITU-T E.804]
Unavailability describes the probability that a network or service is not offered to a user (see section
6.4.1 of [ITU-T E.804]). This is a basic notion of network performance for all stakeholders (for
operators it is a sign that network operations failed somehow, whereas for end users or regulatory
authorities it is the symptom that the promise of access to the services is not kept) and a QoS criterion
independent on the considered service. This means that it has a much more global application scope
than others parameters defined in section 7.2 of [ITU-T E.804] and must be considered in priority.
With this definition, this is not possible to distinguish between the case when the operator is
systematically and deliberately not offering the network or the service, and the one when they should
be offered there but aren’t. These different cases are linked to potentially separate regulatory
requirements, and to separate expectations which are part of at least QoS if not QoE.
Radio network unavailability is highly linked with underlying radio technical KPIs and can be derived
from the computation of statistics on these KPIs.

From a practical point of view, some end-to-end service KPIs (such as the one mentioned in section
7.2.2 for telephony, for instance) implicitly contain the element of unavailability. The result will
basically be the same if there is no network at all or if there is a network which just cannot handle the
setup attempt. Finding which was the case would then be part of “drilldown”.

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 5

6.2 Other parameters
Others interesting parameters are well defined in [ITU-T E.804] section 7.2 and do not require any
further definition. These are (the numbers in bracket correspond to the respective section in [ITU-T
E.804]):
- Network selection and registration failure ratio (7.2.2.1)
- Network selection and registration time (7.2.2.2)
Furthermore, as exposed in section 6.1, other service independent QoS parameters defined in section
7.2 of [ITU-T E.804] are more specific to a limited aspect of service delivery, represent a sub-part of
radio network unavailability, or can be redundant with service-specific parameters closer to real users’
perception. This is not a priority to implement them. These are (the numbers in bracket correspond to
the respective section in [ITU-T E.804]):
- Attach failure ratio (7.2.3)
- Attach set-up time (7.2.4)
- PDP context activation failure ratio (7.2.5)
- PDP context activation time (7.2.6)
- PDP context cut-off ratio (7.2.7)
- Data call access failure ratio (7.2.8)
- Data call access time (7.2.9)
- DNS host name resolution failure ratio (7.2.10)
- DNS host name resolution time (7.2.11)

7 QoS parameters for direct services

The parameters exposed in this section are taken from section 7.3 of [ITU-T E.804].
Out of the 14 services identified in [ITU-T E.804], three of them represent the large majority of usage
and network throughput and will be addressed in this section. These are (the numbers in bracket
correspond to the respective section in [ITU-T E.804]):
- Audio-visual streaming (7.3.5) 2
- Telephony (7.3.6)
- HTTP-based data services (7.3.8)3

7.1 Audio-visual streaming

This service is not defined in [ITU-T E.804]. Instead, the triggers for a streaming session are given,
in terms of signaling messages (mostly RTSP), which is normally enough for defining metrics based
on detection of events. It has to be noted that the name of this service has been modified in this
document in comparison with [ITU-T E.804] (where the term “video streaming” is used with its
common meaning including audio-visual contents) in order to avoid misinterpretation and to highlight
the fact that video contents are not the only ones concerned; indeed audio signal is also streamed at
the same time (and [ITU-T E.804] is proposing metrics concerning the audio medium).
Section 7.3.5.3 of [ITU-T E.804] provides a description of a typical audio-visual streaming session.
This description requires some updates or complements as follows.
- Signalling flows, exposed in section 7.3.5.3.1 and figure 7-21, are representative of a part of
use cases only, since they take the assumption that the audio and video payload is carried by

2
Note that the respective section of [ITU-T E.804] is entitled “Video streaming”
3
Note that the respective section of [ITU-T E.804] is entitled “Web browsing”

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 6

 RTP packets over UDP. In reality, HTTP packets on TCP are often used in most cases.
Anyway, the protocol for the transport of real-time packets has no influence on the metrics
and the way to measure them.
- The different phases of the session, exposed in section 7.3.5.3.2, and associated trigger
points t0 to t3 in figure 7-22 can be completed to reach a more complete view of the session.
The figure 1 below, taken from [ETSI TR 101 578], provides this. Trigger points t0 to t3
correspond respectively to each of the 4 first vertical bold (plain or dashed) lines (starting
from the left) in figure 1.

- Figure 1: Typical phases of IP-based audio-visual services, as from [ETSI TR 101 578],

7.1.1 Streaming service non accessibility

See section 7.3.5.4 of [ITU-T E.804].
This parameter corresponds to the probability that a request for a session started at time trigger t0
never reaches the status defined for time trigger t1 (as from figure 7-22 from [ITU-T E.804]).
The technical description of trigger point t0 is currently restricted to WAP. In reality, this protocol is
almost never used for streaming audio-visual contents. A more generic description is provided in
table 2 of [ETSI TR 101 578] and must be used instead: “The corresponding event in the application
or browser”.
7.1.2 Streaming service access time
See section 7.3.5.5 of [ITU-T E.804].
This parameter corresponds to the time between time triggers t0 and t2 in figure 7-2 from [ITU-T
E.804].
The information provided in section 7.1.1 on trigger point t0 applies also here.
[ETSI TR 101 578] proposes a timeout associated to this metrics at a value of 60 seconds.

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 7

7.1.3 Streaming video quality
See section 7.3.5.8 of [ITU-T E.804]. This section states that there are no standardized solutions yet.
Since the adoption of [ITU-T E.804], this situation has evolved and several relevant ITU-T
Recommendations can be mentioned:
- The P.1201 and P.1202 series of ITU-T Recommendations apply for UDP-based streaming,
with specific documents for mobile applications:
o [ITU-T P.1201.1] is parametric (it uses packet header information only) and provides
audio, video and audio-visual quality estimates.
o [ITU-T P.1202.1] is bitstream-based and addresses only video quality. This model is
dedicated to performance and quality assessment of live networks including the effect
due to encoding and transmission errors.
- The P.1203 series of ITU-T Recommendations is bitstream-based and applies for TCP-based
streaming, with 4 different modes of operation depending on the level of encryption of the
application, and provides audio ([ITU-T P.1203.2]), video ([ITU-T P.1203.1]) and audio-
visual ([ITU-T P.1203.3]) quality estimates.
- The P.1204 series of ITU-T Recommendations applies to adaptive streaming with video
resolution up to 4k (but includes formats adapted to a usage on mobile phones):
o [ITU-T P.1204.3]: Video quality assessment with access to full bitstream
information
o [ITU-T P.1204.4]: Video quality assessment with access to full and reduced
reference pixel information
o [ITU-T P.1204.5]: Video quality assessment with access to transport and received
pixel information
- In case access to image information is given, image-based measures are preferred (J.343.x,
P.1204.4 and .5). In case no access to image is available, bitstream measures as in P.1203.x
can be applied
Streaming video quality is estimated between trigger points t2 and t3.
Note: Model architecture is modular, and in cases building blocks from different Recommendation
series should be possible to combine (for instance a model for video quality could be combined with
a model for global audio-visual quality from another Recommendation). However, there is not
description of how that could and should be done in current Recommendations, and it is not just a
straight forward plug and play.

7.1.4 Streaming reproduction metrics

See sections 7.3.5.6 (cut-off ration), 7.3.5.10 (start failure ratio) and 7.3.5.11 (start delay) of [ITU-T
E.804].
The most up to date definitions of some of these metrics can be found in [ETSI TR 101 578]:
- Video access failure ratio: The overall failure ratio for the video access between t0 and t2
- Video Playout Cut-off Ratio: The overall cut-off ratio for the video playout between t2 and
t3.

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 8

Although these definitions are given for video only, they can easily be expanded for the whole content
of audio-visual clips.

7.1.5 Streaming rebuffering metrics

See sections 7.3.5.14 (failure ratio) and 7.3.5.15 (time) of [ITU-T E.804]
The most up to date definitions of some of these metrics can be found in [ETSI TR 101 578]:
- Video freezing: The proportion of the accumulated video freezing duration in relation to the
actual video playout duration (including freezings) for successful playout starts, considered
between trigger points t2 and t3.
Furthermore, [ETSI TR 101 578] proposes timeouts associated to rebuffering events:
- Rebuffering Timeout (Single) 30 s.
- Rebuffering Timeout (Total) 75 % of session time.
- Max Allowed Rebuffering Frequency 20 rebuf/min.

7.1.6 Other parameters

Others interesting parameters are well defined in [ITU-T E.804] section 7.3.5 and do not require any
further definition. These are (the numbers in bracket correspond to the respective section in [ITU-T
E.804]):
- Streaming audio quality (7.3.5.7)
- Streaming audio/video de-synchronization (7.3.5.9)
Furthermore, other QoS parameters defined in section 7.3.5 of [ITU-T E.804] are less important to
implement and supervise, since they concern the process of audio-visual session termination (i.e. after
the usage of the service). These are (the numbers in bracket correspond to the respective section in
[ITU-T E.804]):
- Streaming teardown failure ratio (7.3.5.12)
- Streaming teardown time (7.3.5.13)

7.2 Telephony
From end-user’s point of view, the IMS multimedia telephony (MTSI) direct service (see section
7.3.12 of [ITU-T E.804]), corresponding to packet-switched IMS voice over 4G networks (VoLTE),
is a sub case of telephony and does not need to be separated from it. All parameters identified and
defined in this chapter apply also to this service.
For additional considerations on end-to-end quality of service for voice over 4G mobile networks see
[ITU-T G.1028] and [ITU-T G.1028.2].

7.2.1 Speech quality metrics

See sections 7.3.6.3 (quality on call basis) and 7.3.6.4 (quality on sample basis) of [ITU-T E.804].
These sections recommend the use of either [ITU-T P.862] or [ITU-T P.863]. In what follows,
guidance is provided on the choice of the right algorithm and operation mode.

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 9

The most challenging question when it comes to speech quality measurement is about test campaigns
where various systems and networks are tested in sequence or in parallel. Some of them can be purely
restricted to narrowband telephony when others can use wideband audio or even upper audio
bandwidth. This is the practical case in most mobile networks, where 2G (NB and sometimes WB),
3G (NB and WB) and 4G (NB, WB and now also SWB) networks co-exist, with no prior knowledge
of when exactly they will be considered during a test.
In this situation it is possible to apply separate models or separate modes of the same model (here
[ITU-T P.863]):
- NB mode for tests where one can make sure that 100 % NB calls are involved (e.g. 2G only).
- FB mode for ALL other tests.

The fact is that the results of these various tests will be expressed in terms of statistics on MOS scores
that can’t be compared with each other, since the respective benchmarks are different. Practically, a
condition with a MOS score of 4.5 in NB mode will be scored only 2.8 in FB mode. If the probability
exists (even if it is very small) that the results of tests performed on network with different radio
technologies are compared with each other or with a single performance threshold, then a solution
combining tests with different modes of [ITU-T P.863] is no longer valid.
For simplicity reasons, it is therefore recommended in such case to adopt the mode of [ITU-T P.863]
with the widest scope, i.e. the FB mode. Anyway, the scope of the FB mode of [ITU-T P.863] fully
englobes the scope of the NB mode, as illustrated below by Figure 2.

Figure 2: scopes of [ITU-T P.863] application modes

This Recommendation is also justified beyond simplicity considerations, i.e. from a QoS perspective
which also includes the user’s expectation. In a contemporary mobile network, users have experience
of, and are likely to expect, high voice quality. So, if they don’t get it due to any reason, respective
MOS values are justified and cannot be hidden by the usage of another MOS scale.
In complement with the information provided above, ITU-T Recommendations P.862.3 and P.863.1
provide clear guidance on the application of the standard models for voice quality prediction:

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 10

 - Sections 7.2 and 7.3 of [ITU-T P.863.1] state clearly in which cases the NB and FB modes
and scales of [ITU-T P.863] must be used.
- Section 10.5 of [ITU-T P.863.1] explains also what happens when a narrow-band conditions
is assessed with the FB mode of [ITU-T P.863].
- Section 8.8 of [ITU-T P.863.1] gives a check list for the selection of appropriate reference
signals. Such signals, as mentioned in section 8.1 of [ITU-T P.863] can be found in ITU-T
Recommendation P.501 [ITU-T P.501].
- Section 12 of [ITU-T P.862.3] explains well the differences between the applications of the
NB and the WB scales of [ITU-T P.862.1] and [ITU-T P.862.2] respectively.

To be valid and applicable, any specification from network operator or national regulator addressing
the estimation of speech quality with objective measurements must comply with the following
instructions:
The objective estimation of the end-to-end quality of a telephone communication is made using MOS-
listening speech quality objective (MOS-LQO) scales. These scales describe the opinion of users with
speech transmission and its troubles (e.g., noise, robot voice, dropouts, etc.) according to:
- [ITU-T P.863] (NB or FB mode), to be preferred,
- or alternatively [ITU-T P.862], in conjunction with [ITU-T P.862.1] (NB mode) or [ITU-T
P.862.2] (WB mode).
The selection of the most appropriate algorithm, application mode and reference files must comply
with rules explained in ITU-T Recommendations [ITU-T P.863.1] and [ITU-T P.862.3]. One can also
refer to [ETSI TR 103 138] on which speech samples to use and how.
- In most cases, where telephony service is available or will be available for users in WB audio
or beyond, or when the quality of the service is expected to be compared with the quality of
another service running in WB audio or beyond, ITU-T Recommendation [ITU-T P.863] FB
mode is the only algorithm to apply, with FB or SWB reference speech files.
- NB modes of [ITU-T P.863] and [ITU-T P.862] can be applied, with NB reference speech
files, only in very limited contexts where there is the absolute certitude that only NB
conditions will be tested, and that the results of the tests won’t be compared with other tests
performed on conditions where WB audio (or beyond) is used.
- For the time being, it is not recommended to use the WB mode of [ITU-T P.862.2].

In any case, the algorithm used ([ITU-T P.862] or [ITU-T P.863]) and its application mode (NB, WB
or FB) must be reported.

7.2.2 Other parameters

Others interesting parameters are well defined in [ITU-T E.804] section 7.3.6 and do not require any
further definition. These are (the numbers in bracket correspond to the respective section in [ITU-T
E.804]):
- Telephony service non accessibility (7.3.6.1)
- Telephony set-up time (7.3.6.2)
- Telephony cut-off call ratio (7.3.6.5)

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 11

For this last parameter, it must be mentioned that a variant, mentioned in [ITU-T E.807], uses the
number of call-set up tries in the denominator (when the one from [ITU-T E.804] the number of
actually established calls), can be preferred in some cases. The choice of either parameter will have
an impact on numbers reported and corresponding targets. Furthermore, the information on cut-off
call ratio is incomplete without information on the test call duration. This needs to be reported
alongside any such results.
Furthermore, another QoS parameter defined in section 7.3.6 of [ITU-T E.804] is less important to
implement and supervise since it addresses an additional, non-mandatory, feature. This is (the number
in bracket corresponds to the respective section in [ITU-T E.804]):
- telephony CLIP failure ratio (7.3.6.6)

7.3 HTTP-based data services

The metrics reported in this section apply to web browsing as well as to other HTTP-based services
such as HTTP based data transfer (down/upload). This larger scope, compared to [ITU-T E.804],
corresponds to the actual usage of mobile data services. This is in accordance to the introduction of
the present Recommendation, where it is stated that there are special test types which do not directly
represent a user scenario, but are understood as a useful proxy for a wide range of actual usages. The
most prominent member of this category is data downloading and uploading using the http(s) protocol.
Appendices I and II to this Recommendation provide examples of approaches based on this larger
understanding of HTTP-bases services.
Section 7.3.8 of [ITU-T E.804] introduces a difference between the service (i.e. the corresponding
PDP context, covered in sections 7.3.8.1 to 7.3.8.2), the IP service (i.e. the TCP connection, covered
in sections 7.3.8.3 to 7.3.8.4), a session of the service (covered in sections 7.3.8.5 to 7.3.8.6), and the
data transferred during sessions (covered in sections 7.3.8.7 to 7.3.8.8). The concept with more
relationship to the end-user perspective is the session, it must be considered in priority.
Relevant KPIs must refer to events relative to successful (or not) events in the session. For each event,
the logical KPIs are then the success rate and the timing.

There are two practical ways to test http-based data services.

- The most classical one consists in transferring a fixed amount of data and measuring the
transfer time. For this first method, the session events are:
o when a visible reaction of the service occurs (session set-up),
o when the session is fully completed (session completion, or session failure).
- An alternative method consists in using a fixed time window for transfer without specifying
the amount of transferred data. Here, the event “session completion” and the corresponding
KPIs are no longer considered. Instead, the mean data rate is computed via the data volume
which has been transmitted during this time.

Scenarios using a fixed amount of data face a practical problem coming from the fact that in different
radio access technologies there are huge differences in maximum data rate. This leads to the situation
where, for a data volume sufficient to determine data rate in e.g. 4G, this volume would always cause
a time-out in 2G or even 3G. This is the reason for introducing the alternative method. In practice,
this means transmitting a data file of very large size, and stopping the transmission after the given

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 12

time is reached. A detailed description of the rationale associated with this alternative approach is
given in Appendix III to this Recommendation.

7.3.1 HTTP set-up time

See section 7.3.8.2 of [ITU-T E.804]
Although not directly related to the notion of session, this is the most important metric for this type
of service from an end user perspective. It must be monitored and reported. It corresponds to the
measurement of time of the first event (session set-up).
The time trigger point for stopping the measurement of HTTP set-up time corresponds to the starting
point for the measurement of HTTP session time. Thus, the sum of both corresponds to the total time
spent by an end-user when experiencing this service.

7.3.2 Other parameters

Others interesting parameters are well defined in [ITU-T E.804] section 7.3.8 and do not require any
further definition. These are (the numbers in bracket correspond to the respective section in [ITU-T
E.804]):
- HTTP session failure ratio (7.3.8.5). It corresponds to the measurement of success rate of the
second event (session completion) in test scenarios with a fixed amount of data.
- HTTP session time (7.3.8.6). It corresponds to the measurement of time of the second event
(session completion) in test scenarios with a fixed amount of data.
- HTTP mean data rate (7.3.8.7). It corresponds to the measurement of data volume transferred
in test scenarios with a fixed duration. It can also be derived from HTTP session time in test
scenarios with a fixed amount of data.
In [ITU-T E.804], there is no parameter to characterize the success rate of the event “session set up”.
It is recommended to implement such a parameter in complement to the four parameters selected
above with similar triggers as for HTTP set-up time.
Furthermore, other QoS parameters defined in section 7.3.8 of [ITU-T E.804] are less important to
implement and supervise. These are (the numbers in bracket correspond to the respective section in
[ITU-T E.804]):
- HTTP Service non-accessibility (7.3.8.1)
- HTTP IP-service access failure ratio (7.3.8.3)
- HTTP IP-service set-up time (7.3.8.4)
- HTTP data transfer cut-off ratio (7.3.8.8)

7.3.3 Summary of parameters

The table below presents the proposed parameters in section 7.3 with their practical application area:

Parameter Measurement approach HTTP-based data services

HTTP session set-up success Both Web browsing ; UL/DL
ratio
HTTP (session) set-up time Both Web browsing ; UL/DL

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 13

 HTTP session failure ratio Fixed amount of data only Web browsing ; UL/DL
HTTP session time Fixed amount of data only Web browsing ; UL/DL
HTTP mean data rate Fixed duration UL,DL

7.4 Other direct services with lower usage and less priority for QoS supervision
This Recommendation does not give any emphasis to other services that the three ones studied in the
sections 7.1 to 7.3 above, and therefore provides no further definition of corresponding parameters.
These services are (the numbers in bracket correspond to the respective section in [ITU-T E.804]):
- File-transfer protocol (7.3.1): no big usage on mobile devices
- Mobile broadcast (7.3.2): similar characteristics to video streaming
- Ping (7.3.3): not a service
- Push to talk over cellular (PoC) (7.3.4): almost no usage
- Video telephony (7.3.7): almost no usage
o For additional considerations on end-to-end quality of this service see [ITU-T
G.1028.1].
- Web Radio (7.3.9): similar characteristics to video streaming (audio part)
- WLAN service provisioning with HTTP based authentication (7.3.10): contributes to the
access and usages of other services, but not a service by itself
- Wireless application protocol (WAP) (7.3.11): almost no usage
- IMS multimedia telephony (7.3.12): included in Telephony service in section 7.2 above
- E-mail (7.3.13): no specific parameters, only service independent QoS parameters are
identified in [ITU-T E.804],
- Group call (7.3.14): included in Telephony service in section 7.2 above

8 Store and forward services

The parameters exposed in this section are taken from section 7.4 of [ITU-T E.804].
Out of the 3 services identified in [ITU-T E.804], only SMS (section 7.4.3 of [ITU-T E.804]
represents a broad usage and will be addressed in this section.
The most relevant store and forward applications corresponding to the actual usage in mobile
networks are rather over the top (OTT) applications or RCS. These services (with features making
them also pertaining to the category of direct services) are not considered in [ITU-T E.804]. A need
has been identified to specify test cases and triggers in an application-agnostic way. This will be
addressed in a future revision of this Recommendation.

8.1 Generic parameters

Section 7.4.1 of [ITU-T E.804] defines several parameters that apply to any store-and-forward service.
Indeed, the users’ experience is similar for all these services.
It has to be mentioned that all these parameters correspond to the use case where the users is at the
origin of the interaction with the service. There is no parameters in [ITU-T E.804] to quantify the
performance in use cases where the application server is initiating the interaction (by sending

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 14

notifications to the user). A need has been identified to specify parameters for such use case. This
will be addressed in a future revision of this Recommendation.

8.1.1 Message upload session metrics

See sections 7.4.1.2 (failure ratio) and 7.4.1.3 (time) of [ITU-T E.804].
A store and forward session is characterized mostly by the two dimensions of completion and duration,
reflected respectively by these two metrics.
These metrics rely on the notion of message upload session. A session is considered successful (or
successfully completed) when some conditions are met. From an end user perspective, this means
that a message is received back from the network within a given delay by the party originating the
upload to indicate this success. The nature of this message is highly dependent on the type of service
and on the way information is provided to users. For instance, an SMS upload is successful for a user
on a smartphone when the sending time is displayed before a given timeout on the screen of his device
(this corresponds also to the reception of the delivery report message 7b in Figure 7-59 of [ITU-T
E.804]).
Such a distinction between services is not necessary for download sessions. The reception of the
required payload and the possibility to use is not an observation specific to a given service.

8.1.2 Other parameters

Others interesting parameters (addressing also completion and duration) are well defined in [ITU-T
E.804] section 7.4.1 and do not require any further definition. These are (the numbers in bracket
correspond to the respective section in [ITU-T E.804]):
- Message download session failure ratio (7.4.1.16)
- Message download session time (7.4.1.17)
- End-to-end failure ratio (7.4.1.24)
- End-to-end time (7.4.1.25)

Furthermore, other QoS parameters defined in section 7.4.1 of [ITU-T E.804] are less important to
implement and supervise: they are generally specific to a limited aspect of service delivery or of the
customer journey, or can be redundant with other parameters closer to real users’ perception. These
are (the numbers in bracket correspond to the respective section in [ITU-T E.804]):
- Message upload access failure ratio (7.4.1.4)
- Message upload access time (7.4.1.5)
- Message upload data transfer cut-off ratio (7.4.1.6)
- Message upload data transfer time (7.4.1.7)
- Notification start failure ratio (7.4.1.8)
- Notification start time (7.4.1.9)
- Notification download session failure ratio (7.4.1.10)
- Notification download session time (7.4.1.11)
- Notification download access failure ratio (7.4.1.12)
- Notification download access time (7.4.1.13)
- Notification download data transfer cut-off ratio (7.4.1.14)
- Notification download data transfer time (7.4.1.15)
- Message download access failure ratio (7.4.1.18)

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 15

 - Message download access time (7.4.1.19)
- Message download data transfer cut-off ratio (7.4.1.20)
- Message download data transfer time (7.4.1.21)
- Notification and message download failure ratio (7.4.1.22)
- Notification and message download time (7.4.1.23)
- Login non-accessibility (7.4.1.26)
- Login access time (7.4.1.27)

8.2 SMS
See section 7.4.4 of [ITU-T E.804]
From end-user’s point of view, the Multimedia messaging service (MMS) (see section 7.4.3 of [ITU-
T E.804]) is a sub case of SMS and does not need to be separated from it. All parameters identified
and defined in this chapter apply also to this service.
The interesting parameters for SMS are well defined in [ITU-T E.804] section 7.4.4 and do not require
any further definition. These are (the numbers in bracket correspond to the respective section in [ITU-
T E.804]):
- Service non-accessibility (7.4.4.2)
- Completion failure ratio (7.4.4.4)
- End-to-end delivery time (7.4.4.5)
Furthermore, other QoS parameters defined in section 7.4.4 of [ITU-T E.804] are less important to
implement and supervise. These are (the numbers in bracket correspond to the respective section in
[ITU-T E.804]):
- Access delay (7.4.4.3)
- Receive confirmation failure ratio (7.4.4.6)
- Receive confirmation time (7.4.4.7)
- Consumed confirmation failure ratio (7.4.4.8)
- Consumed confirmation failure time (7.4.4.9)

8.3 Other store-and-forward services with lower usage and less priority for QoS supervision
This Recommendation does not give any emphasis to other services SMS, and therefore provides no
further definition of corresponding parameters. These services are (the numbers in bracket correspond
to the respective section in [ITU-T E.804]):
- E-mail (7.4.2), mostly used on fixed access, less often on mobile phones
- MMS (7.4.3) ), included in SMS service in section 8.2 above

9 Summary
In Table 1 below, the currently proposed list of services and parameters to be selected from [ITU-T
E.804], as exposed in sections 6 to 8 above is recapitulated. For each parameter in this table, the
corresponding reference inside [ITU-T E.804] and trigger point are given.

Service Service QoS parameter Reference Further or new Trigger points

type in [ITU-T information in (see definition in
E.804] this section 7.1.4 of
Recommendation [ITU-T E.804]

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 16

 Service-independent Radio network 7.2.1 Yes 2G, 3G: see
(section 6) unavailability section 7.2.1.3 of
[ITU-T E.804]
4G: same as 3G
(S-criteria based
on RSRP and
RSRQ instead of
Ec/No and RSCP)
Network 7.2.2.1 No See section
selection and 7.2.2.1.3 of [ITU-
registration T E.804]
failure ratio

Network 7.2.2.2 No See section

selection and 7.2.2.2.3 of [ITU-
registration time T E.804]

Direct Audio- Streaming 7.3.5.4 Yes Timers t0 and t1

services visual service non as from figure 7-
(section streaming accessibility 22 from [ITU-T
7) E.804]
- t0 defined in
table 2 of
[ETSI TR
101 578] (the
corresponding
event in the
application or
browser).
- for t1, see
section
7.3.5.4.3 of
[ITU-T
E.804]
Streaming 7.3.5.5 Yes Same as last
service access parameter
time
Streaming 7.3.5.6 Yes Timers t2 and t3
reproduction as from figure 7-
cut-off ratio 22 from [ITU-T
E.804]
Streaming audio 7.3.5.7 No See section
quality 7.3.5.7.3 of [ITU-
T E.804]

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 17

 Streaming video 7.3.5.8 Yes See section
quality 7.3.5.8.3 of [ITU-
T E.804]
Streaming 7.3.5.9 No See section
audio/video de- 7.3.5.9.3 of [ITU-
synchronization T E.804]
Streaming 7.3.5.10 Yes Timers t0 and t2
reproduction as from figure 7-
start failure ratio 22 from [ITU-T
E.804]
Streaming 7.3.5.11 Yes Timers t0 and t2
reproduction as from figure 7-
start delay 22 from [ITU-T
E.804]
Streaming 7.3.5.14 Yes Timers t2 and t3
rebuffering as from figure 7-
failure ratio 22 from [ITU-T
E.804]
Streaming 7.3.5.15 Yes Timers t2 and t3
rebuffering time as from figure 7-
22 from [ITU-T
E.804]
Telephony Telephony 7.3.6.1 No User’s point of
service non view: see section
accessibility 7.3.6.1.3 of [ITU-
T E.804]
Protocol part:
- 2G, 3G: see
section
7.3.6.1.3 of
[ITU-T E.804]
- 4G: see table 3
of [ITU-T
G.1028]
Telephony set- 7.3.6.2 No User’s point of
up time view: see section
7.3.6.2.3 of [ITU-
T E.804]
Protocol part:
- 2G, 3G: see
section
7.3.6.2.3 of
[ITU-T E.804]
- 4G: see table 3
of [ITU-T

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 18

 G.1028] and
associated
comment
Speech quality 7.3.6.3 Yes User’s point of
on call basis view: see section
7.3.6.3.3 of [ITU-
T E.804]
Protocol part:
- 2G, 3G: see
section
7.3.6.3.3 of
[ITU-T E.804]
- 4G: see table 3
of [ITU-T
G.1028]
Speech quality 7.3.6.4 Yes User’s point of
on sample basis view: see section
7.3.6.4.3 of [ITU-
T E.804]
Protocol part:
- 2G, 3G: see
section
7.3.6.4.3 of
[ITU-T E.804]
- 4G: see table 3
of [ITU-T
G.1028]
Telephony cut- 7.3.6.5 No User’s point of
off call ratio view: see section
7.3.6.5.3 of [ITU-
T E.804]
Protocol part:
- 2G, 3G: see
section
7.3.6.5.3 of
[ITU-T E.804]
4G: see table 3 of
[ITU-T G.1028]
HTTP- HTTP set-up 7.3.8.2 Yes See section
based data time 7.3.8.2.3 of [ITU-
services T E.804]
Protocol part:
starting point
command will

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 19

 depend on access
technology
HTTP session none Yes Same as last
set up success parameter
ratio
HTTP session 7.3.8.5 No See section
failure ratio 7.3.8.5.3 of [ITU-
T E.804]
HTTP session 7.3.8.6 No See section
time 7.3.8.6.3 of [ITU-
T E.804]
HTTP mean data 7.3.8.7 No See section
rate 7.3.8.7.3 of [ITU-
T E.804]
Store- All (generic Message upload 7.4.1.2 Yes See section
and- parameters) session failure 7.4.1.2.3 of [ITU-
forward ratio T E.804]
services
Message upload 7.4.1.3 Yes See section
(section
session time 7.4.1.3.3 of [ITU-
8)
T E.804]
Message 7.4.1.16 No See section
download 7.4.1.16.3 of
session failure [ITU-T E.804]
ratio
Message 7.4.1.17 No See section
download 7.4.1.17.3 of
session time [ITU-T E.804]
End-to-end 7.4.1.24 No See section
failure ratio 7.4.1.24.3 of
[ITU-T E.804]
End-to-end time 7.4.1.25 No See section
7.4.1.25.3 of
[ITU-T E.804]
SMS Service 7.4.4.2 No Timers t1 and t3
non-accessibility as from figure 7-
58 from [ITU-T
E.804]
Protocol part: see
section 7.4.4.2.3
of [ITU-T E.804]
On 4G, SMS
relies on CSFB to
2G or 3G.

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 20

 Completion 7.4.4.4 No Timers t1 and t9
failure ratio as from figure 7-
58 from [ITU-T
E.804]
On 4G, SMS
relies on CSFB to
2G or 3G.
End-to-end 7.4.4.5 No Timers t1 and t9
delivery time as from figure 7-
58 from [ITU-T
E.804]
On 4G, SMS
relies on CSFB to
2G or 3G.

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 21

 Appendix I

Experience of a multinational operator

(This appendix does not form an integral part of this Recommendation.)

The example below is submitted based on an operational feedback of a multinational mobile

operator present in many different countries.
Quality of service is an essential matter for all stakeholders:
- for national regulation authorities, it is a crucial lever in the framework of their development
policies towards telecommunication and information services,
- for end users, it helps doing the right choices in terms of services, depending on their
respective use cases (domestic, traveling, professional), taking into account other criteria
than only the price,
- for operators, this give a lever for differentiation, with a potential competitive advantage
against competitors, eventually with better users’ satisfaction and less churn.
In the specific case of an operator present in several different countries, it is facing a large variety of
national and market contexts, with a potential high heterogeneity between them. Such an operator
will have difficulties to monitor all performance indicators required in all of these contexts (mostly
through national operating licenses). These difficulties will consist in:
- operational (manpower) and economical (purchase of tools) issues to implement holistic
supervision policies and provide regular and relevant feedback,
- lack of proofs about the reality (or at least the magnitude) of the impact of all required
indicators on perceived quality, and therefore issue to decide and implement relevant
correction or optimisation actions.
[b-ITU-T E.806] provides a collection of best practices on how to address these challenges in terms
of measurement campaigns or tools and sampling methodologies as well as some useful information
to build a comprehensive answer.
For a better efficiency of QoS monitoring, this multi-national operator took in 2019 the decision to
base actions in all countries of its footprint on a simple approach, where less indicators are
measured (not more than 3 KPIs per service, with possibility of instantiation depending on the
context, e.g. between indoor and outdoor situations), if possible taken from [ITU-T E.804],
covering the widest spread of usages and services, but only indicators with a strong link with end
users’ perception and allowing a quality feedback towards operations and end users’ information.
Depending on specific national context, this core of meaningful metrics can be complemented by a
few number of other metrics selected locally.
Radio coverage
In this domain, it is important that an indicator
- Reflects the efforts of operators to deploy and optimise their networks.
- Can be easily checked and counter-measured by any third party.
- Can be easily used to build coverage maps accessible to end users.
To do so, the best parameter is the simplest one, i.e. the signal power, expressed in decibels, and
depending on radio technology.
- 2G: RxLev (dB)
- 3G: RSCP (dB)

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 22

 - 4G: RSRP (dB)
The targets for this parameter (absent in [ITU-T E.804]), as well as the measurement profiles, must
be adapted to various contexts: outdoor vs. indoor, cities vs. countryside vs. roads, etc.
Although signal power is the metric selected for the core set of metrics for reporting in this
example, other RF metrics are actually supervised for troubleshooting and network optimization
purposes.
Voice telephony
Here, the goal of end users consists in having straightforward conversations, easy and instant to
access, in good quality conditions. The list of parameters from [E.804] to apply is therefore:
- Service non accessibility (section 7.3.6.1 of [E.804])
o Other possible name and definition: Call Set up Success Ratio, the ratio of the total
number of successful call attempts to the total number of call attempts made
- Cut-off call ratio (section 7.3.6.5 of [E.804])
o Other possible name and definition: Drop Call Rate, the probability that a successful
call attempt is ended by a cause other than the intentional termination by A- or B-party
- Speech quality on call basis (7.3.6.3 of [E.804] completed with section 7.2.1 of this
Recommendation)
o Other possible definition: values on a predefined scale that subjects assign to their
opinion of the performance of the telephone transmission system used for listening to
spoken material.
SMS
SMS users simply want to send and receive messages, quickly and safely. The list of parameters
from [E.804] to apply is therefore:
- Service non-accessibility (section 7.4.4.2 of [E.804])
- Completion failure ratio (section 7.4.4.4 of [E.804])
- End-to-end delivery time (section 7.4.4.5 of [E.804])
Data services
Three use cases of data service have been selected in priority, with associated metrics:
- Web browsing: seamless consultation of internet pages
o HTTP session time (section 7.3.8.6 of [ITU-T E.804])
o HTTP session failure ratio (section 7.3.8.5 of [ITU-T E.804], completed with section
7.3.2 of this Recommendation)
- Data exchange: quick and safe upload and download of data in HTTP protocol
o Download mean data rate (no direct correspondence in [ITU-T E.804], but section
7.3.8.7 is providing an applicable definition)
o Upload mean data rate (no direct correspondence in [ITU-T E.804], but section
7.3.8.7 is providing an applicable definition)
- Video streaming: quick access and safe viewing of contents
o Video access time: The time it took for the video to start displaying - also known as
“ time to first picture" (no direct correspondence in [ITU-T E.804], but section 7.1.2
of this Recommendation is providing an applicable definition)
o Video playout duration: The time it took for the video to playout (no correspondence
neither in [ITU-T E.804], nor in this Recommendation)

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 23

 Appendix II

Example of a recent work on best practices, as proposed by ETSI

(This appendix does not form an integral part of this Recommendation.)

The material in this appendix is based on recent work contributed at ETSI by actors from the field of
QoS monitoring and benchmarking of mobile networks and services. It reflects the current state of
the art (as for 2019) in this area, when latest version of [ITU-T E.804] has been published in 2014.
ETSI, in the Technical Report [ETSI TR 103 559] entitled “Best practices for robust network QoS
benchmark testing and scoring”, provides several practical principles to help designing efficient
mobile QoS benchmarking. The recommended tests concern telephony, video streaming, data
throughput and more interactive applications such as browsing, social media and messaging.
According to this technical report, the main governing principles for mobile benchmarking are:
- Fair play: Steps should be taken to ensure that the measured results are truly representative of
the real customer experience.
- Coverage extent: Benchmarking should be performed in such a way that it highlights coverage
differences in the results.
- Radio technology in use: Benchmark scoring should account for operators who offer
performance differentiation through early adoption of new technologies by way of a 'bonus'
for such deployment.
- Test device selection: Care should be taken in the selection of such devices to ensure they do
not favour one operator's network over another in the results.
- Test server selection: Test servers should be selected so they do not favour one network
compared to another. Web pages should be selected such that they represent a cross section
of pages commonly used by customers and that they ensure a representative performance
comparison.
- Transparency: Given the importance of the clear interpretation of benchmark results, all
results should be accompanied by a declaration containing information about factors required
to understand them fully.
The recommended test metrics are the following:
Telephony
Telephony tests are tests with a fixed call length where two terminals, either both mobile or one
landline and one mobile call each other. To consider unsustainable quality in a call, for a low
speech quality score (e.g. MOS < 1,6) or silent periods for consecutive measurement samples (e.g.
> 20 s), the call can be counted as unsustainable, and as an unsuccessful call or treated by a separate
indicator.
- Telephony success ratio: The success rate of the voice service independent of access or relay
technology is the Telephony non Accessibility and the Telephony Cut-Off Call Ratio in [b-
ETSI TS 102 250-2], clauses 6.6.1 and 6.6.5.
- Setup time: Defined in [b-ETSI TS 102 250-2], clause 6.6.2. It starts with the initiation of the
call and ends when the alerting of the called side is indicated. Alternatively, the time when
the acceptance or successful setup of the call is signalled to the user can be used as the end
trigger.
- Listening quality: The value is calculated on a per sample basis as described in [b-ETSI TS
102 250-2], clause 6.6.4 where Recommendation [ITU-T P.863] in FB mode needs to be used.
Video testing

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 24

Video testing is in the standard case IP based video streaming. A Smartphone app based testing as in
Figure 1 of [ETSI TR 101 578] is used. In order to collect details of the transport and reproduction,
the length of the observation period of the video should reflect the relevant delivery mechanisms and
the typical usage profile of a mobile user.
- Video streaming service success ratio: The end-to-end success ratio of the requested video
stream. It starts with the request of the video and ends with the end of the playout. This is
derived from the metrics in [ETSI TR 101 578] as a combination of Video Access Failure
Ratio and Video Playout Cut-off Ratio.
- Setup time: The time from stream request to the display of the first picture and start of
playout. This is Video Access Time from [ETSI TR 101 578].
- Video quality: A comprehensive measure for the perceived quality that combines the impact
of parameters such as freezing, frame-rate, resolution and compression depth and scheme by
the codec is the mean opinion score (MOS) scale and is done according to clause 6.5.8 in [b-
ETSI TS 102 250-2]. In the case of video streaming with a respective app on the smartphone,
an encrypted stream and a range of different resolutions (up to HD) is expected. The Video
Quality parameter in [ETSI TR 101 578] reflects such measure. In addition to this, Video
Freezing Time proportion in [ETSI TR 101 578] provides an insight about the proportion of
the accumulated video freezing duration in relation to the actual video playout duration.
Data testing
For data testing the throughput bandwidth for the user is tested. This is done by downloading and
uploading incompressible files over HTTP. In clause 6.8 of [b-ETSI TS 102 250-2] the up and
download of entire files is described. The description of an upload and download using fixed
duration is in clause 5.2 of [b-ETSI TR 102 678]. Both approaches can be used, either alone or
combined, for the purpose of evaluating throughput bandwidth.
- Success ratio: The determination of the success ratio for HTTP uploads and downloads is
included in clause 6.8 of [b-ETSI TS 102 250-2] and in clause 5.2 of [b-ETSI TR 102 678].
- Throughput (upload and download): The determination of the mean data rate or throughput
for HTTP uploads and downloads is included in clause 6.8 of [b-ETSI TS 102 250-2] and in
clause 5.2 of [b-ETSI TR 102 678]. The best view of the actual download bandwidth is
provided by a multi-threaded HTTP download test.
Service testing
Besides the browsing of web pages, services like social media and messaging systems (SMS is not
considered) are not described or standardized for mobile testing. Some overall interesting aspects of
all of these services are:
- Success ratio.
- Duration or timing of the interaction.
The Technical report provides also some recommendations on the selection of test areas (cities, roads,
complementary areas) and of typical user profiles. It includes also examples of weighting of
individual test results depending on test area type, tested service type and metrics for each tested
service. These metrics can be the raw ones described above, or more elaborated ones like percentiles
or proportion of individual measurements above a given threshold. Finally, it elaborates on statistical
confidence and robustness of results.

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 25

 Appendix III

Alternative approach for performance measurement of HTTP-based data

transfer services
(This appendix does not form an integral part of this Recommendation.)

These practices have a long history of usage in commercial network performance measurement as
well as in regulatory contexts, and they are covered in various technical documents issued by
standardization bodies. For instance, [ETSI TR 101 578] describes the principle, and [b-ETSI TR
103 501] takes this further towards KPI computation based on events taken from the application plane
or corresponding operating-system API levels. This takes care of the fact that in https, transport layer
events (e.g. http resp. TCP-layer events) are not visible anymore even if packet capture tools are used.
Also, on the background that testing is typically done using smartphones, using packet capture tools
requires system-level (root) access. This would in any case create a gap of direct comparability to
measurements done by crowdsourcing, where such access is practically unfeasible and also not
desirable because it would remove crucial security mechanisms.
[b-ETSI TR 103 501] also introduces a consistent terminology for different methods of upload and
download testing: Using transfer of a fixed amount of data, typically a data file, and measuring the
transfer time is named Fixed-size method. Transferring data for a given amount of time and measuring
the data volume transferred is named Fixed-time method. The data rate is in both cases calculated by
the quotient of data volume and time.
For the actual calculation, a spectrum of additional rules is common practice, which are aimed at
dampening extremes due to the packet nature of data transfer. The purpose of such measures is to
increase the applicability of results in the aforementioned sense, i.e. usability as a proxy to assess the
performance of actual services.
Commonly used is time windowing which excludes the first data packet of content, or a certain
amount of time at the beginning of the transfer. This has the purpose to get a better approximation to
steady-state data rates by at least partially removing the ramp-up phase which is e.g. typical for TCP
(slow-start).
The definition of the Stop method A in Rec. E.804 (ETSI-A in the original TS 102 250-2) is a
standardized example of the former approach where the reception of the first packet of content also
serves as a trigger point for respective KPI definitions, making the measurement easily implementable.
Also, when data is taken in a time-resolved way (e.g. recording samples of data volume every second),
other techniques are used such as removing the sample with the lowest and the highest data rate from
the set before averaging.

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 26

Bibliography
[b-ITU-T E.806] Recommendation ITU-T E.806 (06/2019): Measurement campaigns, monitoring
systems and sampling methodologies to monitor the quality of service in mobile networks.
[b-ETSI TS 102 250-2] ETSI TS 102 250-2: Speech and multimedia Transmission Quality (STQ);
QoS aspects for popular services in mobile networks; Part 2: Definition of Quality of Service
parameters and their computation.
[b-ETSI TR 102 678] ETSI TR 102 678: Speech and multimedia Transmission Quality (STQ); QoS
Parameter Measurements based on fixed Data Transfer Times.
[b-ETSI TR 103 501] ETSI TR 103 501: Speech and multimedia Transmission Quality (STQ);
Guidelines for the Measurement of Data Throughput on Devices connected to Mobile Networks.

Rec. ITU-T E.804.1 (10/2020) – Prepublished version 27