Network Acceptance testing methods

v1.0
25.10.2012

For internal use

1 © Nokia Siemens Networks Presentation / Author / Date
Content
•Testing methods and KPIs
•Case example: Post processing challenges with XCAL-Actix combination

For internal use

2 © Nokia Siemens Networks Presentation / Author / Date
LTE drive test KPIs for network/cluster acceptance

Target:
• Few selected KPIs that can be post-processed automatically (Nemo Analyze,
Actix analyzer) from drive test logs.
• Acceptance testing executable with one drive (=one test case/script)

Different test methods:

1. Each session started from EMM-DEREGISTERED state
2. Each session started from RRC-IDLE EMM-REGISTERED state
3. Each session started from RRC-CONNECTED state

For internal use

3 © Nokia Siemens Networks Presentation / Author / Date
LTE drive test KPIs for network/cluster acceptance

1. Each session started from EMM-DEREGISTERED state

• Attach – Application data – Detach
• Several data applications can be included in the same session (FTP DL/UL, HTTP, Ping)
• Possible to measure Success Rates/times: Attach, ERAB setup, RRC setup, Application
setup, HO
• Not reflecting end user behavior, as users typically start sessions from EMM-
REGISTERED state. sesssion1
Time
13:56:34.094
Subchannel Direction
Uplink
Message
ATTACH_REQUEST
13:56:34.094 Uplink PDN_CONNECTIVITY_REQUEST
13:56:34.097 CCCH Uplink RRCConnectionRequest
13:56:34.127 CCCH Downlink RRCConnectionSetup
13:56:34.132 DCCH Uplink RRCConnectionSetupComplete
13:56:34.366 DCCH Downlink SecurityModeCommand

• Example script
13:56:34.368
13:56:34.369
13:56:34.390
DCCH
DCCH
DCCH
Uplink
Downlink
Uplink
SecurityModeComplete
UECapabilityEnquiry
UECapabilityInformation
• Attach 13:56:34.410
13:56:34.414
DCCH
DCCH
Downlink
Uplink
RRCConnectionReconfiguration
RRCConnectionReconfigurationComplete
• FTP DL (5MB) 13:56:34.416
13:56:34.416
Downlink
Downlink
ATTACH_ACCEPT
ACTIVATE_DEFAULT_EPS_BEARER_CONTEXT_REQUEST

• FTP UL (3MB) 13:56:34.424

13:56:34.458
DCCH Uplink
Uplink
MeasurementReport
ATTACH_COMPLETE

• HTTP page DL (x5) 13:56:34.458

13:56:34.459 DCCH
Uplink
Uplink
ACTIVATE_DEFAULT_EPS_BEARER_CONTEXT_ACCEPT
ULInformationTransfer

• Ping 32-1024B (x10)

13:56:35.464
13:56:56.606
DCCH Uplink
Uplink
MeasurementReport
DETACH_REQUEST

•
13:56:56.607 DCCH Uplink ULInformationTransfer
Detach 13:56:56.662 DCCH Downlink DLInformationTransfer
13:56:56.663 DCCH Downlink RRCConnectionRelease
Session 1 stop 13:56:56.665 Downlink DETACH_ACCEPT
For internal use 13:56:57.142 BCCH_BCH Downlink SYSTEM_INFORMATION_BCH
4 © Nokia Siemens Networks Presentation / Author / Date 13:56:57.142 BCCH Downlink MASTER_INFORMATION_BLOCK
LTE drive test KPIs for network/cluster acceptance

2. Each session started from RCC IDLE, EMM-REGISTERED state

• RCC setup - Application data – Wait – Release due to inactivity
• Several data applications can be included in the same session (FTP DL/UL, HTTP, Ping)
• Possible to measure Success Rates/times: Service request, ERAB setup, RRC setup,
Application setup, HO
• Challenges in practical measurement setup, especially with inactivity timer >10s. Requires
specific firewall settings to measurement laptop to block all outgoing packages during wait
period. sesssion1
Time
14:25:52.537
Subchannel
CCCH
Direction
Downlink
Message
RRCConnectionSetup
14:25:52.540 DCCH Uplink RRCConnectionSetupComplete
14:25:52.574 DCCH Downlink SecurityModeCommand
14:25:52.576 DCCH Uplink SecurityModeComplete
14:25:52.577 DCCH Downlink RRCConnectionReconfiguration

•
14:25:52.581 DCCH Uplink RRCConnectionReconfigurationComplete
Example script 14:25:52.594
14:25:53.634
DCCH
DCCH
Uplink
Uplink
MeasurementReport
MeasurementReport

• RRC setup/service request session2

14:26:06.540
14:26:17.957
DCCH
CCCH
Downlink
Uplink
RRCConnectionRelease
RRCConnectionRequest

• FTP DL (5MB) 14:26:18.021

14:26:18.023
CCCH
DCCH
Downlink
Uplink
RRCConnectionSetup
RRCConnectionSetupComplete

• FTP UL (3MB) 14:26:18.059

14:26:18.061
DCCH
DCCH
Downlink
Uplink
SecurityModeCommand
SecurityModeComplete

• HTTP page DL (x5) 14:26:18.061

14:26:18.066
DCCH
DCCH
Downlink
Uplink
RRCConnectionReconfiguration
RRCConnectionReconfigurationComplete

• Ping 32-1024B (x10)

14:26:18.077
14:26:19.117
DCCH
DCCH
Uplink
Uplink
MeasurementReport
MeasurementReport

For internal use

• Wait until released by eNB session3
14:26:57.044
….
DCCH Downlink RRCConnectionRelease

5 © Nokia Siemens Networks Presentation / Author / Date

LTE drive test KPIs for network/cluster acceptance

3. Each session started from RCC Connected, EMM-REGISTERED state

• Application data only
• Several data applications can be included in the same session (FTP DL/UL, HTTP, Ping)
• Possible to measure Success Rates/times: Application setup, HO
• Only mobility and re-establishment/drop related signaling is seen.

• Example script
• FTP DL (5MB)
• FTP UL (3MB)
• HTTP page DL (x5)
• Ping 32-1024B (x10)

For internal use

6 © Nokia Siemens Networks Presentation / Author / Date
LTE drive test KPIs for network/cluster acceptance

Example list of KPIs that could be measured with different methods:

KPIs 1 2 3 Comment
Attach SR x Attach request -> Attach complete
Attach time x Attach request -> Attach complete
Session setup SR from Deregistetered x RRCC request -> ULinformation transfer (Attach complete)
Session setup time from Deregistered x RRCC request -> ULinformation transfer (Attach complete)
Session setup time from RRC idle x RRCC request -> RRCC reconf. Complete
Session setup SR from RRC idle x RRCC request -> RRCC reconf. Complete
LTE intra freq HO SR x x x
LTE inter freq HO SR x x x
Call/Session drop rate x x
# drops/minute x x x
Re-establishment success rate x x x
FTP/HTTP/PING SR x x x
RTT PING x x x
FTP DL throughput x x x
FTP UL throughput x x x
HTTP page download time x x x

Recommended method for mass acceptance, closest match to user experience

Could be used, if Attach SR measurements are mandatory

For internal use
7 © Nokia Siemens Networks Presentation / Author / Date
Content
•Testing methods and KPIs
•Case example: Post processing challenges with XCAL-Actix combination

For internal use

8 © Nokia Siemens Networks Presentation / Author / Date
Case example: Post processing challenges with XCAL – Actix
combination (by Tommi Mecklin 24.10.2012)
Latest version of the report:
https://sharenet-ims.inside.nokiasiemensnetworks.com/livelink/livelink/overview/D483100912

•Accuver XCAL has been widely used in LTE pilot projects in NSN
•From time to time there have been indications of some issues when post-processing the logfiles
with Actix Analyzer
•This short presentation is based on the observations collected with a trial project using following
versions:
• Accuver XCAL version 3.2.16.253 with Qualcomm LTE chipset
• Actix Analyzer 4.05.268.606 (September-12 update build) and 4.05.271.973 (October-12 build)
• Only limited amount of data was available
• Not enough for HTTP or FTP applications
• More emphasis in CSFB issues and XCAL properties in general
• This covers only the post-processing part
• The usability of XCAL on the field is not considered here
For internal use
9 © Nokia Siemens Networks Presentation / Author / Date
Observations 1/3

•The latest Actix build did not have officially support for the used XCAL version
• This is typical problem with Actix-XCAL combination. XCAL changes too fast and Actix decodes are
always a bit behind.
• In this case logfiles were opening without errors in Actix (but see below)
• Difficulties creating working call scripts for performing Attach-Detach sequences
• XCAL performs automatically detach preventing it (details not fully known)
• Packet capture (IP part) of the logfiles requires enabling IP Data Analysis in Preferences in
Actix
• Logfiles might get really huge if user plane data is activated in XCAL. Quite soon IP Data Analysis
must be switched off to enabled loading the files in Actix  resulting just in huge logfiles that are not
in the end providing any additional information
• This is very different from Nemo Outdoor and TEMS 14 where packet capture files don’t create such problem
in Actix as logfiles are a lot smaller and work without separately enabling IP Data Analysis in Actix.
• Ping RTT was reported from time to time as 0 ms
• It looks like it actually means that particular ping had failed (OK, not too elegant, but can be handled
in the analysis)
For internal use
10 © Nokia Siemens Networks Presentation / Author / Date
Observations 2/3

• Ping Packet Size is not show correctly for larger (>=512 byte) pings
• According to Accuver this is natural and happens due fragmentation, but it seems strange that the
packets are not rearranged and shown correctly when received  raises suspicion of the data
application implementation (this has not been seen in other DT tools)
• FTP Downloads might not be properly detected in Actix
• Visualization of downloaded or uploaded application bytes doesn’t work in Actix (problem in
Actix)
• It would be needed to troubleshoot the data transfer applications
• Actix events for CS Callback are not working properly
• This is due XCAL reporting BCCH messages when moving into 3G causing CSFB failures and drops
even in case setup succeeds and/or call continues normally
• There are no settings in Actix to solve this
• Requires a custom event that has tolerance for BCCH messages, but it can never bee 100% correct in case
of real failures do occur

For internal use

11 © Nokia Siemens Networks Presentation / Author / Date
Observations 3/3

• The most serious problem with XCAL data is that there are no direct application layer events
(vendor specific events decoded in Actix) that would give precise view what is happening in
application itself (setup attempts, failures, drops, state in general, etc.)
• XCAL seems to provide merely just a dump of the trace from the chipset plus some additional data
events (that are masked/not displayed in Actix?)
• Vendor specific events would allow making more accurate KPIs and analysis as event detection
doesn’t have to rely on signaling only (generic events in Actix)
• More robust detection of real attempts, failures and drops can be done with the combination of signaling and
vendor specific events.
• Top->Down approach (from application layer to lower layers) is more accurate in many cases than the
opposite way around that is implemented with Actix generic events (trying to detect from signaling what the
applications are doing).
• Other drive test tools (Nemo, TEMS and even SwissQual) do provide vendor specific events in Actix
 accuracy and handling in Actix can be easily enhanced.

For internal use

12 © Nokia Siemens Networks Presentation / Author / Date
Summary

• Using XCAL data together with Actix requires almost constant push towards Actix to update
the file decode
• Basic L1 and signaling decodes are working fairly well (as it is in practice just a raw dump from
the chipset)
• Lack of vendor events could be a major factor in cluster acceptance work where accuracy of
the KPIs is a key factor
• This is why DT tool and project specific KPI reports have been done before as generic events based
on only signaling alone will never work accurately enough
• Furthermore this puts NSN a bit ahead of others as using just the generic events is available for every one,
but NSN specific ones are protected are limited to NSN only.
• There might be some issues in the basic implementation of data applications in XCAL.
• This could be corrected relatively fast if there is input from different use cases.

For internal use

13 © Nokia Siemens Networks Presentation / Author / Date