Author: Anil Rama Rao, Director – R & D & Application support , Rohde & Schwarz India
Visibility of subscribers’ quality of experience is something operators have sought to measure since the early days of mobile networks. In this discourse, we consider the emergence of 5G, the new use cases and services it brings, and the necessity to rethink the approach to measuring network performance and the quality of experience (QoE) of 5G networks.
From the emergence of GSM to the global adoption of LTE, operators’ focus has been on ensuring the quality of service (QoS) delivered to subscribers. Consequently, test solutions were developed that enabled the verification of network quality and provided specific indicators to identify and diagnose issues and to optimize performance.
Historically, testing mobile networks has been built up from spectrum and physical layer parameters, such as RSSI and SINR, to OSS and trace data and, latterly, smartphone-based solutions that measure QoS and QoE, such as voice and video, at the point of delivery to the end user.
Quality of experience has become the ultimate parameter for operators to quantify subscriber satisfaction. As the name suggests, it is a complex metric that characterizes the human experience of the service delivered over the network to an end user’s device.
The emergence of 5G brings not only a leap in network performance and capabilities but also the increasing likelihood that the end user will be a machine, not a human. That means that existing test methodologies based on measuring voice, video and data delivered to subscribers using a mobile device should be expanded to encompass machine-type communication; a revised concept for what we mean by quality of experience (QoE) and new techniques for testing performance must be developed.
5G demands a new QoE paradigm
It is important to remember that 4G services will continue to evolve but also to recognize that the implementation of 5G will be very different from previous generations of cellular technology, particularly in the RAN. It starts with a consideration of the 5G use cases of eMBB (enhanced Mobile Broadband), mMTC (massive Machine Type Communication) and URLLC (Ultra Reliable Low Latency Communication).
These use cases bring new dimensions in terms of real-time, higher bandwidth, scalability, etc., and require a revised methodology and set of parameters to measure QoE. This, in turn, drives the performance of the network needed to support these use cases.
Finally, an expanded test methodology is needed to measure new parameters more accurately at higher frequencies and wider bandwidths and meet the challenge of how to quantify QoE – particularly where the end user will not be using a smartphone. We, therefore, need to rethink our approach to what constitutes QoE and how to measure it.
We need a new interpretation and understanding of what constitutes quality of experience and how to map the consequences of changes in QoE for each application. Smartphone-based test cases to measure the quality of eMBB will remain but for mMTC and URLLC, the end user will be a machine and the service will bring a new set of demanding parameters to measure. For machine-type communication, the traditional concept of QoE will no longer be appropriate because the end user is not a human and will not experience the service in an emotive way.
Implications for operators
The services and applications for these 5G use cases, such as automotive, industry 4.0, augmented reality, remote surgery, etc., are often more critical, and the consequences of degradation or loss of service become more serious, potentially life-threatening. Therefore, operators must monitor networks with greater accuracy and in finer detail to pre-empt QoS issues.
The challenge facing operators moves from ensuring subscriber satisfaction and compliance with SLAs to being able to guarantee the delivery of often critical services and legally prove that they meet this obligation to address questions of liability.
With each application potentially supported by a dedicated network slice, each with its own QoS definition, metrics and thresholds, operators’ network monitoring requirements expand from a single measurement for their entire network to multiple, simultaneous performance and quality measurements.
The 5G QoE measurement challenge can be summarized as follows:
- 5G use cases, particularly mMTC and URLLC, demand a new way of measuring QoE for each individual application.
- QoE has a different meaning in machine-type communication. A machine or connected thing’s QoE needs to be interpreted.
- Each application brings a specific set of network performance characteristics that need to be monitored.
- Monitoring becomes more important to operators for critical 5G applications.
- 5G testing adds a new dimension for both data acquisition and post-processing.
- Active testing and understanding of QoS per use case, allied to network operations, will be a key enabler of future 5G business cases.
Without knowing the underlying service and device requirements, network quality engineers do not know what aspects and metrics should be treated as important key performance indicators (KPI). So we need to compile the set of parameters and KPIs, and define thresholds, creating good/bad KPI limits for each application and use case. This is still at a very early stage, but one approach would be to start at the PHY and logical layers and work up to the apps and use cases, defining what needs to be measured.
One positive aspect is that this sort of understanding of use cases QoE can drive the Network Operations Center (NOC) to Service Operations Center (SOC) transformation that operators are making. The SOC is important because it is via this service-driven environment that operators hope to differentiate their customer experiences. For example, a car manufacturer can understand which operators’ network is best suited for its connected or autonomous cars. So we can see that active testing and understanding of QoE per use case, allied to network operations, will be a key enabler of future 5G business cases.
Implications for test solution providers
Considering the operators’ challenges , test equipment providers must produce solutions that can measure multiple virtual networks at the same time and the same location with different methodologies.
An examination of the metrics required to characterize 5G use cases quickly reveals that measuring QoE becomes more complex and more demanding in terms of the data acquisition of additional measurement parameters with greater precision in the RAN. This also drives the need to provide post-processing analytics that encompasses new models for quality of service (QoS) and QoE measurements for network benchmarking, optimization and monitoring.
The consequences of degradation or loss of service for some mission-critical 5G use cases go way beyond an unsatisfactory voice call to a friend or a YouTube video freezing when you stream it on your phone. The critical nature of some applications demands that test solutions must be independent, transparent and traceable to certified international standards and not aligned to proprietary techniques or individual network equipment vendors.
Testing 5G QoE
5G introduces a new dimension and type of use cases; not only the physical test equipment required to sample the network, e.g., a wider bandwidth scanner, but also the methodology of what parameters to test for a specific application and how to post-process the data. There will be new KPIs that contribute to the evaluation of QoS and other factors that feed into QoE. QoE can be built up from the lower layers and use a model to define how QoS maps into QoE.
The key question is what does good QoE look like for a sensor in an industrial IoT deployment, or a connected car, or a VR device, or any specific 5G use case? In these use cases, we need to have a way of understanding whether QoE is good or bad and what the thresholds are. For a simple example, take call setup time. What is an acceptable “setup time” for a sensor alarm, or an autonomous car, or in remote medical use cases?
What may have been well defined in previous use cases, for example, a subscriber streaming a movie on their smartphone, may well not be transferable to 5G use cases. Attention will have to be given to the range of acceptable values of QoE for each specific application, below which it becomes a problem and above which it brings no additional value.
Therefore, it is apparent that testing 5G QoE, particularly for applications other than enhanced Mobile Broadband (eMBB), will require more metrics to be acquired with greater precision that will need to be post-processed more quickly and with greater complexity.
Standardization of testing
International standards organizations, such as the ITU and ETSI, are actively evolving their test models to cover the changes demanded by 5G, and this is something R&S is very actively engaged in. However, building 5G methodologies and standards is going to be complex when we contemplate all the use cases and remember that operators already have 200-400 core key performance indicators (KPI) to monitor.
The sheer amount of KPIs makes the understanding of quality of experience (QoE) in a granular way very difficult. Therefore, this is another change in the quality of service (QoS) environment driven by 5G, where there will be many more parameters to monitor, often in real time.
To evaluate and benchmark networks, KPIs are required that truly reflect the network’s performance so that based on such KPIs it is possible to define a fair and transparent performance scoring method. ETSI has taken the driver’s seat to discuss and define best practices for network benchmarking and scoring that enables the network to be characterized in a single, unified metric.
The method provides the operator with visibility of the status of their network and identifies the factors that influence quality. The factors and weightings that influence the scoring method will be adapted for each 5G use case and application. The robust fundamental methodology will provide the industry with an independent reference against which 5G QoS/QoE can be measured.
The test approach should cover two aspects. First, to release test solutions that enable measuring the technical aspects of 5G networks such as coverage, performance and operation; and second, to enhance our existing QoE methodologies to encompass 5G use cases.
The solutions for testing technical aspects of the 5G RAN are already being used by operators as they move from trials to the commercial deployment of 5G. The QoE solutions from lower layers up to signaling are being developed in partnership with the operators and standardization bodies.
In conclusion, we need to establish an understanding of what the requirements of each 5G use case are. Then we can build out key parameters and KPIs required to meet those requirements and their range and limits. Once we understand why we are testing certain parameters, we will have the test methodology to quantify QoE according to those KPIs.