5G NSA (Non-Standalone) has been around for some time now and launches of 5G SA (Standalone), as well as 5G devices supporting SA, are rapidly increasing. These deployments are driving a myriad of new 5G services and OTT applications. Specifically, services and/or applications characterized by demanding requirements, such as high bandwidth (e.g. 4K video streaming) and very low latency (e.g. mobile cloud gaming, remote drone control). Being exposed to this new and large variety of services and applications, each one more sophisticated than the last, raises user expectations of their mobile network experience to significantly challenging levels for the operators.
The increased complexity of the network, encryption of services, proprietary codecs/clients and even delivery protocols, and the need to minimize network operational costs, calls for highly evolved testing techniques. These techniques need to be easily adaptable to a diverse set of services and applications and must provide a trustful user experience evaluation. The metrics assessed need to cover familiar services/applications (e.g. voice, audio, video) as well as emerging ones (e.g. e-gaming, remote drone control) which require new, previously untested quality metrics such user interactivity.
Introducing a generic testing approach
Understanding this unprecedented challenge has led Infovista to introduce a generic testing approach, both for the data collection procedure as well as the service/application test configurations. The approach innovatively combines generic testing techniques, sophisticated real live service traffic pattern emulations and machine learning (ML) algorithms. 4G and 5G mobile operators can now test not only their networks’ performance with these services/applications, but also the user experience of them. This generic approach enhances our user experience testing portfolio which includes solutions for carrier voice services such as VoLTE and VoNR, OTT voice services, OTT video/media streaming and interactive services including e-gaming, remote drone control and video conferencing.
With extensive expertise in drive testing, active testing and QE modeling and estimation, Infovista understands the possible risks of a generic testing approach. A generic approach mimics the service/application traffic, behavior, and/or configuration (codec/client) rather than using the real service/application. Naturally, questions regarding the usefulness and accuracy of a generic approach will arise. Infovista’s research, as well as ongoing standards initiatives, have proven that if the generic approach closely mimics the real service/application, then the generic approach delivers trustful and representative results. To closely mimic a real service/application, the generic approach needs to trigger the same network resources, use the same protocols, and encode and adapt as real services and OTT codecs and clients would.
The benefit of a generic testing approach is that it can still deliver cost-effective network focused performance evaluation, root cause detection, analysis and optimization without the need to test a large variety of complex OTT applications. Additionally, it provides operators with continuous monitoring to ensure that the minimum performance requirements for services/applications are met. In this way, operators spend their efforts on what they can fix and optimize (the network) rather than on the OTT service/application which they cannot control. Finally, by using a well-designed generic framework for data collection, operators can test a set of the most commonly used real live OTT services and applications characterized by a limited set of KPIs, to provide the most important information on user experience within those services/applications.
The three pillars of Infovista’s generic testing approach
Below we introduce the three pillars of our generic testing approach to help operators understand the user experience for 5G services and OTT applications:
1) Generic OTT voice/video clients
Testing all mobile OTT voice and video streaming services/applications is practically impossible. This is due to the large number and variety of mobile OTT applications, multiple different platforms and device-based operating systems (OS), as well as proprietary codecs and clients, and error concealment schemes. Perhaps the most challenging aspect, however, is the level of encryption within these applications, which in many cases is device OS dependent as well as continuously, and even dynamically changing.
These testing challenges call for the use of a generic client mimicking the behavior of an OTT voice client (e.g. WhatsApp) and/or video streaming client (e.g. video on demand category like Netflix, or live video streaming category like Breakout News). In this case, the testing involves a cloud-based public and/or private voice application server and on-device client completely controlled by the test set-up. This generic client, which behaves like all the most commonly used ones, provides the ability to test only one OTT application, one version and one set of fully accessible KPIs (free of encryption). The result is a reference of network performance for the OTT application type for which the generic client was designed.
Infovista offers a generic OTT voice client solution with our user experience voice quality machine learning based predictor sQLEAR (a.k.a ITU-T P.565.1). The generic OTT voice client was designed based on one of the most used OTT voice applications, WhatsApp, which has the advantage of using an open-source codec (OPUS) and jitter buffer (PJSIP). A thorough analysis of real WhatsApp application traffic patterns and jitter buffer handling across different network conditions, ranging from very poor to excellent, was used to determine the generic OTT voice client KPI configuration for the codec (e.g. encoder expected packet loss, packet loss concealment scheme) and jitter buffer (e.g. packet size, mode, jitter size). Following this, an extensive validation of the generic OTT voice client was performed by simultaneously running WhatsApp and the generic voice application in a broad range of network conditions. Comparing the voice quality scores showed a statistically significant similarity between the two.
To learn more about sQLEAR please visit the Infovista Resource Library for some white papers on the subject.
2) Generic OTT service/application traffic patterns and user interactivity
The ability of 5G to deliver very high bandwidth and very low latency provides the foundations for a new category of highly interactive services/applications such as mobile cloud gaming and remote drone control, for which testing of interactivity becomes critical. However, these services/applications come in an even larger variety and diversity of genres, flavors and application use cases than their OTT voice/video counterparts. This again necessitates a generic testing approach.
The generic testing approach for interactivity testing uses generic traffic patterns which emulate both the traffic behavior as well as its adaptability to the network conditions in the same way as a real application would. In this case, the generic application server is an adaptive two-way active measurement protocol (TWAMP) server, which has been designed to change the reflected packet patterns according to the network conditions as measured on the device-based client. The generic traffic sever can be placed in a public and/or private cloud.
Infovista’s first generic solution in this area addresses mobile cloud gaming (a.k.a e-gaming). Based on the idea that the generic testing solution needs to provide meaningful information on the network performance in the most challenging conditions, the real live e-gaming application used as a model is the highly interactive and resource intensive First-Person Shooter (FPS) game genre (e.g. Counter-Strike). Infovista’s generic solution validates if the network under test can provide the optimal performance for a great gaming user experience. The novelty of our generic testing technique (using traffic emulation) lies with the real-time adaptivity of the emulated traffic to the network conditions. Consequently, the networks resource usage matches what the real service would consume. In addition to assessing network readiness for e-gaming services, Infovista provides operators with the means to identify potential improvement actions needed to enhance user experience. This is done by measuring e-gaming KPIs directly on the emulated traffic patterns, which are then mapped to the user experience scores of the e-gaming interactivity KPIs. The mapping function can use a deterministic formula or a machine learning based technique; the latter using concepts similar to our voice quality predictor sQLEAR.
Both the traffic pattern profiles, and the mapping were determined through extensive research of real live games and subjective testing of user experience on gaming interactivity. In addition, Infovista developed a robust procedure to verify that the generic traffic profiles stressed the network in a similar manner to real live applications, in the areas of resource allocation, streaming client-based adaptability, and error concealment reaction to the network conditions.
3) Generic framework for OTT media application testing
The third pillar of Infovista’s generic testing approach is the generic framework data collection for real live OTT media application testing. The solution provides operators with the capability to run a “sample” test of OTT media applications to validate their findings based on the generic testing approach presented above. The “sample” test is a specific set of OTT media applications with a limited set of ETSI defined KPIs, which can be used to infer the actual user experience. As mentioned above, this limitation exists due to OTT application encryption, which drastically impacts KPI availability for measurements.
The generic framework takes care of one of the key challenges of OTT media application testing, that is the continuous changing of their configuration parameters. The login or logout and behavior of the application can change without notice, and can differ between devices, platforms, countries and even networks. Infovista’s generic framework solution solves this by providing user interface (UI) automation when setting up the tests, while the test methodology and KPIs remain generic across all evaluated OTT media applications and in accordance with ETSI specifications. With only one drive and one test script required, Infovista’s generic framework approach allows operators to quickly test any new OTT media application with consistency and confidence.
In Conclusion
The multitude of OTT apps and the ever-expanding range of devices, means it is simply not practical or financially viable for operators to test every device, every OTT app and every interactive service. That said, Infovista has developed a solution to cope with these challenges. Our generic testing approach provides results that are highly correlated to real live testing. Using this approach provides operators with the confidence that their network will deliver the expected user experience for even the most challenging applications.
There is a lot more to say on the topic of generic testing and we will be publishing several white papers and blogs on this topic over the coming months. Please sign up to our newsletter below to be notified when we release these.
