5G has resulted in the emergence of a large number of diverse services and Over The Top (OTT) applications, each with demanding network bandwidth and latency performance requirements. This, correlated with the increased complexity of the network, encryption of services/OTT applications, proprietary OTT codecs/clients/delivery protocols and the need to minimize network operational costs, calls for advanced testing techniques. These techniques need to be easily adaptable to a diverse set of services and OTT applications and must provide an accurate evaluation of user experience. The metrics assessed need to cover familiar topics such as conversational OTT voice/video apps and OTT audio/video streaming. They also need to evaluate previously untested areas such as user experience within interactive services such as OTT cloud gaming over mobile networks, remote drone control and video conferencing.
What are generic testing techniques?
Genetic testing techniques have emerged as best suited to rise to these challenges. These techniques address both service/OTT application test configurations and performance evaluation, as well as the data collection methods. The generic approach innovatively uses generic OTT services/clients, sophisticated real live service traffic pattern emulations and/or machine learning (ML) algorithms to test services and OTT applications including OTT voice/video calling and conferencing, cloud gaming over mobile networks, OTT video streaming, remote drone control.
The benefit of a generic testing approach is that it can deliver cost-effective network-focused performance evaluation, root cause detection, analysis and optimization without the need to test a large variety of complex and encrypted OTT applications. Additionally, it provides operators with continuous monitoring and benchmarking to ensure that the minimum performance requirements for services/applications are met. In this way, operators spend their efforts on what they can fix and optimize (the network) rather than on the OTT services/applications that they cannot control. Finally, by using a well-designed generic framework for data collection, operators can still test a set of the most popular live OTT services and applications characterized by a limited set of KPIs, to provide the most important information on user experience within those services and apps.
A generic approach mimics the service/application traffic, behavior, and/or configuration (codec/client) rather than using the real service/application. Naturally, questions regarding the usefulness and accuracy of a generic approach arise.
With extensive expertise in drive testing, active testing and QoE modeling and estimation, Infovista understands the possible risks of a generic testing approach. Therefore, Infovista’s technology research team worked with standardization bodies (ITU-T Study Group 12, ETSI STQM, Service Transmission Quality Mobile) to define specifications and recommendations for these techniques. Our experienced team also provided guidance for data-driven proof that if the generic testing approach closely mimics the real service/application then the generic approach delivers trustful and representative results. Closely mimicking a real service/application means that the generic approach triggers the same network resources, uses the same protocols, and encodes and adapts as real services and OTT applications/codecs and clients would.
What are the three generic testing techniques and what is their scope?
There are three generic testing techniques:
- Generic OTT voice / video clients. Testing all mobile OTT voice and video streaming services and applications is practically impossible. This is due to the large number and variety of mobile OTT apps, multiple different platforms and device-based operating systems (OS), as well as proprietary codecs and clients, and error concealment schemes. Perhaps the most challenging aspect, however, is the level of encryption within these applications, which in many cases is device OS dependent, and continuously, and even dynamically changing.
These testing challenges call for the use of a generic client that mimics the behavior of an OTT voice client (for example WhatsApp) and/or video streaming clients such as video on demand category like Netflix or live video streaming category like Breakout News. The testing involves a cloud-based public and/or private voice application server and on-device client completely controlled by the test set-up.
This generic client, which behaves like all the most used applications, provides the ability to test only one OTT application, one version and one set of fully accessible KPIs – all free of encryption. The result is a reference of network performance for the OTT application type for which the generic client was designed.
- Generic OTT service/application traffic patterns and user interactivity. The ability of 5G to deliver very high bandwidth and very low latency provides the foundations for a new category of highly interactive services/applications such as mobile cloud gaming and remote drone control, for which testing of interactivity becomes critical. However, these services and apps come in an even larger variety and diversity of genres, flavors, and application use cases than their OTT voice/video counterparts. This again necessitates a generic testing approach.
The generic approach for services’ interactivity testing uses generic traffic patterns which emulate both the traffic behavior as well as its adaptability to the network conditions in the same way as a real application would. In this case, the generic application server is an adaptive two-way active measurement protocol (TWAMP) server, which has been designed to change the reflected packet patterns according to the network conditions as measured on the device-based client. The generic traffic server can be placed in a public and/or private cloud.
In this case, the result represents the network-centric performance for all interactive services and OTT applications; rather than for a specific service/OTT application for which the performance is also dependent on the streaming server, client, codec and individual subscribers’ devices. This helps operators optimize the network and ensure the minimum performance requirements are met for a broad range of interactive services/OTT applications, delivering significant testing cost efficiencies.
- Generic framework for OTT media application testing. The third generic testing technique refers to the data collection procedure and it consists of a generic framework for data collection for real live OTT media application testing. The solution provides operators with the capability to run a “sample” test of real OTT media applications to validate their findings based on the generic testing approach presented above. The “sample” test is a specifically pre-selected set of real OTT media applications for which only a limited set of ETSI-defined KPIs can be measured. These can be used further to make some inferences about the actual user experience. As mentioned above, this limitation exists due to OTT application encryption, which drastically impacts KPI availability for measurements.
The generic framework takes care of one of the key challenges of OTT media application testing, namely the continuous changing of their configuration parameters. The login or logout and behavior of the application can change without notice, and can differ between devices, platforms, countries and even networks.
In this case, the results can be used to perform, with a real OTT application, a sampling validation of the network optimization executed based on the previous two generic testing techniques. In addition, this generic solution can be used to quickly run informative tests whenever a new OTT application comes out. The depth of the knowledge regarding the performance is dependent on the level of encryption which can allow access to more or less ETSI KPIs measurement.
What are Infovista’s generic testing solutions?
Infovista’s generic testing solutions cover all three techniques described above. While the first two techniques have been developed by Infovista within the context of standardization activities, as author and co-author, the latter one regarding the generic framework for real OTT application testing is an Infovista proprietary solution.
Infovista offers a generic OTT voice client solution with our user experience voice quality machine learning-based predictor sQLEAR (a.k.a ITU-T P.565.1).
To learn more about our generic OTT voice client, download our whitepapers:
- Designing a generic OTT telephony application
- OTT telephony application testing: A data driven validation
Infovista offers generic traffic patterns emulation and interactivity scoring as defined in ITU-T G.1051. However, in the case of cloud gaming over mobile, Infovista introduces a novel aspect which is the real-time adaptivity of the emulated traffic to the network conditions. This ensures that the network resource usage matches what the real service would consume. In addition to assessing network readiness for e-gaming services, Infovista provides operators with the means to identify potential improvement actions needed to enhance user experience. This is done by measuring e-gaming KPIs directly on the emulated traffic patterns, which are then mapped to the user experience scores of the e-gaming interactivity KPIs. Both the traffic pattern profiles, and the mapping were determined through extensive research of real live games and subjective testing of user experience on gaming interactivity.
To learn more about testing and optimizing the user interactivity experience, check our whitepaper on mobile gaming user experience.
