Oxford languages, the leading dictionary publisher, defines empathetic (/ɛmpəˈθɛtɪk/) as "showing an ability to understand and share the feelings of another." It is only logical to assume that organizations would be testing their products with customer empathy in mind.
However, most product organizations seem to have various treatments in action when it comes to customer empathy.
These organizations assume empathy to be the responsibility of the design teams. After all, empathy is the main attribute of the customer experience, and customer experience falls in the design teams' gambit. By limiting empathy to the design team, these organizations invariably limit the customer/user experience to inherently lagging A/B testing.
A/B testing is often driven by the design teams, with the critical objective of validating the preset design hypothesis. One of the essential flaws of this approach is that customer experience in varying customer environments is often not effectively evaluated. In the OTT video context, where connected experience is critical, design teams do a limited analysis of the varying bandwidths' impact on customer experience.
Such research is often assumed to be done by the technical teams as a part of the performance resulting in the design team taking partial ownership of the empathy "problem".
Product managers have an unsaid task of filling the gaps in the design and technical teams' assumptions. Strongest and the most powerful tool in the product manager's arsenal are the KPIs. A good product manager would have cascaded down the nonfunctional requirement KPIs to the technical team. Such NFR KPIs are the reflection of the customer experience.
Due to the sheer diversity of the customer preferences and customer environments, Product managers stick to the quantitative way of attributing value to the NFR related activities. While this is an efficient approach, this approach's inherent flaw is that this is a feedback loop and not a feed-forward approach. In video and OTT applications, QoE measures are the critical KPIs that reflect closely with the customer feeling and experience.
Product managers invariably consider the lagging QoE KPIs, thereby reacting to the customer experience and not proactively defining the customer experience.
While it is not very close, NFR or performance testing is often considered somewhat related to empathy. NFR test scenarios are expected to be the reflection of customer user experience. Test leads or test architects (a rare breed) often design the test strategy, which is then executed by the tester either automated or through the manual testers.
Traditionally this test suite is the smallest of various test suites. From our experience working with over top OTT video organizations, the average test suite size of the NFR is a mere 11 test scenarios. That's is the reflection of the importance of customer empathy provided by the technical teams.
In connected applications like Video OTT, last yard environment factors play a significant role in defining the customer experience. While CDNs play a crucial role in addressing the last mile challenges, the last yard challenges are often inadvertently missed. In a uber-competitive world like OTT, brand loyalty is not a factor. Close to 39% churn rate is connected to customer experience. Churn rate is a matrix mover KPI for any product manager.
Product managers need to acknowledge that having leading indicators on these KPIs is exceptionally critical for product sustenance and potential growth. Structuring the customer experience KPIs as actionable ones for both design teams and development teams is very important. It's equally crucial for product managers to embrace "Shift left" when it comes to customer experience KPI design. Development and test teams need to consider putting together processes that allow the NFR test suites to be enhanced to address the last yard issues such as Wi-Fi changes, network changes, etc
With the sheer number of devices and customer environments, Manual testing can never match up the customer needs. Automated multi-device QoE and customer experience testbeds are essential. Enhanced multi-device automation strategy is critical. The development team should look at a more comprehensive data-driven development and testing strategy to improve customer experience. The design team needs to explore beyond traditional A/B testing by incorporating experiences for last yard challenges and interventions.
To achieve a more empathetic approach to testing, product managers, design teams, and development teams need to have a shared goal of improving customer experience.
End-user Environment simulation, automated multi-device QoE testing, and enhanced NFR KPIs are the critical pillars of empathetic testing.
Multidevice automation
Playback testing
Last-yard QoE testing
Content Monitering
QoE KPI Monitoring and Validation
Smart Automation Tools
Comprehensive Reporting
