How to create best in class digital products using facial expression analysis

Understanding subconscious emotions is important to anyone building a product in the B2C or B2B space.

When those of us who work in the product design world think about our users, we naturally consider the rich marketplace our products will compete in and the way our brands, products, and digital experiences will compare to others. 

What we are less likely to consider is the rich emotional reality that our users feel that also impacts their interaction with our products.  

Self-reported emotional experiences are important and oft-recorded and analyzed in product research, but mounting evidence suggests that it is subconscious emotions that are a more reliable predictor of purchase behavior than self-reported emotions and preferences. Therefore understanding subconscious emotions is important to anyone building a viable product in the B2C or B2B space. 

This article will focus on the ways that remote research methods, accelerated by the COVID-19 pandemic, have laid the groundwork for increased opportunity to incorporate the analysis of subconscious emotions into the product development lifecycle.

Subconscious emotions

Generally, emotions are conceptualized as either discrete feelings (e.g., happy, sad, angry, surprised) or biphasic experiences (e.g., negative feelings versus positive feelings). Conscious emotions are ones that we can accurately self-reflect and self-report on. 

Subconscious emotions, however, are feelings that exist out of reach of our immediate conscious awareness. The data types that can help reveal subconscious emotions include (but are not limited to) brain waves, skin conductance, voice expression, and facial expression, and these data types each exist on a spectrum of ease of use and accuracy. 

Recent developments

In the past, recording and analyzing subconscious emotions has been limited to advanced labs and technology often found in academic or medical settings. However, developments in software-based solutions that can be acquired relatively quickly have made access to subconscious emotional data and analysis more feasible. 

While more accessible than ever, there are still trade-offs to consider with each type of biometric data that record emotions. With all of the various pros and cons associated with these data types, facial expression analysis (FEA) offers the most simple implementation that still provides rich and distinct emotion data.  

FEA requires video recording and specialty software that analyzes the micro-expression action units that are associated with the discrete emotions of happiness, sadness, anger, and surprise.

As the UX research (UXR) industry moved towards increased remote testing out of necessity due to the pandemic, the advantage of using FEA paired with easier logistical access to the data has helped researchers gain deeper insight into their consumers’ feelings and attitudes beyond what they can reflect themselves in a think-aloud protocol or by answering survey questions.  

This opportunity to understand customers’ true emotional experiences when interacting with products has helped customers build deeper understanding and empathy, and it can also aid in the prioritization of design updates. 

Expressing the inexpressible

There are almost always more updates and design recommendations than most product, engineering, and design teams can execute at any given time, but without an understanding of how customers are impacted by each update, teams are left with their best guess as to which to address first.  

For example, in any given digital experience that has complex steps, users can only express what they can parse and identify themselves, and if the entire interaction involves learning, then it is even more challenging to identify the specific moments that require the most attention. 

But with FEA, researchers can identify the interaction or task that causes the most anger or frustration against a slew of other feelings, this provides invaluable information about the path forward that a customer cannot express. 

Use cases

While FEA provides valuable information for anyone building customer-centric products, when considering the return on the investment of time and energy using FEA there are some scenarios and use-cases that are ideal. 

These use-cases include, but are not limited to:

  • A study whose objectives includes an understanding of discrete emotions (for example, understanding the emotional journey of customers on an e-learning platform as they engage with challenging content; Multi-step processes within a digital experience in order to understand moments of frustration vs learning)
  • When there is a discrepancy between product analytics and users’ self-reported experience (for example, there is a high abandon rate during an important interaction experience but customers are not reporting issues qualitatively) 
  • New ideas and concepts that may elicit varying levels of excitement or delight (for example, your team is trying to prioritize which delightful feature will bring users the most happiness)

These example use-cases provide a silver lining to the way our research processes have had to adapt in 2020 (and beyond). They are providing more researchers the ability to justify the investment and attention in technology that will continue to allow for increased remote research beyond the pandemic. 

In the case of FEA, our increased repositories of video-recorded research sessions that involve a customer looking directly at a screen (as opposed to the dynamic nature of research lab recordings) are the perfect testing ground.  

Start demonstrating digital success

In our inaugural UX 360 report, you'll discover who is accountable for UX, what value it brings to businesses, how that value is measured, and what are organizations doing to ensure UX delivers on its goals.

Implications

While FEA has many benefits and exciting data that can be unlocked, it is important to consider the ethical implications.  While many industry researchers don’t have a formal Institutional Review Board, identifying the best way to gain consent and debrief a customer when their data will be used with FEA software is important for several reasons. 

First and foremost, transparency is key to the technology industry's ability to be respectful of our customers and their choices. Second, in a culture that has increased surveillance associated with consumer experiences, providing information about how emotions can be recorded builds trust and frankly, is the right thing to do.  

These are not issues to be taken lightly and should involve a meaningful discussion and process that best suits the way your team plans to use FEA with consumers.

Conclusion

FEA isn’t just a tool for academic and market researchers anymore, its use in small sample UX research can unlock rich insights about your consumers and products.  

To get started, identify the use cases that most suit your team’s product roadmap, investigate the market options that are best for your budget, work with a remote software partner who can provide strong moderated and unmoderated video access, and begin using some of your videos to pressure-test the applicability of the technology for your research objectives.