Screen Shot 2016-09-06 at 11.10.13 AM

Are users getting what they expected out of the product?

This is something a lot of companies want to know the answer to, which is why it’s not uncommon to see this question in a script: “Does this feature work the way you expected it to?”

It’s a very natural and human response to ask the question in this way, but we know that what users say isn’t always what they do or mean – especially people who don’t want to hurt anyone’s feelings. 

Rather than asking the question above, companies can run a usability test and see by the success rates if their feature sets are working correctly. If your users are not successful then they might be having trouble based on whatever mental model they came in with, which is something you can ask them about. Most users aren’t designers, though, so they may not look at the interface the same way you do. This is also why they may not answer “Does this feature work the way you expected it to?” accurately enough to generate actionable insights that identify how to prioritize your current feature list. 

So, how can you be sure that product features are actually meeting user’s expectations in a measurable, meaningful way? 

First, understand their goals

I often emphasize to clients that the first step to understanding user expectations is to understand users’ underlying goals.  While you may already have some idea as to what the main goals of your users are (as you have designed interactions to meet these needs), asking users to rank their top 3 to 5 goals in the context of your product and your industry is a quick and easy way to get a top-tasks analysis.

For example, I use my mobile banking app to:

  • Check my current statements
  • Review my account balance
  • Transfer money between accounts
  • Pay my bills
  • Etc.

Also, asking about their goals helps to set the context to ease them into thinking about features in the product that you are intending to design for your users.

Once you understand user goals you can start developing specific kinds of studies to better measure and understand their expectations on different product features.  

The Most Used Method – Card Sorting

In this type of classic UX card sorting, you want to see what items (features) users would group together as the most important. This is the most oft used method for measuring user’s expectations in regards to product features.

As an example, you can ask users to list features and drop them into 5 categories:

  1. Very important to me
  2. Somewhat important to me
  3. Nice to have
  4. Somewhat not important
  5. Not at all important to me

The reason why companies tend to use this method is because it’s easy for users to group together and prioritize features in a very natural way by dragging and dropping cards. However, it’s important to note that analyzing card sorting data can be challenging since the answers are not always clear enough to draw actionable insights from. This means there is the potential for subjectivity when it comes to what each grouping means.

If you want different method options that have less subjectivity, you should look into the Kano Methods.

Feature Prioritization Analysis – Classic Kano

Sometimes you may already have a product that is in use and you want to evaluate the current state of that product and how it’s doing. This type of feature prioritization is called a Classic Kano model, which relies on the correlation of two areas –  Importance and Satisfaction. It should be noted that for this method to truly work it requires that your users have experienced your product before. 

For Example:

How important are the following feature sets to you? 1 – Not at all important, 5 – Very important

  • Download my account statement
  • Check my account balance
  • Transfer money between accounts
  • Pay my bills via the mobile app

How satisfied are you with our mobile banking app features? 1 – Not at all satisfied, 5 – Very satisfied

  • Download my account statement
  • Check my account balance
  • Transfer money between accounts
  • Pay my bills via the mobile app

By simply charting the highest two scores (4’s and 5’s) of these features along side with their satisfaction ratings, we can see how the product currently measures up. 

screen-shot-2016-09-07-at-2-39-13-pm

 

Feature Prioritization Analysis – Functional/Dysfunctional  Kano

The Functional Kano method tests the presence and absence of specific features that are proposed in a new interface. This means that users don’t require any previous experience using the product.

The results of such data can be very rich and include:

  • What features would delight users
  • Which features users expect
  • If the feature was absent it would upset users
  • Features that are not important, but don’t bother users
  • Features that would annoy or frustrate users if it were present

screen-shot-2016-09-07-at-2-44-43-pm

Examples of a Functional and Dysfunctional Kano:

(Functional) What are your feelings when the feature is included in the service/application?

    • I like it this way
    • It expect it this way
    • I am neutral
    • I can live with it this way
    • I don’t like it this way

(Dysfunctional) What are your feelings when this feature is NOT included? 

    • I like it this way
    • It expect it this way
    • I am neutral
    • I can live with it this way
    • I don’t like it this way

Conclusion

If you want to really know what users expect out of your product / product features, you’re going to have to get more in depth than simply asking them “Does this feature work the way you expected it to?” The most common method to do this is via a card sort, but like all methodologies, it has its benefits and drawbacks. This is why having more tools in your toolbox is invaluable, and the Kano Methods are great options to get actionable insights that will impress your stakeholders. And while this isn’t an exhaustive list of all methods to measure user expectations, it’s a good place to start.