Making UX measurement manageable

Christian Rohrer, Vice President of Experience Design at TD Bank shares his expertise in UX measurement.

Christian Rohrer, Vice President of Experience Design at TD Bank shares his expertise in UX measurement, specifically the PURE Method. Pure stands for Pragmatic Usability Rating by Experts and is a usability-evaluation method that attempts to quantify how difficult a product is to use and provides qualitative insights into how to fix it, without costing a lot of time or money.


Alfonso de la Nuez: Welcome to UXpeditious! A show that brings you quick, insightful interviews with design, product, and UX leaders.

Dana Bishop: In each interview we dive into how UX research impacts user insight; shaping the design and business strategy of some of our favorite tech tools and products.

Alfonso de la Nuez: I’m Alfonso de la Nuez, Co-CEO and Co-Founder of UserZoom.

Dana Bishop: and I’m Dana Bishop, VP of Strategic Research Partners at UserZoom.

Alfonso de la Nuez: And we are your hosts.

Our guest today is Christian Rohrer, Computer Scientist, Designer, Researcher, author, and leader in the User Experience field. He’s worked at a number of major companies in the design and experience industry: Yahoo, McAfee, Capital One, and eBay to name a few.

Dana Bishop: In today’s episode we’re gathering insights from Christian on a subject, you could say he’s an expert on; the business of measuring and quantifying UX.

Alfonso de la Nuez: Thanks Christian for doing this with us, man.

Christian Rohrer: My pleasure.

Alfonso de la Nuez: I was thinking about topics for the podcast and UX measurement was definitely going to be one of them. We'd love to hear from you and your passion for measurement. Why it is an important process for teams and companies to consider as they mature in user experience.

Christian Rohrer: Let's break it down a little bit because if we're measuring user experience, the first thing we should do is talk about what user experience means. If you look at the definition of user experience, from various authorities, whether it's the Nielsen Norman Group or IXDA, or whoever, the essential definition says that it's all aspects of every interaction with a brand or a company or a product and the services it provides.

That all encompassing definition makes it very hard to then operationalize how you measure, because how do you measure everything? Are you going to go out and measure every single touchpoint that you have with a customer, are you going to pull that all together and have some Uber score? That may be a good way to do it, and certainly an effort worth trying.

With an all encompassing definition like that, let's decide which element or aspect of user experience we're going to measure first, and then we can contract down into how that would best be done methodologically or pragmatically. While there are lots of elements of user experience, I tend to think of it in a simple way. I tend to think about user experience has to really start with meeting a user need, providing some kind of value that the user gains from the company.

It actually has to be the most central thing that we provide, it's the value proposition, and then outside of that are the more traditional layers of the digital user experience that people think about. There might be the tasks that people perform, the ease of those tasks, how we get those things done, or there may be the information architecture, and the content, and the way we speak to the customer, or there's the visual design, the way we present ourselves in a visual manner.

They all work together, those outer layers definitely work together to allow the customer to reach that value that we offer. They're a little bit different in that the value depends very much on the domain. Right now, I'm in banking and the value you offer to those customers is going to be very different than my last job where we were providing security.

The other elements, like how good the visual design is, how clear the content is, how easy it is to perform tasks in order to get my needs met as a user, those are governed, in a way, by pretty universal principles. Once you know that, okay, the interaction and the use of a product, there are some principles behind that. Then you can start to figure out, well, how do we measure those? Whether those principles are available or not. What I like to do in measuring user experiences is to say, which aspect or element are we talking about first? Now let's go zoom in on how you actually do that.

Dana Bishop: Can you give us a breakdown of the PURE Method, starting with what pure stands for?

Christian Rohrer: PURE stands for practical usability ratings by experts, and it is intended to be an alternative way, if you will, to measure the ease of use for a product or service, for a specific type of user, and against a limited set of tasks.

The way you generally conduct a PURE Methodology is, you first start by being clear about, who's the target user type we're talking about when using this product or service, and that's necessary for a lot of reasons, but mainly because it's important to know who your most important customer is.

Then, you'll see when I explain how the rating method works, you need it to be able to do it successfully. Then you limit the number of tasks that you are able to score, or provide a score on, just to make it practical, to make it usable, I should say, to make it doable for the practitioners that are creating it.

Dana Bishop: How do you determine what those tasks are? You're limiting the tasks and what method do you use to figure out which of the tasks?

Christian Rohrer: There are a couple ways to do it. You may have heard of the term Top Tasks, if that may be a methodology that's pretty popular. That's a way where you determine what the Top Tasks customers have by actually surveying them and finding out what matters to them most.

The way I do it, is by having a conversation with the design leaders and the product management leaders, and I try to ask them, what is fundamental for this user type to be successful and for our business to be successful. I try to say the five to 10 fundamental tasks, or if it's a more complex product, like an enterprise product, maybe up to 20. A typical response from a product manager will be, well, all of our features are important, it's everything.

Then you really have to press them to reduce that set down, just to make it really pragmatic. You can't apply the method when it's every single possible use case and flow that you could be reviewing. That leads me to another point I want to make, which is, while it sounds like the PURE Method is a research method, it's actually not. It's an analytical method, meaning it's a way to produce a score, kind of like consumer reports will show you how good or bad a product is against various criteria, but this is a way to provide a score that reflects its likely ease of use for the target audience.

Alfonso de la Nuez: When you say it's not in research, it's because you're not actually involving end users, right?

Christian Rohrer: That's right. It's not empirical, meaning we're not gathering data from users. Now that sounds like blasphemy, I know that sounds like we should never do that, but there are other analogies you could think of that makes sense here. One analogy I like to use is, if you were to review something like a gymnastics performance or an ice skating performance, that performance is judged a certain way, there is a set of rules and a way that judges, a panel of judges, reviews it and scores it, depending on the difficulty and how well the athlete performs the moves that they have.

Similarly, a product, if you look at it, the way it presents itself to a user, in the fundamental tasks that have been defined, there are principles, going back to the UX principles, that it either exhibits, it either has or doesn't have.

Those principles are what help you see whether this would be for the target user, likely to allow them to easily use it or not. This now tells you why it's important to know who the target audience is, because if I told you we're going to evaluate from the perspective of anybody or everybody, it would be very hard to come up with a score that would have reliability amongst different raters and be valid because someone who's very technically adept is going to be more able to accomplish certain tasks than someone who's less technically adept.

Dana Bishop: I think there's a key difference that I found really interesting between a heuristic evaluation and PURE Method. If you wouldn't mind talking about kind of where that, they're both expert involved, rather than end users, but there's some key differences that are really interesting.

Christian Rohrer: There sure are. The main thing is that the heuristic evaluation methodology has a certain way of being done where the raters, the way it was defined by Jakob Nielsen anyway, the raters are independently reviewing a product, on their own, and that's good for the purpose of finding and evaluating and really reviewing anything and everything that might be a potential usability issue. What it does is it means that they see very different things, and in a different level of timing, if you will.

One thing that PURE does to help alleviate the fact that different raters will find really, really different experiences, is that it lets them see the same experience at the same time, just like a panel of judges is seeing the same performance at the same time, they're not seeing a different performance, they're seeing the same one, and that allows them to have the same context.

Alfonso de la Nuez: How does the PURE Method impact business and product strategy?

Christian Rohrer: We know that measures and measurements are very impactful to business and strategy and for decision makers, this is a fact, we can't get away from it. They need and want numbers and okay, but the challenge that we often have, and you know from your years at UserZoom, how getting regular numbers that reflect ease of use, or user experience, can be challenging. It's not at all impossible, it's very doable, but let's say that we want to do a benchmark of our core product, and we want to know how well it's doing in terms of ease of use.

The way most companies are set up, it's impractical to get that done that frequently, maybe it's done yearly or every six months. That's a fairly intense kind of study, and it does require a good number of participants. You may be running through your panel of participants quickly, if you do that, and if you reuse people, then you have the problem of them knowing the task already, and then now doing it again, so there's some effects there you want to avoid.

The PURE Method isn't better because anything empirical, I think, is better than something like PURE, but PURE does have a strong overlap, and when you have it on a regular basis, like you could have a PURE score produced in less than a week. That allows you to apply it to the regular cycle of business, and then that allows you to insert those numbers and score cards into regular business conversation, and then that allows decisions that are strategic and important to be made.

Dana Bishop: I think one of the things that also is interesting, thinking about heuristic evaluations, which I think we've both done many of over the years, and you're looking at everything and you're trying to pick apart everything and you're trying to get down into the weeds and the details of everything.

Where as, the PURE, you're keeping those top tasks, and one of the struggles in my past has been trying to present such detailed information from a heuristic review to executives. It's not as tangible, and it's not as connected to the business goals. Whereas, you're focusing on those core top tasks, I would imagine it's a lot easier to understand, and it has a much higher impact on that executive, rather than a research team. You're talking to a different level.

Christian Rohrer: You're right that the executive team doesn't really want to nerd out on the details of the interaction and the heuristic that was violated and what severity it is. That's kind of almost too much detail for them.

What they do like is that high level scorecard, it shows them exactly where to focus, what to worry about, and then what we do with the PURE Method, in addition to providing a scorecard that summarizes the ease of use of the top tasks for the most important user, is behind every single score and each little score is a step in one of the tasks, there's a companion.

Usually we use some kind of a slide show, like a PowerPoint or Keynote, there's a separate slide that shows either one or two screens of that step, and then it has all these call outs that describe, what is the actual issue with this particular step or what caused it to get a red three score or a yellow two or a green one? What could be improved?

What that does is that combination of the scorecard up front, with the lengthy deck afterwards, with all the details, it gives people two things they need. Do I have a problem? Yes. Okay, I see where the problems are. What do I need to do about it? Go here, go here, go here, and then you flip the page and you go right to that slide, and there you are. It gives them and the design and the product team, a way to go to get out of the trough of despair, which you might be creating with a lot of red.

Dana Bishop: Right. It's simple in its presentation, the visual red, yellow, and green, a single score. But it's actionable in that you then had the details behind it of, okay, here's the issue, here's what you do with it, and I would imagine that addresses more audiences than just the executives. It also gives the UX team somewhere to go.

Christian Rohrer: It sure does. We know that UX teams and designers, they really thrive on qualitative insights and information about the root of the underlying problem they're trying to solve. While this doesn't really qualify as an insight coming from research, it does kind of indirectly, because when you do the PURE Method, you have to do it with a panel of trained raters or experts.

Alfonso de la Nuez: Christian, Thank you so much.

Dana Bishop: Thank you for being here.

Christian Rohrer: Thank you, Alfonso and Dana. It was my pleasure.

Alfonso de la Nuez: That's Christian Rohrer, Vice President of Experience Design at ​​TD Bank.

Dana Bishop: Thanks for listening to UXpeditious. Make sure to continue listening to our new episodes each week for quality insights from UX industry leaders. If you like what you heard, help us out by rating and reviewing the show on your favorite podcast platform.

Alfonso de la Nuez: UXpeditious is produced by UserZoom in partnership with Pod People. Special thanks to our production team: Christopher Ratcliff from UserZoom; and the team at Pod People: Rachael King, Matt Sav, Aimee Machado, Hannah Pedersen , Colleen Pellissier and Jason Mack.