We had a great webinar last week on how to measure the ROI of user experience with Susan Weinschenk, the Founder and Principal of Weinschenk Institute, LLC. Unfortunately we didn’t have enough time to answer all of the questions about measuring the ROI of UX that our audience asked during the webinar. So, we wanted to do a follow-up blog and share Susan’s answers with all of our readers and webinar attendees.
1. If I want to use an ROI calculation in order to show a prospect that they could be saving a lot of money doing UX on their site (already online), what’s the best way to do it?
Susan: Decide on what the KPIs are, and estimate the data you need to do the ROI calculation for those KPI’s. Or talk to them first to find out what the KPIs and data estimates are. For example, exactly how will they save a lot of money? Because people will buy more? Because visitors will take less time to buy? You have to decide on the KPIs in order to start the calculation.
2. In what context do you try to bring up the discussion of ROI with stakeholders? Do you try to come up with ROI thoughts before you have that discussion?
Susan: I would definitely put my thoughts together before starting a conversation, but I would consider my ideas, for instance on what KPIs might be important, to be just starting ideas. It will get the conversation started. If you say, “I think an important KPI is “x”, but then I thought perhaps “y” is more important, that is usally all you need to get the conversation started. People usually have lots of opinions and ideas if you get them started.
3. How to measure ROI for conducting a usability test? My management focuses on creating user interface which is impossible to skip, but they would like to skip the user research part.
Susan: There are several ways you could do this. So assuming that no one gets design perfectly correct, and it’s even less perfect if there isn’t user research, then what is the likelihood that important interface features and issues will get messed up without user research? How damaging is that? How long will it take to correct later? These are the KPIs I would use in this situation.
4. How do you handle companies coming in with arbitrary KPI’s, a 50% reduction in drop-off for example, but not having done any real ROI research before hand? Do you often recalculate it on your own and give them a more realistic number?
Susan: I’ll run the ROI with whatever data or estimates they want to use, but if I believe their estimates are flawed I will point that out to them, as well as the risk of doing the calculation with wrong data. I would probably suggest they re-calculate it. I don’t think I’ve ever gone off and done a calculation without their participation.
5. Have you ever calculated the ROI of a drop in brand perception? e.g. customers will think less of the brand because of the issue they are experiencing?
Susan: Yes, I have. It really is the same. The key is to figure out how you are going to measure the effect of the drop in brand perception. Are you going to measure that as whether there is a drop in visits to the website? Reduced sales? Lower scores on a satisfaction survey? That’s the tricky part. It’s deciding on what measurement would be the most meaningful to you.
6. If you are creating B2B software applications that your clients have to use, what are some key performance indicators that my business would care about ? They care about selling products . They usually want them to be “good enough to use”. In an agile environment, there is pressure to get things out quickly.
Susan: I think sometimes as UX people we have different criteria and different KPIs that we think are important than our clients do. But you’ve got to use KPIs that they think are important, or they will just dismiss your ROI calculations. If they care about sales, then use sales, but maybe you could ask if they want to consider satisfaction, calls to the help desk, wasted time, etc. If they don’t, then they don’t, and “good enough” is what everyone will get. That may just be the reality of it. They may not care enough to create a more optimal user experience.
7. If your site is new, how much data do you recommend to have before you attempt any UX work? For example, if I am measuring conversion, a new site has no real benchmark to compare to.
Susan: That’s true, that there wouldn’t be an actual benchmark, but certainly someone has an expectation. If 100% of the visitors convert that would be wonderful, and 0% would be terrible, so what’s reasonable and what’s considered successful? Is it 10%? 20%? 50%? So you can use this questioning to get a starting number to work from until you have real data.
8. What do you think about using abandonment surveys to benchmark your eCommerce site?
Susan: It’s one datapoint. I would hope to have other data points rather than just abandonment surveys. If that’s all you have then use it. Better yet, combine it with other KPIs (time on task? return rates?).
9. How about intranets where companies don’t really care about time (users are not employees) used? What would be a good ROI calc to convince a client it is important?
Susan: I would turn that question around… It’s not what ROI or KPI you could use to convince a client that intranet UX is important. I would interview the client and find out what they think is important in general, and then figure out from that what KPI they actually care about (that they may not realize they care about) that is related to UX work.
10. As a psychologist, why do you think people are so reluctant to invest in preventative or proactive measures and prefer to wait until something severely breaks before they react?Your presentation made me think of preventative healthcare versus reactive care and how hard it is to convince people of the ROI of illness prevention. Why do you think this is?
Susan: Well, this is a big question! It’s all about the frame of reference you are using, and also, who you are speaking with. If you have a mid level manager who is recognized and rewarded for certain things, then those are the things he or she is likely to pay attention to. If you want people to invest in proactive or preventive then you will have to use a message that speaks to one of the more unconscious factors… check out my book How To Get People To Do Stuff for info on this. Or read an overview of it at my blog: www.theteamw.com/blog
11. Isn’t the lack of marketing of the UX profession and how it can help business, a part of the challenge in selling ROI?
Susan: Yes, I think that this is one of the main reasons to calculate ROI, and one of the reasons you need to do it with the business side. If you want to market UX in the organization then you have to show how UX is benefitting the organization. That’s what calculating ROI can do for you.
12. How long does it take to typically calculate the ROI?
Susan: It can be a few minutes if you have one KPI and you have all the estimated data. Typically this takes from 3 hours to 3 days if we have to hash out what the KPIs should be and if we have to go dig up some data.
13. What would you say is the number 1 point to make when UX is new to the organization (to prove its value)?
Susan: There isn’t a number 1 point because it depends on what’s important to that organization and to the person you are speaking to. So it might be more revenue, or it might be not wasting the salespeople’s time, or it might be pressure from competition… Basically at this point UX is your “face” to the world/customers/employees… so what’s important to you about the face you present?
The webinar recording and slides are now available to view at your leisure.