The ROI of UX Research – Questions and Answers
ROI Questions and Answers with Dr. Susan Weinschenk.
During our recent webinar with Dr. Susan Weinschenk on the ROI of UX: How to Win Funding and Secure Buy-in, people had a lot of great questions about the value of user research. Given the popularity of the topic, and the frequency with which we hear these questions being asked, we transcribed them to share with our readers. Enjoy!
Q: I need to convince our product and design directors that we need to do foundational research as we need to do more than just small scale projects like usability… usability only helps with problems AFTER a product is designed or built, but does not help us in the discovery and early stage … So how do we convince our decision makers about the “type” of research we should do or how do we show the ROI of doing early stage research so we know what to build before we build it? As you know, doing foundational research costs a lot of money, especially if you outsource to an agency like yours…
A: I didn’t mean to imply that the ROI calculations were for just small scale user studies. The same applies to doing larger user research. You have to find out what it is that the product and design directors are worried about/care about. Interestingly, if they are only worried about meeting a time and budget then large-scale user research may be hard to do. In some cases you may have to make your case above them – talking to someone who is concerned about having a product that fails, or falls below expectations, or has to be re-worked because the user research wasn’t done.
In this case I would ask about assumptions. If the team is going to design without user research, then they must be making assumptions about who their target audience is, what the target audience wants to do or will do and so on. Take one or more of those assumptions and map out the consequences of those assumptions being incorrect.
For example, I had a client that was making changes to their website based on a new demographic they wanted to attract. They had assumptions about what the new people (women who were younger than their usual demographic) would want or not want. They also had assumptions about how their current demographic would react to the changes. But they hadn’t done much research. They had contacted me about working on the re-design. But the first question I asked was, are you sure? How sure are you about what you are telling me about the new and current target audience groups, and how bad will it be if you are wrong? We calculated what would happen if the changes upset the current customer and DIDN’T grab the interest of the new ones. We came up with a number that represented the possible loss of revenue if their assumptions were incorrect, and then did the calculations for ROI using a fairly large number to conduct user research on both groups.
Q: What if you are adding new functionality and workflow that does not exist – what would your approach be?
A: You have to figure what is the UX work you think you need to do and why. If it’s new functionality and workflow, are you sure you know what the workflow should be? Are you sure you know how people want to go about doing that task? Are you sure your design you’ve come up with is a design that fits the mental model of the target audience? If not, then what is the UX work you need to do? User research? Task analysis? Prototyping? Collaborative design? What is the cost of that work? And what is the risk of not doing that work? Is it having to fix it later (rework)? Is it not being able to sell the software (if indeed it is sold)? Is it loss of productivity of the workforce that has to use it? Pick a key indicator and figure out how much you are saving by doing the UX work.
Q: I do UX for a company that makes its money by selling hardware to customers. We supply accompanying software for free, so it’s difficult to draw a straight connection between improving usability on the software and ROI (since we don’t charge for the software). Any suggestions for calculating ROI for usability of the software in this situation?
A: If the software is poor what are the consequences? Is it that people won’t buy the hardware? Or people won’t recommend it to others? Or people will call the help desk?
Q: How long does it take after UX research is done to see the ROI in a short time frame?
A: It depends on what you are measuring. If you are measuring conversion at a website that’s pretty fast, same with calls to the help desk. Other measurements (productivity, cost of re-work) may take longer. You don’t have to wait to make your ROI case. It is the RISK that you are avoiding by doing the UX work.
Q: Can you drill down into the dollars associated with *user* time savings – and how broadly that metric is accepted in corporate America?
A: User time savings is usually calculated as the time saved (in seconds, minutes, hours, weeks, whatever makes the most sense) times the hourly/weekly salary of the user. However, there are other ways. One time I calculated the impact of saving the time of the salespeople. In that case I calculated it as sales lost because they were spending their time trying to use the software, and I used that metric.
It’s only accepted if people care. Sometimes people don’t care if their employees waste time. Or their customers. So then that will not be a good key indicator to use for your pitch. Always ask, “What is this person worried about? Where is the risk from their point of view?”
Q: I LOVE this, but where does one get these numbers? Especially for non-e-commerce instances. Who estimates how much potential rework will cost? What about internal tools where the “return” is less tangible, like employee productivity or increased morale?
A: Sometimes you have hard data, but sometimes you have to make estimates. So make estimates and then run the numbers by the people and see if they want to adjust the numbers. Better yet, ask them to collaborate with you. SOMEONE knows the numbers or can estimate them. You have to find that person, and if there is no one, then you can do it! It’s better to estimate and calculate than to do UX work not even knowing if it is making an impact to the bottom line!
Q: Susan, can you share some of your titles on the bookshelf behind you? I see several I recognize but many I don’t!
A: There are so many! Habit, Cost Justifying Usability, Gamestorming, The Lean Start-up, Interviewing Users, Blink, Strangers To Ourselves, The Paradox of Choice, Forms That Work, Rise of The Robots, Don’t Make Me Think, and of course my own books, including my NEW one: 100 MORE Things Every Designer Needs To Know About People, 100 Things Every Designer Needs To Know About People, How to Get People to Do Stuff, and Neuro Web Design. Some of the books on the shelves are my books in other languages.
Q: Examples of the right stakeholders to present and work the ROI with?
A: Whoever has the money and/or authority to bring you in to do UX work on the project.
Q: From a product design approach, how would one quantify the ROI for the use of UX Testing within the various design phases? Is this the Productivity example? Is the testing worth it?
A: Which ROI calculation you do depends on which Key Indicator you are going to use. It doesn’t go by the type of UX work, it goes by what’s important to the people in charge of that project. So key indicators for a user test might be conversion, or productivity of employees, or time savings for users, or avoiding re-work by developers.
Q: Have you been able to get the actual ROI (after the implementation) of any of your projects and compare the estimate to reality?
A: Yes, sometimes we actually calculate the ROI. It is usually much LARGER than what we estimated, but that’s because I usually use conservative numbers in my calculations.
Q: What is your approach for quantifying non-financial information? We work with nonprofits who have members, yes, but a lot of their work is “education.” What would you look to quantify in that case?
Also, how do you calculate ROI for something that is not purely transactional? Say you have many enterprise users of an app. How can you evaluate something like increased satisfaction and ROI. (Well, increased Usage Time is one metric. What else?)
A: I focused on money calculations, because that is the “traditional” way to calculate ROI. You could, however, use a different metric, for example, you could calculate the likelihood of increasing usage. My colleague, Guthrie, specializes in doing these kind of “behavioral economic” calculations… What is the probability that certain behaviors will change?
Q: If these calculations include some assumptions, how reliable are they? Do we use the results to convince stakeholders only – OR – can we treat them as official numbers for business decisions?
A: You can use them for both. They are as reliable as you can get data for. You can make estimates, then adjust when you have actual data. Many business decisions are made on estimates. Decisions based on estimates are better than decisions made without any calculations at all.
Q: Do you include salaried employees costs into your cost equation? Or is it just services purchases (like from a third party service provider?)
A: You can include salaried employee costs if that is important. You and your team decide what is important.
Q: How do you calculate the cost of a UX team’s time?
A: This could be # of hours * hourly rate, either as hourly $ paid to the employee, or as hourly rate the department charges.
Q: I’m interested in trying an ROI calculation but I don’t know how to accurately estimate rework costs or usability testing costs.
A: Find someone that knows how to do each of those things and ask them for help. And/or take a stab and then run it by someone who knows how to estimate them.
Q: What if I don’t have access to these “numbers”?
A: Estimate them or find someone who can.
Q: As a consultant, I can see how you can make assumptions for the numbers before – but how do you get the numbers after? (ex: 25% increase in conversion)
A: It’s a good idea to figure out ahead of time how you will get the numbers. What data is currently being collected? Adjust your metrics to the metrics that are readily available. Figure out who has data of that kind and then find out from them what data they have or can get.
Q: How to estimate ROI of enhancements that don’t translate well to savings in time or rework, such as things that simply make the product more attractive?
A: Ask, “What is the value of the product being more attractive? What behavior will that change?”
Q: What do you do when usability issue cost is not measurable or conversion data is not available?
A: Either pick key indicators and costs that are measurable or where data is available, or estimate them.
Q: How would you sell UX to a CTO that thinks their software works good enough?
A: This happens a lot (which is why there is still poor software out there!). Find out whether there is anything they are worried about and work from that. One time we just all had to wait for the CTO to leave the company!
Q: What if we’re a B2C SaaS company, and I’m looking to show the value through LTV as opposed to CRV – How could we estimate the impact of new features that have not been built yet?
A: You have to decide what is the impact of the new features, and then figure out the key indicators for that. Otherwise one could argue that if you don’t have the impact of new features estimated you should be building them.
Q: What advice do you have for someone at an agency who wants to help prospective clients with ROI but may not have access to the inputs/data?
A: Show the client a sample calculation and ask them for the data. Or tell them what you are trying to do and see if they will do so collaboratively.
Q: I’m wondering how to calculate ROI for software that gives general insight into customers, like UserZoom and/or Google Analytics? Thanks!
A: Think about the value of having the information and data. What will you do differently if you have the insight? For example, if you have data from your target audience and then you design based on that, a key indicator might be that you don’t have to re-work the product because it was designed right in the first place.
Q: How did you come up with the costs for ROI (eg. cost of UX technique and cost of rework)?
A: I just estimated for example purposes. It depends of course on how you are doing the work. Doing user testing yourself? Hiring it out to a UX firm? Doing remote user tests or in-person? You would need to calculate your estimated costs.
Q: How were you able to estimate that your abandonment will be reduced by 25% I assume you needed that ahead of approval of work?
A: In that particular example we had already done a user test and we based our 25% estimate on the user test. It was a conservative estimate (since almost everyone we tested had the same problem and said they would have abandoned!
Q: How do you know that you’ll get back ALL of the money you’re losing? Couldn’t it just make it 50% better?
A: Yes, so you can use a more conservative number. You can actually say, I think we can make it 50% better. And/or you can change that probability column in the spreadsheet to .5 instead of 1
Q: Some teams give the classic “we’re going to launch it to a small group of people and they’ll give us feedback, and we’ll react then. We don’t have to do a user test – we’ll just get live data.” Can calculating ROI help with this type of argument?
A: Yes. If you get live data and it shows a problem, now you have to go do re-work. So compare that ROI calculation to the one where you tested earlier and therefore had less re-work to do. Or maybe they are right. That’s why doing the calculation will be useful.
Q: I’ve sometimes, astoundingly, had people reject ROI as justification for our work. I use UIE’s $300M button story a lot (and Expedia’s $12M form field) as examples, then suggest they see only 1% of those returns and note that the ROI is still positive. No dice.
A: Yes, I have too. The question is always what is it that people are concerned about. Where is the risk.
Q: How can you get information on salaries?
A: Sometimes if you are working with the right people they will have that data, otherwise you can look up typical salaries for jobs with the Bureau of Labor Statistics if you are in the US.
Q: How do we convince our decision makers about the “type” of research we should do?
A: You could use ROI for that, but it probably goes beyond ROI. I do it terms of: If what you want to know is X then you need to do Y. For example, If what you want to know is whether our target audience assumptions are correct then you should do interview. If you want to know how people use our software now then you should do interviews and observations. If you want to know how people feel about shopping online at your site then you should do behavioral science interviews. If you want to know what the usability issues are that you don’t even know you have then you should do user testing. And so on.
Q: What was the most difficult ROI calculation that you’ve done?
A: Probably the first one with the bank!
Q: How would you approach a more traditional corporate structure, that may be a bit hard headed, to release the data needed to do an ROI spreadsheet? (Kind of like your initial example about the VP at the Bank.)
A: I would show them an example so they can see what it is I’m trying to do. Then even perhaps put in some estimates which they will roll their eyes at because the numbers are wrong. Then I can ask for their help and/or ask them which key indicators they are even interested in.
Q: I’m interested in trying an ROI calculator and I know it shows cost saving from dev time that isn’t taking in UX work as a consideration, but the dev time is still needed to implement UX findings and that cost can still be high/more than expected. How do you approach this situation when it happens?
A: Get help from someone who would know about the development time.
Q: I *have* had people respond well (again, strange to me in the business world) to altruistic motives, like “You’re dealing with cancer patients, and you have an opportunity to take one burden off of them.” And they do respond to this. Any thoughts on that paradox?
A: Business people can be warm-hearted too! The benefits don’t all have to be in dollars and cents.
Q: Are “expenses” and “investment” always equal in the ROI calculation?
A: In the ones I do, yes.
Q: How do you do ROI on non-purchase items? What about staff applications? Is it just time saved?
A: Time saved, or employees retained, or calls to the help desk avoided, or less training…