UX LIVE 2019: Lee Duddell, Senior UX Director, on ‘Democratizing UX’

Our Senior UX Director Lee Duddell discusses the importance of democratizing UX, while highlighting the mistakes that people make when they begin sharing research empowerment around the organisation and how they can avoid them.

The following video was taken at UX LIVE 2019, and we’d like to give them a huge thank you for inviting us to talk at their fantastic conference. A transcript of the talk follows the video below.

Lee: So we’re gonna talk about democratizing research. There are structures. I’m gonna describe why I think it’s a good idea to democratize, I’ll cover some of the mistakes that people often make when they start democratizing, and we’ll talk about how to avoid them, and there’ll be a Q&A as well.

Before we start, though, just a quick show of hands… Well, actually, even before quick show of hands, let’s just define what we mean by democratizing research. Democratizing research means enabling people who are not researchers to run tactical design tests. How many people are already doing this in their organizations? Show of hands? A small percentage. Okay, so you’re already doing it, fantastic. How many people are thinking of doing it? A few more. Okay, cool. And then I guess other people are here because perhaps you think it was another room, another stream, or because you want to maybe weigh up the benefits and cons of it before going ahead.

Anyway, please listen to me if you can. I know you’re in here and you’ve got social media and a number of other distractions. If you’re a researcher, please listen to me, because I’ll explain the reasons why I think it’s a good idea to democratize testing. Note I’m now using the word testing rather than research. And I’ll explain how many other companies have done it. And if you’re a mere mortal, so you’re a product owner or designer… Is that patronizing? It might be. I don’t mean it to be patronizing. But if you’re not a researcher, and you’ve not done research and testing before yourself, then you’ll learn why it might be good idea for you to do this, and to encourage your researchers to support you to do that, and some of the common mistakes you’ll need to avoid when you start doing it.

Two other reasons to listen to me. I have the experience of doing this. The team that I work in has enabled democratization of testing at over 500 organizations around the world. And I timed this to finish in about 20, 25 minutes, depending on questions. It might mean that we’re first to the food. Then I reminded myself this morning, after I sent my slide in, what conference food is often like. So, but it’s been sponsored by Balsamiq, so I’m sure it’ll be amazing. Anyway, we might get out of here early.

So we know what democratizing testing is, some of us are doing it, some of us are thinking about it. But why would you do it? Why democratize? Democracy, by itself, has been a bit of a letdown recently, I’ll be honest. I don’t know if you feel the same. But if you think about this guy and things like Brexit, it’s been horrible. But there were really good reasons to democratize testing. Three of the main ones that I hear from our customers. The first one is that it reduces risk of launching bad product. The second is that it ends debates within product teams. And the third one is it increases the influence of researchers in the organization. The third one is the main reason why researchers do it and make it happen in their organizations.

Let’s explore each of these. Lots of people are working in agile now. That means the product is getting shipped faster and more frequently than ever before. And there’s not that many teams out there. Agile teams that are out there saying, “Right, we’ve built this, it’s ready to ship, but it hasn’t been through enough rigorous testing with users, so we’ll just park it until the researchers are ready to run a lab session to see if it’s usable.” It doesn’t happen. What happens, unfortunately, is stuff gets shipped. It goes, it goes without testing because we can’t necessarily, as researchers, keep up with the demand for that type of testing.

A quote from Forrester, if you need to quote it to senior people, they listen to the likes of Forrester and Gartner. What actually happens is that, “Agile has meant that things get shipped possibly with even less testing than there ever was before.” We need to address that as researchers. That’s not great. Perhaps you have enough researchers in your organization to run all of this testing. Most people don’t. We found in a recent…in our State of UX in the Enterprise survey report that we ran in 2019 that now, which seems to support this, now the biggest challenge that researchers face is, yes, we still need to source the right participants, secure budget. In 2019, this is a big change from previous years, inclusion within the product development life cycle is kind of the biggest challenge. A lot happens. It doesn’t get tested, because it’s happening so much faster and at more volumes than before. So that’s reason number one why I think, and why our customers tell me, that democratizing testing is important. It’s to be able to satisfy demand and to reduce the business risk of launching product that’s untested.

The second reason, this is one that’s real relevant for mortals, is that within teams it enables endless tiresome energy sapping debate about version A, version B, variant A, variant B to be ended. Because if you test within sprints, you don’t need to discuss whether A is better than B or listen to somebody who’s, “I’ve been working here for 15 years. I know what users want.” You can actually put it out to end users, get the results in, and focus on what’s important. And, for designers, a lot of them who started running their own design tests, they say that’s really important and kind of quite liberating for them.

But the main reason, certainly, that researchers tell me, to democratize tactical design testing is so that they can run studies and research that’s more influential and more strategic, and thus have greater value to the business. They are spending more time on discovery-stage research, driving product decisions rather than reacting to them, than ever before, if they put tactical design testing in the hands of other people. And they can run more strategic types of research, things like measuring UX and benchmarking against competitors, that gives them greater influence and, to coin a phrase, a seat at the top table. This is the main reason why researchers democratize testing.

So how do we do this? Well, what you don’t want to do is give your mortals access to a platform and say, “Off you go. Start running your own tests. It’s really easy.” Because what will happen? Well, let’s have a look at what would happen if you do that. Let’s start with a persona. Let’s talk about Matt. So he’s somebody, he’s a designer, a UX designer who wants to test, because he knows that it will reduce the amount of debate and discussion in his product team. A little bit about him, he doesn’t wear socks, even this time of year. I mean, luckily, there’s bright lights so I can’t pick on anyone. Yeah, we’re okay in the front row, we’re all right on the sock front, but there will definitely be people here wearing ankle-length socks this time of year.

That’s Matt. This is just to help us characterize him. He loves his craft beer, he spends a fortune on those kind of beard grooming products as well. That’s Matt. That’s our Matt persona. But, the most important thing, he’s never run any of his research before. He’s always gone to the researcher, maybe with some research questions or some challenges, or just like, “I’d like to get some feedback.” He’s never done it before, but suddenly we’re gonna let him do some research.

And, at the moment, the sprint he’s working on in the moment is about improving the voucher code, it doesn’t matter too much what it is, but voucher code journey for, like, an online food ordering service. So what does he do? He’s just been given a login to UserZoom. So he logs in, and he’s like, “Right, I’m gonna start doing some tests on what I’m working on.” Obviously, there are other platforms available other than UserZoom. So the first thing he does is he builds a screener and a screener question. So the screener question is gonna go to the panel to get invited, to make sure he’s got the right people.

The screener question is, “Do you like sushi? Yes or no?” And you’re gonna have to shout. Any thoughts on that as a screener? Any from anybody. Is that like a great screener, or is it…? Put your hand up if you think it’s a great screener. Put your hand up if you think it’s a bit rubbish. Okay. It’s a bit rubbish, isn’t it? Because it’s quite obvious, it’s almost too obvious what the right answer is. So somebody who might be motivated to participate by money is probably gonna go, “I’ll choose yes.” So that’s kind of his first mistake, but, you know, it’s probably not the end of the world.

He then goes on to design the rest of his study. And in this example here, he’s uploaded an image of something he’s working on, the design version that he’s working on for the voucher code system. And he sets a task for people, which is, “Where would you enter the voucher code?” Yeah, and then people would click on that flat image to say where they’d enter the voucher code, to indicate where they’d do that. And this is the first screen that the participant sees. So they might’ve got, “Do you like sushi? Yes or no?”, and then they see this. Anybody got any thoughts? You have to shout out, put your hands up and then shout out, about whether that’s a good thing or a bad thing. Any thoughts on this? Not many. Okay. I’m gonna assume that nobody really wants to give feedback on this.

Basically, there’s a bit of a problem with this, isn’t there? Because if you’re a participant, and you’ve just kind of got an email, and you’ve just logged in to, say, UserZoom take part, and then suddenly it goes, “Where would you enter the voucher code?” What are we missing? We’re missing kind of a scenario here, aren’t we? And I’m not making up these tragedies of testing. This is what we’ve observed when some people just jump on and start running their own studies for the first time.

Somebody does an internal test and says, “Hey, before you launch this out, you really need to set up a scenario.” So he’s like, “Understood. Yeah. It seems I’ve got to set up a scenario, just get them in, give them a little bit of context so they know what they’re doing.” So he sets one up, and this is his scenario. It’ll take a while to read it, I think, but this is it. I’m not gonna read the whole thing out, but, “It’s Friday morning and you’re at work in a swanky office in Soho.” It doesn’t say it’s raining outside, it could easily say that. “The team you’re in, they’ve been working very hard and you want to reward them. They’re…,” and it goes on and on. And this is a common mistake. Sometimes people don’t put a scenario in, sometimes they just create “War and Peace,” basically. And this is confusing for a participant who’s come in. And, actually, the ultimate study is just to see whether they can find a voucher code box in a checkout area.

Okay. He’s moved on, he’s now decided he wants to compare one design version versus another, because he got that thing. I mean, perhaps he’s here now, perhaps he listened to me and thought, “Yeah, ending debate is really good, and we spent a lot of time deciding between version A and version B. So what I’m gonna do is I’m gonna put version A and version B into UserZoom and say to participants, ‘Which do you prefer?’ And then we’ll see which one comes out top, and that’s the one we’ll go with.” Some people are laughing. I’m gonna take that as feedback that maybe this isn’t the best approach.

So he’s gone a bit literal on it, really, hasn’t he? He’s just gone, “Hey, do you prefer this or that?” And while, on the face of it, I admire that he’s, you know, attempting to remove debate. This approach is not great, is it? This isn’t A/B-testing, really. It’s like a bit of opinion about some designs or about what’s on the screen. Yeah, it’s not even a valid A/B test. We can see that. So that’s almost like literal translation of a business problem into a task for participants.

Here’s the fifth and final one. So he’s now progressed the design, he got those studies out, and now his next iteration, his next iterative test is on an InVision prototype that’s starting to stitch together a little bit of the journey. So he sets people a task, which I can’t read because I didn’t bring glasses. And the task is, “You want to order some sushi for the office.” And, in this type of study, people are gonna think out loud and give their spoken thoughts as they kind of navigate through. The task he sets people is, “You want to order some sushi for the office.” Now, on the face of it, you might say, “Sounds all right. That is kind of what they want to do. You want them to interact.”

I mean, it’s a bit of a nuance, but it’s not really. If you’ve done this type of testing, there’s not really a task. It’s just saying what you want to do, “You want to do this. You’ll do that. You’ll do this.” It’s not, “Order some sushi for the office. Stop when you get to the payment page,” which is what he should write. So these are five of the most common mistakes that we find when newbies jump on and start running their own design tests. It’s a good sign when cameras go up, isn’t it? Sorry, cameras, nobody knows what that is. When phones go up. I think easy-to-guess screeners, there’s no real scenario, just jump them right into the study. Or there’s too much scenario that’s really confusing at the beginning, unnecessarily. Asking about design, asking opinion, opinion, opinion rather than observing behavior, and writing tasks that aren’t really tasks that don’t really ask people to do a thing, basically.

So how do we help Matt? So if we’re gonna democratize testing so that we, as researchers, can be more influential and run more strategic type studies and basically have a greater impact on the business, how do we help him? Well, some of the things that we’ve done and some of our clients done are, unsurprisingly, to train people to develop internal case studies and to provide a guardrail. Let’s look at each of those.

This is like an agenda from one of our training workshops. But, at UserZoom, we have a team, it’s a team that I work in, who train end-users in person to run their own studies on the platform, and end-users who are not researchers. And how do we do that? Well, the first thing we talk about is an internal case study, I will talk about that in a moment, so we’ll skip that. We talk them through the testing process. And we spend a lot of time, almost as much time talking about research questions and the type of things you should do, than we do, actually, on the platform doing stuff. We help them identify what is a valid tactical design question that they should run themselves, should expect to run themselves. We don’t expect them to run discovery-stage research, for example.

We map those research questions to types of studies. And then, in the workshop, we have them built, we have them build and launch a couple of studies. Here’s an example of how we map questions to a type of study, I won’t go into too much detail here, but the idea is that you just help people frame what they can and can’t do on the platform, the worst thing. So, earlier, when we were looking at the mistakes that Matt was making, the big one he could have made was sending a study out to 10 people and saying, “What should we do to improve our product?” That’d be a huge mistake. And, of course, we spent some time working with them on the results end of things.

So once we trained him on the types of studies that he should be running, the types of tactical design question he should be answering, we then aim to humiliate him. Why do we do that? Well, one of the hardest things, if you’re new to research and testing, is framing tasks, particularly tasks, less so questions, but particularly tasks, in a way that makes sense to participants, and is not leading, and is unbiased, and that’s really hard. And those of us that have worked in research, it kind of comes quite naturally to us.

So the best way we found for Matt and his colleagues to learn how to do this is, the first thing, is to allow them to build a study. So I have this on UserZoom, other platforms are available. They build a study and then share it with their colleagues in the session, at which point we then step through it and say, “Right. Question number one, you’ve asked this. Task two, that’s not a task, that doesn’t mean anything. Your scenario is too long and confusing.” By allowing Matt and his colleagues to make some mistakes, and then pointing them, out and sharing them, that starts them on the road to writing great tasks and questions. And we provide them with a checklist moving forward as well, but it’s really important in the training session.

I’d say the two most important things, number one, describe, discuss, and agree with what the types of studies are that should be run. Secondly, build, and critique, and really focus in on getting those tasks right. Even if you’re not using the UserZoom platform, there is a freely available resource called UserZoom Academy that’s short videos that is designed primarily…it’s designed for researchers as well, but is useful for mortals, so non-researchers, to learn about running testing. So that training is really important for Matt.

The second thing, I mentioned this in the…it’s something we referred to at the beginning of the training. So let’s say you’re convinced you want to democratize testing. The first thing to do, unsurprisingly, is not to roll it out to the entire organization, and try and train everyone in one go, and try and get everybody enthused in one go, because it just won’t work, that’ll fall flat. What’s a good idea is to pick on one team, maybe one product team that you already know, but with the right kind of people that are open to this idea, that maybe are already coming to you with lots of research questions that are of the right nature, and who are likely to be advocates, and run a series of studies with them that initially may be used to help them build. But then they build the next few, and you use that as your internal case study. So one of them comes at the beginning of every training session and says, “Hey, this is what we’ve been doing, this is why it’s a benefit, and it’s straightforward, and these are some of our common pitfalls.”

It’s crucial that you do that. Trying to roll out to everybody in one go is difficult and hard to manage. The other important thing is to set up a guardrail, so I’ve kind of talked about this a little bit already. And the guardrail may manifest itself in kind of limiting Matt and his colleagues to maybe one or two types of study. Don’t promise that they can maybe test every question that they have, all the time, from now, and it will be straightforward, and they won’t make any mistakes. Say, “Right, let’s just start with,” so what we looked at earlier, “like a click test,” or something like that.

The second thing is, on the screeners front, in our platform and, I think, in others as well, you can set up approved screeners for people to use. So people like Matt don’t have to think about, “How do I construct a screener that’s not gonna introduce too much bias? Can you get me the right people, and it’s gonna make sense to participants?” No, that should be in there already for him to draw down on. And you can use study templates and task templates to ensure that he doesn’t have to be too creative, because probably the key tasks, key tests that he’ll want to run are maybe just a few of your kind of top tasks, the common things that people want to do on your site.

So three core things to remember. Writing tasks is really, really, really, really hard. Sorry, writing tasks is really easy. You just log in, you write the task, and launch it. Writing decent tasks is really hard for people that haven’t done that before, and people need help to do that. And the help, I think, should be in the form of a training workshop where you build with people that you want to enable. And it’s a good idea to start with maybe just one type of study or one type of research question, that’s focused on design, that your teams should aim to answer, and then build up from there. Try and start with one team as well, and use that as like an internal case study. You’ll also learn from doing that.

I promised I wouldn’t be too long. We’re nearly at 25. I think I estimated 20 minutes. We’re at 23. So I just want to, before I go, I’ve got a slide that I must remember to talk about, which is like a big call to action. I just want to talk about getting to sleep. So I was talking to one of our customers who’s like the senior VP of design of a very large global consumer brand, and we were talking about this because they democratize testing, been doing it for a number of years. And I said to him, “What about quality?” I said, “Don’t you worry about this?” He said, “You know, when my head hits the pillow at night, my final thought as I go to sleep is that maybe some of those tests aren’t gonna win any awards, but the impact my research team are having outweighs it a hundredfold.”

I don’t think about research when I go to sleep, I must admit. But I think that’s an interesting perspective on why it might be a good idea to consider democratizing testing. We’re gonna have some questions in a minute. There’s a crucial call to action. We have a stand, and there’s a game on there, and you can win stuff, and the stuff is t-shirts, stickers, and Google Home thing as well, so some wicked prizes on there. If you get a chance, it’s a bit of a laugh, go and visit. Maybe we’ve got some questions or some thoughts.

Host: Great. Thank you very much. So, yeah, round of applause, please. Thank you very much as well. I always ask the questions. Do we applaud now or later? I’m just buying you a little bit of time to think of your questions, which we’ve got plenty of time for. Lee, thank you very much for that. It’s a thought-provoking thing to talk about democratizing design, democratizing research, and then you put this filter on it saying democratizing testing. And, actually, it’s really nice and clear to show a very simple value there which is allowing the research team to elevate the level at which they’re working, which is what they’re all crying out for. So thank you very much for that perspective.

But, for some of you, that might be…maybe you’re already doing it. And if you’re not, that might be a little bit scary if that’s the first time you’ve had to try to do that. Has anybody got questions for Lee and his experience observing a number of different teams going through this process as a platform provider? Any questions? Put your hand up, I’ll come to you, and then put your hand up again and I’ll find you after the question. If you could just share your name and what your question is. Thank you.

Alana: Sure. My name is Alana. And thank you so much for the talk, it was brilliant. And you told us crucial things about what went well. So in the process of democratizing research, what doesn’t work so well?

Lee: So what doesn’t work so well when you democratize? Okay. Self-serving. Okay. So self-serving means two things. I was gonna do a wordplay on it during the presentation. Thank you for the opportunity to try it out. But UserZoom is a self-serve platform, but sometimes people run studies that are self-serving, so they use studies for them to make a point. That’s quite hard, that can be quite hard to manage. It tends to not happen too frequently, and I think it doesn’t happen when you’ve got product owners on-site. So sometimes you might see, potentially it’s…you know, it’s unfair to say just a designer. But, you know, maybe a designer runs a study that’s like, the research question is, “Just how good is my design?”, type thing. So it’s self serving.

And if you’ve got product owners on board, which you kind of you need to have on board, they’ll usually do a good job at calling that out. That can go wrong. I think the point I made as well about trying to just democratize to everyone straight out, straight off. Unless you’ve got decent research ops set up, then that can be quite hard, I think. The main thing is to learn, start with one team, and move it out from there as well.

Host: Okay. Thank you for that question. Great question. Any more questions? Yes. Again, if you could share your name and your question.

Katelyn: Hi, I’m Katelyn. Thank you so much, I really enjoyed your talk. I’m not really familiar with your product, but my question would be more like, “Do you use this platform also to test more different types of design research, or use that to democratize other kinds of design research, other than testing specific flows of interaction patterns?”

Lee: Yeah, there’s a bunch of stuff. So, if you remember, there was that pyramid that I showed of measure, discover, and tactical-design testing at the bottom. UserZoom is there for all of those. There’s loads of types of studies you can run. I just kind of showed a couple in here throughout the product development life cycle.

Man: Great. Thank you.

Jerry: Hi. I’m Jerry. One other thing that we do use UserZoom in the business. However, when it became global, it wasn’t so easy on the other side of the world when users have different behavior. So how do you do that? How do you manage that?

Lee: Are you saying that you want to test globally? Is that…?

Jerry: Yeah, which we tried. However, APac, the behavior of user don’t react the same, so we end up having to do moderated, so it became a completely different behavior. So we run the test the first time, and we were shocked to realize it came back, and it’s different behavior from the side of the world.

Lee: Right. Okay. So I’m slightly unclear. Do you mean that there was different behavior in the way that they used UserZoom rather? Okay, fine. When was this?

Jerry: In Japan and China.

Lee: When was it, when did you do it?

Jerry: Last year.

Lee: Last year, so in 2018? Okay, cool. I mean, we run a lot of international studies now, more than we ever have, and there have been improvements to the way that we do that. So it might be worth revisiting that now, because we’ve made some significant improvements to the interface, including the participant experience. But we can chat about that after, at the stand, if you want. Cool.

Man: Time for one more question, maybe two, it depends on your question. If you could introduce yourself and your question.

John: Hi. Thank you for that. My name is John. One of the challenges we’re having in this sort of process is that we’re finding it quite difficult for PMs, and devs, and design teams to give up their time in order to sort of take part. So, for example, when we’re sort of having kind of UX interview viewing parties, which take an hour or two, one of the challenges we’re facing is that people just don’t want to give up their time to attend one of them. So my question really is, “How do we fit this process in and get to the point where we can allow people to design their own studies for teams that are challenged with lack of time in our own role as a designer or a PM?”

Lee: Sure. Totally. And every team is challenged with time, aren’t they? I mean, nobody has that luxury. I guess there’s a few things, a few techniques. What I didn’t show you is that, in the actual training workshop itself, there’s a part of it where we just focus on driving home the benefits of running testing and how it can actually do a couple of things. It can actually speed you up, because of that, what I talked about, the endless debate. But a crucial point, particularly for product owners, but anybody working product team, is it reduces the likelihood you’re gonna have to rework, you know. Which, actually, if you take a kind of longer-term view on things, is one of the main reasons why you should do this, because it reduces the amount of rework that happens as well.

The internal case study would help as well. So if, you know, one of the teams is already doing this and can say, “Look, we have to dedicate this amount of time, but we’re actually reworking a lot less, the product KPIs are going up, we’re not debating stuff as much.” Maybe that demonstrates how time can come back into those teams.

Host: I think we do have time for one more question, which was you, sir.

Christian: Hi. My name is Christian. I was wondering, when you set up the mortals… What did you say, the other people that could set up the…?

Lee: Mortals.

Christian: So they set up the test, but do you also leave them with interpreting it or do you need the designers come back in? Because, A/B testing, I think it’s quite easy because you have percentages. But sometimes, in a walkthrough, you just need to think about a design solution then.

Lee: Yes. So our general focus in the workshop itself is to get people setting up a test. It’s one thing at a time, really. You know what I mean? So getting to understand why effective task-writing is important. But then when they actually get to launch a study, we don’t need them to interpret unless it’s a very straightforward study. Yeah. So particularly around think-out-loud testing, when we run that, it’s important that, you know, people don’t get a bit too literal, or more people say, “For instance,…” So we typically provide some separate training session about that.

But the first training session is get them excited, allow them to see the benefits, hear from another team that’s already doing it. Ideally, if they are, get them hands on and let’s just get the tasks making sense first, and then a separate one on analysis. If you’ve got researcher capacity, spending time on analysis with them is a great idea, yeah, a co-analysis session. And the team at UserZoom could do that too.

Host: Great. Thank you very much. Thank you very much for all your questions. It really helps. Everybody is learning here, and gets us deeper into Lee’s experience. So, Lee, thank you very much for joining us and sharing the democratization testing and the value that can add for elevating your game and research. Thank you very much. One more round of applause for Lee.