Erin May and JH Forster of User Interviews and hosts of the Awkward Silences podcast, discuss the value of high-value research participants.
Erin May and JH Forster of User Interviews and hosts of the Awkward Silences podcast, discuss the value of high-value research participants. Depending on where a product is in its lifecycle, an intentional pool of research participants can substantially impact the strategic direction of a product.
ALFONSO DE LA NUEZ:
Welcome to UXpeditious! A show that brings you quick, insightful interviews with design, product, and UX leaders.
DANA BISHOP:
In each interview we dive into how UX research impacts user insights; shaping the design and business strategy of some of our favorite tech tools and products.
ALFONSO:
I’m Alfonso de la Nuez, Chief Visionary Officer and Co-Founder of UserZoom.
DANA:
And I’m Dana Bishop, VP of Strategic Research Partners at UserZoom.
ALFONSO:
And we are your hosts. On today’s episode we're chatting with our friends from the Awkward Silences Podcast, Erin May and JH Forster. In addition to their host duties, they both work at User Interviews, a UX Research recruiting platform.
DANA:
In our conversation, we’re gonna dive into a classic UX Research topic: the importance of finding the right participants to support your UX research.
ALFONSO:
Thank you so much, Erin and John, for joining us today. It's a pleasure to talk to you about this mega super important topic, of course, which is finding participants for user research, probably one of the biggest pain points for everyone out there who's doing research, especially these days where more research is being done than ever before So I'd like to maybe get started, by asking you to first introduce yourselves.
ERIN MAY:
Sure. Hi, I'm Erin. Erin May. I lead Marketing and Growth at User Interviews.
JOHN-HENRY FORSTER:
I'm John-Henry Forster. Everyone just calls me JH. I’m Head of Product here at User Interviews, so I oversee product management and product design. And otherwise, just kind of cohost the Awkward Silences Podcast with Erin.
ALFONSO:
Which is a phenomenal podcast by the way. I'm a big fan.
ERIN:
Thank you.
ALFONSO:
So let's talk about the participant recruitment or finding participants for research, but, in particular, the right participants. So you guys at UI have built a company and a product dedicated to specifically that. There's a lot of panel companies out there.Can you guys talk about that?
JOHN-HENRY:
I think we've always thought about the quality of the participants as being a fundamental part of what we do, so I very much agree with that hypothesis. I think Roberta, who runs our research team internally, her and I have talked about this a fair amount, but if you have an unskilled facilitator or researcher or a product manager doing research and you're talking to a great participant, you can actually still get some good insights out of that. Whereas if you have an amazing researcher facilitating a session, but kind of a dud in terms of the person's not the right user or whatever else, it's actually very hard to get usable data out of that. So it's a really core piece of it.
And something we've been talking a lot about internally is you do research because you want to have impact, so you want to do impactful research that drives better decisions and helps business outcomes. And the two components of that to get really reductive is it needs to be quality research, you need useful insights, but it also needs to be timely. You need to have the research at the right time to impact the decision. And if you don't have the participant piece figured out, you can't really do either of those things. So if you don't have high quality participants, that's kind of a non-starter. And if it takes you forever to find high quality participants, then you're also kind of stuck because you don't have them when you need them.
ERIN:
Yeah, I think another dimension that's really important is access, so cost and speed. There are panels out there and there are old models of, you can find quality participants that match exactly the niche you're looking for, but it might take weeks, it might cost you a lot more than you want to spend. And so, when you talk about democratizing research and flexible research, mixed methods research, research that can iterate and evolve as your product evolves, as your audience evolves, you need something that's affordable and fast. And so, you want to look for solutions that can get you the participants you're looking for, but in a flexible, fast, and affordable way.
ALFONSO:
Absolutely agreed.
DANA:
So what would you say the business value of starting off speaking to that right participant is when conducting research? You started alluding to it, but sort of connecting a little bit more maybe to the business value.
JOHN-HENRY:
I think it's one of those things where if you're just not talking to the right people, it's really hard to make the right decisions. So myself, as a product management background and now heading up the product function here, I'm not a trained researcher. And I think, having talked to a lot of researchers, know my own limitations and where I'm maybe not as skilled as some others.
But in the early days, when we were building our own product and going out and talking to people, especially as we were trying to build our research hub product, which is a solution for managing your own users and keeping track of who is contacting who, we were able to find people who had that pain point so precisely in our user base, they're like, "We're managing spreadsheets, we're doing this," that even me as a non-skilled interviewer could go talk to them and really understand, "Okay, these are the parts of the workflow that are breaking for them and this is a need that they have in the organization." And so, then, we were able to go make better decisions and put prototypes in front of them and figure out how to approach it. And we would've maybe gotten 60% of it right if we had just kind of guessed at it and maybe done less research or not talked to the right people, so we would've been in the right ballpark.
I don't think some of our assumptions would've been all that far off, but it just wouldn't have been as good. And so, when you talk about impact, I do think it's hard to quantify in a precise way what was the benefit of those conversations and what was the impact of those decisions. But just from a qualitative sense, it is pretty easy to feel like we were closer to getting this right our first couple iterations because we had this signal and this feedback, but I know, Erin, you've thought about the impact angle a fair amount.
ERIN:
You can't please everyone all the time. And I think in the early days maybe you're trying to find product market fit. So obviously, that's your core mission in the early days, is to discover that. And you can't do that without figuring out who are the right participants to talk to so that we can have actually just a viable business?
Then, you start moving later in your business, and now we have different segments, different use cases. So you need to be talking to those segments deliberately to understand how do you grow your product to meet their evolving needs. So I think at every stage of the business, knowing who you're talking to and who you're solving for is critically important. Otherwise, you'll never be able to manage all the feedback coming in.
ALFONSO:
I think it's very important in order to make those decisions as product leaders like JH to have the right combination of the participants profiles for the right type of problem we're trying to solve. So the quality of the participant has a direct connection to that business value that Dana was asking in my point of view, as a C-level executive myself.
JOHN-HENRY:
Yeah. Totally.
ERIN:
And it's all about relevance, right? Quality with a participant, it's not a binary good, bad. It's really, are they relevant to what you're trying to figure out? And exactly as you were saying, that's the point of research, is we're trying to unlock new growth opportunities. We're trying to solve existing known problems, whatever it might be. But you have to be talking to participants who can give you relevant information for the opportunity at hand.
JOHN-HENRY:
Yeah. And I think the downside of not doing it well I think is actually pretty costly. And I think this is where you see some people who are really afraid of scaled or democratized research practices because I think they worry that it might be skewing towards the performative a little bit, of a product manager decided that, "Oh yeah, we're supposed to talk to users," and they don't talk to the right people and they kind of do it poorly. And then, that research doesn't have much impact. And then, now, internally there's a reputation that research doesn't do that much. And you get into a very bad cycle there.
DANA:
Yeah. And that's something we've been thinking about a lot is quality in the age of democratization. It's a great opportunity for more people. We've seen the demand. In the last three years, for the amount of research being run and the number of people running it, has gone through the roof. I'm curious about what opportunities and challenges that's presented in terms of participant recruitment, sort of this surge in remote research.
ERIN:
Yeah. Well, challenges and opportunities tend to come together as things scale.
DANA:
Yes. Exactly.
ERIN:
And everyone loves to talk about that with democratization. I know. It seems like we've moved to, I think, a stage where it was researchers really holding onto, "This is bad and I don't want this," to, "This is happening whether I want it or not. How do we make it work?" And so, one of the things, for example, that we think about when we think about democratization in participants is security, GDPR, PII, compliance, just sort of putting guardrails around budgets, incentives. How do you let an entire organization of 10 people, a hundred people, a thousand people, recruit participants at scale? That's why we've built our Research Hub solutions to help solve for some of that.
But there are tools that will help you, but tools always should follow process. How is it that we're going to centralize, whether it be through centralized screeners, a frequency of inviting participants? And I think, also, through sharing existing research and insights so that you aren't repeating research that's already been done. Don't waste participants' time if you've already talked to them and already have those insights. So those are some of the things that researchers are thinking about when it comes to participants and democratization.
JOHN-HENRY:
Yeah. I think another challenge that just comes to mind is, just as the demand goes up for research, how you're sourcing just gets harder. So if you're just doing some small batch research occasionally, you honestly can get by with some friends and family outreach and doing some of your own recruitment and stuff, and it's a little bit more work and you maybe get some bad participants that way, but it's manageable and it's conceivable that you could do it that way.
But if you're trying to do it across a decent sized team with a bunch of different people doing research for all sorts of different niches and needs and methodologies without a more scaled solution, you just can't keep up with it. Or you get into the habit, then, where you're only talking to the same people over and over. There can be good use cases there, but if you're only doing that, that's probably going to be detrimental at a certain point. And so I think that's just where it really... As the ball gets rolling on research, it gets hard to keep up with it.
ALFONSO:
I wanted to, maybe shift gears, but I see that JH is in product and Erin, you're in marketing. And I was wondering...
ERIN:
And we're still friends.
ALFONSO:
You're still friends. That's amazing.
JOHN-HENRY:
Usually.
ERIN:
Most of the time.
ALFONSO:
That's funny. Well, my thought was on how critical user research is for each of your areas. And I'm not just talking about UI, or user interviews, I'm thinking in general for the leaders in other industries, could be a healthcare vertical and you have product leaders and you have marketing leaders. How critical is user research for each of your teams? But again, not just at UI, but in general.
JOHN-HENRY:
Yeah. I'm obviously in a bubble of user research for the last couple years, so I'm a little biased. But I talk to lots of product folks, so I think I have a good perspective here, in the sense of... I think it does matter what life cycle you are in as a business.
So what you need to make good decisions is you need incoming data and you need some feedback loops so you can learn and iterate. And in the early days of a business, if you're trying to start something or you're not that large yet, you don't have much quant data, you're not going to get a lot of incoming signal through your customer support team because you don't have that many users yet. And so in that stage, the user research and the active discovery and usability testing and stuff is your only tool in some ways to really go figure out what problems need solving and how to continue to scale up your product market fit and build a great solution.
When you're later in the stages, you get a little bit more robust on how you can approach problems. So you are going to get a lot more passive signal from sales and customer support and all these channels. And if you're able to catalog that and look for trends, that's really powerful. If you are able, in certain parts of the experience, to start doing some AB testing or experimentation through other ways, that's another tool. But I don't think you ever get away from the research piece being still really fundamental in terms of just why someone is choosing to come to you or where they're getting stuck.
And there's just something different seeing drop off in a conversion funnel or something, people losing your... And then, you go talk to them and you can hear the frustration of, "I got this validation error on this form twice and I couldn't figure it out and I just gave up." And it just is very different, so I think it also then starts to play, in addition to making good decisions and helping what you're building, it also becomes motivating and inspiring for the team to have that proximity and hear it first hand.
ALFONSO:
I do have the feeling that there's a need for education and training around the importance of great design, great UX, and certainly great research, among product managers. What are your thoughts on that?
JOHN-HENRY:
I think that's come a long way over the last couple of years. I think one of the most recommended books I've seen within product circles lately has been Teresa Torres' Continuous Discovery Habits, which really talks about that feedback loop and trying to talk to somebody weekly and instilling some of those habits I do think though what is helpful with product management and design and research and stuff is where you can find that healthy tension and that thoughtful collaboration, I think if you have that healthy trust and dynamic, then that's the best case. Right? Cause then you're going to kind of compromise and figure it out.
But absent that, I do think there's just been a good kind of awakening within product management that you can have more impact if you understand your user base better. And I think, just almost by default, that doing user research, whatever kind of methodology you might be deploying, is one of the best methods for doing that, probably the best job. So I don't know. I haven't spoke to many product folks recently that don't have it as part of their team's system to some degree. I think that you see varying levels of how committed they are to it, how much they do it themselves, how much they have a researcher do it for them.
ALFONSO:
I've also heard much more of leaning towards quant analytics and surveys and not as much focus on the qual, which would be a moderated study, or even unmoderated, with think out loud. That's kind of the experience I've had. And I think if you can convince product managers that in order to design great experiences you need to have a multi-method approach and not just make decisions based on quant, I think we're making progress.
JOHN-HENRY:
I think, within the PM community at large, the quant side is a little bit more embedded for sure. I think you see larger companies, kind of like an MBA, track into product management and that's coming from a lot of quant side methodologies and skills. The data tools just have gotten ahead of a lot of the research tools, so there's been kind of a fast revolution there and it's been something that's been pushed on for longer. So there's more self-service data tools that are pretty easy to use, mixpanel amplitude, other things, right? So I do think those are a little bit more woven into the toolkit and maybe there's a bias in that direction, but I've been really encouraged with how people have been picking up on the qual side
ALFONSO:
Absolutely. Erin, what about on your end in marketing and growth?
ERIN:
At User Interviews, our Growth Team sort of sits at the intersection of product and marketing. So Growth runs like a product team in that we have engineers, designers, data analysts, but focused on creating and accelerating loops within the product itself. And then, marketing does marketing things. And I think, what's a team in an organization that can't benefit from user insights? So I'm sure you all talk about, the analytics will tell you what, the qual will tell you why. And I think there's always this pendulum switch sort of back and forth in marketing of we're going to be direct response and totally quant and that's how we're going to run things, and then it's, well, actually, when the cookie apocalypse happens and we can't track everything anymore, maybe that's not going to work as well. Maybe we should get back to some first principles thinking and understand our users and what makes them tick and their motivations and why they might to us in the first place.
And I think we're very much in that kind of moment now, not instead of quantitative, but we can't forget about that stuff and do good marketing. And so, it's hard to imagine doing that well without good research . What we do a lot at User Interviews and what I see is you centralize the research being done and then the entire organization can take advantage of it. Who in the organization doesn't need to know who is our users? What are their pain points? What are their motivations? How did they discover us? Why did they come to us? And how can we fulfill that promise?
It's very important that product and marketing be aligned on what those things are so that you can deliver on that promise across every touchpoint within and outside the product. So I'm seeing a lot more in marketing communities of people saying, "If you want to get to know your customers, you could talk to them." And so, I'm seeing a lot more of that on teaching marketers how to actually go out and do that themselves or to take advantage of that if it might already be happening in the organization.
DANA:
I have heard more marketing teams talking about, coming right from the top down in the organization, the culture of we need to be talking to our customers all the time.
ERIN:
Yeah.
DANA:
We can't ever get too far away from them and we need to keep the dialogue and the understanding of their wants and needs and experience close. Right?
ERIN:
Yeah.
ALFONSO:
Exactly. Another topic that I like very much about the business value, once again, Dana, you brought it up, of research is innovation. If you actually talk to people, talk to real users, you might get some really interesting ideas of how to change some of these products, whether it's information architecture or whether it's how you position it.
ERIN:
Yeah. Talk about stepwise growth and not incremental growth. That's what innovation's all about. And I think creativity comes from disrupting your day to day, from getting out of your own head. And what better way to do that than to talk to users? Because you never know what's going to click and kind of unlock that next big idea that it's too early to validate, there isn't a market for it yet. Right? It's the faster horse problem. So I think yeah, absolutely. Talking to users is a great way to innovate.
JOHN-HENRY:
This gets back to when we were talking earlier about the participant quality piece, I think, because maybe it's worth doubling down there a little. Quality is very contextual. And so I think maybe people are like, "What's a quality participant?" And somebody will just think, "I don't know. Somebody who's very articulate or something." Maybe the obvious thing you might go to. But if you're trying to do… Like imagine at that bank, you have two different teams. One team is trying to optimize some deep setting page of how you manage your account or something and another team is trying to find some new big business ideas. You're not going to want to talk to the same people for that.
The quality participant for those different things is going to be highly differentiated because for the account settings page, you need power users who are in there all the time and making sure that you're not going to break something that's an important use case to them and it's going to be very different. Whereas when you're trying to find an unlock, big growth idea, you're going to want to probably talk to people who maybe are not your customers, but sort of look like your customers, but for some reason, they don't use you and they're doing something else. And maybe in that space, you can kind of understand, "Oh, they have this other pain point, this other need." And those are both quality participants, but if you mixed them, they wouldn't be. Right? And so you need to be really intentional there, in addition to a bunch of other factors. But I thought that was a good example as you were getting into the innovation side.
DANA:
That's a great point. And I think people get very focused on the idea of what the right participants are. But to your point, and that was a great example of the right participants for very different kinds of research, one about improving the user experience of existing experience for customers, and the other one really about innovating. And design teams sometimes get locked into, or product teams, into just improving the current experience without thinking outside of that box or looking at what others in the market are doing or how they should be expanding.
ALFONSO:
So just would like to take the opportunity to thank you very much once again for joining us. It’s been super interesting to hear you today talk about the quality of participants, the importance of quality when finding participants for research.
ERIN:
Yeah, thanks for having us. This has been great.
JOHN-HENRY:
Cool. Yeah, this was fun. Thanks for having us.
ALFONSO:
That was JH Forster and Erin May, hosts of the Awkward Silences podcast and SVP’s of Product and Marketing at User Interviews.
You already know at UserZoom we thrive on collecting user insights to improve digital experiences, and the same is true for this podcast. We would love to hear from you, our listeners, on how we can improve the show. Please follow the link in the episode description and take our quick survey. We’d love your feedback, as well as any recommendations for future guests or topics.
DANA:
Thanks for listening to UXpeditious. Make sure to continue listening to our new episodes each week for quality insights from UX industry leaders. If you like what you heard, help us out by rating and reviewing the show on your favorite podcast platform.