How to Recruit for UX Research & Usability Testing – Webinar Q&A

Leah Kaufman answers your recruiting questions.

We recently had a webinar with Leah Kaufman of Lenovo to discuss how to recruit participants for user research and usability testing. During the webinar people had a lot of great questions. Given the popularity of the topic, and the frequency with which we hear these questions being asked, we shared your questions with Leah and transcribed her answers in order to share them with our readers. Enjoy!

Q: How do you avoid competitors receiving a user study that might give away product direction? Competitors might see your survey on social media, or be a part of your customer email database if they’ve bought your product.

A: Using a vendor might help as there will be a lower likelihood of a competitor being in their subject pool, and hence, your study. I don’t think it’s possible to do a large scale study with your customer database and avoid this possibility. If it’s a small-scale study you may be able to use LinkedIn to check the participants professional associations and identify anyone who works for a competitor.

Q: Do you tend to use different recruiting methods for quantitative vs. qualitative studies?

Quantitative studies typically need a lot of bodies; this means that you need to find big pools of people to recruit from. It also means you may have looser recruiting/screening criteria so that more people are eligible for your study. With quantitative studies I am more likely to use a vendor or company sources (sales, marketing, CRM, Eloqua) for recruiting so I can get the volume I’m aiming for. Also, the more people I need, the more time I allow for recruiting.

The qualitative studies that I run typically have 8-12 participants per profile/segment and up to a max of 30-40 people per profile/segment. With these smaller numbers I can afford to be a lot more strict when it comes to screening.

To me that’s the main difference. Otherwise, the steps are still the same: Who do we need to see in the test? What are the profile(s)? How many? Where can we find them? How much time and money do we have for recruiting? Who’s going to manage recruiting and, if needed, scheduling?

Q: What would be an appropriate incentive for a 45 min live interview call with a B2B customer?

For incentives I would recommend either a high value Amazon gift card or a donation to a charity/organization they belong to. Keep in mind, however, that an incentive may not always be necessary. You can always try to appeal to their vanity: “You are an expert on this; I think your ideas on [X] could change how my company does [Y].” Giving them respect and showing that what they tell you is valuable and will have a direct impact can help get someone from ‘maybe’ to ‘yes’.

I’d like to reiterate that it’s important to put your focus on recruiting rather than the incentive. You know that B2B people are hard to get for longer interviews. Your best bet then is to get a handful to agree and ask them to recommend others you should contact. This means you can send an email to these contacts saying “Your colleague Joan Smith recommended that I talk to you.” Always use your own email for these invites and mail individual invitations rather than BCCing a large group; the invitation should come from you as an individual, asking another individual for their time and input.

So, if I was doing this I would try the following (depending on time of course):

1) Talk to the organizer of a local Meetup group for this type of professional; ask if you can post something to their site, or even better, get a few minutes to introduce yourself at their next meeting.

2) Find an online forum or community where they exchange ideas and post a single interesting question related to your project and get a discussion going. Follow up with those who join the discussion and ask if they have the time and are willing to talk to you in more detail.

3) Use LinkedIn to contact and recruit people. Be sure to include a detail you found in their LinkedIn profile in your email – this will make you someone they want to talk with. Make the invitation a personal one from you to them.

Q: Have you found one method of user research to be more effective than another?

I like having a ‘toolbox’ of UX research methods that I can choose from depending on the study, goal, type of participants and resources (time and money). The goal is to pick the research method that is most effective for the specific project.

It’s like picking the right shoes for the right occasion; you have hiking shoes for when you’ll be on the trail, sneakers for the gym, dress shoes for a night out, casual shoes for the weekend and something better for the office. Now, you probably don’t wear Birkenstocks to the opera 🙂 Likewise, you don’t use eye-tracking to interview SMBs about their business needs. No specific test method is objectively more effective – it’s more effective relative to the specific study and what you want to learn.

Q: How do you frame to stakeholders that screeners are necessary?

Ask them if they would hire someone without seeing their application or resume. When you hire someone for a job, they fill out an application and hand in their resume so you can see if they have the right background and experience for the position. The screener has the same function for a usability test – not all people are equally qualified to give you feedback you can use. If your product is intended for stay-at-home moms, then construction workers aren’t the best people to to give you feedback. Stay-at-home moms are. Ask your stakeholders which data set they would trust more.

If they believe that everyone/anyone should be able to use your site or product, that’s a different discussion. You need to have a deeper dive into whether this is accurate. It is possible that this is the case and that the the goal of your studies will be to identify ANY issues, not issues specific to a particular group.

Q: Wouldn’t spam filters choke on the $20 in the subject line?

Good question. When I was doing this type of recruiting two years ago, it wasn’t a big issue. You may have to simply rephrase your subject line to “Get a cool gift card.”

Keep in mind that if you use Amazon gift cards you aren’t allowed to put ‘Amazon’ in the subject line because this implies that Amazon endorses the study or company running the study.

Q: Pop-ups are generally hated by people (mostly because they block tasks users are trying to do). Do you have some best practices or tips if you must/want to recruit users from your website/app with pop-ups?

Consider timing and placement of your invitation layers and whether your site already goes overboard with pop-ups. Ideally your site doesn’t use pop-ups or only uses one and you can temporarily swap out the standard one for a recruiting one.

The main trick is to try to do the pop-up in the place that makes the most sense to recruit. If you want to improve your search or search results, pop-up the invite on the side of a search results page. “Tell us what you think about this page and get a gift card” or “Got 5 minutes? Answer 3 quick questions and get a 25%  off promo code.” Also consider a banner in the place you want to recruit. “We’re re-doing this page – click here to be part of the redesign.”

The bottom line is that yes, pop-ups are annoying but given the large volume of visitors to a site they can work well for getting people into a screener and a subsequent study.

Q: Do you have any recommendations on how to recruit the same participants several times, such as for an ongoing benchmark study?

Let them know upfront that you are looking for people who can do several sessions over a given period time. Let them know the payments are broken up over time and that there’s an added bonus for completing the entire set. Recruit more people than you will actually need, at least 20-30% more, because there will be those who drop out even with the staggered incentive.

You should also follow up with a ‘thank you’ to everyone who participates, stay in touch with these participants in between sessions, and even send then unexpected small gifts or treats in between sessions to reinforce your relationship with them. If someone does drops out, ask if they have any feedback about the survey/project  in order to learn what you can from them to ensure that others don’t drop out, too.

Q: Can you make a recommendation if we’re recruiting customers but aren’t getting the response rate we need and the sprint is almost up? 

Three suggestions:

1) Look at your profile/screener – are you losing possible participants because you’re profile is too narrow and excluding people who could be reasonable participants? See if there are any aspects of your profile that you can loosen up on.

2) Change recruiting methods completely – go to a vendor, go to an online usability test tool like fivesecondtest, find a meetup or online group that attracts the kind of people you want in your study.

3) Offer the previous participants a gift card for every person they recommend (with the same/similar profile) who completes your study. Have a cap on the number of people you’ll use from them so you don’t go bankrupt 🙂

Q: We’ve traditionally only used panel companies to recruit, how can we start building a customer list to use for ad hoc recruiting?

Easiest way to start is by sending an invite to be part of the consumer panel to your entire customer base. Ask them to forward the email to others that they know. Put a blurb about this on your homepage, on forums, on your Facebook page, and in as many places as you can think of where it will be visible by the people you want to recruit.

Prior to doing this, put a lot of thought into what qualifies someone to be part of the panel: what you’ll expect them to do, how often and how they’ll be rewarded. Be explicit about all of these aspects in the materials you use to recruit for the panel. I would also recommend having an immediate reward for signing up and another within 1 week of signing up, and/or give them their first opportunity to participate. Definitely don’t get them in and leave them waiting.

It’s also critical to understand that your panel will need constant recruiting and maintenance. You want to add new people to replace those who signed up but don’t participate and those who drop out. Intermittent rewards in between study invites are great – send them a 10% off coupon just because, give them movie tickets, Starbucks cards or other small gifts, opportunities to buy your company’s products at a discount, or sneak peeks at new products. Think of how to make them feel special and respected as individuals so that they continue to want to be part of the panel and the majority respond when you ask for participation.

Thanks Leah, and thanks to all of our webinar participants for your great questions! In case you’re sharing this with colleagues who couldn’t make it to the webinar, you can watch all of our webinars on-demand.


258 / 571