17 Questions Answered about Agile UX
Jon Innes and Jeff Sauro, both UX pros, discuss the best practices for Agile UX testing:
1. When do you recommend running user tests in Scrum?
Throughout the process, on an ongoing basis. 3×3 prototype studies are great for sprint 0. RITE is very suited for in sprint testing with working code. You can also test a sprint behind, but that tends to lead to UX debt, the accumulation of usability issues, so in that case make sure you have buy in on using metrics and goals to track what’s going on.
2. Can you talk a bit more about making sure the results of the tests are visible and the team is engaged with automated testing?
If the team has a physical scrum board, I of make a column called “user tested” at the end, and move story cards over when we test them, and I either note on the card the metric we are tracking (say task completion rate), or I move it only when it meets our predefined metric based goal. I also post screenshots with markups for issues showing recommendations. Follow up by filing bugs so they are tracked as part of overall quality.
3. Where is the major time savings in comparison to remote moderated testing & A/B testing?
Moderation takes time, so clearly both automated and A/B testing saves time. Unlike moderated testing, much of the data collected in an automated test is precompiled making analysis and reporting faster. In automated user testing standard task and sat metrics are calculated, as are clickstream graphs and heatmaps. Some of that stuff would take forever to do by hand. In contrast to A/B testing, both moderated and unmoderated can save development time. People often overlook the time it takes to develop and deploy stuff for A/B tests. Also, in many cases running follow up tests is needed with A/B testing to understand complex interactions. That can often be avoided with automated tests which give you more qualitative data.
4. What mistakes do researchers typically make when first doing remote automated testing?
First, test planning and design are much more important. Once you pull the trigger it’s harder to adjust for any problems. That’s why pilots are so key. If you’re used to doing only qualitative testing, only reporting issues with no metrics, you’ll find you need to play more attention to study design.
5. What is an appropriate recruiting strategy for agile testing?
Define the recruiting strategy early in the project, not just in time. Depending on your research methods, you can use a combination of private lists, panels, and intercepts. The key is to predict when you will need participants, how many, and of what type. Allow at least a week for panel & private list participants to respond to remote study requests. Also keep in mind that with iterative testing, comes the need for participants on an ongoing basis. Often more participants are needed as well. Waterfall recruiting strategies won’t work, you’ll get bottlenecked.
6. What effect does UX testing have on the product backlog?
Tests find that stories aren’t really “done.” That means you’ll have to accept that some tasks may be added to stories that the PO thought were complete. This is no different that finding other types of software bugs on an agile project. Make sure you’ve agreed to how to prioritize UX defects of any type with the team early on and defined metrics and goals. Otherwise you’ll end up with unmeasured “UX debt”, similar to Ward Cunningham’s concept of technical debt.
7. How long do you run remote unmoderated usability tests? Do you ever run them longer than 15-20 minutes?
I’d say avoid longer than 30 mins, but you could do longer in some rare situations.
8. Does the UX team need to be a part of the agile team? If so, how do you make it work? What types of UX skills are necessary?
You need at least some UX representation on the team to be effective. The challenge is that it’s hard to have all the skills required on such a small team. I’d recommend having a dedicated UX lead on each team that has the skills most likely to be needed, as well as some supporting UX staff that can help out. One researcher might be able to cover multiple teams in many cases. As to what types of UX skills are necessary, it really depends on the project. That’s like asking what kind of development skills are needed. I wrote an article on this previously titled Defining the user experience function: innovation through organizational design.
9. One of the presenters mentioned that the definition of done can only be determined by the user. Would you suggest testing at the end of each sprint?
That was me. Yes, I’d recommend reviewing the usability test findings during the sprint review.
10. Working in 4 weeks sprints how do you integrate UX testing when users are spread all over the world?
This is where remote testing really helps. Run studies based on the prioritized market segments. Keep in mind that many usability problems are not location specific, but just as likely to impact any user. If faced with limited resources, prioritize the testing by market segment, focusing on high risk personas and stories. One technique you can use to prioritize is what I call the UXI matrix, you can find a description of it in the blog post Integrating UX into the Product Backlog.
11. Can you elaborate a bit on automated user testing for stories?
Take your list of stories, and define which ones you want to test as you scope each sprint. For each story to be tested, you create a user task in a remote usability testing software. Once you do that you can run that test as many times as you want. The only additional cost is your time to review the results and any incentives for panel recruits.
12. Jeff, you mentioned testing with only a handful of users. But shouldn’t statistically significant results be based on larger samples? How many do you recommend for a site that receives millions of visitors in a week?
This is a common question. You can actually achieve statistical significance with only a few users. In the examples I used we had between 10 and 15 users and the results were statistically significant. It helps when you have the same users attempt the same tasks on comparable interfaces (called a within subjects study). Statistical significance refers to results which are unlikely to be due to chance alone.
With smaller sample sizes we are limited to detecting relatively large differences between designs (large differences in preference and performance). In early stage designs however, it is those larger noticeable differences we are most interested finding for decided the better design.
Even for a website with millions of visitors (the examples also came from websites with millions of monthly visitors) or just a thousand visitors the math works the same. For determining which sample size to use in an evaluation depends on what you are doing (comparing, estimating or finding usability problems). For guidance on finding the right sample size see: 20 Questions Answered about Unmoderated Usability Testing Question #3 and Chapters 6 and 7 in Quantifying the User Experience.
13. Can an outside consulting team be engaged in a client’s agile process? How does the engagement look like?
In Scrum, the ideal is that everyone works in the same space. If you’re an outside team (not just a person) that gets harder. You can put one person on site (a lead) that coordinates with the rest of the offsite team. Another approach is that the entire offsite team just dials into the daily standup. Both compromises have drawbacks, but people have made both work in practice. Keep in mind that being offsite means you often will introduce overhead in the process due to the lack of in person communication. Stories in Scrum are meant to be discussed, so you’ll need to figure out how to ensure the communication happens with any offsite team. It can be done, but it’s more difficult.
14. What would be the recommended number of staff to effectively support this approach?
I’m assuming “this approach” means automated remote testing. One of the benefits of automating moderation is that it allows a single user researcher to do more. If you are using remote automated testing, combined with A/B testing, you could probably get away with having a single researcher cover 3-5 teams depending on the nature of the work. That estimate would of course be different if moderated testing was used to do RITE tests. With moderated forms of RITE testing, you’d need a dedicated researcher per team in most cases.
15. How do you do remote usability testing on a mobile app or site?
Alfonso de la Nuez:
There are a variety of ways and tools to go about remote mobile usability testing. With our own Mobile Voice of Customer tool, UX professionals can gain insights into who, how and why actual users visit their mobile websites or apps plus their satisfaction and likelihood to return and recommend the mobile site.
Basically, the way it works is the following:
- Tag your site or app with JS code.
- When visitors come to your app, they’ll get an invitation to participate in a test.
- The test can include a task and/or a questionnaire.
- When users are finished with their session and leave the app, they get a notification inviting them to fill out a survey (their browser will open if they agree).
16. If we’re making decisions rather than paperwork, and your content is subject to audits, how would you present your processes and rationale?
Discuss the process at Sprint planning, reviews, and retrospectives, as well as in the daily stand up meetings. I’d make sure any results were posted publicly for anyone to “audit”.
17. Is there a way beyond automated tools to integrate UX testing into agile?
Yes, the first three methods I spoke about during the webinar work well in agile. 3×3 tests are regularly done in 2 to 3 weeks from start to finish and ideal for sprint 0 or spikes/sprint ahead. RITE testing as an ongoing process is practiced by many firms today. A/B testing methods are also very commonly used. We focused on automated testing since it’s less well known.