Soma Ray & Stephanie Agotborde from Booking.com on making UX research a team sport

In the following video and transcript of their talk at BetterUX London 2019, Soma and Stephanie discuss their belief that research is a team sport, rather than a monolithic process that sits in isolation. And also how, through the coaching of others and democratizing user research throughout the company, Soma and Stephanie massively scaled the number of user needs and pain points solved.

Soma: Hi, everyone. To loosely borrow from a famous phrase, “It’s 10:00 a.m. somewhere in the world,” so none of us have a food coma right now and we are all wide awake. And with that, I hand it over to Stephanie.

Stephanie: Hi. So, you might recognize The Avengers. Different superheroes with different superpowers from all over the world to work together and prevent the villains from taking over our planet. We may not have such a difficult task, fortunately. But as the Avengers, we need to work in multidisciplinary teams to achieve our goal of building products that improve the lives of other humans. And sometimes this collaboration can turn into a challenge, especially when it comes to solving user needs at scale.

Let’s think about Iron Man for a moment. The super cool guy that made his own superhero suit that wants to figure out all the problems by himself, he creates his own plan and then wants to defeat the villain all alone. Not the best team player, right? So now, we’ll ask you to raise your hand if you ever experienced one of these situations. How many of you have this memory of sitting in front of your laptop with a bunch of research requests in front of you trying to prioritize, plan, and execute all of them just by yourself? Raise hands, don’t be shy, come on. Okay. And because of all this work, how many of you had to say no to a research initiative, not because research was not needed, just that resources were not enough? Come on, okay.

Well, and while Iron Man is trying to execute all these tasks, where is the rest of our superheroes? And most of the times, they are just waiting for a research request that is somewhere being planned or executed, uncertain about what to do next or what direction the project should take because they need the research findings to make those decisions. Or they are even launching a new feature without any qualitative feedback, because, “Hey, we need to be fast.” And Iron Man is so busy that he cannot always make it on time.

So, at Booking.com, we have a huge UX community that are identifying and addressing multiple user needs in simultaneous. We are multidisciplinary teams with a mission to deliver people the most meaningful and delightful experiences in the travel field. And even though each of us has their own superpower, we all need access to the work of our users to make the best out of our work. And we need to be fast. But in the context of limited research resources, how can we achieve this? Let’s start by analyzing the problem.

So, when we look at the way UX teams were doing research across the company, we identified three key cultural problems.

First, there was no clarity about the expectations that each role should have when we were doing research. This basically means that we haven’t set any agreements in advance on how we expected other UX roles to collaborate and take part in the research process. These unclear roles cause for researchers having to own all the stages of the research process. This means that, by the end of the day, researchers are the only one that is responsible that tons of projects get done at the right time. And finally, also the quality of the research results were affected, mostly due to not set standards of outcome. So, when teams wanted to do research by themselves, they were mostly on their own, without any guidelines or any support on best practices.

So, how do these three cultural problems manifest in the way we were doing research as researchers in the company? Let’s imagine this for a minute. So, a new research request has been announced among 10 others. Obviously, it’s the beginning of the quarter. So, the researcher that supports those teams has to sit down and ruthlessly prioritize. And even though we know there are some product managers that have been exposed to how to write a research plan in the past, they still know how to collaborate with us and they are really not expecting to have that possibility.

And then when these requests finally gets kicked into action, well, most of the times, the researcher is leading all the activities of the research process. From logistics to planning, recruitment, data collection, synthesis, and sharing the findings. But we also have other UX roles like designers or copywriters that they do have experience in moderation. But still, it’s the researcher’s main responsibility to also their team has to wait until that researcher is available to move forward. And also in a situation where the designers of the team are running some usability tests or some Guerilla Testing, we know that the results could have had a better impact if they were to follow best practices on how to avoid bias, synthesis, and also share those findings to a wider audience.

Soma: So if you identify with this scenario of a big UX team in the organization and massive requests for solving user needs at scale, we have something to share with you today. In the next 20 minutes, Stephanie and I will try to show you how we at Booking have approached this problem and what are our key takeaways from it. But before that, let’s quickly introduce ourselves.

So this is Stephanie. Stephanie is from Argentina and she’s moved to Amsterdam, where she lives now, with her two cats and a partner. She has a background in sociology. And she’s worked with market agencies for market research. And then she discovered the world of UX research, our user experience, and promptly fell in love with it. Since then, for about six odd years or so, she’s been working with e-commerce companies as a UX researcher. And part of the time apart from, obviously, producing really kickass insights, she also spends time strategizing on how to involve other UX peers and train them into the art of UX Research.

Stephanie: Thanks so much. And Soma here lives in Amsterdam by way of Bangalore, San Francisco Bay Area, and Austin. Having worked with tech enterprises like Dell, Nvidia, IBM, we are lucky enough to have her at Booking.com. And she’s passionate about driving UX through high tech avenues. And she also loves tinkering with organizational processes. And also, she’s a big fan of system thinking.

Soma: Yes, system thinking is real great.

Stephanie: So, we both believe that research is a team sport. And if we plan a strategy and coach others, we can massively scale the number of user needs and pain points to solve. Yes, like the Avengers, it’s all about combining superpowers and setting the right game plan.

Soma: So what is this game plan that Stephanie is talking about? So in our case, we decided to look at UX research, not as a monolithic process that sits in isolation but decided to approach it in phases or stages if you may. This allowed us to basically strategize for democratization into three distinct forms or phases. And those are form, perform, and transform. So let’s spend about a minute each on what these look like at Booking.com.

We believe that forming the right foundation for user research is the key to having the biggest impact and delivering maximum value for the products. This is a stage that needs critical thinking and prioritization. Not only that, around the time when project roadmaps or product roadmaps are being created, if we start including UX research goals and deliverables into the discussion, we believe that we can balance the business goals with the user need at the same time.

While performing research, we are the closest to the user in this stage and have the ability to gain maximum empathy. The researchers in our organization believe that in this stage, if they include all our UX peers along with us in the journey, then we are democratizing their, sort of, reach to the entity as well.

Using research outcomes to transform our products and services that Booking.com also delivers, can have a direct impact on the user and the business outcomes. At this stage, the more lenses, due to the different job roles and profiles, that are involved in getting from the objectives to the observations to the insights, the better are the quality of those insights. And not only that, during this stage, the more people that are involved, the richer the buy-in and the more long-lasting is the impact of that insight into the product development process.

To bring the thought of this framework into action, we decided to approach it from a systems perspective. Instead of looking at isolated problems and, you know, finding tactical solutions for those problems, we wanted to take a step back and look at it from a holistic point of view. For example, just having a one-time classroom training for research skill sets or some methodologies is not enough because short-term memory loss is a thing. Or neither is, you know, if a researcher has conducted an experiment or has conducted a research project and they’re just hearing it with a limited group of, sort of, audience, just in a one-time sharing mechanism, is also not enough. Because who knows? There might be other teams and organizations that need it and at different times.

Instead of addressing each of those individual areas from a problem solution perspective, we decided to start to connect the dots and find overarching patterns. So for example, how about if we balance the one-time classroom training with ongoing one-to-one mentoring sessions? And then enable the people that are being mentored to also carry forward the torch. Systems mapping actually opened up cause-and-effect relationships across the whole research framework that were more holistic and hence, more reliant.

So how does this look in practice for us? Through trainings, mentorings, and one-on-one sessions, researchers have been working with designers, product managers, copywriters, language specialists to identify research problems, create the right research questions, and identify right methodologies. This is the stage that usually needs the most researcher involvement because setting the right foundation is the key. And it might need a whole gamut of research experience.

Another approach was to synthesize our existing, sort of, research insights that were sitting in different pockets and creating artifacts that the teams could use in a living and breathing way. For example, we created this journey map and journey card framework that UXers use not only to ground their sort of future ideation in holistic thinking but also start identifying these new gaps in user needs and unmet needs and pain points and opportunities that they can further address.

When carrying out research at scale, one of the best investments that Booking.com has made is in enterprise usability testing tools. So for example, tools like UserZoom and others in the market, have helped our UX peers to kind of conduct research, whether it is evaluative testing or information and architectural card sorts or even concept explorations at time. And do it at scale, do it with a wide reach of participants, and have a really fast turnaround time. With the help of these pieces of training and guidelines from researchers early on in the phase, we have noticed that this kind of activity or this phase needs the least amount of time investment from researchers if everyone else is set up for success.

During the performing stage, there are matters that also need trained researchers. And these are typically in the range of exploratory and generative research. But even at this stage when the researcher is primarily running the show, we heavily rely on collaboration. That there is note taking, organized debrief sessions, or even sometimes call moderation. Especially for note taking and debrief sessions, we feel that that kind of reduces silos of bias thinking and includes more roles in contributing to the outcome. But for the co-moderation activities, what we really realized is a lot of our UX peers feel very, sort of, happy and find it really fruitful that they’re adding on new skill sets and being part of the process.

A quick question. Who here has heard of the Warren Buffet quote, “You can’t get a baby in one month by getting nine women pregnant?” So in most cases, this might be true. However, we like to believe that during the synthesis session, this probably does not apply. So in our case, we have noticed the more hands on deck, the better is the quality of the outcome and the better is the buy-in. What really helps in this, sort of, this phase is the speed and transformation that occurs and how fast our insights can then make into product development.

But also, let’s assume that we are doing a project and we’re delivering on the insights. And by having the team members include in that, we gain future buy-in from stakeholders, because now they are more familiar with the grey areas of research and things that are not well documented necessary elsewhere.

As researchers, we also, obviously, do love creating reports. It helps us tell like an end-to-end story and represent our users in a more human way. And oftentimes, we balance that by facilitating designs prints or conducting design brainstorm so that we can kind of get from the insights generation process to the product development process in the fastest turnaround time as possible.

By splitting and juxtaposing research stages with design and product development lifecycle, we want to ensure that research is not thought of as a monolithic process, that people think of research as equally iterative just as, you know, a design sprint cycle or design life cycle or even a product development Agile process. And we don’t want our team to think of research as just the bookends of product making. Like, we want some research as the beginning and then we want some evaluative testing at the end but like continue throughout the journey with us. And at all times, our eye is always on the price, which is to deliver user needs or insights to user needs at scale. And also at the same time, enable our UX peers to do a really good quality job at it. Like Avengers, we believe that we can solve bigger and better problems when we approach it this way.

Stephanie: So, where has this led us? Basically, in the past six months, we have trained a good number of different UX roles. And we even trained some other roles like developers or language specialists that we didn’t think they might be interested in doing this. So that was a surprise. And we’ve been running these trainings without any promotions or they are not mandatory. It’s mostly about word of mouth. And we can see that there was clearly big demands for our UXers to taking part into the research process and collaborating more with us. But also, besides the amount of people that we have trained and the amount of research studies that now we can run, we also see an impact on our peer’s self-development. Here, you can read some quotes about whether their designers or some other roles are saying about the training and also about these frameworks that we are exploring together.

So through this process, we believe that as a community, we are all learning new tools, sharing best research and UX practices, and also exploring new frameworks together.

Soma: So this has made it possible for researchers to guide and lead and sort of, you know, “touch more projects” than ever been possible. And this is without spreading too thin. So for example, in 2018, if you just consider the guests facing or the customer-facing research in booking.com, there were around 30 researchers that had a direct or indirect impact on about 370 projects. And this is more on the lower side of estimation. Democratization has made us able to kind of extend our reach in this manner. For example, you know, almost like adding more superpowers to our repertoire.