James Barley, Head of UX Research, on ‘UX Research for the Masses’

Head of UX Research, James Barley, showcases how he’s worked to democratize research at Auto Trader over the last couple of years, by making it more visible and understandable throughout the whole business.

In the following video and transcript of his talk at BetterUX London 2019, James covers the reasons why they started, how they engaged the business and what the impacts have been.

James: Hello, good morning. First of all, I’ll introduce myself. I’m James. I’ve been heading up UX Research at Auto Trader now for the last couple of years. Actually, truth be told today, I kind of feel like one of the early acts on stage at Glastonbury. You guys have got a lot of a lot of insights to follow me as well. So I hope you have a great day. I’m also I’ll just like to say thanks. Thanks to the guys at UserZoom for inviting me. It’s always good to kind of take a step back and reflect on a topic. And on a personal level, actually it can be an indicator of what you’ve been doing and how much you’ve progressed. I’d also like to thank you guys as well, for coming along to listen. I’m feeling quite humbled to be staring at a room full of all these faces. So thank you.

Just for a bit of context, where myself and my team sit in Auto Trader, we follow kind of a similar model to Spotify. So we’ve got squads and tribes.

And we live essentially in the buying experience tribe. So our main focus is the consumer’s buying experience in finding a car, finding one that they feel confident in, and then following that up and buying it. We tend to focus on the core business priorities of the tribe. And these are split into what we call blue cards. They’re not like your traditional product teams. We tend to focus on what the two, three, four big bets the organization has got for the next 12, 18 months.

So I tend to live with the new car squad. So I’m working with a team who are trying to make Auto Trader to ‘new cars’ what we are to ‘used cars’. We also have a finance squad. So those guys are really looking at the transactional side of inquiring about a car, and potentially working towards buying one online. And then we have a team that’s focused really just on continual optimization. We call them iBET. Because literally that’s what they do. They place bets, they run tests, and they’re generally just involved in improving what is the core experience for most of our users. You know, we don’t want to leave that and to gather dust, or to grow old. So our main focus, yes, is the consumer. But obviously, we have customers as well. We have people who are paying to advertise on our platform. So we’re equally as focused now on the service design behind the new products that we develop. Because if there’s going to be an impact on the customer, we really do want to know because they’re paying the bills. So that hopefully kind of gives you a context for where me and my team operate.

Since I started a couple years ago, we’ve worked really hard to democratize research. And what I mean by that in this talk is to make it accessible, to make it visible, and to make it understandable throughout the wider business, not just within our tribe but across the whole of Auto Trader. Today, I’m going to cover three main areas, quite logical, really. You know, why did we start? There was really an underlying intent. What we’ve been doing to achieve that. So we have been engaged in various activities on various different levels. And I’ll run you through those in a bit more detail. And then also the impact. You know, what is the outcome of doing all of this great democratization? And have we achieved or are we beginning to achieve what we set out to do?

So, why did we start? Well, I mean, of course, I’m sure there’s many researchers in the room. You know, it is great to tell people how awesome our job is and how great the findings we make are, and how fun it is. But actually, a strategy behind why we make research accessible is really just a means to an end. It’s there to help us achieve what we set out to do more. Initially, we just started out by wanting to be on the map really, to be on people’s radar. When I started at Auto Trader, we were really at ground zero in terms of research. I was the only researcher for a start. We didn’t have many tools. I mean, we had our own in-house lab but we didn’t really have much else to play with to gather the insights that we needed.

And also the organization liked to pay a lot of lip service to research. It was putting a tick in a box. It was doing research, but actually the data it was getting out and the data it was using wasn’t great. And also, it was a very tech-heavy culture, a very delivery like culture. So when researchers come in and start to talk about being evidence-led or be more experimentational, it can be quite a difficult message to sell to programmers who are just programmed to program. So the key thing initially was to just get people inquisitive, to get people interested. People are naturally interested in people. So we try to get onto people’s radar by really sparking that natural curiosity. The goal of that was to start people talking, was to hopefully create some advocates around the business, people who through word of mouth, that actually talk about the great work we were doing.

The second goal was really just to educate people about research. It’s not just about asking people what they like or what they would do next. In fact, it’s not just about testing prototypes or, you know, the live product. It was really a case of educating them that it’s never too early to get research involved that, “Do you know what? We can start learning things before the guys have even looked at sketch or even put a sharpie to a poster.” We also wanted to educate them that the research is there to help them. It’s not just to put a tick in a box, but if we gather evidence, and we’ve got robust evidence that’s driving our ideas, driving our hypotheses, we’ve got a much higher chance of that being a winner. We’ve got a much higher chance of that being successful out there in the real world.

And ultimately, the goal was and the goal still is to, slay the hippo. Auto Trader is no different from a lot of other organizations. The hippo wanders here and there. And really, we wanted to make sure that the evidence we were generating was there is actually a stronger argument and actually a good argument for making evidence-led decisions. And actually, truth be told, this was the ultimate motive that we were following. The primary objective was to make sure that actually as a team, we had a purpose. Actually as a team, the great work we were doing was going to land. It wasn’t just being cherry picked and it wasn’t just being used to put a tick in a box. So yeah, categorically, we hadn’t intended to replace the researcher in any of the work that we’re doing. What we intended to do is actually amplify the work that we were doing, and actually get it used and acted on much more frequently.

So, in terms of what we’ve been doing, first and foremost, the mantra right from me starting was, “Yeah, half of our job is to dig out insights. Half of our job is to find those actionable pieces of data that are going to help the organization. But the other half is to turn around, yes, we need to face outwards.

Yes, we need to understand that consumer, our customers, we need to understand their motives. But actually equally, we need to understand the folks that we’re working with. We need to empathize with them. We need to communicate their language. And also, we need to be able to frame you know, our message to them around how it will benefit them in their role and as individuals.”

So, as I mentioned before, we work in squads. And, as I’ve said, I’m aligned to the new car squad. In that squad, we got product-lead, design-lead, tech-lead, engineers, QA, delivery leads. And the way that we conduct research is that we don’t just bring them on the journey, we don’t just drag them along. Actually, we involved them in the journey right from the off. It’s really important that we get them involved, and actually get them participating in the research before, during, and after any study that we conduct. It’s really an exercise I suppose, and as researchers this is what we do anyway, in creating common ground amongst the team, and getting them to share an understanding that, “Okay, we’re writing a line of code, but actually the real reason we’re writing that code is because of that person sat there interacting with it.”

It’s a great source of collaboration, and more importantly, moving forwards in terms of acting on the data, inspiration as well. So it’s a key ingredient really, in getting the squad to fully engage in actually what our OKRs are and what our goals are. So this is literally the nitty-gritty. So before any piece of research, any study, be that a remote study, a one-to-one piece, I’ll work closely with the product-lead to make sure that actually, we’re addressing the right things, we’re addressing the right features. It doesn’t just stop there, then we’ll call the wider team round. We’ll call the leads around usually product design and tech. And we’ll start to really drill down on a piece-by-piece basis what are those questions they’ve got? If we’re looking at a design, you know, specifically, what are the metrics that we’re trying to track? What are the areas that they’re unsure of? What’s the understanding that they don’t have here about that design?

So we’ll take that away and we’ll shape it. But then the collaboration doesn’t end there, the team will then be called back in and we’ll run through it. And it’s a chance for those guys to say, “Actually, James, you know, you’ve misunderstood that.” Or if we still got any blanks, we can start to fill those in. Or if they’ve got any additional questions, it’s still their chance to chip in. And they just feel that ownership of the study right from the beginning. When I first started, we deliberately over-indexed on lab studies for about the first 12 months. And this was not just the fact that was pretty much the only tool we had. It was also a really great way of engaging people in the research. People are interested in people and if you’re bringing the consumer into your business to set and be observed within the business in the heart of the business, that’s really quite engaging and just quite interesting because it sparks a natural curiosity.

And what we do as well during a study, folks don’t just sit there, you know, potentially having Outlook, or twiddling about on their phones. Everybody has got a role to play. If you’re not here to observe, then you know, you’re not here to kill time. You’re here to participate as part of the team. All of you guys are my note takers. Obviously, I’ve got representative of UX making some detailed notes. But by actually getting people to engage and make their own observations is a really great way of them really understanding what it is we’re seeing and potentially how we can act on the back of it. And it doesn’t just stem with the squad. You know, a lot of our projects touch many stakeholders. So we made sure that we invite as many people and as many people get exposed to the study as possible.

Now afterwards, actually quite frequently what we’ll do, we’ll transcribe all of the notes together at the end of each of the sessions. And that means that, well, we can actually see the patterns and the themes emerge before us. And as a team, then if we’re testing a design in the lab for example, we’ll use a little trick that we’ve learned for each of the tasks called Delta Next Framework. And that’s literally just a matrix with four quadrants. And in each quadrant, the team then starts to chip in. They start to actually point out the assumptions we got wrong, the assumptions we got right, the new questions. Because obviously, the more we learn, the more we realize we don’t know. So the new questions that come off the back of that piece. And then just as importantly, you know what are our ideas for solving the problems or the pain points that we’ve just observed in that study?

When that’s not possible, so we might have been running remote study, for example, or doing a piece of field work. What we’ll do is we make sure that will pull the team back, and we’ll play that back to them. But the researcher never makes a recommendation and researcher never prescribe something at this stage. But if anything that’s really going to wind your designers up because that’s [inaudible 00:12:15] It’s really important that we kind of put the observations over to the team and let them inclusively generate the ideas. And what we found is by actually having that real range of stakeholders in those sessions, in those ideas, that we don’t just generate more ideas, but they’re richer ideas, they’re more diverse ideas. So it actually plays back into I suppose our organizations that you know, we embrace being an inclusive and diverse place to work. And this really does work in the research arena as well.

I suppose the second line of activity that we’ve been undertaking is to demystify what we do, is to literally coach people and help them to understand what we do why we do it. Since I’ve started, we’ve been running internal coaching sessions for one of a better word. And these just cover the basics in one-to-one moderated user testing. They’re bi-monthly, they’re voluntary, and I think since we started, we’ve had about 80 colleagues trained up through this. It’s now actually been adopted as part of the Graduate Induction program as well, because they feel it’s really important that any new comer to the business understands why we’re customer-centric and how we do it.

And we’ve done other things. We’ve for example, spun up a guild. I don’t if anybody is aware of, you know, like the Spotify models Lingo and Jargon, but we spun up an experimentation guild. So anyone in the business, not just in our tribe, can get involved on a weekly basis in discussing their experimentation, their A/B test. Now half of the purpose of the meeting is operational. No, we don’t want to have conflicting tasks. We don’t want people polluting each of those results. But the other half, which is really why we started these sessions is to provide the support and guidance to folks who were not from an optimization background into how to run a robust and viable test. So it’s a great chance for us to support and guide, and kind of nudge them in the right direction and you know, do some of that knowledge share on an ongoing basis.

The other aspects of what we’ve been doing in this theme is giving people access to some of the new tools that we brought on board. You know, we’re very lucky as a team. We’ve brought UserZoom in for our remote testing. And also we have a UX metrics tool called Content Square. So we’ve got two very powerful, sophisticated tools that allow us to get insights very quickly. And also, we’ve been very fortunate in our partners in that they’ve got great account teams. Those account teams have come on site several times and help me train up folks from the product design data, Q/A delivery backgrounds in how to use them. I think we’ve had 20 or 30 people trained up in them so far. And it’s not that we want people, you know, running off and spinning up their own think out loud study or pulling together their own detailed reports. What we want is for them to be aware of, “Actually, we can get answers quickly. We didn’t know that you could do that. And actually can we do that before we even begun design?’

So now folks are actually aware of when we can get involved, how quick it is, how easy it is. And essentially now we’ve got lots of eyes and ears around the business that can give us a little tip off if research is required, if they need us to help coach in some of the far flung corners of Auto Trader, we can do that. Another element, has anyone heard of Google’s 20% time? I expect you have. We’ve got 10% time. So I don’t know, is that twice as good or half as good? And we’re encouraged to use it each and every one of us, I mean, here today. So it’s really for us to help develop ourselves, grow ourselves, and really feed into our own PDPs.

What we found is folks around the business who actually, they want to take that coaching session a little bit further. That they want to understand a little bit more about, or maybe they’ve been to a lab a few times and they want to get more involved. We take those folks on. I think I’ve got four mentees at the moment, each of them giving 10% of their time. So I’ve nearly got half a person. And essentially, we’re giving them like projects to work on. We’re not just giving them, you know, like a school project or anything like that. They’re actually helping us with live work. And that’s great because on a personal level, I can feel quite, what’s the word? Quite warm inside that I’m helping somebody, you know, potentially develop new skills, potentially look at a new career. But also, from a resource perspective, it means that actually our team can continually grow, and do so in a managed way.

The last thing that we’ve been doing has been basically finding any opportunity to plug in a laptop and tell people about what we do. We’re very lucky at Auto Trader, we have showcases at every level. So we have a company showcase, a tribe showcase, tribe days, squad showcases, steering group committee presentations, even an opportunity to talk to the board. And we’ve taken all of those opportunities because it’s really been a way of amplifying not only what we’ve been doing, but also the great results. And yes, we have our own internal network, social network. So we have been hammering Yammer as well. And this, again, is a great way of socializing the results after you’ve maybe delivered a playback or that you’ve delivered a showcase.

The final aspect of I suppose this concept of sharing knowledge and really just you know, giving people real insight into you know, what research is, is working with other teams. We’ve actually started to help them using our own techniques solve their problems. So by using the same co-creation or participatory design techniques, we’ve been able to really get under the skin of some of the business problems. Our people team are big fans of this approach. And what you can see here is actually a session we run around our return ships program. And we actually co-created that with a group of people who were our target audience. And that’s gone into the syllabus for next year.

So in terms of the impacts, you know, after all, I said that we are doing this for a reason, we’re doing this for a purpose. There is an end and this is a means to an end. I mean, first of all, we have built our profile now. We are the go-to people in the product and tech teams for understanding consumer behavior and good UX. And quite often we end up shrugging, but, you know, we are seen as a source of expertise. And we also get requests from around the businesses to whether we can help other squads, other teams do testing, especially our sales and service teams, and they look after all of the franchise’s and all of those franchises who’ve got websites. And the amount of times I’ve been asked, “Can we test my customers website in the lab?” The amount of times I’ve had to say no.

But actually it is a good measure as to I suppose, the profile that the team has built that. You know, I suppose the measure is how many people who I have never heard of contact me each month to ask for a bit help here or a piece of research there. I’d say there’s at least two, three, four requests every few weeks. This for me though is the main result of making researchers transparent and as democratic as possible. The fact that the squads that we work with in now are fully engaged and actually they’re switching their behavior. They’re moving into more experimentational approach. They want to use the evidence. They understand why we’re gathering the evidence, and they know it can give them the bigger winners.

Key to this has been getting 25% of each of the teams working at a grassroots level on board. A key to it was really just observing which of the team have got a natural interest in this? Which of the team want to be data driven? And then we spent time, each of the folks in the team building relationships within each of the squads that they’re aligned to. Once we’ve got about 25% on board, you’ll generally find that people will follow. For me, the tipping point in this, so I suppose one event that could really epitomize this turn around within certainly the squad I was working in, was one test that the guys ran without my knowledge, without the design team’s knowledge, in fact, without the product team’s knowledge.

This was a few engineers who thought, “Do you know what? We want to increase the amount of new car searches.” Because that’s one of our key KPIs. How many people search for a new car? How many people look at one and how many people generate a lead for one? And they did something very simple because they understood that actually the happy path we designed didn’t upgrade afford, and so we were putting it in the wrong place. So they did something as simple as just putting a button on our search form that said, ‘New Cars’, and in fact, they didn’t just stop there. They played with the placement. They played with the color. And then they came to me afterwards and they said, “Have you heard about this test we’ve been running?” I was like, “No.” “Well, you know, have a look at this.”

They increased new car searches by 155%.

Now, we haven’t rolled this out ‘cuz it will get very political. You know, we’re stealing audience from the used car teams. And actually, it wasn’t that result that was the thing I was most pleased about. It was actually, “Wow, you guys, engineers, not even front-end developers have set this test stop, and run it all off your own back. All based on the evidence that we had.” That was when I kind of knew we were making positive steps and definitely moving in the right direction.

Actually, to kill any suspense there, when they call it the, ‘Button in Red’, we increased searches by 400%. So these guys were hungry. They were hungry to learn. And they were using all of the evidence that we gathered to feed into this. So it boded really well for the future. And I suppose the second largest scale event that’s happened is, there’s been a bit of a switch around now in our tribe. I think the first year that it was formed, everybody was given a roadmap with features and widgets and things that they had to deliver and everyone was like, “Oh God.”

Now, on a 12 monthly basis, all we got are a list of goals and chaos, that’s it. These are the outcomes we want to achieve, you guys go and find the best way to deliver that.

Which from an experimentation and an evidence-lead perspective is kind of music to my ears. Which means, me, my team, the product teams, can be generating hypotheses throughout the calendar. And then when the time is right, when we’re approaching a theme or when we think, “Actually, we’ve got a bit of a hotspot around here. We’ve got all of these insights pointing in this direction for that micro-goal,” we’ll pull those down into individual experiments or projects.

I think also, actually, just anecdotally and the squad that I worked in six months ago, it was a big deal for 30% of the guys time to be set aside to improve in the UX of the existing tool, to improving the performance of it and the optimization. For the last three months, we’ve had nothing new to build. We’ve just been focusing on, “Okay, what if we got? Let’s make that as good as possible.” Which is, again, quite a turnaround from a company that just wanted to build one shiny, shiny new thing, and then move very quickly on to the next shiny, shiny thing. Actually going back and polishing these shiny things now, which is fantastic from my perspective.

So this kind of leads me up to the present day. And we have a backlog, a huge backlog full of ideas and hypotheses that have got evidence on depending them. And not only this, this is actually yielding much more tests, much more frequent tests, much more winning tests. So for example, the guys understood, “Do you know what? If we’re going to get people to look for new cars, we need to kind of work with how they behave.” And they like filtering, they don’t really want to use different calls-to- action within the search journey. Lo and behold, that hypothesis increased searches by 30%.

They finally listened to me when I started talking about pricing, and anchoring, and typographic, etc. Lo and behold, we had another relatively big winner.

So we’re in a really good place now, folks have taken on kind of the message. And I’d say, democratizing research, actually giving people the transparency, the access, the understanding has been a key ingredient to that. Not making recommendations, letting them generate the ideas, letting them be involved in the process, letting. us all find common ground together. That’s what research really is there to do is to facilitate that. Yes, there will always be a role for the researcher. Yes, we always need to be objective. And we’re there to really play devil’s advocate and to maintain that.

But by opening it up to the team, we’ve actually increased, I suppose, the impact and the rate that the evidence that we’ve generate, and how that’s landed quite significantly. And here we are, we’ve broken down our journey into a series of micro-goals. And each of those micro goals now, has got between three and six months of ideas, experiments, et-cetera, that we can start to feed into our experimentation program.

This year, if you can just about make out all of those post-its were generated from a session that was just about getting people to click more on an advert in our search results. That’s what we wanted people to do. And the evidence was basically the enabler for all of those ideas from all of the squad.

So from that perspective, we’ve been relatively successful. We’re making really, really positive strides. But this is really only at one squad and at one tribe level. We need to keep moving. We need to maintain not only what we’re doing, but to step it up a gear, to crack climb to the next wrung just so that we can fan it out now more broadly across Auto Trader. So yeah, in summary, we started with a simple goal to get our research, to land, to make a difference. We’ve done that. And if you take one thing away from my talk today, just take this, that mantra, “50% evidence, 50% engagement.” That’s really important.

And yeah, we’ve made good progress, really positive progress, and going back to what I initially said, you know, pulling a talk like this together, actually, you know, times can be hard. They can be quite choppy as a researcher, but it’s good to take back and take stock. But never rest on your laurels. There is always the next phase, the next goal that you can move towards. That’s it.

Thank you.

Moderator: So you spoke about ‘Slaying the Hippo’. And you’ve mentioned you’ve got a lot of good results and getting teams involved. How does that translate at the ‘Hippo’ level in terms of buying at the top?

James: I think, I mean, what reflects that for me was that they left those to just get on with it this year. They left those to, basically, they set the goals and the OKRs out and then we work towards those. Yes, it’s tough,and yes, you sometimes have to try and get a leopard to change it spots. You know, you have to change people’s habits and behaviors. But if you actually just start, you know, just saying, “Well, that’s a great idea. Why don’t we just test it?” Now, that’s one way, when people have evidence put in front of them, it’s quite hard for them to make another argument. So yeah, we just try to divert those conversations around to that. But with this kind of working at a grassroots level as well, it gets harder and harder for people to maintain that habit.

Moderator: Okay. Thank you. The Hippo is officially dead at Auto Trader.

James: Not quite.

Moderator: Oh, nearly dead. Sorry.

James: Approaching extinction.