Katja Borchert & Pietro Romeo from FlixBus reveal their user research successes

Katja Borchert, Senior Product Owner & UX Researcher, and Pietro Romeo, UX Researcher, reveal practical and leftfield examples of user research at FlixBus in their presentation ‘Don’t Guess It, Test It!’

In the following video and transcript of their talk at BetterUX London 2019, Katja and Pietro discuss how they democratized UX and took a journey towards a corporate UX majority, by running two concurrent teams: Team Hive, which is focused on the development of design systems, and Team Swarm, focused on user research and insights. And it’s these two teams that drive UX across the organization.

They also reveal some intriguing user research successes, including building a fake bus in the office to test the offline experience and a tasty focus group project called ‘UX Croissant’.

Katja: So, we are from FlixBus, a German mobility provider. So let’s start. We want to quickly introduce ourselves to you. So, my name is Katja. I joined FlixBus in 2018 as a Senior Product Owner and researcher. So I’ve been working in a lot of areas in Berlin’s IT for more than eight years now, for travel, for weather or real estate portals.

Pietro: I’m Pietro, I’ve been a UX researcher of FlixBus for about six months now. My background is actually in computer science. I graduated a few years ago with a Bachelor in Software Engineering from the University of Glasgow, and now, I am about to finish my Master’s degree in Human-Computer Interaction. Back to you.

Katja: I want to say a few words about FlixBus as I’m not quite sure how familiar you are with the company. So maybe you can raise your hands. Who has heard of FlixBus before? Okay. Interesting. And now, who has traveled with FlixBus so far? Okay. I hope you had good experience with us, otherwise, we can talk later maybe. So FlixBus is a provider who wants to do smart and green mobility for everyone to experience the world.

The idea started in 2012, and our three founders got the first bus on the street actually in 2013. And since then, many countries have become green, as you can see here. So FlixBus serves around 29 countries, with more than 2,000 destinations and over 350,000 daily connections. So that’s quite a lot. We also recently joined the US, and yeah, trying to compete with big companies as Megabus or Greyhound. And I think you here in the UK are rather familiar with Megabus.

So, we are going to talk today about democratizing UX and our journey to corporate UX majority and what we have learned along the way. So, first, I’m going to talk a little bit about our set up of FlixBus and the evolution in terms of UX at FlixBus, followed by what we did to get UX outside of our team. Pietro is then going to show you what works well for us and a few interesting showcases. And we finish with our little summary conclusion, with a few takeaways.

So, our set up at FlixBus. Of course, as in a lot of other companies or talks we’ve had today, we have many product teams at FlixBus. And we call our Flix, our IT department, FlixTech. And there are many product teams, and we have two special teams. One team is Team Hive, which is focusing on the development of design systems, kind of a living style guide for our consistent front-end development across the company, which has a huge part of making our product usable.

And we have us, as Team Swarm, focusing on user research, customer insights. And as two teams, we’re driving UX across the company and the organization. Our mission is to support teams, to test and validate their assumptions and hypothesis to user research, so they can make more confident product decisions and avoid building products no one needs or no one uses in the end. So I’m curious, as we heard a lot of talks from researchers today, how many of you are here today as POs? As product owners? Okay, I can see a few. How many of you are here as researchers? Okay. Let’s say that’s the majority. And as designers? Okay. Also quite a lot. From the business side? Someone? Okay. Development? Okay. All right. So I think in terms of democratizing UX, I would think in a few years from now, I would hope that there are more fingers raised when I ask about POs and about other worlds because I think then we’ve made it, then we have really democratized UX.

Okay, let’s move on. I’m gonna tell you a little bit more about UX research, then and now at FlixBus as well. I think you have heard that at your companies, maybe once in a while, “Oh, we don’t need research. We know our users, of course.” Right? Okay. So, we had this challenge as well. So, what did we do to overcome this challenge? So, step one was, in our case, we had to make sure, what maturity stage are we in? Our company? So this was very important in order to know which steps we have to take next.

So as you can see here, stage one means no awareness of UX. There’s an absence of UX or even, yeah, not a really good opinion about UX, that it’s not really needed. And step five or status stage five is, UX is part of a global strategy. It’s internalized by the teams, it’s part of the development process.

So how did UX research look like two years ago from now, at FlixBus? There have been some activities, of course, there have been designers, but it has mainly seen and understood as aesthetics, as I just said. Something like, “Let’s build it now, we test it later. We have to release it now, we don’t have time to test it. We fix it.” Wow, okay, in the end, that’s not really the way to go. And I can especially tell you because I’ve been working as a PO for so many years, so I realized at a certain point, it has to be different. So, in our team, we had to create more awareness in order to adopt UX as a process within the product development and the whole organization.

So, how did we enable the process within around one and a half years? We started asking questions in the beginning, in each step or in a lot of steps of product development. Kind of, “What do our users do? What are their pains, what are their gains? How did you solve the problem? Can they use the product? Do they need it?”

Another thing we did was we initiated on-site tests, we shared the findings, we invited developers, product owners, other people from the companies to join as observers. And we started with design sprints because design sprints are a very powerful method. You have a cross-functional team, so you invite a lot of different people. You take a challenge, and you have this collaborative method to create a shared understanding across this team or across-functional team. And it’s really important for them then to see and experience this powerful method. And in the end, on the fifth day of a design sprint, you test the solution. You really see the customer feedback and all of the people who have been included, worked on this challenge, and on the solution.

So, and then to scale, last year mainly, we hired more researchers because at a certain point of time, one or two researchers in our company haven’t been enough. So, last year, we hired, and we hired with a focus on different backgrounds. For instance, I have a product owner background, Pietro has a computer science background, we have someone in the team with a design background, we have another team member with marketing background. And this enables us to basically swarm. As our name says, we are Team Swarm and we market us as Team Swarm to really serve all the different areas.

And today, we have a few things. We want to share with you what we have learned along our way, to become more mature and to spread UX. So the first thing we learned is it makes sense to divide and focus, in terms of research. And our thing, FlixBus, is a growing organization and a very fast-paced organization as well. So we had to kind of define our own working guidelines. So to split and divide for two reasons: one, to be more efficient, less context, which is in the research team. Second reason: to build strong relationships with the product teams and to establish trust, because, in the end, it’s about trust.

And as I said, our name is Swarm, we try to swarm out, to support the other teams. The other team’s name is Hive. And yeah, we, together, kind of help driving UX. So in our daily business, we split and closely collaborate with designers and POs of the product teams. And we, as a team, update each other at our dailies. And as you can see, we have many different domains of FlixBus, and we help our self-organized Scrum product teams with our research. So, second one, we learned it’s very important to have workflows and a good collaboration in place. As you can see here, we work with Scrum in our research team, as also our product teams.

So, we understand their workflows, they understand our workflows, and we have kind of adjusted the Scrum workflow to our needs. So for instance, we are joining their reviews, they are joining our reviews, we invite the people. The researcher is very important too. And, yeah, so far it works quite well. And we also set the focus in these two weeks, on… Yeah, we set it differently because we have to focus in these two weeks time frame.

Furthermore, we align with POs and the road maps. We align on important, quarterly defined epics initiatives, and to not lose the focus basically. And we work with all departments and all teams. We started 2017 with web applications, mobile applications, and sites, as we’ve been just one to two people. Then in 2018, we scaled the research and tested more products and services along the customer journey, in 2019, even more, in terms of marketing and branding. And 2019, for us, is also the topic to spread UX, to make the teams enabled to run their own research, to basically, democratize UX.

Number three is about shared goals. Of course, it’s very helpful to spread UX outside a team if you have a shared goal. Our vision at Flix is smart and green mobility, to experience the world. So our mission in the FlixTech department is that we sustainably deliver the highest customer value to innovate use of technology. And we defined, as a collaborative method, basically, in our FlixTech department, [inaudible 00:12:58] goals. [inaudible 00:12:59] goals we know won’t be achieved tomorrow. But, as the name says, they are our overall goals. So we want to lead the mobility industry in holistic user-centric product development, and kind of this serves as a foundation for all tech teams.

Okay. Point four is about transparency. Transparency is very essential because it’s important to reduce confusion or prejudices regarding UX research. And it’s very important to us to know that everyone knows what we are doing, what are the goals of the research? What’s the topic? Everyone connects as is on Confluence or a Google Site. And we spend all of our findings, as well as our mindset, on different levels in the company. Kind of, we join different kind of meetings, as you can see here, with a higher frequency on a product and stakeholder level, and on a less frequency, on a company and management level. Also, the level of details are different.

So, all in all, our goal is to make research findings accessible to everyone in the organization, to support data-driven decisions, increase the UX majority, and our overall plan is also to create a platform for all our research findings, for also, all our quantitative and qualitative data and kind of have a single source of truth. Right now, we have a lot of our findings in Confluence, but some people don’t use the data often or don’t really have access. New people join the company, Confluence gets messy. I think some of you know that as well. So it would be nice to have something, overall, that’s easy to use.

Okay. So, creating some noise is another very essential part of it. We really learned that it’s important to share our findings right away to help creating awareness. We have a Slack channel, which is highly being used. As you can see here, we share our presentations, usually, it’s Google Slides, quick summary. And we start conversations, discussion, with the stakeholders or other people involved. It works quite well, and everyone can join, from the company, the Slack channel. Furthermore, we use our external blog to promote what we do. Also to maybe new joiners or people who would like to work for the company, we use our Flixnet, which is our intranet, and we also post, once in a while, on our social media Yammer, interesting findings, some fun pictures about interesting studies we did. And overall, we host meetups. We try to do that. We have a certain area in our building, in Berlin, as well as in Munich, and we talk at conferences or meetups if possible.

So the last but not the least, as we heard today as well, is about sharing knowledge. So, what we do in terms of sharing knowledge? As well, we’d like to do workshops and training because we know we can only enable others if we spread what we already found out. So we offered info, how to run research. We host workshops, as for instance, last year, empathy map. We then did that with one product team, and in the end, we hang it on the wall so everyone can see what, for instance, Sofia thinks, does, feels, and says. And last year, for instance, I did the Lean Service Creation Workshop with our PO community, just to try out different UX methods, also in terms of what can help product owners to become more user-centric, to include UX methods, and in the end, develop products customers like and would love, hopefully. Yeah, now it’s Pietro’s turn.

Pietro: Thank you, Katja. So, now I’m gonna talk about what works well at FlixBus. First of all, we found that the most effective way to get people to understand user-centricity and our UX work is to get them involved directly into the user research sessions. I think some of you might know about this, like the feeling you get when you watch your users use the product for the first time or when you just watch your customers struggle with your service. And this is quite eye-opening, I would say.

So how do we do this? We have different types of research methods across the company. User lab is how we call our standard usability studies, so one-on-one in-depth interviews, we think aloud. And usually, we have one moderator from our own team, so a researcher. And as another survey, we invite someone from the product team or from the project team. So could be a PO or a developer or a designer or someone working on the project.

Then less often, we conduct focus groups which we use to collect different types of feedback and point of views on a specific topic. And in this case, we might want to get them police feedback as well, so we invite the police as participants. And sometimes we also conduct guerrilla testing usually at the very early stages of the design process. So it would be useful to get the designs involved at this stage, where we just go to the station and approach people to collect quick and dirty feedback. We also like to experiment with new formats. This is something that we call UX Croissant because we get breakfast and then we invited… We tested this week, new employees who just joined the company. We did some kind of speed dating style user labs. So we invited them to test multiple topics for a really short time. So every 15 minutes they would swap table and go onto a different topic with a different researcher. And the next step for this would be, since it worked pretty well, to get the POs and the designers to also join as moderator of this session.

And finally, Katja already mentioned, we conduct the science prints, which are essentially five days collaborative workshops, and we put together a cross-functional team, so made again with POs, developers, designers, and researchers, usually to tackle a really big or complex challenge or a problem that needs to come already to some kind of solution or prototype that is tested on the last day of this five-day sprint. So the outcome of this would be an understanding of this challenge or this problem, through a prototype or a solution that’s already tested with users.

Now I’m gonna show you just a few showcases from user studies that we conducted. And these are pretty different from each other. I would start with a basic usability study that we conducted for our website, the navigation menu for our German website, flixbus.de. So you can see that already… This was just one month ago. We had kind of a more complex, and I would say, messy navigation, with 28 menu items in our drop-down menu. And now through user research, we managed to clean it up and reduce number items to 11.

So, how do we do this? Well, the content management team was the one that takes care of navigation. But we had actually more teams involved in this because already before this user study had been conducted on the website, we had some requests from other teams, such as strategic operations and customer service, for user research related to other topics that were not specifically related to navigation. But we found through this user studies that customers were struggling with navigation, they were struggling to get in touch with customer service. And we brought these insights to the content management team in order to show them that there was a problem. And then through user team, we conducted more studies, specifically, on navigation, using a tree test. And we found… Well, we didn’t just validate that our customers were struggling, we actually showed that they were really confident while failing a task. So they were really confident while failing to find what they were looking for, as you can see in this lower quadrant on the right.

So in combination with this test that we conducted and more test from the analytics team, we managed to get to the new improved version of the website. Moving on to a second study, this is quite interesting. It’s called Tip the Driver, and it’s a feature that would allow… It’s not live yet. But it would allow our customers to leave a tip to the bus drivers through the smartphone. And this is interesting because there were a lot of different product teams involved in these projects, specifically, the payment team, the two mobile teams, Android and iOS, and also the driver app team because this is a feature that would be used, not just by our customers, but also by our drivers. And these teams came to us, and they had just no idea what this feature should look like or how it should behave. So it was really an exploratory study for us. And the aim of this was to collect requirements for the product teams in order to create this feature, to have them make decisions in their development process.

So we conducted quite a few studies for this one. Actually, we have more studies going on right now. There is an ongoing project. But we started with guerilla testing, and again, we went to the bus station in Alexanderplatz in Berlin. And we started approaching people who were waiting for the bus or they just drop off from their FlixBus. And we were basically just commenting live time on Slack. Just creating a discussion, saying, “Oh, we talked to a French person, they don’t like tipping.” This was something that we found out. And then from this, we… Is any of you French? Can you confirm? I mean, the first finding from this would be that we shouldn’t roll out to France, but we decided to kind of focus on different types of markets for the US and Germany, where people were a bit more welcoming of this feature, to say so. And we conducted more remote, moderated studies with UserZoom and also Userlabs. For these ones, we tried to have the POs from the three customer-facing teams, so payment, iOS and Android, as a service for the sessions we did with the customers in the office. And then we had the PO for the driver app, who joined the session to be conducted remotely with a bus driver from a competitor company in the US.

One last study. This is really interesting because it came from the travel experience team, which does not deal with digital products. They actually work on the offline products that you find on the buses. So this is for a self-check-in machine, the aim of which would be to reduce chaos at boarding. So the situation right now is that the bus driver has a lot of responsibility when customers board into the bus. The bus driver has to take care of their luggage, but also to check them into the bus, and it can get quite chaotic. So we wanted to find out whether the self-check-in machine would help.

Again, quite a lot of different teams involved in this one: travel experience, strategic operations, driver app, once again. Our own team of UX, UI, we had one designer working on the interface for this machine, and we also had the machine manufactures, who are not a part of FlixBus. It was an external company, but we still invited them to join the user study to see how our customers would interact with their machine.

So how did we test this? We tried to make it tangible, and we created what I like to call, an experienced prototype, basically a fake bus out of benches and beer crates. We printed fake tickets, fake documents. We gave the participants some luggage, and we literally asked them to board into the bus. So, they had to come into the bus. We split them. We had 14 participants, 6 employees, and 7 customers. And we asked them to board into what we were two scenarios, for domestic and cross-border trips. For cross border, they will need to have a valid traveling document, which some of them did not have. And yeah, we also had an intern acting as a bus driver because we wanted to know whether a customer would still need to rely on the bus driver while boarding on the bus. So, after doing that, after boarding in… This is the insides of the bus. They would have to fill out this individual questionnaire, and then we split them together in two groups, one with the employees and one with the customers, to conduct focus groups.

As I said, this was interesting for us because when you conduct a focus group with the employees, they have a completely different point of view about the topic. And in this case, we did not test it with just employees from Flixtech, but we had people working in the ticket shops or with the bus partners. So they all had more insights to contribute too. And also, I must say, this was the one study that brought us a lot of publicity. A lot of people were talking about it. They were writing articles, writing on Yammer and Slack, and in general, talking about it. So people got to know our team quite a lot because of this study.

So just a quick recap of all the things that Katja and I went through regarding how we try to democratize UX at FlixBus. First of all, we are aware of our own maturity level, UX maturity level. We always try to aim at getting to the next stage. We set a clear focus and device strategy for our own team in order to better support all the other departments, all the other teams in the company since we have so many different products and services. Our shared goal is user-centricity, which means that we think of ourselves as advocates for the users. We work in Agile, so we always have this fast-paced, always-changing environments that we need to adapt to. And we rely heavily on transparency and accessibility. We always try to make our own methodologies to explain them as much as possible and to make our results accessible to everyone in the company. Even if they don’t have Slack, even if they lack Confluence, everyone should be able to find the fruits of our work.

We try to make noise. We’ve been really pushing, especially in the beginning because not a lot of people knew only 26 months ago. And I think we managed to grow a lot since then, by pushing a lot, for telling people what UX is, what we do, and why it’s important. And finally, we want to share our knowledge to enable others. We don’t want to be the only ones conducting research. We want people to maybe do their own research study and then come to us just for feedback.