Borderlands 3 User Research: Old and new approaches in franchise testing | Jonathan Cohen

00:14 - So a little bit about myself– Again, hi, I’m Jon Cohen.

00:16 - I am the user research manager at Gearbox Entertainment, down in Frisco, Texas just a little north of Dallas. And a little bit about myself. I am a big tabletop game player.

00:25 - However, over the past year or so, I’ve been doing a lot less of that and a bit more of this, as we all learn to navigate gaming online.

00:33 - So with this talk, what I’d like to do is speak a bit about the use of research that we conducted for Borderlands 3 during its development, as well as the talk implies, we’re visiting both old methods  and talking about new approaches to testing.

00:51 - So quick outline of the talk. We’re going to start by discussing user research and user research team at Gearbox, then we’ll give you a quick overview of the Borderlands franchise.

00:59 - Then we’re going to dive into some case studies, looking at both established methods that we’ve used from the past and testing this franchise, as well as the new approaches we’ve integrated into our testing portfolio.

01:10 - And then, we’ll end with some takeaways and looking ahead, both back and ahead I suppose now, at how things have been working for the user research team  since we started working in a remote environment.

01:22 - So the goals for this talk are really about methods.

01:25 - This is a discussion of combining methods things that have worked for the user research group in the past; the established methodologies we’ve relied on, as well as the new approaches that we’ve begun incorporating into our testing based on inspiration from academia, literature etc.

01:42 - Now to do this we’re going to illustrate these methods with Borderlands 3 case studies, and we split this up into three case studies looking at established methods, and three case studies looking at new approaches.

01:55 - So I do want to talk for a moment about the takeaways that I have for you folks in the audience at the outset of this talk, just so that it’s in the back of your mind, helping to frame and improving on your established methods.

02:06 - The first is always continue iterating and improving on established methods.

02:09 - There’s always more that you can learn, more that you can incorporate, things you can refine.

02:14 - Second thing is look to existing research outside of games for inspiration for user testing in games.

02:20 - And the third, you know, incorporate these new methods when it’s appropriate to answer questions that you have, that your development teams have, and so forth.

02:29 - All right, let’s now talk about Borderlands 3 user research; the user research team, the project, and kind of some introductory overview material before we dive into those case studies.

02:40 - So whenever I give a presentation to the company, let’s say a quarterly meeting, I always like to start by discussing the mission statement of the user research team, just to help remind everybody what it is we’re doing.

02:50 - So that mission statement is to understand the user’s experience with all of our products.

02:54 - And to do this, we apply hypothesis-driven methods and research best practices, like all of you do.

03:01 - This, of course, is to inform development and publishing efforts, and to help deliver the best possible experience to our customers at the end of the day.

03:10 - Now the user research team covers a lot for all of our projects.

03:15 - The bread and butter of what we do is really playtesting; we also focus on usability, we do some A/B testing, some kind of low-tech paper tests, and card sorting tasks.

03:23 - We also work very closely with Gearbox publishing and our marketing groups when helping to evaluate potential projects, and also looking at how projects are landing after they’ve gone out the door.

03:36 - Now we tend to break our testing down into iterative testing and evaluative testing.

03:42 - Iterative testing is the testing we tend to do earlier in a project that is very focused around specific objectives, or features mechanics; so for example, looking at a specific level or a specific character, or a usability issue within a menu.

03:57 - We do this under very controlled conditions with our participants.

03:59 - This is much more along the lines of laboratory testing.

04:03 - Now the customers of these data are the designers and developers themselves. So we work very closely with individual producers and designers on the teams to make sure that we’re answering those targeted questions.

04:13 - Now as production continues and more content becomes available to test, we kind of release the reins a bit on some of these controlled aspects of our testing, where we get to evaluative testing toward the end of a project.

04:26 - Now this testing looks at full playthroughs, more natural gameplay with fewer controlled variables, and toward the end of a project, we hire what we like to call  ‘playtest interns’ to provide us with a lot of longitudinal and deep data.

04:39 - The customers of our evaluative tests not only include the producers  and developers of the project, but also the executive team, publishing when it’s appropriate, the marketing groups, folks that are our external publishers; So the audience begins to grow a little bit as the scope of the testing increases as well.

04:59 - The team consists of three of us; Kyle Beasley, Michelle Garza, and myself. These are our pub emojis as they are called; fantastic artwork by our marketing art group at Gearbox.

05:10 - And the user research facility in Frisco consists of a fantastic lobby space, a couple of really cool lab spaces, a great group discussion room, all of which are equipped for streaming all of our gameplay footage and the participants’ video out to the studio at large; I’ll talk a little bit more about that later.

05:31 - I very much miss this place; it’s been a long time since we’ve been in the lab.

05:35 - I hope to get back there at some point soon.

05:39 - Those chairs in the lab, I have been told, are the most comfortable chairs in the entire studio, by many of our developers who come down to observe and participate in internal testing.

05:50 - So as I mentioned, we’ve been working from home, like all of you, since last March.

05:55 - We have been doing remote testing since then with developers, working with our devs as participants and, of course, the team has been providing team reviews and expert reviews on content for all of our projects.

06:07 - And you know, we’re going to get back to the studio one day; we’re looking at developing plans right now about getting back.

06:15 - However, this talk is about the before times; in fact, years ago, during the development of Borderlands 3. So let’s talk about the Borderlands franchise.

06:26 - What is Borderlands? Borderlands is a looter shooter.

06:31 - So this is a game that combines first-person shooter action and gameplay with roleplaying game mechanics, for as far as leveling skills, abilities, classes go.

06:42 - So it combines staples of both genres. It’s set in a post-apocalyptic, sci-fi, fantasy universe.

06:50 - It’s got a huge array of characters. It’s got wonderful, deep RPG mechanics, as well as a very dark sense of humor.

07:04 - So Borderlands first came out in 2009, followed by a sequel in 2012.

07:08 - And in 2014, we released the pre-sequel, a game taking place in between Borderlands 1 and 2.

07:14 - And our collaborators over at Telltale Games released Tales from the Borderlands, a narrative-driven game set within the universe.

07:23 - And then Borderlands 3 came out almost two years ago, now in 2019.

07:29 - The Borderlands franchise has a number of staples that tend to appear in each one of the games in the franchise.

07:35 - For example, our friendly, yellow robot Claptrap, who kind of serves as a mascot for the franchise, along with some of our other characters like our Sirens; Lilith and Maya, in this case.

07:44 - The Sirens are characters who have almost supernatural magical powers within the universe, and tend to function as protagonists in the game.

07:53 - Guns. Lots and lots of guns. Bajillions of guns, even.

07:57 - Borderlands is a game known for the number of guns in the different permutations, and a variety of guns that are available.

08:02 - It is a game that has guns built based on procedurally generated algorithms that are done in real-time as you play the game.

08:09 - This is a big feature of the game, and players always look forward to seeing what new guns each new version of the game has.

08:15 - And, of course, we also have… we also feature plenty of large monsters like the Warrior, in this case, to use said guns on.

08:24 - There’s also plenty of high octane, frenetic action if it’s a first-person shooter with tons of guns.

08:30 - There are pretty deep RPG mechanics, in this case, skill trees that are also featured, of course, on many other RPGs and other games and genres.

08:37 - And to coincide with all of the guns, you know, we have loot.

08:42 - We have the loot chase, typically in the form of guns, but also other items like shields, grenade mods, class modifications, etc.

08:48 - But the loot chase is a big component of the kind of core gameplay loop and aspects of playing Borderlands.

08:56 - Now in Borderlands 3, we introduced a number of  new features and mechanics to the franchise, and we also innovated or iterated upon others that previously existed.

09:04 - So for example, we introduced new forms of movement, like mantling or sliding as shown here, and we’ll talk a little bit more about this in a bit when we get to our case studies.

09:12 - We also changed how the action skill system worked, and the associated skill trees with those abilities.

09:17 - Action skills are these unique, kind of character-specific abilities in the Borderlands franchise.

09:23 - And we also traveled to new worlds for the first time.

09:26 - Borderlands 1, 2, and the pre-sequel all really took place in and around the world of pandora.

09:30 - And for the first time, we traveled via spaceship to worlds like Eden-6 with its dinosaur… fauna.

09:37 - And the city world; the sci-fi city world of Promethea, with the Maliwan robot army as primary antagonists.

09:48 - Now when we started testing Borderlands 3 pretty early in development, we began by focusing on characters, and concept art, and story elements; We then shifted to looking at usability, both in terms of in-game usability issues, as well as menu flow, and front-end work.

10:04 - And then we started focusing on individual features.

10:07 - This is really part of that iterative playtesting that I mentioned before.

10:10 - And as we continued into production, we started with content focus playtesting quite a bit.

10:15 - And I’ll break this down a little bit to talk about the general flow for a playtest in our lab, but this tended to focus on individual maps, missions, characters, those action skills that I mentioned before.

10:27 - And then towards the end of development, when we reached a critical point where there was plenty of content to test, we started running full playthroughs with playtesters that we call our ‘playtest interns,’ and I’ll discuss those in more detail later in the talk.

10:43 - So there represented a number of interesting franchise challenges when we started testing Borderlands 3.

10:48 - The biggest, of course, is how do you introduce new players to this game in this franchise without fatiguing returning players? So you want to make sure that you teach what you need to teach to the new folks, but if people are coming to this franchise, perhaps after, you know, just finishing another playthrough of Borderlands 2, you don’t want to bore or fatigue those players.

11:07 - But you have to teach the new mechanics, or you have to teach the mechanics that have changed since the last time people have played to both audiences equally.

11:15 - Now this is really critical because  the baseline differences in expectation and comprehension between the folks who are coming to Borderlands 3 as their first Borderlands experience, and the people who have been with us since 2009 are drastically different.

11:28 - And of course, to do this, we employed a number of different methods.

11:32 - So getting back to the original point of the talk, this is really about established tried and true methods that worked for us in the past, as well as the ways in which we incorporated new methods into our approaches to testing Borderlands 3 to tackle these unique issues and topics.

11:47 - So returning briefly to the talk goals again, this is about combining methods; what worked before, and what else is out there that we could potentially capitalize on? And in a moment, we’ll start talking through those Borderlands case studies, three with the established methods, and three with new approaches.

12:04 - So, what are the case studies? First up, we’re going to talk about three case studies involving a revisitation and iteration on old methodologies.

12:13 - By old methodology, again, I mean the methods that we rely upon day in and day out when testing our products.

12:20 - So these three case studies are player movement tutorials, action skills, and the mechanic called Second Wind.

12:27 - After that, we’re going to turn over to innovating new approaches, so three case studies that involve taking approaches that were not commonplace for our laboratory, and using them to answer specific questions that the development teams had; and these involve gun audio, navigation through the hub environment, and some interesting narrative insights.

12:47 - So, revisiting old methods. As I mentioned before, the bulk of what we do is playtesting.

12:54 - And I want to take a moment and talk through what those old or traditional or established methods tend to look like for us.

13:01 - So when testing Borderlands 3, we drew upon a number of mixed methods and data sources, and I’m sure, you know, folks who are watching other talks during the course of the summit, will encounter these terms quite a bit.

13:13 - But these are the methods that we employ  and the data sources that we that we tend to look at and take advantage of.

13:18 - So the biggest two at the top: observational data and survey feedback.

13:22 - When we are testing our products in our lab, we are constantly watching what our players are doing, and taking notes, making observations using those data to inform inferences about their behavior, use it as a prompt for follow-up questions during group discussions or interviews, and ways to supplement the qualitative feedback  we get from surveys.

13:43 - We utilize a series of survey tools that are often either prompted by moments in gameplay or benchmarks during a playtest or sometimes they’re self-motivated where we train our participants how to provide survey feedback whenever they encounter something that merits it.

14:02 - Now in addition to these survey tools or interviews and discussions that are either conducted individually or in groups, we often look at task metrics, often in the form of video analysis.

14:13 - So we may give a participant a series of tasks to perform and after the fact, post hoc, we can look at the video, code it, and analyze the metrics associated with success or failure or time to completion, and so on and so forth.

14:25 - And when we have the opportunity to look at game telemetry, we like to because it provides really great objective feedback and often quantitative measurements that provide an additional vector of data that can support a lot of the qualitative feedback we’re getting from our participants.

14:40 - Now the other part of this is that we broadcast all of our playtests from our lab to, not only our control space where we can observe this directly– getting that observational data– but also to the entire studio at large that all of our developers can watch in real-time what our participants are doing.

14:57 - So we have utilized a great service that our IT department has created for us to broadcast every single player’s gameplay footage, and their face in a little picture and window, just like you’re seeing right now with me, so that they can observe what our participants are doing in the lab.

15:14 - We also utilize Slack and Microsoft teams to communicate with our developers in real-time, so they can provide observations themselves because the more eyes, the better.

15:24 - They can also ask us questions, and prompt us during group discussions because we also broadcast that.

15:29 - So those two approaches, this mix method with multiple, both qualitative and quantitative data sources, as well as this, you know, broad communication to the studio at large, is the way in which we tend to approach playtesting.

15:44 - Now this diagram here is something I communicate out to our developers during all of our quarterly meetings.

15:50 - This showcases how everybody at the studio Gearbox is a part of user research.

15:55 - We use this as a way of communicating that we really need developers to get involved before, during, and after a playtest, to kind of help that close that feedback loop.

16:04 - But this reinforces the idea that they should be taking part by observing, by asking questions, by getting involved through our streaming tech, as well as through Slack or Teams.

16:16 - Now the average participant, in the before times…

16:21 - It’s been a while. We’ll enter into our user research facility via the lobby where we will onboard them, they’ll sign a non-disclosure agreement, we’ll talk a little bit about physical security, informational security, and then we’ll bring everybody involved in the playtest into our group discussion room.

16:36 - Now we start our playtest here because it gives us an opportunity A) to have people store their belongings in lockers, which are just offscreen in that image, but it also gives us a chance to talk more in-depth about the non-disclosure agreement, and we can also discuss the methodology of the day.

16:50 - So we can discuss the procedure of the game that we’re playing, if we want to tell them ahead of time– sometimes we keep that secret and we also can go into the kinds of feedback we’re looking for, the kinds of feedback we’re not looking for, the ways in which they’ll be providing feedback, so we can set up expectations accordingly, and we can answer any questions to level set all the participants before we bring them in groups into our two different lab testing rooms.

17:15 - So here’s an image, of course, of Lab 1. Lab 2 is identical to it, just farther down the hallway.

17:20 - Each one can support up to eight participants at once.

17:22 - We have PCs and consoles in each one. And players, you know, utilize headsets so we minimize noise disruption.

17:29 - And whether we’re doing multiplayer testing  or singleplayer testing of any kind, we will often fill up both rooms to increase our sample size.

17:38 - So participants will engage in whatever the test is of the day; it could be a 15-minute menu analysis, it could be a four-hour game playtest, it could be a three-day test where at the end of each day, participants go home and the expectation is they’re going to come back the next day to continue their gameplay.

17:52 - Most of the time, they show up again, which is great.

17:54 - Participants are rewarded with swag of various kinds; things that they can only get through playtesting with us.

18:00 - And we wrap our playtest back in that group discussion room, principally because we want to run a group discussion, and you can see the big screen TV, and that picture there is the webcam that we use to broadcast the group discussion out to the rest of the studio.

18:14 - Sometimes we’ll bring individual participants in one at a time to do, sorry, to conduct interviews with them, rather than running a full group discussion.

18:22 - And on occasion, we’ll also have developers visit the lab and take part in group discussion; you can see a couple of couches on screen there, so the beginning and the end of each playtest happens in that room, with the actual playtesting itself tends to occur in our lab rooms.

18:37 - That just gives you a quick showcase through the various spaces in the user research lab.

18:41 - And I really am looking forward to getting back there.

18:45 - All right, let’s talk case studies. This is the meat and bones of the talk.

18:48 - So let’s first talk about player movement tutorials.

18:53 - So just so keeping in mind all the old methods, that tried and true stuff, we just talked about.

18:59 - Let’s dive right in. So I mentioned earlier that we introduced some new movement capabilities in Borderlands 3; things like sliding and mantling. So here’s an image of sliding.

19:10 - We also brought something back from Borderlands: The Pre-Sequel called the ground slam, or the butt stomp, where a player can jump from a very tall height and hit the crouch button and slam  down, make a melee attack into the ground doing some area of effect damage.

19:25 - Now these were with the exception of the ground slam, sliding and mantling, you know,  are brand new to the franchise.

19:30 - So we conducted a number of playtests very early in development, to kind of  getting into production where we took some observational notes  and we got a bunch of survey feedback that indicated that while there was  a tutorial present in the very first mission of the game,  where players received these brief prompts to mantle into slide,  things were not reinforced.

19:49 - After they received those prompts the first time, they never appeared again,  and there was nothing that was reinforcing or requiring that gameplay to really, you know,  teach the fact that these were new mechanics and new mobility features in the game.

20:03 - So we actually found through observation that players had difficulty mantling up  almost required, golden path places multiple hours into the game.

20:13 - And of course this is problematic if  players actually have to mantle later on and they just don’t know how to do it.

20:18 - So the solution here from the development team is quite elegant.

20:21 - They actually built some more content for us.

20:23 - They developed a brand new side mission that helped with additional training providing reinforcement and reward.

20:28 - So what was the side mission? Let’s call it Bad Reception.

20:31 - It’s available very early in the game  right after you complete the kind of core tutorial area in the first boss fight.

20:38 - And in fact it requires new movement, mantling, sliding, ground slam, in order to complete objectives.

20:44 - In this case here you have to ground pound  or butt stomp this specific location in order to complete the objective.

20:51 - And this provides reinforcement of course. So players understand “oh I actually have to do this in order to complete  certain things in the game,“  so they’re more likely to engage in it moving forward.

21:00 - And of course like all good Borderlands missions, it provides you with a reward at the end.

21:03 - Hey look it’s Claptrap! He gives you the mission, he wants a bunch of weird hats, and in order to get them, you  have to mantel, and slide, and butt stomp, and he gives you some cash at the end.

21:13 - So, great side mission, provided that reinforcement reward, and what we found  is with additional reinforcement we had greater engagement.

21:22 - So interestingly enough, the novelty of these new movement features was not enough for our returning franchise fans to engage in that content.

21:29 - One might think that, ‘Oh the presence of mantling or sliding for the very first time in a Borderlands game would really capture the attention of those returning fans. ’ Wasn’t the case. So in fact, we had to reinforce that.

21:41 - So observational player reported data in the form of surveys, in interviews and group discussions as I mentioned before, revealed this issue and helped inform a brand new mission.

21:51 - All right, moving on. Second case study. Now let’s talk about action skills, those unique abilities.

21:56 - So again, these are franchise staple. They are special abilities unique to each character, and they look like something like this.

22:03 - So for example, you might be able to use six powerful Siren arms to slam into the ground causing an area of effect attack.

22:11 - You might be able to deploy a drone or a shield or even duplicate yourself as Zane the Operative.

22:17 - They often help define the nature of a character and they can really shape and inform gameplay.

22:22 - They provide that kind of cornerstone to a player’s strategy, and they can base their entire kit around a particular action skill, instead of abilities.

22:34 - Now in Borderlands 1,2, and the pre-sequel, each character, each playable character had one action skill available to them. So one character had their action skill that was unique to them.

22:46 - In Borderlands 3, there are three action skills per character. So we increased complexity quite a bit, and not only that we introduced additional augmentations and attunements that were available to each action skill that could change the way an action skill behaved.

23:05 - So a bunch of complexity from two different factors.

23:09 - So what we found from a whole lot of playtesting, we’re talking playtesting from individual missions, even up through full playthroughs, that while players really understood their passive skills, these abilities that were always functional once you place skill points into them, they understood how those worked arguably because they were the returning piece from the franchise.

23:31 - Every other Borderlands game has passive skills.

23:33 - Players understand you put points into them, they augment you somehow.

23:37 - Typically numbers go up, you do more damage, your reload speed increases, you move faster, you heal faster.

23:44 - And you know, they’re also common in many other games, so it stands to reason that this was the piece that made sense.

23:51 - What didn’t make sense to those players were how the action skills worked.

23:55 - Mainly because it required you to not only pick one, but slot one into a loadout.

24:02 - Now this was a brand new step that the previous games did not require.

24:06 - Because you only had one action skill, as soon as you gained access to it, it was there, it was yours. You didn’t have to do anything with it.

24:13 - Well now when you have a choice of three, you need to pick which one you want and toss it into your loadout. There’s an additional step required to even make it function.

24:22 - And as far as those augments go, the different ways you can modify an action skill, players had no clue how to unlock them, they had no clue how to slot them in, they had no idea what they did. It was a black box.

24:36 - So the solution here, thanks to our fantastic UI and UX groups, were some new tutorials that walked players through how to handle and approach each of these pieces of the action skill system. Now at first, they built tutorials that unloaded all this information onto the player at once.

24:54 - Unsurprisingly, this was a bit too much. Now when players unlock their action skill, it’s actually up to an hour or more until they get their first passive and potentially even longer until they get their first augment or what we call a wing skill.

25:07 - We’ll show you why in a moment. So version 2 transformed these and parsed them out into contextual, almost benchmark sensitive tutorials. So a separate tutorial for the action skills, the passive, and the augments respectively.

25:22 - Well let me show you what I mean by this. So when you reach level two, you unlock your action skill and get this great notification.

25:29 - And from there, the game tells you a little bit about your skills and what action skills do. This is character specific, here we’re looking at the siren Amara. And when you enter into your skill screen for the first time, it walks you through how to equip an action skill. It tells you how to pick one of the three, and equip it in your loadout. And then once you do it, hey! You get a great notification that says “you’ve done this. fantastic! And then, come back later when you’ve when you’ve received additional skill points so we can teach you about other things later.

” So when you come back after you get your skill point, what happens? Ah! Another tutorial pops up. So it tells you “Hey leveling up grant skill points.

26:05 - Time to spend them on your passives. “ Now this is the part the players tend to understand from previous games, so cool, you can spend some points on a passive ability.

26:13 - There we go. And here it tells you that as you progress, you’re going to unlock new action skills and augments.

26:21 - Great, so that reinforces the idea that they’re going to be more choices later on. When you return after having spent enough skill points to unlock your first Wing Skill, and here we call it a Wing Skill because they are located on the edges or the wings of each of those trees, this is where typically the augments are found; a third tutorial prompts you to equip one of those for the first time, and it will explain that you can mix and match them.

So, we drastically evolved the core feature of the game, introduced a ton of additional complexity by tripling the number of action skills available to players per character, adding in augments and Wing Skills and changing how the creation of a loadout worked.

27:04 - So we had to teach the new; the new pieces of this.

27:07 - So with the observational data we collected, indicated that our players were not interacting with these features, they had no idea how to slot their action skill, and the survey and the interview data that coincided with this indicated that they didn’t understand the feature.

27:20 - It makes sense; if you don’t understand something, it’s kind of hard to interact with it correctly.

27:24 - So this helped inform the tutorial iteration that we just went through, and of course after the tutorials were implemented, we saw that both comprehension and use of these systems improved quite a bit.

27:35 - All right, our third case study our traditional playtesting methods: Second Wind.

27:40 - What is Second Wind? Second Wind is a self-revive mechanic in Borderlands.

27:45 - So when you take enough damage to be knocked down, you’re given an opportunity to revive yourself by getting a kill on an enemy before a certain amount of time has passed.

27:55 - If you do that, you’ll revive and you’ll get a Second Wind.

27:58 - A portion of your health is restored and you’re told, ‘Hey. cool. you got a Second Wind. time to get back into combat. ’ So this is actually really core to combat and is really key to success and engagement in the core loops of a Borderlands game.

28:12 - Losing all of your health doesn’t mean you’re done, it means you need to refocus and get a kill very quickly, otherwise, you will in fact die and respawn, and have to repeat some content.

28:23 - So this is critical and core to the franchise.

28:25 - It has been a staple from the very beginning.

28:28 - Now knowing that, let’s talk about a very specific case where Second Wind was not being done the way that the developers intended it.

28:38 - So, the Rampager. What’s the Rampager? The Rampage was a big boss fight. Big vault monster.

28:43 - The Rampager is encountered at the end of the very first act of the game.

28:47 - It’s a HUGE boss fight; typically these vault monsters in previous games, come at the very end of the game.

28:52 - Now Borderlands 3, just like it increased the number of action skills available to each character, also increase the number of vault monster fights that the player encountered.

29:01 - And the Rampager is unique as well because it has multiple phases that it enters and it physically transforms; its abilities change at some point it begins to fly.

29:10 - Well, floating around the Rampager are these luminescent birds.

29:17 - Now the intention for these luminescent bird-like creatures was that they’re supposed to be used as a Second Wind because they can be shot and killed very quickly.

29:26 - So again, if a player loses all of their health, all they need to do is look up and fire out one of these birds and they can get that Second Wind; they keep fighting the Rampager.

29:35 - Well we found that players weren’t using birds that Second Wind; time after time, we would watch a player get to the Rampager, and watch with bated breath as we hoped that this player  would actually look up and use those birds, and time again, we found that that was not the case.

29:53 - There were all the other ground-based enemies that were available for players to focus their fire on and get Second Wind, which they did, however, those required multiple shots, multiple rounds, whereas the birds would take one shot; they were the easy pickings for a Second Wind. What the heck was going on? We inferred that they were not connecting those birds to combat; they weren’t considering those birds to be valid targets to get Second Wind off of.

30:20 - This may have been complicated by the fact that during the Rampagers phase changes where it physically transforms, it enters into this invulnerable state and an animation plays where it draws in these birds like it’s somehow powering up using them.

30:33 - So we surmised that players were associating the birds with that phase change and not being available as a Second Wind.

30:42 - Well, what do the developers do to address this? This is one of my favorite examples.

30:48 - So we showed them how to do this by example; this is great.

30:53 - So in this section of the game on Promethea where you are racing to the vault and eventually you encounter the Rampager, you are accompanied by an NPC, Maya.

31:05 - Now Maya was one of our Sirens. She was a playable character in Borderlands 2, returned as an NPC in Borderlands 3; much beloved character in the franchise.

31:14 - Maya accompanies you during this massive stretch of vehicle combat, and racing through tunnels, and fighting enemies, and then she’s there with you during the Rampager fight.

31:22 - So the developers said, ‘Okay, we have something that we can use to show the players that these bird creatures can be shot at, and that they are functionally, you know, available as targets for Second Wind.

31:35 - So right before the player jumps down a chute and enters into the boss arena space where they fight the Rampager, they enter into a brand new map and you spawn into this map looking straight ahead at Maya.

31:48 - So what the developers did is they added some of these birds, and they had Maya shoot them right in front of you.

31:55 - Your attention is not drawn elsewhere, there are no enemies, you’re looking straight ahead, there’s nothing threatening you, and most of the time, you’re going to be able to see Maya firing at these birds, and in fact, when she shoots them, loot drops out.

32:08 - And that if nothing else, is the telltale sign that, ‘Hey these things are available as targets. ’ to the player.

32:16 - So after that change was implemented, we saw both through observational data, which we later confirmed with direct questioning of our players that ‘Yes, they understood that these creatures could be used for gaining Second Wind during the fight with a Rampager. ’ So these data helped inform a small change that had a huge impact on the player.

32:37 - All right, so where are we so far? we’ve talked about three case studies involving a revisitation of old methods, these tried and true methodologies, these playtesting methods that we utilize.

32:47 - So with player movement tutorials, we saw that the data we collected through surveys, group discussions, observational data, comments from developers that, ‘Hey, players needed additional tutorials and reinforcement, so that informed some side mission training. ’ With our action skills, we saw that the increased complexity from our action skills system in Borderlands 3 required additional contextual tutorials, and with our Second Wind mechanic, we saw that through showing by example, we could increase player awareness of how to gain a Second Wind during a critical moment with a boss fight.

33:20 - Now let’s turn our attention to the latter part of the talk, which is all about innovating with new approaches, from outside traditional user testing.

33:29 - So, again we have three case studies, and the first one involves gun audio.

33:34 - So let’s flashback a couple of years to pretty early in the development of Borderlands 3, where our audio team came to us with an interesting question about the audio system they were developing for our guns.

33:46 - Now there are robust gun systems that underlie the core mechanics of how guns are built in Borderlands games.

33:54 - These guns are constructed out of multiple parts, they have different identities, different manufacturers with different abilities, and the combinatorics of all of these different pieces lead to the sheer number of guns that players can encounter during a playthrough.

34:09 - So the audio team was working on a system that was equally robust so that there would be some distinction sound wise between all these different guns. They asked us this interesting question of, ‘Do some players like or appreciate the different sounds that our guns make?’ They wanted to understand if it was worth continuing down this road of building this robust system.

34:28 - Well, the user research team countered with, ‘Do they even perceive the difference?’, because if you can’t perceive the difference between guns, you probably aren’t going to be able to appreciate it.

34:36 - so to answer this question, we looked to a methodology called signal detection.

34:42 - With signal detection, we wanted to understand whether players could perceive the difference between two stimuli when that difference was present, but also correctly indicate when those differences were absent.

34:52 - So let’s unpack this a bit more. We took sounds from shotguns and pistols in Borderlands 3, and we played them together.

35:04 - so participants in this study listened to sounds that shotguns made and sounds pistols made without any visuals.

35:14 - And they would hear pairs of sounds. So either a pistol and a shotgun or two shotguns or two pistols and they would make a response, ‘same’ or ‘different. ’ So if the gun sounds that they heard were, in fact, different from one another and they replied, ‘different,’ well, that was a correct response; that was a hit.

35:33 - However, if the sounds were the same but they answered ‘different,’ well, that’s a false alarm.

35:37 - That’s not correctly noting when the differences are absent, and misses and correct rejections are essentially inverse of that.

35:45 - So we had participants listen to these gun sounds and make these responses to try to get an understanding of whether there was a perceptual difference on the audio level between these different types of guns.

35:55 - The results of this study indicated the answer was, ‘Yes they can actually distinguish the difference. ’ We had very high accuracy ratings amongst our participants; they could, at 100 accuracy, tell us the difference between shotgun sounds and pistol sounds, and at 97 and 94 percent respectively, the difference between two different pistol sounds or two different shotgun sounds.

36:15 - In addition to accuracy, we computed a sensitivity score, this is called d’ (d prime) within signal detection, and this is a measure of that sensitivity, so, accuracy, but also taking into account those false alarms.

36:27 - We found that there was very high sensitivity with both the pistol and the shotgun sounds; pistols were slightly higher, so in addition having a slightly higher accuracy, the people were a little bit more sensitive to the differences between pistol sounds.

36:41 - So with the study, we looked at the signal detection paradigm, which is a brand new methodology that we had never incorporated into testing Borderlands content before, to look at the perceived differences between gun sounds; those pistol sounds and shotgun sounds.

36:54 - And these data inform the continued development of this system; the audio team was thrilled to get some feedback that indicated that players could, in fact, perceive those differences.

37:04 - Our second case study involving new methods was related to the hub map in Borderlands 3.

37:11 - So Borderlands 3 has a hub called Sanctuary, or Sanctuary 3, which is a spaceship that you use to travel to different worlds  and interact with different characters.

37:19 - And this is the map that you return to very frequently as a player.

37:23 - Not only is it the means by which you travel to new worlds, but it’s also where you pick up a number of side missions, where you interact with NPCs, where you upgrade your character in various ways, so this is the space that we knew players are going to be coming back to a lot, and we wanted to provide some information and data to the design team that could help inform the layout of the space, the way it was decorated, the signage that went in place, and where those NPCs were located.

37:48 - Now for this study, we looked at various research paradigms in navigation research; specifically, how people navigate through novel environments, and encode information about those environments.

37:59 - So here, we had our players freely explore Sanctuary 3 for a period of time, typically about 10 minutes.

38:06 - We gave them a series of points of interest and NPCs that they had to visit and they indicated they had reached that location by meleeing that location, so we could demarcate that in the data, and after they finished exploring the ship and learning where all these places were, we give them a series of errands.

38:21 - Now these errands took the form of multiple points of interest that were strung together with a bit of narrative that tested their knowledge of the space and also simulated an in-game experience.

38:32 - Let me give you an example. So for example, players were told, ‘Hey, you just returned to Sanctuary 3 after finishing a mission.

38:39 - What you need to do first is go to Sir Hammerlock in his room, and turn in that mission to get your reward.

38:44 - After that, you want to go do a couple of things around the ship in any order so, do these four things in any order that you’d like.

38:52 - Go visit this NPC, talk to them. Go to the shooting range, and try out a new gun.

38:57 - Go to your personal room and customize it. And go visit Claptrap to get your next mission.

39:03 - When you’ve done all four of those tasks, return to the bridge of the ship.

39:06 - So this was the type of errand we would give players to test their understanding of where these spaces were relative to one another.

39:14 - These are the data for that errand. Now, this alluvial plot shows each player’s path through the ship; each of those gray bars you see on the running across the x-axis is an individual participant’s route through the ship as they completed each task in their errand set.

39:30 - Now across the bottom of the chart, you see the order in which the rooms are visited, so first, second, third, fourth, and fifth, and then leading back to the bridge.

39:38 - So what we saw is that, you know, the majority of participants after visiting Sir Hammerlock to turn in their mission would go to the shooting range.

39:44 - A few of them went to visit that random NPC, the one we called the Red Shirt, and one person went right to their room.

39:51 - So paths like this show us kind of the majority family of paths that existed, which helped us understand how players were relating the various locations in the ship to one another.

40:03 - On top of that, we got plenty of great qualitative feedback that told us a little bit more about where they were running into some problems navigating this hub space.

40:10 - So for example, ‘I kind of forgot where my room was. ’ Well that’s very telling.

40:14 - Another player said it was annoying they couldn’t get back up to the balcony where the ladder and Zer0’s room is to get back to the main floor.

40:21 - So this told us a little bit about how they were utilizing a navigable space from one of their task locations back up to the main deck of the ship.

40:33 - So this navigation paradigm provided some great supplemental feed- supplemental data to the feedback we were collecting from players about where they were struggling, where they ran into friction points, where they got lost.

40:46 - And these families of paths told us that some areas were, in fact, easier to find than others, which helped inform the placement of NPCs, the deco and signage, and art that went into this map.

40:58 - The images you saw in all of those screenshots were fully arted out from the final game, but in fact, this test was run before any art was added to Sanctuary 3; it was pretty much a gray box map.

41:10 - So these data were very helpful for informing how that space was decorated in the final product.

41:17 - And what was great about this is we used some existing research models that were out there in the literature to inspire how we approach this.

41:24 - It wasn’t a one-to-one direct translation, but we were very much inspired by work and navigation research.

41:31 - Now our final case study involves the narrative of Borderlands 3.

41:36 - Specifically, the comp- player’s comprehension and engagement with the plot it’s pacing the characters.

41:43 - Now at the very beginning of the talk, I talked about playtest interns.

41:46 - Now these are folks that we bring aboard when we hit a critical threshold of content.

41:51 - Now these playtest interns represent a methodology under themselves.

41:55 - We treat them very differently than playtesters who arrive for a 15-minute menu test or UX test or a four hour playtest on a given afternoon.

42:05 - These are people who are with us for one to two weeks at a time, and we focus very much on full, playthrough longitudinal feedback.

42:12 - So these folks give us consistent written feedback.

42:15 - We conduct numerous interviews and discussions with them.

42:18 - We go deep with the developers into various topics; in fact, we structure entire group discussions around single topics at times.

42:27 - And of course, we get lots and lots of summative feedback when they’re finished with their playthrough.

42:31 - And this helps us highlight and understand issues that would not come to light with, kind of, slices of the game that we would be able to playtest earlier in development.

42:43 - This is also seen as having a high value at the studio.

42:46 - We have massive, massive buy-in from the development teams, and the developers are actively involved in our playtest intern experiences.

42:54 - So here is a screenshot from when we were testing Borderlands 3 with our interns.

42:59 - We had a Slack channel with our developers, and at one point, we had 160 people from the development team who were actively involved with the playtest intern program.

43:09 - They were observing players via the streaming tech that I talked about earlier, they were communicating with us directly, asking us questions, making observations; in fact, sometimes even solving problems in real-time.

43:21 - So the narrative structure of Borderlands 3 is a prologue followed by four acts.

43:26 - and each act takes place on a different planet.

43:28 - So you start in the world of Pandora. You get your spaceship; you leave.

43:31 - You travel to a couple different worlds; eventually, come back.

43:34 - So what we’re really going to focus on right now is everything up to act two, which takes place in the world of Eden-6.

43:42 - So in the prologue and in act one, you as the player are focused on the antagonists called the Calypso Twins, Troy and Tyreene.

43:52 - Now these are the ultimate bad guys in Borderlands 3.

43:56 - They are Sirens, those superpowered, almost magical characters; they are the main antagonists and at the end of act one, they do something pretty heinous.

44:02 - They kill a character that you’ve grown attached to.

44:06 - And we saw with playtesting that yeah, people ended act one out for revenge, unsurprisingly.

44:12 - Well, our playtest observations in data took an interesting turn when players began act two; they traveled to the swampy world of Eden-6, and an interesting thing happens.

44:23 - The Calypso Twins kind of vanish. Eden-6 has its own subplot involving some of our other much-beloved NPCs, but the Calypsos themselves, they’re not really seen or heard from for a little while, and this pretty much killed pacing for players.

44:39 - Their engagement and interest in the storyline for Eden-6 took a major nosedive.

44:45 - They told us point-blank in group discussions and in interviews that  they weren’t- they didn’t understand what was happening.

44:52 - Why did the Calypsos disappear? What was going on with them? Why were they engaging in in these other subplots? And these interviews and group discussions with our narrative team led to a really great, elegant solution.

45:06 - Simply, add more Calypsos to Eden-6. During the latter portions of development, the Calypsos made additional appearances and had more voice over and dialogue in Eden-6’s plot; in fact, there were additional side missions that were added, some lore, items, and moments that brought the presence of the Calypsos back to a section of the game that didn’t quite focus on them, front and center.

45:30 - So this greater presence saw better engagement.

45:34 - With later playtest groups during our intern program, we saw that once these changes had been implemented, many of the pacing issues with act two started to disappear.

45:43 - They weren’t gone entirely because the Calypsos weren’t the focus anymore, but these mitigation strategies kind of assuaged some of  the larger concerns about what was going on with players engagement with the storyline in act two.

45:56 - So while this wasn’t necessarily a brand new methodology, we definitely improved our playtest intern approach when testing Borderlands 3.

46:03 - We increased the number of interviews that we conducted with our playtesters over time.

46:07 - We added more group discussions with developers, and we did more deep dives into narrative content– the stuff that is hard to grasp if you’re only looking at a small section of the game, but if you’re playing through the entire thing, you can start to understand the ebbs and flows of the storyline and where some of those critical issues are occurring.

46:24 - So this testing highlighted pacing and comprehension issues with the story and some of these decisions that were made about how to fix the issues in Eden-6, and many other issues that were addressed during this part of development were done in real-time.

46:39 - Again, the studio had large buy-in to our intern program.

46:43 - There were many developers conversing with one another with us, while the playtesters were playing content.

46:49 - So many of these decisions and changes were done in real-time while the players were in the lab.

46:57 - So to summarize the latter three case studies– with our gun audio question, we looked at signal detection and found that players could hear the differences between guns.

47:07 - For our hub navigation study, we looked into navigation research and had players conduct errands around the Sanctuary 3 spaceship, which helped inform the layout and deco of that environment.

47:17 - And finally, we improved upon our intern program to get information and data about pacing and engagemeny with the narrative of Borderlands 3.

47:28 - So if we return to the goals of this talk, we can see that through combining established methodology, the stuff that’s worked for us in the past, and by incorporating some new approaches from what’s out there beyond just user testing and games, we can improve upon how we test a game like Borderlands 3;  something in a new entry in a longstanding franchise.

47:49 - And we illustrated with six different case studies, three with established methods and three with new approaches, how exactly we did that.

47:56 - So this is echoed in the takeaways that we first discussed at the beginning of the talk.

47:59 - So iterate and improve on your established methods; those methods that you rely on day-to-day in your research program.

48:05 - Look to existing research outside of games user research, and incorporate those methods into your research portfolio to answer questions that you or your development teams have.

48:16 - So over the past year, (sighs) we haven’t done any of this stuff because we’ve been working from home.

48:21 - So looking back at what we’ve been doing since last March, and ahead to the next couple of month, I want to talk a little bit about what we’ve been doing to innovate more methodologies while we’ve been remote.

48:33 - So since working from home, we have been working exclusively with developers as our participants.

48:38 - And to do this we have created kind of two new ways of testing; One: I have dubbed play at your pace; and what this means is we provide instructions and a build for one of our products to our developers, and they have a window of time, typically a week, to playtest content.

48:55 - Now this is done asynchronously because everybody’s busy.

48:58 - Developers have full-time jobs and they can’t necessarily devote four hours in a given chunk to some content, so we give them more flexibility and more time to play content and provide feedback.

49:07 - And we bookend these player pace tests with kickoff meetings and group discussions to both level set and answer some additional questions at the beginning and also collect some group feedback at the end of the playtest.

49:18 - And so far these have proven to be very effective for getting feedback from our devs, while everybody’s busy working from home.

49:25 - Now on top of that, often times, we will gather some of our developers in our Discord server to conduct one-on-one sessions, where developers who are not playtesting can actually jump in and observe, and this is a bit more like how things are run in our lab with that streaming tech.

49:40 - So a developer shares their screen via Discord and securely and privately, we the user research team and some of our other devs can watch what they’re doing and talk to them both during the playtest and afterwards, in an interview format.

49:53 - So we are working on plans right now to return safely to the lab; discussing with all the people that we need to about what might need to happen in order for us to return to testing in person with outside participants.

50:05 - We don’t anticipate this happening very soon, but we’re looking forward to the day that we can actually bring people back in and collect feedback from external users.

50:15 - And we’re also very eager to learn from all of you.

50:17 - If you have thoughts and ideas and things that you’ve been doing, both to collect feedback while you’ve been working from home remotely, as well as the plans that you and your lab are working on to bring people back in safely; we’d love to hear it and love to learn from you.

50:31 - That brings us to the end of the talk, I want to thank a number of people.

50:35 - First and foremost the user research team at Gearbox, Kyle and Michelle, as well as the Borderlands 3 team in all of Gearbox Entertainment, 2K games and their user research team, everybody here at the Games User Research Summit, my amazing partner Linda and our cat, TK, and of course, all of you.

50:53 - So please keep the conversation going; you can get in touch with me here.

50:55 - I look forward to speaking with all of you and learning from you.

50:58 - So enjoy the rest of the summit; have a great afternoon, and I look forward to talking with you. Thank you all again!.