Pathfinder: Kingmaker Update #43

The latest Kickstarter update for Owlcat Games’ upcoming cRPG Pathfinder: Kingmaker informs us that the game is about to begin alpha testing, available to the $95 and above tier backers. The usual warnings of missing and work-in-progress features, and possible bugs apply, so if you want a more polished experience, you’re better off waiting for the beta or the full release. And apart from that, the update talks at length about the ins and outs of Kingmaker’s UIX and focus testing, making it quite clear that the Owlcat folks are serious about making their game as polished as possible.

Here are a couple of short videos that illustrate the high-tech testing process, and the following brainstorming sessions:

And a few paragraphs about the testing itself:

After we have selected our respondents, testing begins. On average, each test takes three days, with three respondents per day. Each player needs approximately three hours to undergo the testing – introduction takes a half an hour, playing the game is two hours, and the last half hour is the final interview.

The testing itself looks like this:

• We meet the respondent volunteer and treat him or her to something yummy. 🙂
• We introduce the respondent to our UX lab specialist (a developer is always present during the testing, taking the opportunity to watch the respondent’s game in “live” mode).
• We give the respondent a quiet room to play in with no distractions – just you and the game. Oh, and a chair. 🙂
• We tell the respondent about the game, but do not mention a word about the goal of our testing (This is to keep the results unbiased.)
• We tune the equipment: eye tracking, cameras, microphones. Using these devices, we record everything that happens in the testing room and broadcast it live to the observation room: the player’s words, their facial expression, gaze direction, and so on.
• Then the testing starts. During the game, the player can share their experience in any form they see fit.
• After that comes final interview with the UX specialist who will ask questions about your gameplay experience.

During the testing, all the player’s comments are written down by both the developer and the lab specialist. Together with the video records, these notes will later be used to create a final report.

After the testing is complete, UX specialists compile all the data they’ve collected and make a summary report. Its form depends on the initial goals of the testing. In most cases, they just take the list of these goals and add their extended remarks, screenshots and thoughts on what’s working properly and what’s problematic.

[…]

The final stage comes when the team reviews the reports, discusses new-found problems/inconveniences, and decides how to solve them.

UIX testing is an extremely useful tool, and it gives detailed feedback on how users perceive our game. But we realize that this method has a substantial drawback – the number of testers is too small, and this testing is extremely time-consuming. As a result, representation and statistical validity of the results we get can be questionable. That’s why we use UIX-testing only to detect some nuances and non-apparent bugs of usability in interfaces that have already passed several iterations of testing.

To collect more representative feedback data on our game, we use another classic approach – focus testing.

To perform focus testing, we invite the same categories of players we use for UIX-testing, but the numbers are bigger – 10-20 people in each category. It’s important to note that in this case territoriality doesn’t matter much , so we can invite people from other countries to participate in our testing.

To perform the testing, we assign one developer to each player. It’s important to us that every game developer – and especially every game designer – take part in an iteration of focus testing at least once. Imagining how people would play your game is one thing, but seeing how they do it in real time – is something completely different.