Carbine Studios Producer Letter

Carbine Studios senior producer Eric DeMilt has penned a letter to the community about what the team has been working on in their MMO lately, the things they’ve learned during usability testing, where the game stands in development, and more. Difficult puzzles are apparently a go:

We’ve set up a usability testing lab on site, where we can get groups of players in the game and have the dev team observe them from one of our conference rooms. We can watch both what’s happening on the screen and the player’s reaction to it.

We combine this direct observation with a short survey at the end of a play session and brief Q&A, and a lot of listening. Did this quest make sense? Is the story interesting? Could you find your way around? We’ve gotten some great information and some great reactions. It’s been really rewarding to be able to see our game and its features through the eyes of new players.

All of this usability testing has been really eye opening. It has caused us to prioritize important fixes and make changes to things we simply didn’t notice, or that had become dull annoyances to us. Dev teams can get used to things being broken after a while and begin to ignore them, real users don’t! We believe extensive usability testing keeps us honest about the game we are making and keeps us focused on things that matter to real users. If you want a polished product you cannot get too focused on what’s up next on the schedule and ignore things that need doing, right?

In addition to helping us identify important issues, these sessions also encouraged us to do more of the things that people liked. We’d sit in the conference room, agonizing as we watch a player struggle with a particularly hard puzzle, wondering if we’d just made the most frustrating piece of content ever. We’d cheer when they finally solved it, but would still wonder what we should change to make the puzzle easier. Later during the Q&A we discovered the player really enjoyed the difficulty, and felt a real sense of accomplishment when he solved it. This type of scenario played out in our testing so many times, rather than take it out or nerf it, we decided to do more.