 |
|
|
| |
Analysis: What Do Games Have In Common With Jam?
by Jamie Madigan [PC, Console/PC, Exclusive]
|
|
| |
|
September 3, 2010
|
| |
[Psychologist and gamer Jamie Madigan analyzes the human decision-making process for Gamasutra, explaining how research on jam reviews, of all things, might have surprising lessons for video game reviews.]
For every one of us, making decisions is part of our daily human existence. Most of them are of little consequence –what to eat, what movie to see, what video game to buy– so we have developed an astonishing array of mental short-cuts to make these kinds of decisions comparatively quick, easy, and not too mentally taxing.
We may eat what we have eaten and enjoyed in the past, and by and large we use simple decision rules such as "I like this genre" or "I like this developer" to choose movies or games.
Other decisions, though, are either much more important or much more public and thus we put more work into it. Whom should we date? What college should we attend? Which house should I buy?
When faced with questions like these, many of us have probably drawn two columns on a piece of paper, labeling one "Pro" and one "Con" and then listing things in each column.
When trying to decide whether to marry or stay a bachelor, famous biologist and five-time Counter-Strike world champion Charles Darwin did exactly that, producing the list below:
The list admittedly looks a bit sexist by today’s standards, but it illustrates the idea well. But is this sort of thing always a good idea? When video game reviewers ruminate over the merits of a particular title, they are often asked to consider standardized lists of features –graphics, sound, fun factor, multiplayer, value, extendability, controls, and so on. Should they always try to analyze decisions across every possible variable? Is that the right way to review a game?
Researchers Timothy Wilson and Jonathan Schooler probably wouldn’t think so, or at least they could imagine situations in which this type of approach could lead to poor evaluations. And here’s the best part: jam was involved. Delicious, strawberry jam.
In their study the duo were intrigued by a Consumer Reports ranking of 45 different brands of strawberry jam. Panelists in the study were trained sensory experts (i.e., professional food tasters) who sampled each of the condiments and rated them on 16 characteristics including sweetness, aroma, bitterness, spreadability, and others.
This is the kind of thing Consumer Reports does all the time, but Wilson and Schooler were curious about something. Some of their previous research had suggested than when asked to analyze their reasons for making decisions or ratings, people tend to screw things up. The theory goes that we are often aware of our preferences for products (or art, or whatever), but when asked to explain WHY, we often feel obligated to include the most salient (that is, apparent) and plausible explanations. Even if we would have otherwise ignored them.
So if asked to explain why you like Red Dead Redemption so much, you may think about what should be included in the checklist for evaluating an action game, come up with “the weapons,” and then feel compelled to award or take away points for how the game’s weapons feel and work.
The problem is, the most salient and plausible factors may not be the ones that are really responsible for how much you enjoy the game. The weapons in Red Dead Redemption are largely unremarkable –the game’s appeal lies almost entirely in other areas and any weight given to how cool the weapons are is inappropriate at best.
So, thinking along these lines, Wilson and Schooler wondered what would happen if they asked normal people to recreate Consumer Report’s jam ratings. And what would happen if you asked them to ponder the reasons for their ratings the same way the Consumer Reports experts did?
So they fed some college students the 1st, 11tth, 24th, 32nd, and 44th best jams from the report to find out. Those in the control group were just asked to taste the treats and make their ratings. Those students actually did pretty well –their rankings were very close to the professional taste testers’ rankings. But the group that was asked to write down the reasons for their ratings did far worse. They may have favored the jams that the experts thought were gross and scrunched their noses up at the ones the experts thought were great.
Why? Because the subjects started focusing on factors that didn’t really matter. Smucker’s had more chunks of fruit in it, so it gets a higher rating. Wait, what? Is chunkiness really important for them? Doesn’t matter; it sounds plausible so it got factored in. When the XBLA shooter Monday Night Combat came out, some people lamented the small number of maps. Same thing –one could argue that because the gameplay requires a very specific setup, you don’t need –or even want– a lot of maps.
Puny humans are pretty bad at combining an array of weighted factors so as to arrive at a rating or decision –it’s just not how our minds were designed. Jelly or game review guidelines that require us to over analyze our decisions or check them off against a standardized list of factors (graphics, sound, etc.) can exacerbate this limitation and lead us to consider what should be irrelevant information when making our ratings. This corrupts the rating process and takes us farther from our “true” feelings or evaluations.
This is one reason why I prefer more organic, experience-based evaluations of games from message boards or podcasts rather than formal game reviews. I feel like I can listen to someone talk in an unstructured way about how much the enjoy a game and get a much better idea of how much I might like it. Just consider what’s important and ignore the rest.
Now, go get yourself some jam – whatever kind you think tastes good.
References:
Wilson, T. & Schooler, J. (1991). Thinking Too Much: Introspection Can Reduce the Qualities of Preference and Decisions. Journal of Personality and Social Psychology, 60(2), 181-192
Strawberry Jams and Preserves (1985, August). Consumer Reports, 487-489
|
| |
|
|
In this case, a (possibly) interesting footnote is that this problem of of overvaluing some qualities from a list and undervaluing others has been addressed using fuzzy logic.
The problem is one of weighting. If you and I each make a ranked list of the five most important factors in determining jam quality, when we compare our lists we may not have even thought of the same five qualities. And for the qualities we both both thought of, we probably won't have assigned the same importance to each one.
Fuzzy logic offers a way out of this by replacing the binary, is-it-there-or-not checkbox for some quality of a thing with a larger number of assignable values. It might be a zero to ten scale, or a 0-100 percentage; in any case it's possible to assign a more graduated weighting to some quality than just yes/no.
In a fuzzy database, for every item you enter you list the factors describing that item, and you assign to each factor a weighting (more on that in a moment). Once all records are entered, anyone can then write queries against that database that specify what factors matter to them, and (this is the important bit) how much each factor matters.
For example, based on an article in Byte magazine in about 1983, I wrote a fuzzy database system for tracking my paperback books. As I entered each book into the database, I listed all the descriptive terms I could think of that applied to each book -- "science fiction," "fantasy," "time travel," "depressing," "omnibus," "long," "illustrated," "rating" (or "unread"), and so on. And for each descriptive term, I entered an associated value -- a weighting. For a six-hundred-pager, I might associate "80" (percent) with the descriptor "long", while for a 100-page book I might enter a value of "20".
Once all my books and their descriptors and values were entered, I could then perform a search against that database that could identify particular books down to a fine level of satisfaction. For example, I might ask something like, "show me all science fiction paperbacks; I really want something long, and I want something about time travel but I don't care too much about that":
SELECT FROM "BOOKS" WHERE "SCIENCE FICTION"=100 AND "LONG"=80 AND "TIME TRAVEL"=40;
The database code takes this query and does a "fuzzy AND" for each value entered against the value in the database for the same descriptor -- a fuzzy AND is defined as the minimum of the two values. So if one of the books in my database has the "long" descriptor and an associated value of 50%, and my query weights the "long" descriptor at 80%, then that book would be rated at the minimum of the two values: 50%. (ANDs within a query are resolved similarly; ORs, NOTs, and IMPLIES are also valid fuzzy connectors.)
Eventually what I'll get from my query is a 0-100 ranking for each book. I then group those rankings into roughly ten sets. Based on my query, I might get two books at the 90%-100% range, twelve books at the 80%-89% range, and so on. Then I can drill down into the range that interests me to see the books in that group, which should be the books that most accurately satisfy my specific interests.
It actually does work pretty well... for me. But what about someone who might have rated the various descriptors differently for each book?
To be a good general solution, what we'd really need is for a lot of people to enter descriptors and associated values for each item, such as a book or a game. The system would then average all the values for every descriptor applied to an item, and that average value would be what the descriptor value in anyone's fuzzy search would test against.
Imagine if Metacritic, instead of having a single overall 0-100 rating for every game, allowed journalists and developers and critics and gamers to specify any descriptive terms they wanted for a game, along with a value indicating the degree to which each term applies to that game. In such a system, when a user looks for a game, he wouldn't just see an average "good/bad" score -- he'd be able to supply a set of *specific and personally-weighted factors* to craft a search that is highly specific to what he really wants from a game.
Of course, you could do the same thing with flavors of jam if you really wanted to. But wouldn't it be interesting to see something like a Fuzzy Metacritic that largely solves the problem of including factors that don't really matter in satisfaction analyses?
I have had thoughts along these lines recently, too. Often, people really don't know why they like something and with games it can be really hard to discern why you got enjoyment out of it. It is my belief that with all of the detailed talk of the fluff that surrounds the gameplay, people are somehow deluded themselves into thinking these things are more important than they are. Of course, it is very presumptuous to go up to someone and say "Hey, you don't know what you think so let me tell you", but experiments such as the one described here and also game sales data will go some way to revealing reasons for behaviour.
Even reviewers, who make a living of articulating why a game is good are poor at it. I used to read game reviews with the aim of finding out how good a game is. Now I read them only to see what the state of game reviews is. I don't know if I can blame them, however. Playing games for a living, they must be so burnt out on gameplay that it is a relief when something comes along that lets them sit back and enjoy some production values. But for those who work all day, this is just a waste of time.
Personal accounts do indeed tell you so much more about a game than a "professional" review. I find that forums such as those on gamefaqs are also a good place to gain an understanding about a game to decide whether its worth buying or not, but a month or so after its release date after the hype has died down. After this honeymoon period, if people are still discussing game mechanics and meanings then there is something of merit here. Many highly praised games fail this test.
I understand why I do or don't like something. Also comparing Jam to games is ridiculous, games are essentially movie-like pure imaginative worlds that you participate in. You can't compare the two.
Most complaints about games come down to aesthetics, uninteresting worlds and pacing. Since almost every part of a game is covered by aesthetics, control aesthetics (how a game feels to control it), animation/art aesthetics - how animations amplify or reduce emotional feedback when executed after hitting button or causing an effect.
The problem with the whole "people don't understand what they like or don't like about games" is that gamers can quickly identify why a game sucks. Almost any expert gamer will tell you metroid prime 1 tended to be boring and drag on in parts. When gamers can pinpoint the problem like that exactly like when a game tends to drag, or the controls need fixing - the best thing you could do is invite those gamers to replay the game and then identify while the parts and time when the game begins to drag, when a game starts to get uninteresting.
Patterns emerge over time in game development and the same things are said by gamers similarly across genre's, that doesn't look to me like gamers don't understand how the game made them feel.
I am agree that check list don't contribute to transmit the gaming experience, but more organic and experience-based evaluations don´t do it too.
To review a piece of art (symbols and signs es put together in order through a precise grammar to create something that has no previews existence), you need to understand the Language behind it. The problem with video games is that we still don't exactly understand how the game grammar works, how all the Language behavior.
That is why reviewers agree to use checklist and personal experiences, they don't have choice.
As we keep exploring game language precise reviews will come up.
You can review a Movie using a experience-based or checklist evaluation, but the true review is through the analysis of the Language.
In the other hand... Yes, If you use Food as a material arranged through the art language food could become part of a piece of art.
Jam is both an art and a product. Games are both an art and a product.
There is nothing more or less valid about looking at either in either way.
I find Left 4 Dead a extremely fun and great game. Nevertheless if your look at a Graphics, Sound, Story, Game Play metric it would not score so high. The graphic is a half a generation from the leading edge, the sound is good but kind of repetitive, the story is basically only a excuse for gameplay and too short for a high replay value and the game mechanics are mostly a standard FPS. Score? Maybe 60-70...
Why do I like Left 4 Dead so much? It is the pacing, the environment the whole thing.
Btw. Left 4 Dead is not Art...
In my post I did´t make that statement.
Also, I don't mean art in the Intellectual narrow minded stereotyped way, I am talking about art as a language. Games has their own language different from other forms of art, forms like comics, movies, literature, visual arts, etc.
Repeating what I said before: The problem with video games is that we still don't exactly understand how the game grammar works, how all the Language behavior.
That is why reviewers agree to use checklist and personal experience, they don't have choice.
Finlay I´m not saying if a game is fun or boring, that is totally subjective is linked with the emotional expirience. But Language is not.
For example in Literature you can identify if in a sentences you has a correct use of grammar or not. But you can still like or not like that sentence.
"Also, I don't mean art in the Intellectual narrow minded stereotyped way, I am talking about art as a language. Games has their own language different from other forms of art, forms like comics, movies, literature, visual arts, etc."
Food is an art in a similar way. There is an immense and varied language of and taste that predates video games by thousands of years. The article is relevant to video games. People can be confused about quality by drawing their attention to certain aspects.
Reviews can often be useless and I always wish I knew one basic thing about the person reviewing the game, namely, which other games is he inevitably comparing the game to?
All games will have flaws and will be imperfect, but sometimes those imperfections can be ignored for the greatness presented elsewhere in the game, sometimes in the most inconspicuous places.
GS did an article on this too, for those that missed it: