The Daily Worker Placement

Thursday, April 26, 2018

Behind the Numbers: Consensus and Conflict in Game Ratings

by | published Wednesday, March 28, 2018

Keeping up with the board game hobby can be challenging.

There’s been a good deal of discussion lately both on Twitter and right here on DWP about the desire to follow trends and play what’s hot. It takes an educated consumer to navigate the sea of board game releases, but one key skill is learning how to assess hype.

Does the gaming community buy into consensus, or are diverging opinions more common? Are your tastes more mainstream or niche? Knowing the answers to these questions may inform how much research you should conduct before buying into that next hot game.

As a data nerd (albeit an amateur one), I’m always tempted to analyze a problem using hard numbers. What better source of data than BoardGameGeek? Bounding the data set a bit, let’s look only at 2017 releases with 100 or more ratings. This criteria might seem overly restrictive, but it still applied to a whopping 527 games. Of those games, three stats were collected: average rating, standard deviation of ratings, and total number of ratings.

Before we begin charting one metric against the other and analyzing the extremes, one bit of data jumps out: sorted by quantity of ratings, there is an incredibly steep drop off from the year’s handful of hits. If gamers are keeping up with the rapid pace of releases, bouncing from one hot game to the next, there should be a crowded field at the top. Yet the numbers fall precipitously after the top 1, 5, and 20 games. The top 50 games are a good summary of the year’s buzz-worthy titles, but after that the games barely have a pulse.

As I attempt to win the Golden Geek for worst infographic of 2018, allow me to indulge in an interplanetary metaphor. If every rating gave a game 100,000 miles, you’d get the header above.

This solar system  map isn’t even to scale, so I’ve included the real chart below it, showing that The Year of Gloomhaven really was in play, with its 12,766 ratings dwarfing Century’s 7,202. The runners-up are in the 5,000-rating range, with a steady drop off in their wake. If this had been a volcano infographic,  Gloomhaven would be ejected into orbit, with the Top 20 games providing a nice ash cloud, and the rest of the year’s releases flowing over civilization like hot lava before quickly cooling off and never getting played again (because every died in the volcano blast? Or because the games aren’t good? You be the judge.)

With that out of the way, let’s compare some factors and look to identify games that exist at the extremes of both spectrums. For purposes of this “experiment” let’s hypothesize that a high rating with low standard deviation indicates a universally loved game, while high ratings with a high standard deviation are cult hits. Games with low ratings and low standard deviation were universally panned, while low ratings and a high standard deviation are games still struggling to find their niche audience.

First let’s take a look at the plot, so we can see the groupings we’ll identify in each corner:

Do these assumptions hold true? Sometimes, but mostly not.

Wait. Coaster Park. What did you do?! With a 4.15367 average rating, it turns out that if you promise people they can do roller coaster loops with marbles, you’d better deliver.

The “Universally Loved” category didn’t net the year’s classics as expected. Instead, the category contains Twilight Imperium 4 and the year’s best expansions (for Spirit Island, Clank!, Aeon’s End, Xia, and Fields of Arle). It turns out that gamers only put their ratings into a tight grouping when they are a pre-selected audience: those who enjoyed the base game enough to purchase more.

“Universally Panned” didn’t quite pan out either, with the category holding Karuba: The Card Game and some European-published remakes and reprints that didn’t garner enough buzz to publish in North America (Santo Domingo, Cartagena). While I haven’t played it myself, Karuba’s card game approach of providing a more complex take on a family game, rather than the typical other way around, did seem a bit odd. It would seem that the German audience providing these initial ratings agrees.

The “Cult Hits” category held some truth, with Red Dragon Inn 6 and the Runewars Miniatures Game. These fall into the “did we really need this?” category, but they certainly have their fans. These games were also grouped with true hits such as Gloomhaven and Pandemic Legacy Season 2. The conclusion here is that with high variance among the ratings of bigger titles, the board game community isn’t falling into groupthink around the hottest games.

Lastly, the “Searching for Niche” category reads more like the true list of this year’s flops: Coaster Park, Bears vs Babies, Unlock! The Nautilus’ Traps, and Rail Raiders Infinite. It turns out that when folks despise a game, the one thing they can’t agree on is to what extent they despise it.

While my time wrangling spreadsheets didn’t draw any shocking conclusions, it does provide reassurance that user-generated game ratings are in a healthy range. Gamers are effectively fighting back the tide of too many games to focus largely on playing the top 50 titles, yet these folks aren’t afraid to provide diverging opinions.

This won’t help most consumers focus their purchase researching, as I’d hoped, but it does indicate one area of specific use. As a gamer with personal skepticism towards expansions, it was interesting to see how they tightly expansion ratings were grouped together. Comparing ratings of future releases to expansions of the past, regardless of base games, could provide a useful assessment of which expansions are outperforming the pack. Filtering to “expansions only” is not currently an option on BGG’s advanced search, but perhaps it should be!

3 thoughts on “Behind the Numbers: Consensus and Conflict in Game Ratings

  1. Andrew says:

    Interesting conclusions. There are definitely too many games for even the dedicated gamer to sample a slice of them in one year. I played perhaps a dozen, perhaps two dozen, and that itself mostly hits from the top 50 you mentioned.

    Are there ways that board game reviewers can more responsibly surface the ‘hidden gems’? Do we have any obligation to do so? What’s wrong with everyone playing (and loving) Gloomhaven and Century: Spice Road.

    Interesting thoughts. . .there’s definitely so much quality being produced right now, but so much of it that it’s difficult to truly discover even a piece of it.

    • says:

      Yes, for sure. The wealth of options out there mean there is a lot of great titles to choose from, but it can also mean that some good games will slip through the cracks.

  2. Moonbi Taylor says:

    The ‘hot’ new games certainty get their share of attention on BGG and in the community. I wonder if the ratings of classics (ie games with good ratings which have been around for a while) can be similarly analysed. Are their high ratings still influenced by their first release flush of excitement, or are they still gaining good scores? I don’t know if you can break down the BGG data to that degree, ie do the ratings numbers drop off after the first year, and are high ranked games coasting on old scores?

Leave a Reply

Your email address will not be published. Required fields are marked *