A few weeks ago, we posted a blog post describing some of our thinking about user reviews, some tools we added, and some of the next areas we were planning on addressing. If you haven’t yet, it’s worth a read to understand our thinking about reviews and the goals we think user reviews should be addressing.
Today we’re making a few more improvements to user reviews to address some concerns and feedback that we’ve been hearing, and to ensure that the system remains a useful tool for you to identify games that you will enjoy. In this update, we’re focusing on how we determine the helpful reviews you see on each game’s page and the order in which they are shown.
What is a helpful review
Reviews should help paint a picture of what it’s like to play the game and how well the game has been enjoyed by the people who have already played it. A good review typically describes some of the factors that directly impact the experience of playing the game, which can include a wide variety of things like how well the matchmaking works, how buggy the game is, or whether the game represents a good value for the price.
While everyone that has played a game probably has opinions on how much they enjoyed the game or not, some people are better than others at writing thoughtful descriptions that are useful for other potential players. We think it’s fine for players to be able to indicate whether they liked a game or not, but not every review is useful to show in greater detail. A helpful review then is one that includes enough information to aid in your understanding of whether you are likely to find the game fun, or whether you should avoid the game and explore more alternatives.
As of this writing, there are over 36,579,839 reviews posted by players across all of Steam. Some games have a handful of user reviews while others have hundreds of thousands. Regardless of the number of reviews, we want to make sure that the most helpful and relevant reviews are the ones that you see first when you are looking at a game’s page in the Steam store.
Sorting helpful reviews
Below each review you’ve probably seen the question of “did you find this review helpful?” along with a set of buttons to indicate yes or no. You can use these controls whether you have already purchased the game or not, because it is usually prior to purchase that you are looking at user reviews and considering which of them are helpful in making your purchase decision.
These inputs are then used to determine which reviews people generally find most helpful. Up to now, our system simply looked at how many people had rated each review as ‘helpful’ and how many people had rated it as ‘not helpful’ and then highlighted the ten reviews that the most percentage of people found helpful. Since games can change (sometimes dramatically) over time, we introduced a change a while ago that prioritizes showing recently-posted helpful reviews, as they are more likely to reflect the current state of the game. Of course these are just the defaults that are shown on a store page, and you can still easily browse all of the reviews for a particular game if you want to investigate further.
In a perfect world, people would truthfully mark a few reviews that were helpful for deciding to purchase or not purchase the game and we could use that data to directly determine the ten most helpful reviews. Alas, it turns out that not everyone is as helpful as we would like. Instead, we are seeing more and more feedback from players that the helpful reviews shown on store pages aren’t representative of how well people are actually enjoying the game.
Taking a closer look
So we took a closer look at the patterns and behaviors of people that are rating reviews. Of the 11 million people that have used the helpful buttons, most follow a reasonable pattern of usage: Typical players rate a few reviews as helpful or unhelpful while deciding whether to make a purchase. However, we found a small set of users on the far extreme that are clearly trying to accomplish something quite different from normal players, and are rating more than 10,000 reviews as helpful or unhelpful on a single game. This behavior is not only humanly impossible, but definitely not a thoughtful indication of how ‘helpful’ each of those reviews were. These users also tend to rate up just the negative reviews while rating down the positive reviews (or vice-versa) in an attempt to distort which reviews are shown by default.
Because of how many reviews these users are rating, they each have a disproportionate amount of influence over the display of helpful reviews and cause certain reviews to appear more prominently than they should be. This can result in a confusing appearance where the default set of reviews shown are negative, even when most players have posted positive reviews and clearly enjoy the game.
What we are changing
Up to this point, we’ve tried to maintain just showing the simple math behind how we calculate whether a review is helpful or not–the percentage and number of people who indicated a review as helpful. We like systems that are transparent and easy to understand, as they are also easier to believe and trust. Unfortunately, this has resulted in a system that allows a small group to manipulate reviews to a degree that is clearly decreasing the value of Steam for many other players. So we’re making two main changes.
- Firstly, our system will use a new method of calculating the helpfulness of each review, taking into account the users that are trying to manipulate the system. One way we’re doing that is by counting the helpful ratings on reviews differently for users that are far outside the norm. Ratings from users that follow normal patterns of rating will continue to be counted the same way that they have, whereas accounts that rate an excessive number of reviews on an individual game will see the weight of each individual rating count for less and less.
- Secondly, store pages will now show the default helpful positive and negative reviews in a similar proportion to that of the overall review score for the game. For example, if the game is reviewed positively by 80% of reviewers, then the ten reviews shown by default on the store page will be 80% positive, showing eight positive and two negative. This should keep the reviews shown on a game’s page from being so easily manipulated by a few determined players and should more accurately represent the overall sentiment of the people playing the game.
We’re rolling out both of these changes as a beta today. You can turn the new method on and off to see how it impacts the default display of reviews on any given store page. Note that these changes only impact the default listing of reviews (Called “Summary”) and the “Most Helpful” display option.
Still much more to do
There are still a number of further changes we’re considering for the user review system. One thing we’re looking at, is how review scores on games change over time as games develop or languish, and ways to better indicate how players are enjoying the game right now. We also want to better indicate when players are reviewing issues in a game that only pertain to a particular region (such as server locations), or particular language (such as poor translations). We’re also exploring ways to calculate a personalized review score for each player and thinking about how that would look.
We know the review system is important for players and developers and we’re going to continue making improvements to ensure that user reviews remain useful and trustworthy. Please let us know your thoughts on these latest changes.
-The Steam Team