Ratings Scandal Numbers

A big story this week in XBLIG has been the manipulation of ratings by fans of College Lacrosse 2011.  The story started with Robert Boyd noticing a sudden drop in the rating of Cthulhu Saves the World at the same time as an increase in position of College Lacrosse 2011.   Achievement Locked has an excellent write up with links to the relevant posts.  The short story is the fans of College Lacrosse 2011 not only rated the game up, but also rated all the top rated games like Cthulhu Saves the World one star in an attempt to lower their standings and boost College Lacrosse 2011 even higher.

The XBLIG dashboard lists, including the top downloads, top rated, and new releases, equal significant sales revenue for developers.  Being on the first screen of any of these lists equate to more sales than a favorable review by a well know site like Kotaku.  It is also the face of XBLIG to many gamers.  If these lists are full of the quick cash grab apps released on the service, gamers will get the false impression XBLIG is nothing but $1 gimmicks and not real games.  That these lists can be manipulated is quite disturbing, as is Microsoft’s total silence on the issue.

Truth is, it’s very easy to manipulate Microsoft’s Xbox Live rating system.  You do not need to play or download a game, and you do not even need an Xbox.  Simply visit Xbox.com, make an account, and begin rating games.  You do not even need a valid email address – there is no verification of accounts for Xbox Live done by Microsoft.

On the latest podcast I mentioned it was unfortunate we didn’t have real numbers to point at, just anecdotal evidence from developers who’ve seen their position change quickly in the top rated games list.  I was pleasantly surprised when Dylan mentioned we had our database backups!  These would effectively be snapshots of of game ratings and vote counts, and could provide some data to go with the suspicions.

Understanding The GameMarx Data

You didn’t think it was gonna be that easy, did you?

The GameMarx/XboxIndies database updates nightly, but only a hand full of games are updated.  We take the first 75 games by the date they were last updated, and only update those each night.  Weekly, all games are updated.  The reason is new releases and recently updated games are more likely to have changes whereas the majority of games, even ratings and vote totals, don’t change daily.   Because of this, the database doesn’t have all daily changes, and we can only look at weekly changes of rating.

The second issue is we do not have the actual rating of the games.  Xbox.com and the Xbox Dashboard rounds the rating to the nearest quarter vote, but if you look at the order of the list it’s clear they are sorted on the actual rating.  XboxIndies.com sorts the games by rating, then vote totals, and this is why our lists do not match Microsoft’s.  This means we don’t have an archive of list position as it would be on the Xbox and Xbox.com

Because of the impact this has to developers, and because I don’t have days to pour over the numbers, I’m making the dataset available for download.  If you want to play with the numbers or see the raw data behind the charts below, download the ratings dataset here.

On to the charts!

Ratings

First, this chart is in a 3D perspective because most games in the top lists are at a 4.5 rating and you wouldn’t see the lines.  Of the top games I grabbed, only two have dropped to 4.25.  Since I don’t have the real vote total, this chart is of limited use.  A rating change of 4.52 to 4.43 would not be recorded, but would probably mean a loss of a position or two.

Votes

This second chart is interesting.  Cthulhu has experienced a jump in the number of votes this last week, and it lines up with the fall in rating.  College Lacrosse 2011 is getting a disproportionate amount of ratings, beating even the top game on the service.   Let’s also put some perspective on the numbers.  Cthulhu had 1,482 votes when it held a 4.5.  To go from 4.5 to 4.25 with 1,482 you need 114 one star ratings.  In the last week Cthulhu received 174 ratings.

Okay, where does this leave us?  With not much I’m afraid.  There is no evidence that Lacrosse fans down voted games on the service to boost College Lacrosse 2011.  We also don’t have enough data to see if the mass influx of votes increased their score.  There is data though to support that something is off in the Cthulhu rating – a 4.5 game doesn’t just get a series of one star ratings on it’s own.

Microsoft has the data to look into the real cause.   If there are a rash of new accounts that are up voting Lacrosse games and down vote others it will be easy to spot.  They will also have a better trend line to look as since they will not be rounded to the nearest 1/4 rating.

All the more reason Microsoft’s complete silence is disturbing.


Leave a comment