The Real March Madness – XBLIG Rating Manipulation Continues

Last week I posted some numbers on the Xbox Live Indie Game ratings scandal.  Since then little has changed.  Microsoft has only issued a tweet that “top men” are on the case with no follow up as of yet, and the ratings continue to show signs of manipulation.  If you want the drama, I recommend reading the Armless Octopus article on Crosse Studio considering legal action against Zeboyd Games.  Yes gents, we’ve gone plaid.  While the claims of legal action are entertaining, I’m more curious by what story the numbers tell.

Because news sties like Joystiq and IGN Bloggers cannot manage to research before posting, (hat tip to The Consumerist for updating their story after I emailed them) here are the key points:

  • This mess started after A “How to Vote” video was posted for Lacrosse Facebook fans that didn’t have an Xbox
  • A mass of votes came into the XBLIG channel that not only boosted Lacrosse games, but down voted other games
  • There is no security on Xbox.com ratings by Microsoft – this is over looked in the gaming “press” covering the story.  You do not need to download the game, own an Xbox, or even create an Xbox Live account with a valid email to rate games.  You can simply go to xbox.com, make up a fake account, and rate.
  • The rating of a game isn’t as important as it’s position on the Top Rated list.  Falling down or gaining a few spots in the top 20 best rated games has a significant impact on sales.  It will take far less ratings to manipulate the Top Rated list order than it will to change the visible game rating.
  • The only thing to prevent a rating bot is one captcha – the same one Microsoft uses for sites like Hotmail.com.  If you can fool the captcha, automating a ratings bot is simple.  I’ll leave it up to others to judge how strong this captcha is.

Last time I looked at the numbers, Cthulhu Save the World was down and College Lacrosse 2011 was up.  In the past week  CStW is down another 1/4 point, and CL2011 is also down a 1/4 point.  I decided to dive into the numbers this time, and look for titles with 100 votes or more this past month that have had a visible ratings change.  Please remember from last week, there are numerous issue in the data available, so do not take the charts below as absolutes.

Here are the games that have received more than 100 votes in the past month, and have seen their ratings change:

rating_change

What stood out to me is of the game going down in rating, CStW, CL2011, Bird Assassin, and Epic Dungeon have a good chunk more than the others in terms of March votes.  Bonded Realities and FishCraft both have seen a spike in vote and their ratings have improved.  Before anyone jumps to conclusions though, I have another chart:

new_rating

This chart shows two calculated numbers based on the data I have.  The blue bars show the average vote the games needed to reach their new rating.  We can figure this out because we know the rating change, and the number of votes involved (Solve for X FTW).  The second red bar is showing the “pull” of the votes.  It is calculated by the difference of the average rating (blue bar) and the game’s initial rating.

To read this chart, CStW has received an average rating of 3.06 in the month of March, which is 1.44 stars lower than it’s initial rating of 4.5.    So the votes have been “pulling” CStW down.  Looking at the two gaining titles I mentioned above, Bonded Realities and FishCraft, we can see that even though these games saw an increase in ratings, the average rating wasn’t “pulling” that much.

“Mike, something is wrong with your chart!  ZP2K9 cannot get -1 Star votes, and Get Rich or Die Gaming can’t get 6 Star votes!”

I have two possible answers for this, one pretty plausible and a second I really hope isn’t true:

  • Because the data is limited to 1/4 vote increments, a game could really hold a 4.38 and be rounded to 4.5, then fall to 4.37 and be rounded to 4.25.  It would take a small number of votes to do this and to us would look like a .25 drop instead of a .01 drop.
  • It may be possible to negative vote and over vote – if you had a bot you would be able to send in any value for rating you wanted.  If Microsoft isn’t checking the value is one of the values of 1, 2, 3, 4,or 5 then it’s possible.  Microsoft isn’t verifying emails addresses used to create live accounts, so it’s possible they have been lazy here as well.  I really hope not!

I think what I draw from these charts is that there is a significant amount of games getting manipulated.  4+ Star games that have been out for months or years should not have a month of 2 Star ratings.  My personal hunch is that there is a bot or two out there and that this is not the result massive fans coordinating ratings or manually creating accounts to skew the list.  This comes from years of experience as a web developer and seeing most email spam and comment spam coming form just a few sources, that when block clear up the problem.

Since pretty charts are nice but hard to get exact valued from, here is the data used above:

Game 3/1 Rating 4/3 Rating Rating Change 3/1 Votes 4/3 Votes Vote Change Average New Rating Difference from Start
Cthulhu Saves the World 4.5 4 -0.5 1260 1933 673 3.06 -1.44
Epic Dungeon 4.5 4 -0.5 1555 1942 387 1.99 -2.51
Aphelion 2 4.25 3.75 -0.5 545 791 246 2.64 -1.61
radiangames Ballistic 4.25 3.75 -0.5 323 428 105 2.21 -2.04
College Lacrosse 2011 4.5 4.25 -0.25 1084 2318 1234 4.03 -0.47
Bird Assassin 4 3.75 -0.25 473 1060 587 3.55 -0.45
The Impossible Game Level Pack 4.25 4 -0.25 718 1028 310 3.42 -0.83
Globe Clicker 4.25 4 -0.25 2076 2336 260 2.00 -2.25
Aphelion 4 3.75 -0.25 1072 1314 242 2.64 -1.36
Arkedo Series – 03 PIXEL! 4.5 4.25 -0.25 2117 2354 237 2.02 -2.48
Techno Kitten Adventure 4 3.75 -0.25 1127 1362 235 2.55 -1.45
ZP2KX: Zombies & Pterodactyls! 4.25 4 -0.25 1164 1399 235 2.76 -1.49
Solar 4.25 4 -0.25 2720 2940 220 0.91 -3.34
Missile Escape 4.25 4 -0.25 1782 1996 214 1.92 -2.33
The TEMPURA of the DEAD 4.25 4 -0.25 488 701 213 3.43 -0.82
ZP2K9 4.25 4 -0.25 3950 4144 194 -1.09 -5.34
radiangames Inferno 4.25 4 -0.25 922 1078 156 2.52 -1.73
Soulcaster 4.25 4 -0.25 1234 1383 149 1.93 -2.32
radiangames JoyJoy 4.25 4 -0.25 1185 1318 133 1.77 -2.48
Battle High:  San Bruno 4 3.75 -0.25 203 304 101 3.25 -0.75
Avatar Hamster Panic 4 3.75 -0.25 730 830 100 1.93 -2.08
Bonded Realities 4 4.25 0.25 322 1095 773 4.35 0.35
FishCraft 3.75 4 0.25 160 738 578 4.07 0.32
Get Rich or Die Gaming 2.5 2.75 0.25 2456 2628 172 6.32 3.82
SFG Office Brawlers 3.75 4 0.25 519 685 166 4.78 1.03
Zombie Football Carnage 3.5 3.75 0.25 208 369 161 4.07 0.57
Clover: A Curious Tale 3 3.25 0.25 120 242 122 3.50 0.50
M2AF: Message 2 All Friends 3.25 3.75 0.5 205 401 196 4.27 1.02
Dreams of Witchtown 3.25 3.75 0.5 263 397 134 4.73 1.48
Zombie Sniper HD 2.75 3.5 0.75 311 454 143 5.13 2.38

Leave a comment