Last week I posted some numbers on the Xbox Live Indie Game ratings scandal. Since then little has changed. Microsoft has only issued a tweet that “top men” are on the case with no follow up as of yet, and the ratings continue to show signs of manipulation. If you want the drama, I recommend reading the Armless Octopus article on Crosse Studio considering legal action against Zeboyd Games. Yes gents, we’ve gone plaid. While the claims of legal action are entertaining, I’m more curious by what story the numbers tell.
- This mess started after A “How to Vote” video was posted for Lacrosse Facebook fans that didn’t have an Xbox
- A mass of votes came into the XBLIG channel that not only boosted Lacrosse games, but down voted other games
- There is no security on Xbox.com ratings by Microsoft – this is over looked in the gaming “press” covering the story. You do not need to download the game, own an Xbox, or even create an Xbox Live account with a valid email to rate games. You can simply go to xbox.com, make up a fake account, and rate.
- The rating of a game isn’t as important as it’s position on the Top Rated list. Falling down or gaining a few spots in the top 20 best rated games has a significant impact on sales. It will take far less ratings to manipulate the Top Rated list order than it will to change the visible game rating.
- The only thing to prevent a rating bot is one captcha – the same one Microsoft uses for sites like Hotmail.com. If you can fool the captcha, automating a ratings bot is simple. I’ll leave it up to others to judge how strong this captcha is.
Last time I looked at the numbers, Cthulhu Save the World was down and College Lacrosse 2011 was up. In the past week CStW is down another 1/4 point, and CL2011 is also down a 1/4 point. I decided to dive into the numbers this time, and look for titles with 100 votes or more this past month that have had a visible ratings change. Please remember from last week, there are numerous issue in the data available, so do not take the charts below as absolutes.
Here are the games that have received more than 100 votes in the past month, and have seen their ratings change:
What stood out to me is of the game going down in rating, CStW, CL2011, Bird Assassin, and Epic Dungeon have a good chunk more than the others in terms of March votes. Bonded Realities and FishCraft both have seen a spike in vote and their ratings have improved. Before anyone jumps to conclusions though, I have another chart:
This chart shows two calculated numbers based on the data I have. The blue bars show the average vote the games needed to reach their new rating. We can figure this out because we know the rating change, and the number of votes involved (Solve for X FTW). The second red bar is showing the “pull” of the votes. It is calculated by the difference of the average rating (blue bar) and the game’s initial rating.
To read this chart, CStW has received an average rating of 3.06 in the month of March, which is 1.44 stars lower than it’s initial rating of 4.5. So the votes have been “pulling” CStW down. Looking at the two gaining titles I mentioned above, Bonded Realities and FishCraft, we can see that even though these games saw an increase in ratings, the average rating wasn’t “pulling” that much.
“Mike, something is wrong with your chart! ZP2K9 cannot get -1 Star votes, and Get Rich or Die Gaming can’t get 6 Star votes!”
I have two possible answers for this, one pretty plausible and a second I really hope isn’t true:
- Because the data is limited to 1/4 vote increments, a game could really hold a 4.38 and be rounded to 4.5, then fall to 4.37 and be rounded to 4.25. It would take a small number of votes to do this and to us would look like a .25 drop instead of a .01 drop.
- It may be possible to negative vote and over vote – if you had a bot you would be able to send in any value for rating you wanted. If Microsoft isn’t checking the value is one of the values of 1, 2, 3, 4,or 5 then it’s possible. Microsoft isn’t verifying emails addresses used to create live accounts, so it’s possible they have been lazy here as well. I really hope not!
I think what I draw from these charts is that there is a significant amount of games getting manipulated. 4+ Star games that have been out for months or years should not have a month of 2 Star ratings. My personal hunch is that there is a bot or two out there and that this is not the result massive fans coordinating ratings or manually creating accounts to skew the list. This comes from years of experience as a web developer and seeing most email spam and comment spam coming form just a few sources, that when block clear up the problem.
Since pretty charts are nice but hard to get exact valued from, here is the data used above:
|Game||3/1 Rating||4/3 Rating||Rating Change||3/1 Votes||4/3 Votes||Vote Change||Average New Rating||Difference from Start|
|Cthulhu Saves the World||4.5||4||-0.5||1260||1933||673||3.06||-1.44|
|College Lacrosse 2011||4.5||4.25||-0.25||1084||2318||1234||4.03||-0.47|
|The Impossible Game Level Pack||4.25||4||-0.25||718||1028||310||3.42||-0.83|
|Arkedo Series – 03 PIXEL!||4.5||4.25||-0.25||2117||2354||237||2.02||-2.48|
|Techno Kitten Adventure||4||3.75||-0.25||1127||1362||235||2.55||-1.45|
|ZP2KX: Zombies & Pterodactyls!||4.25||4||-0.25||1164||1399||235||2.76||-1.49|
|The TEMPURA of the DEAD||4.25||4||-0.25||488||701||213||3.43||-0.82|
|Battle High: San Bruno||4||3.75||-0.25||203||304||101||3.25||-0.75|
|Avatar Hamster Panic||4||3.75||-0.25||730||830||100||1.93||-2.08|
|Get Rich or Die Gaming||2.5||2.75||0.25||2456||2628||172||6.32||3.82|
|SFG Office Brawlers||3.75||4||0.25||519||685||166||4.78||1.03|
|Zombie Football Carnage||3.5||3.75||0.25||208||369||161||4.07||0.57|
|Clover: A Curious Tale||3||3.25||0.25||120||242||122||3.50||0.50|
|M2AF: Message 2 All Friends||3.25||3.75||0.5||205||401||196||4.27||1.02|
|Dreams of Witchtown||3.25||3.75||0.5||263||397||134||4.73||1.48|
|Zombie Sniper HD||2.75||3.5||0.75||311||454||143||5.13||2.38|
Leave a comment