Tuesday, September 1, 2009

Blizzard's Raid Testing: Practically Nonexistent


Looks like I'm taking a ride on the WAAAAHmbulance this week. Another fun little tidbit from Blizzcon that bunched my panties up but good, taken from a wow.com liveblog of the Dungeon and Raids panel :

1:55PM Hard modes get iterated a lot because they're difficult to test.

1:54PM How extensive is your raid testing, and did anybody internally kill Yogg+0 pre-nerf? Daelo: Encounter team is 5-people, but we bring in other leads and raiders on staff for the 10-man. It's hard to get 25 developers together, but QA does those. "It's pretty much impossible for 25 people on QA to reach the skill level of a good raid. They didn't beat Yogg+0, they got spanked pretty hard by Yogg+4."

Yeah. Internally, they don't even put together a 25-man raid good enough to clear Ulduar.


My favorite part?

"Encounter team is 5-people"

Our millions of $ at work. Considering how many people play WoW, these 5 better be paid like pro athletes and worth every cent.

And:

"It's hard to get 25 developers together"

Gives you a good idea of how big their dev team is.



Don't get me wrong: the game is still worth paying for. But it could be so much better if they'd loosen the belt a notch and hire a few more people. And there's really no excuse. The content patches are part of the promise they make us when we pay the game. If they weren't doing these "free" patches, we wouldn't keep paying the $15, period. So now it's just a matter of the patch frequency and size. Both can afford to increase, given the amount of money we pay Blizzard all told.

I'll get into my Cataclysm impressions soon, but I do want to temper this post with the admission that I'm impressed with the promises they made for the xpac, and am hoping that the execution bears out my current suspicion that content is sparse right now because overhauling Azeroth is such a large undertaking. Please let me be right. Blizzard is burning through my customer goodwill at an alarming rate.

Champions and Aion, HOOOOO!

6 comments:

Peter Moss said...

Very interesting...

I always thought Bliz would have a crack 25 man team, and at least an employee that was highly skilled in each role in the game.

The PTR must provide a lot of data for them, but to not be able to beat their own content (without a Martin Fury buff) seems a little piss weak...

Hirvox said...

I'm not surprised. Some top-of-the-line guilds have long claimed that Blizzard employees watch their raids live and GMs respond almost immediately if they have technical issues on world-first attempts, especially on PTRs.

Also, it's somewhat depressing that people still fail to grasp these simple truths: Blizzard is a corporation. The primary goal for a corporation is to make profit. Profit is what you get when you subtract expenses from revenue. Therefore, the corporation will try to maximize revenue while minimizing expenses. If investments will not result in a proportionally larger increase in revenue, then those investments will not be made. How many new subscribers would a a bigger raid testing team bring into the game? Or how many unsubscribings they could avoid with that raid testing team? As long as people are willing to pay for barely tested content, that content will remain barely tested.

Hatch said...

Sigh...

Thank you for that Startling Revelation that I Never Heard Before because I Just Crawled Out From Under a Rock and Didn't Read Ayn Rand Freshman Year of College and Take Econ Classes in Grad School or Even Participate In a Forum Discussion In My Entire Life.


Yes, everyone knows corps are here to make profit. But it's not as simple as "we will do whatever we want as long as you keep paying". If it were that simple, then things would be a lot worse. WoW would have to get a LOT worse to convince a noticeable number of people that it isn't worth paying for. And yet they keep making cool stuff that they don't HAVE to make in order to keep us (patch 3.22, for instance). Why would that be, in your simplistic "Blizz is only motivated by whether or not you are still paying subscription" worldview?

There's also customer goodwill (which CAN be monetized, and Blizzard does monetize very effectively). There is trust (translating as first-day sales of new games). There is overall perception of the game, which does translate into money, but is hard to track because people aren't going to list for you all the myriad things you did wrong to finally get them to unsubscribe.

If I unsubscribed every time they did something I didn't like, then why bother? They make a million choices a day, of course I'll hate some of them.

Just because I keep paying doesn't mean I should stop complaining or judging. And neither should anyone else. Keep letting them know when you piss them off!


I want to publicize this crap because, if enough of it builds up and people know about it, then they WILL stop paying for things, starting with Blizzcon toys and ending eventually with subscriptions and boxed games.

For the amount of money we pay them, we deserve better.

What I would like to communicate to Blizzard, if they are listening, is that this shit IS going to cost them money. I may still pay for my main account, but I did cancel my second account (from when I leveled RAF) specifically because of the underfunding of the game, and I also chose not to buy the authenticator I was considering.

But hey, I'm just another idiot who doesn't understand that corporations are here for profit, right?

I guess I have to spell this shit out every time I talk about this, because there's always some internet economist who thinks they are handing down some revelation when they say companies are here for profit and not to serve my whims.

Hirvox said...

The strawmen are especially thick in this time of year, it seems.

But it's not as simple as "we will do whatever we want as long as you keep paying".
No it isn't, but it's not far off. It's "we'll do the the most cost-efficient thing that'll keep as many as possible paying."

Why would that be, in your simplistic "Blizz is only motivated by whether or not you are still paying subscription" worldview?
That's not my world view, but I'll bite anyway. Because there's new competitors (Champions and Aion, HOOOOO!) in the horizon, because Icecrown Citadel is still a work-in-progress, because it's a marketing ploy with minimal implementation cost, because they need to keep throwing bones to keep customers interested? Again, you said it yourself: If they weren't doing these "free" patches, we wouldn't keep paying the $15, period.

But back to the raid testing.. Because Blizzard in effect gets free testing from the top-of-the-line raiding guilds (who probably as a whole do a much more through job than a Blizzard raiding team would/could), they can fix the most glaring errors (BWL doors, no-show Nefarian and C'thun's stomach tentacles, anyone?) before those bugs become the proverbial straws that break the masses' backs. PTRs are a great way to harness all that intangible goodwill into tangible savings in testing. Why pay for something when there's a suitable, better, free alternative available? Employing more people to test raids in-house would cost (in terms of opportunity cost, anyway) much more than the current encounter testing, and would not yield comparable increases in revenue or even in perceived quality by the majority of the customers. Therefore, there is no permanent in-house raid test team and won't be, unless PTRs will somehow become more ineffective. Say.. due to customer apathy/disillusionment caused by angry posts, perhaps?-)

If I unsubscribed every time they did something I didn't like, then why bother?
I never said you should.

I guess I have to spell this shit out every time I talk about this
I know exactly how you feel. Captain Obvious signing off.

Ixobelle said...

while we're all just free form postulating, I wonder what would happen if the PTR was boycotted?

would they actually hire testers, or just push untested dungeons out on Live? I mean, the PTR wasn't always around...

Hirvox said...

while we're all just free form postulating, I wonder what would happen if the PTR was boycotted?

would they actually hire testers, or just push untested dungeons out on Live? I mean, the PTR wasn't always around...


That depends on the difficulty curve of the encounters. If there's a c**kblock encounter or two in the way, then Blizzard has a safety buffer between top-of-the-line guilds discovering the bug and the masses reaching said encounter. As long as they get the fix in place before that, Blizzard will be fine.

On the other hand, a buggy encounter easily reachable by the players would be much more catastrophic. Therefore Blizzard keeps that five-man testing team around, because dungeon encounters can be reached as soon as the players reach the proper level. Ditto for quests.

Also, it's notable that Blizzard's currently using a hybrid approach: Early (read: easily reachable) encounters are tested on PTRs while the end bosses are kept off it. Of course, savings in testing is only one of the myriad reasons for that.