How to Judge an MMO

I have had very limited time to play/write recently.  And when I do get those small breaks, I almost always choose to play versus writing.  So if any of you are wondering for the large lulls recently, I am to blame, and I apologize for that.

That being said, I wanted to take a moment to analyze how we judge an MMO. Afterall MMOs, cannot be judged like books, movies, are even other video games.  MMOs are normally designed to be played over several years and are constantly being updated. To make the matters worse, they are far more complicated.  They have varying arrays of large scale systems that imply intricate virtual worlds.  How do you judge an MMO, a style of game, that is so vast, so self-perpetuating, and so incredibly complex?

Everywhere you look people have a different preference.  IGN, Metacritic, MMORPG, or just your daily forum posters.  With Guild Wars 2 having such a massive lead-off of hype prior to it's release, millions of people have been forming opinions of the game.

The problem really begins when those people start trying to write up their opinions as wide-spread fact and try to review the game.  I actually made a similar mistake.  I tried to write up a lengthy review of all of the major aspects of a game, assigning numerical values to each, and coming up with a total score.

The first and most obvious question is: When is it fair to write the review?  At what time point (since beginning playing) can I write knowledgeably about an MMO?

I will largely ignore this first question.  The answer is too subjective for each reader to decide. Some suggest with in a few days, other a few weeks, and some even a few years.  The lines have been drawn depending on the person, and there's not much you can do to change someone's mind on the issue.

I'm of the belief that you should try out every game mode possible, and have substantial experience with each to know the major aspects of the game.  I also normally believe you should reach level cap before being fully capable of writing a lengthy review.

But then the second problem becomes:  How do I actually judge an MMO?  In small categories that sum up to be the review of the entire game?  And if so, what and how many categories?

My first attempt was to base it on the actual pieces of the game:  World, Dungeons, sPvP, WvW, Personal Story, Combat, etc.  But then that brings up the problem of comparison between games.  Many reviewers use numerical scores, and those scores lose significant value if they are incomparable to multiple games.

So then the second attempt was:  Visuals, PvP, Story, Combat, World.  But then the question became, should I include Customer Support, UI, game instability, or should I incorporate overall value into the score?  Afterall a B2P game is almost always significantly cheaper than a subscription based game.

However, let's continue playing this game and say I chose my categories, and we all relatively agreed on them.  Then even more problems arise.  Now more than ever, games can change relatively fast, particularly in MMO space.  With significant patches or expansions, do you need to re-review the game?  If so, when?  And how do you fairly compare two games that came out at different times?  Is it fair to judge WoW-dated graphics to Guild Wars 2 graphics simply because both are releasing some form of content at similar times? And then should you form numerical values for the sub-components of the game, and should they be judged equally?  If you do, what scale should the values be on, and do you include tenths of a point, or only halves?
If your scoring system changes over time specifically on a new release, are you obligated to change your past scores?

I will say this for now.  I do not envy anyone who tries to review a modern day MMO.  No matter what your system is, there will be substantial flaws in your model.