 |
| Not that kind of engagement |
That sentence might seem a bit surprising but I can assure you that it's 100% true. The primary culprit here is engagement bias, which is something you have to consider when you're analyzing a game-as-a-service, like WoW. Suppose 7 million people play WoW in a given week. Let's look at them by how engaged they hypothetically are (as measured by how many days they played that week).
| Engagement | Player Count | % played today | # played today | % DAU in bucket |
| 1 Days | 1,000,000 | 14.29% | 142,857 | 3.57% |
| 2 Days | 1,000,000 | 28.57% | 285,714 | 7.14% |
| 3 Days | 1,000,000 | 42.86% | 428,571 | 10.71% |
| 4 Days | 1,000,000 | 57.14% | 571,429 | 14.29% |
| 5 Days | 1,000,000 | 71.43% | 714,286 | 17.86% |
| 6 Days | 1,000,000 | 85.71% | 857,143 | 21.43% |
| 7 Days | 1,000,000 | 100% | 1,000,000 | 25% |
Here we see that if you look at the people who play on a particular day (DAU - Daily Active User), there is a distinct bias towards users who have a higher weekly engagement. Side note: players who play in a given week are called WAU. Even though the WAU are evenly distributed among the engagement buckets, the DAU are heavily skewed towards the highly engaged. Then again, WAU isn't how Blizzard likely defines 'player' for WoW, they likely use subscribers as the definition of the player, since that's how they get their money and the $15 a low-engaged player gives them is the same as the $15 a heroic raider sends them.
What this means is that the people that you see every day in the game aren't really a good representation of WoW's subscriber base. People aren't as engaged with the game as they appear to be. From a development and design standpoint, the highly-engaged users are the least likely to let their subscription lapse, so features are often made to appeal to the casual crowd/make casual players more engaged. If you look at the history of WoW this is what you'll see. Even heroic raiding was oriented around this because it allowed them to make regular raiding easier and more accessible to the casual player.
This is just one example of engagement bias, which is a recurring problem in user-centric data analysis and therefore is a recurring problem in the games-as-a-service industry. Engagement bias is the phenomenon that more active users are often more likely to be counter/sampled.
Back when I was working on analyzing the results of my 2011 WoW Survey, one question I wanted to answer was "What are the correlations between classes?" meaning that I wanted to know which classes a player was more or less likely to play if they played another class. For example, "Are people who play Warlocks more or less likely to play a Death Knight than someone who plays a Rogue?"
Suppose that the average respondent to my survey listed two different classes among the ones that they play. At the time, this means that roughly 20% of respondents played any particular class (class representation actually varied wildly). When I pulled the percent of Warlock players that ALSO played Paladins I found a much higher number, 40% or greater. This baffled me for a long time. For each combination of classes, this same thing happened, the percentage of X players that also played Y was higher than the percent of the general population that played class Y.
Why was this?
Among the people that I surveyed, they varied widely among the number of characters they played. Some people only listed 1 or 2 characters, some listed 10 or more. When I selected all the players who played a Warlock, the highly-engaged players (those with more characters) were more likely to be in that group than the low-engaged players (those with few characters). So the group of Warlock players had, on average, more characters than the general population. So when I calculated how many of them also played Paladins, I received a much higher number than with the general population.
Of course, there was something else that would skew the results of my analysis. I got my data not via the actual numbers but by getting survey results that mainly came from MMO-Champion. Since these are people that are participating in the WoW community, they are going to tend to be more engaged than the general WoW playing population.
Engagement bias is just one of the many things you have to keep in mind when you're analyzing game players. For example, during my WoW survey, I also found that MMO-Champion users tend to skew more male than respondents from other sources that I've used. For this reason and more when I was doing my analysis I was careful to make sure to state that the numbers were not to be taken as absolute facts, but as being "directional", meaning that it'll likely indicate what the differences between two groups or what may be more or less popular for a group even if the exact values aren't true for the overall population.
This is just one of the slew of problems that you run into when doing user-facing data analysis, something which I'll be covering in a later post.
Engagement bias is just one of the many things you have to keep in mind when you're analyzing game players. For example, during my WoW survey, I also found that MMO-Champion users tend to skew more male than respondents from other sources that I've used. For this reason and more when I was doing my analysis I was careful to make sure to state that the numbers were not to be taken as absolute facts, but as being "directional", meaning that it'll likely indicate what the differences between two groups or what may be more or less popular for a group even if the exact values aren't true for the overall population.
This is just one of the slew of problems that you run into when doing user-facing data analysis, something which I'll be covering in a later post.
