Predict User Churn with Accuracy

At Predikat, we can help you understand your users and customers, while also identifying which users are at the highest rick of churn. Recently, we were able to predict churn with 88% accuracy when looking at the multi-player game,  ​World of Warcraft. 
​
We were able to drive such strong results with the  use of a Customer Churn Machine Learning module to predict whether or not a given user will stop using an application after a certain amount of time, and we did this by observing their past behaviour, and  how the user interacted with the application, and what actions they chose to take during gameplay.

We recently applied our Customer Churn Machine Learning module to   World of Warcraft and an avatar history data set to predict churn at different points of time in a player's journey. We examined the World of Warcraft  avatar history data set to: 

1) Decide a simple set of user behavioural events 
2) Create a robust set of features extracted from those events 
3) Apply a machine learning model on top of the events and features that accurately predicts whether a user will churn or not. 

The World of Warcraft Avatar History (WoWAH) dataset, includes the records of 91, 065 users, including the users game play times, and a number of in-game  and demographic attributes about their character level, race, class, location and social guild during a 1,107 day period between January 2006 and January 2009. 

For our research, and the purpose of this assessment, we used a preprocessed version of this dataset as found in Kaggle with data from 2008. 

In our first experiment, we applied the simplicity principle and dropped the irrelevant information in additional to restructuring the original data to isolate the datasets based on the session length. With these datasets, we looked at the building features and then we created the churn flag for training our models. 

Given that there was a continuous set of sessions spread throughout the timeline, the churn prediction problem could not yet be defined. However, the problem we identified was defined as follows:

• Observe a user for 7 days 
• Wait for a period of time of different lengths (7, 14, 30, 60) 
• Observe if they log back in following 7 days 
• If they didn't log back in, then it's a churn event 

Here, we predicted churn for different future points in time: churn at day 7, day 14, day 30, and day 60. It is important to note that predicting further in the future is more difficult, so accuracy and AUC (Area Under the Curve) are expected to drop to some extent as this horizon increases. 

THE RESULTS 
The results we found are reported for a 5-fold cross validation:

After assessing the data, and looking at the various points of time, our results showed an accuracy of 88% which means that out of the 100 predictions, 88 are correct. 

Since the dataset is imbalanced (roughly 80% of  users are churners in this case), a trivial model predicting churn sets the floor accuracy to 80%. Since there is a known problem with accuracy, the most relevant metric in terms of model quality for the Machine Learning team is the AUC. What we discovered is that it is consistently good (1 is perfect and 0.5 is bad), and our results show us in the 0.78-0.87 range. 

This observation of WoW datasets, shows that with our machine learning software, we can identify the users that are likely to churn, while also predicting the user behaviour to supply actions that they will likely engage in, preventing churn and boosting re-engagement. 

Find out more about predicitive analytics, and how they can help your business in predicting customer behaviour, and reducing churn.  Connect with us today, and  improve your user experience .