Loyalty programme testing


Sky’s loyalty team was set to launch a new customer rewards programme. However, with rewards costs perceived to be high by business executives, the client was under pressure to prove the return on investment of the scheme before going to full launch. The decision was made to trial the programme and we were enlisted to analyse whether the business case for the scheme stacked up.


Measuring loyalty schemes is notoriously difficult. To be robust, we needed to assess the change of behaviour as a result of the loyalty scheme, and the loyalty scheme only.

Often it is assumed that we can compare customers that join the scheme with those that decide not to (assuming both groups have behaved similarly historically). This is wrong. The very fact that some customers decide to join the scheme means they have a stronger affinity with the brand. This may then mean that their behaviours would have changed anyway. We face this conundrum with many clients, however with Sky we were in the fortunate position that we were in a trial period, so they had some hold out customers.

Instead, the challenge was the complexity of the scheme and the myriad of the different reward qualifying behaviours we needed to model within the analysis to create a like for like trial, as opposed to control comparisons. We needed to consider a range of factors including whether the customer qualified for a SkyQ upgrade, the products they owned, tenure, contract status, likelihood to churn and channel of interaction. The majority of these factors were linked to either the KPIs used to build the original business case or the benefits given away as part of the scheme (versus how the customer would ‘earn’ the benefit without being in the scheme).


Given the complexity of the scheme, it was important to model the different motivations that customers may have for joining the scheme and the resulting change of behaviour. To gain better insight into this, we spent a lot of time with a variety of different Sky stakeholders looking at the indicators of churn, how people are ‘saved’ if they call customer services and the upgrade process for different products.

Once we understood this, we split out the different types of customers – trial vs control – for measurement. However, the complexities were so great that the resulting groups were too small for robust analysis. We solved this by using a clustering algorithm. This enabled us to bring similar groups together until we had the right balance of robustness due to volume and robustness due to comparing like with like customers.


We were able to robustly assess the scheme and found that the business case did indeed stack up, although some of the churn metrics required more time to mature as the trial had only been live a few months.

Business Impact

On the basis of our analysis, Sky decided to continue with the scheme and it is now live across all customers.