Category: Blog EN

The traditional survey-based approach to measuring customer satisfaction is no longer sufficient. Using the wealth of available data, companies are already breaking through the limitations of available methods. Data-driven customer satisfaction prediction and proactive actions taken based on it are the future of this area.

Customer satisfaction is crucial to maintaining customer loyalty and is the foundation for business growth in a competitive market. Not surprisingly, this topic has long been of interest to marketing professionals. The fruits of this interest are the many methodologies for measuring customer satisfaction described in the literature and used in practice. Among the most popular are the Customer Satisfaction Score, Customer Effort Score or Net Promoter Score (NPS). A common feature of the most popular approaches is their reliance on surveys, in which the consumer is asked properly prepared questions (or even just one question). It should be noted that measuring and analyzing customer satisfaction with the help of the aforementioned surveys has proven successful in many cases. They have been and continue to be an important element in the market success of many companies. However, these methods are not without their drawbacks. According to a survey conducted by McKinsey&Company, among the most frequently mentioned by CS/CX professionals appear:

• limited range
• delay of information
• ambiguity of responses making it difficult to act on them

Limited coverage

According to the study cited earlier, a typical satisfaction survey collects responses from no more than 7% of a company’s customers. The reasons for this are multiple. Among them are budget constraints (the cost of the survey), lack of an available channel of communication with the customer or lack of consent to communication, low customer interest in answering questions. Importantly, the propensity to respond to a survey may vary depending on certain characteristics or customer experience (e.g., dissatisfied customers may respond more readily). This puts a big question mark over the representativeness of the results obtained and their generalizability to the company’s entire consumer base.

Information delay

Surveys by their very nature lag behind the phenomenon they are investigating. This makes it impossible to take pre-emptive action. We can only take action against a dissatisfied customer after he or she has completed the survey. Practitioners in the field of customer satisfaction often emphasize how big a role reaction time plays in problematic experiences. An excellent satisfaction measurement system should operate in near real time. This will make it possible to take immediate action to solve a customer’s problem or erase a bad impression formed in the customer’s mind. Ideally, it should be able to predict a customer’s growing dissatisfaction before it manifests itself in the form of a negative rating in a survey or a nerve-wracking contact with customer service. Such an opportunity does not exist in survey-only measurement systems.

Additional delays in collecting information result from the limited frequency with which questions can be asked of the consumer. Typically, surveys are conducted after a certain stage of the consumer journey, such as after a transaction. Often, there is no opportunity to ask questions at earlier steps, where issues can also arise that negatively impact the customer experience. In extreme cases, a negative experience e.g. when placing an order may end up with the transaction not happening at all. If we survey only after the transaction, in such a situation we will not get any information, because the consumer will not meet the criterion (transaction) to receive a survey.

Of course, there are companies where survey processes are not subject to such restrictions. These companies conduct surveys after every stage of the customer journey and every time the customer interacts with the company. However, it is important to remember that a survey can be perceived by the customer as an intrusive tool. Thus, an excessive number of surveys alone can negatively affect his experience. Hence, it is necessary to maintain a reasonable balance between the frequency of surveys and the consumers’ willingness to respond. In turn, it is best to think of a slightly different solution, which we will present later in the article.

The ambiguity of the answers making it difficult to take
on the basis of them

This limitation is derived from the desire to maintain a balance between the company’s information needs and consumer satisfaction. It stems from both survey coverage of individual steps in the consumer journey (described above) and limitations in survey length. Short surveys (covering one or a few questions) are not very burdensome for the consumer and may also contribute to better response. However, reducing a survey to a single question can make it difficult to understand what factors actually influenced the consumer’s evaluation of the product in this way and not in that way. If we don’t know what, in the case of that consumer, caused the negative evaluation, it is difficult to take any action based on the result to correct the bad impression.

Limiting the number of moments in which questions are asked works similarly. The rating given by the consumer (for example, “seven”) after the transaction is completed is difficult to attribute to the different stages of the consumer’s path. Determining which stages caused the consumer to deduct points from the maximum score would require asking a series of in-depth questions. This increases the time it takes to complete the survey and reduces the chance of getting an answer. On the other hand, more frequent surveying (e.g., after each step of the consumer) also raises the problems described above.

Predictive modeling as a solution

Using machine learning, it is possible to build models capable of predicting consumer satisfaction at any stage of the consumer’s journey to finalizing a transaction. This, of course, requires integrating data from many different sources usually present in a company. Mention can be made here of sales data, from a loyalty program, from a customer service desk, a hotline, a website, financial data, or, finally, data from satisfaction surveys conducted to date. This data is needed at the individual consumer level. It also requires expertise in data science – you need a team capable of building and implementing predictive models. It is worth noting that in such a solution we do not completely abandon surveys. However, they change their character. They become a typically research tool. In turn, they cease to be a tool for ongoing measurement of satisfaction.

The system works in such a way that on the basis of data collected on an ongoing basis about the consumer, it predicts his current satisfaction rate at a given moment. Moreover, it also indicates which factors positively and which negatively influence this result. This makes it possible, firstly, to identify customers against whom it is necessary to take action, and secondly, to recommend specific actions to be taken against them. The whole thing makes it possible to consistently predict customer satisfaction in near real time and take effective action.

A predictive approach to customer satisfaction is certainly the future. Companies that are the first in their market to implement this type of solution will win the battle for the customer. Even if some service “mishap” happens (and this is inevitable in large organizations) they will be able to respond to it appropriately and quickly. Thanks to an efficient system based on prediction, the reaction can be so fast that the consumer won’t even have time to think about looking for a competing supplier.

Promoting a product, especially one that involves a reduction in its price, almost always results in an increase in sales. But does every increase mean that the promotion was profitable? How often does the promoted product take customers away from other substitute products? How to calculate how much is the actual “incremental” of the action?

The fundamental question to solve the problems identified in the introduction is: what if… Or more precisely: what if there were no promotion. How much would the sales of the promoted product have been? How much would other products (especially substitute products for the promoted one) have sold? On the surface, this seems impossible to determine. After all, we are asking about an alternative reality that we are unable to observe. It is impossible to introduce a promotion and not introduce it at the same time. However, it turns out that based on advances in statistics, data science and artificial intelligence research in recent years, we are able to estimate the aforementioned effects in a scientific, methodical and rigorous manner. The method used is based on so-called synthetic control groups. That is, to put it somewhat simply, comparison groups created by a special algorithm on the basis of available observations of sales of similar products.

We will analyze this using the example illustrated in the chart below. The red line shows the actual sales of the product (in units). You can see that before March, sales of the product were traceable. You can also see a clear weekly cycle with peaks on Saturdays and clear declines on Sundays (related to the trade ban and the limited number of outlets that can conduct sales). One can also see an upward trend in sales since the beginning of March. The black vertical dashed line shows the day the promotion started. The price of the product has been significantly reduced. A clear increase in sales can be seen.

The light blue dashed line is the algorithm’s estimated sales behavior of the promoted product if there were no promotion (alternative reality). It can be seen that even without the promotion there would have been an increase in sales (in line with the upward trend visible since the beginning of March). However, it would not have been as large. Therefore, it can be concluded that the promotion generated additional sales of the product.

The next chart shows a summary of incremental sales for each day. As in the earlier chart, the black vertical dashed line marks the beginning of the promotion period. Most of the bars are clearly above zero, indicating an estimated increase in sales relative to the base scenario (i.e., no promotion). The period preceding the promotion to the left of the dashed line is the calibration period. Based on this period, the algorithm learns the best combination of products that make up the comparison group (the so-called synthetic control group). The closer the bars in the calibration period are to zero, the better the matched comparison group. Of course, in real-world examples (such as the one presented in this article) it is difficult to find a perfect match. Hence, the bars deviate slightly from 0. What is important, however, is that the magnitude of these deviations is much smaller in the calibration period. This lends credence to the conclusion of a real positive effect of promotions on sales.

At this point, one could close the analysis and congratulate those responsible for the promotion. The question arises, however, to what extent the promotion attracted new customers or increased demand from existing ones, and to what extent it merely shifted demand from other complementary products that were not promoted during the period. In other words, to what extent did the promotion scanibalize sales of other products.

To answer this question, we will conduct a similar analysis to the one presented above. This time, however, the red line will represent sales of a substitute product to the promoted product. For this particular product, we want to estimate the cannibalization effect.

As before, the vertical black dashed line marks the date of the start of the action. After the start of the promotion, the red line is lower than the light blue dashed line, which means that the substitute product is sold less than it would have been sold in the baseline scenario assuming no promotion. It is also worth noting that in the period before the promotion (the calibration period) the two lines are very close to each other, which means that the algorithm has correctly calibrated the comparison group.

The chart below summarizes the effect of cannibalization by day. You can see that on each day of the promotional period the product sold less compared to the reality that the promotion would not have taken place.

In the case analyzed, the incremental sales of the promoted product amounted to 407 units. However, when evaluating a promotional action, one must take into account the effect of cannibalization. In this case, the loss of sales on a product substitutable to the promoted product amounted to 326 units during the promotional period. Without taking this factor into account, we could significantly overestimate the financial effect of the action and draw incorrect conclusions as to its profitability. This, in turn, could translate into suboptimal decisions on organizing similar promotions in the future.

The best way to measure effects is to conduct a randomized experiment. However, this is not always possible. It is difficult to imagine how to run a promotion and not run it at the same time and for the same group of consumers. In such situations, modern analytical methods based on synthetic control groups, among others, such as the one presented in today’s article, can be invaluable in marketing analysis.

Customer churn is one of the key challenges facing organizations in today’s highly competitive environment. In order to effectively combat customer churn, companies need to find answers to two key questions: which customers are at risk of churning, and what actions can be taken to stem the process.

With help comes a combination of two methods from the arsenal of data science: prediction and clustering (segmentation), which together increase the effectiveness of retention efforts.

I have already addressed the issue of prediction of the threat of customer churn in an article in the We Love Data So Let’s Date series. Due to the importance of the topic and the interest it has generated, I decided to deepen the issue and present an approach in which, by combining two machine learning methods, we can significantly facilitate the implementation of data-driven anti-churn activities.

Prediction and segmentation as a weapon in the fight against customer churn

The basic tool in countering customer churn is the predictive model. It allows predicting the probability of a particular consumer leaving. However, this may not be enough. Even the best model and the most accurate prediction will be of no use if we do not take appropriate action based on the information received.

Proper interpretation of the data allows us to plan activities and take actions – such as sending an sms message to customers or offering a discount on selected products. We then measure the impact of the action (effect) on consumer behavior and their propensity to abandon further use of the offer. Measuring and observing consumer behavior provides a source of new data that the predictive model can use. Thus the cycle closes, and this is illustrated in the diagram below

The predictive model indicates the likelihood of a specific customer churning. This allows us to prioritize tasks and use the usually limited resources on consumers most at risk of leaving. Information that a consumer has a 75% probability of leaving within the next quarter helps us decide that “something” must be done about it quickly. However, does it provide the knowledge of what to do? How does a company know what action to take? The key is to interpret the data correctly.

The ideal situation would be to be able to take action personalized to the individual consumer. That is, each customer would receive a unique customized offer tailored to their needs and problems. Predictive models built on the basis of appropriately selected machine learning methods make it possible to create an individual profile of each consumer. In addition, they indicate specific factors that in his case are associated with a higher risk of leaving. Despite advances in the area of hyperpersonalization of marketing activities, it is still not yet achievable on a large scale for many organizations. Not all activities can be automated as easily. For example, creative, content production or offer construction can be a limitation.

The solution in such a case is an approach based on advanced segmentation. However, it is definitely not about classic segmentation, which uses only basic variables like age or gender. They do not sufficiently differentiate the base, and the real dividing lines run quite elsewhere. It is important, therefore, that it be a behavioral segmentation, in which we look for similarities in customer behavior, taking into account the same broad aggregate of descriptive variables that was used to build the predictive model. In order to perform such a comprehensive segmentation, clairvoyant machine learning algorithms are required.

The procedure may look as follows:

1. The predictive model allows us to identify the group of customers most at risk.
2. We decide that we have the resources to act against the 20% of the most at-risk customers.
3. We select these customers and, using a clustering model, divide them into segments. Their number should be a product of the model’s indications and the resources we have to serve them (usually it will be several – a dozen).
4. We interpret the segments and plan actions.

The proposed method makes it possible to put into practice predictive modeling and the in-depth understanding provided by data science analysis of consumers at risk of leaving. The approach will also find application in companies not yet technologically and organizationally ready for fully automated hyperpersonalization efforts.

.

After a campaign has been carried out, marketers wonder whether and what kind of profit their action has brought. Summarizing the projects completed so far and planning future actions, they try to find the answer to this nagging question. They calculate the effectiveness in various ways. Usually, the greater the effect, the less inclined they are to reflect on whether their method of calculating efficiency is at all correct. Add to this some obvious, but still frequently made mistakes, and thus overinterpretation of the results is guaranteed. So is there a foolproof way that makes it clear how the effect of an action should be calculated?

Misjudging the effect of the action

Here is a simple example to illustrate the idea. Suppose we organize a campaign, rewarding all customers who have agreed to receive marketing e-mails. Everyone we can contact in this way will receive by e-mail a discount voucher for the amount of PLN 20. The idea meets with approval. The selection criteria are simple. The action is quickly implemented. So fast, in fact, that there was not enough time to think about how its effect would be measured. Somehow, however, it would be appropriate to measure it. After all, one can make a comparison of sales in the group that received the coupon to the group that was not qualified for this action. Well, that’s just it. After all, it’s simple.

With a base of 100,000 consumers, we generated 2700 additional transactions (2.7%100000 = 2700) and increased the value of 2400 transactions by PLN 5.12 (2.4%100000 = 2400). Thus, the total turnover generated was 301 thousand zlotys. The cost of the vouchers used was 102 thousand zlotys (5100 * 20 zlotys = 102000).
So success. Is it?

Such a solution is very simplistic. In fact, one could even use the term naive, and it distorts (usually overstates) the incremental effect of the action. This is because the above method of calculation ignores one important detail. The groups being compared are different from each other, and not just the fact of receiving a voucher. The group with the voucher receives e-mail communication, while the group without the voucher simply does not receive such communication. The assumption that the group that has consented to e-mail communication will be more likely to make purchases, even if it does not receive a voucher, is plausible. If we assumed the opposite, it would mean that communication has no effect on sales, yet we know that this is not the case.

First: plan in advance how to analyze the effects

So what to do in this situation? The best approach, would be to plan how to analyze the effects even before launching the action. That way, an appropriately sized control group could be drawn from among all customers who meet the criteria. This group would not receive a voucher. This would be the only significant difference from the group receiving the voucher. This would make the control group a better and more reliable background for comparison. A similar campaign was indeed carried out for one of our clients, but its effects were measured correctly. Even before the mailing, we selected the appropriate volume of the control group from among those with consent to communicate via email. The results of the correct comparison can be seen in the table below.

We can still see a positive difference in the percentage of the group with a transaction. However, it is much lower in the case of the previous comparison and is only 0.8% According to the new estimates, we generated 800 additional transactions (0.8%100000 = 800) and increased the value of 4300 transactions by PLN 7.79 (4.3%100000 = 4300 transactions). As in the previous case, the cost of the used vouchers amounted to 102,000 (5100 * 20 PLN = 102000). Thus, the total turnover generated was 119 thousand zlotys (85.5 thousand zlotys + 33.5 thousand zlotys). So it was not much higher than the cost of the discount given in connection with this action. So it is difficult to declare success. However, conclusions should be drawn and appropriate changes should be made when planning future actions (e.g., better selection of the value of the discount, adoption of other criteria for the selection of customers). However, should one also conclude that the described method – an experiment based on random control groups (also called A/B testing) – must always be used for this kind of analysis? Yes, but…

What if a control group cannot be distinguished?

Unfortunately, this approach is not always feasible. And this is its main drawback. There are legal constraints in certain situations (e.g., regulatory interpretations that limit the ability to discriminate against customers and mandate that benefits be given to all who meet certain criteria), as well as marketing constraints. For example, a company may not want to run the risk of causing dissatisfaction among customers who are cut off from benefit opportunities as an experiment. The risk is greater the higher the value of the benefit in the eyes of customers. So in such situations, are we doomed to a falsely simplistic approach or must we abandon analyzing the effects of a marketing action altogether?

Who are the statistical twins?

Not necessarily. Fortunately, we do not have to assume such drastic scenarios and give up on measuring the effects of the campaigns carried out. There are advanced statistical methods that allow even without a control group to estimate the actual effect. These methods, to put it simply, are based on the search for statistical twins. Statistical twins are customers with as similar characteristics as possible, among whom only one was subjected to an incentive: he was covered by a promotion, we sent him a text message, displayed an online advertisement to him, or took other actions towards him to encourage him to use our offer. This creates a synthetic “control group” consisting of one twin from each pair, the one who did not participate in the action. The challenge in this case is to identify the key variables that guarantee the similarity of the groups. However, appropriate computer software helps carry out this process. This opens up new possibilities for analyzing data and optimizing decisions in situations where conducting a randomized experiment is not possible, or is difficult or uneconomical.

As we can see, it happens that the effects of marketing campaigns carried out are either miscalculated or the results are overinterpreted. Therefore, even before implementing a campaign, it is necessary to determine how we intend to study its effects. Advanced statistical methods and machine learning help in many problematic situations, such as the inability to compile an optimal control group.