After a campaign has been carried out, marketers wonder whether and what kind of profit their action has brought. Summarizing the projects completed so far and planning future actions, they try to find the answer to this nagging question. They calculate the effectiveness in various ways. Usually, the greater the effect, the less inclined they are to reflect on whether their method of calculating efficiency is at all correct. Add to this some obvious, but still frequently made mistakes, and thus overinterpretation of the results is guaranteed. So is there a foolproof way that makes it clear how the effect of an action should be calculated?
Misjudging the effect of the action
Here is a simple example to illustrate the idea. Suppose we organize a campaign, rewarding all customers who have agreed to receive marketing e-mails. Everyone we can contact in this way will receive by e-mail a discount voucher for the amount of PLN 20. The idea meets with approval. The selection criteria are simple. The action is quickly implemented. So fast, in fact, that there was not enough time to think about how its effect would be measured. Somehow, however, it would be appropriate to measure it. After all, one can make a comparison of sales in the group that received the coupon to the group that was not qualified for this action. Well, that’s just it. After all, it’s simple.
With a base of 100,000 consumers, we generated 2700 additional transactions (2.7%100000 = 2700) and increased the value of 2400 transactions by PLN 5.12 (2.4%100000 = 2400). Thus, the total turnover generated was 301 thousand zlotys. The cost of the vouchers used was 102 thousand zlotys (5100 * 20 zlotys = 102000).
So success. Is it?
Such a solution is very simplistic. In fact, one could even use the term naive, and it distorts (usually overstates) the incremental effect of the action. This is because the above method of calculation ignores one important detail. The groups being compared are different from each other, and not just the fact of receiving a voucher. The group with the voucher receives e-mail communication, while the group without the voucher simply does not receive such communication. The assumption that the group that has consented to e-mail communication will be more likely to make purchases, even if it does not receive a voucher, is plausible. If we assumed the opposite, it would mean that communication has no effect on sales, yet we know that this is not the case.
First: plan in advance how to analyze the effects
So what to do in this situation? The best approach, would be to plan how to analyze the effects even before launching the action. That way, an appropriately sized control group could be drawn from among all customers who meet the criteria. This group would not receive a voucher. This would be the only significant difference from the group receiving the voucher. This would make the control group a better and more reliable background for comparison. A similar campaign was indeed carried out for one of our clients, but its effects were measured correctly. Even before the mailing, we selected the appropriate volume of the control group from among those with consent to communicate via email. The results of the correct comparison can be seen in the table below.
We can still see a positive difference in the percentage of the group with a transaction. However, it is much lower in the case of the previous comparison and is only 0.8% According to the new estimates, we generated 800 additional transactions (0.8%100000 = 800) and increased the value of 4300 transactions by PLN 7.79 (4.3%100000 = 4300 transactions). As in the previous case, the cost of the used vouchers amounted to 102,000 (5100 * 20 PLN = 102000). Thus, the total turnover generated was 119 thousand zlotys (85.5 thousand zlotys + 33.5 thousand zlotys). So it was not much higher than the cost of the discount given in connection with this action. So it is difficult to declare success. However, conclusions should be drawn and appropriate changes should be made when planning future actions (e.g., better selection of the value of the discount, adoption of other criteria for the selection of customers). However, should one also conclude that the described method – an experiment based on random control groups (also called A/B testing) – must always be used for this kind of analysis? Yes, but…
What if a control group cannot be distinguished?
Unfortunately, this approach is not always feasible. And this is its main drawback. There are legal constraints in certain situations (e.g., regulatory interpretations that limit the ability to discriminate against customers and mandate that benefits be given to all who meet certain criteria), as well as marketing constraints. For example, a company may not want to run the risk of causing dissatisfaction among customers who are cut off from benefit opportunities as an experiment. The risk is greater the higher the value of the benefit in the eyes of customers. So in such situations, are we doomed to a falsely simplistic approach or must we abandon analyzing the effects of a marketing action altogether?
Who are the statistical twins?
Not necessarily. Fortunately, we do not have to assume such drastic scenarios and give up on measuring the effects of the campaigns carried out. There are advanced statistical methods that allow even without a control group to estimate the actual effect. These methods, to put it simply, are based on the search for statistical twins. Statistical twins are customers with as similar characteristics as possible, among whom only one was subjected to an incentive: he was covered by a promotion, we sent him a text message, displayed an online advertisement to him, or took other actions towards him to encourage him to use our offer. This creates a synthetic “control group” consisting of one twin from each pair, the one who did not participate in the action. The challenge in this case is to identify the key variables that guarantee the similarity of the groups. However, appropriate computer software helps carry out this process. This opens up new possibilities for analyzing data and optimizing decisions in situations where conducting a randomized experiment is not possible, or is difficult or uneconomical.
As we can see, it happens that the effects of marketing campaigns carried out are either miscalculated or the results are overinterpreted. Therefore, even before implementing a campaign, it is necessary to determine how we intend to study its effects. Advanced statistical methods and machine learning help in many problematic situations, such as the inability to compile an optimal control group.
