What if your uplift metric ranked an imperfect model above the true treatment-effect ranking? In a simulation where we know the answer, Qini can do exactly that, and following its recommendation means choosing a ranking with lower true causal gain.
Let’s start with the classical uplift problem. Suppose a company needs to decide which customers of theirs to send marketing messages to. The best ranking of customers would be based on the expected benefit from sending them a marketing message, which depends on the value of the action we try to nudge them to and the shift in the probability of performing this action due to receiving the communication. Of course, the company doesn’t know these probabilities, so it uses an imperfect ranking based on their estimates.
Let’s take a hypothetical situation in which we know the real probabilities, that means we know the real perfect ranking of the customers in terms of the efficiency of communication.…