Model A has an average score of 0.02. Model B has 0.01. Is A better?
Maybe. Maybe not. You need two numbers to say.
1. The information coefficient (IC)
For a model that ranks stocks, the IC on a given day is the rank correlation (Spearman) between the model's scores and what each stock did over the following period.
from scipy.stats import spearmanr
ic_by_day = [ spearmanr ( scores [ d ], outcome [ d ]). correlation for d in dates ] mean_ic = np . mean ( ic_by_day )
Enter fullscreen mode
Exit fullscreen mode
0 means no skill: the ranking and the outcome are unrelated.
Negative means the ranking pointed the wrong way.
In real markets, an IC around 0.05 held consistently would be strong. Most ideas sit much closer to zero.
2. The t-statistic
The IC bounces around from day to day. The t-statistic asks: is the average big compared with how much it bounces?
Roughly: t = mean_ic / (std_ic / sqrt(n)) , but with one catch.…