Factor analysis

A strategy is usually built from several screening conditions, and each condition is a "factor". This page shows how to measure the link between each factor and the strategy's returns, so you add, drop or change factors based on data rather than intuition.

How the analysis works

First turn the strategy into two tables, then pass them to the functions in finlab.tools.factor_analysis:

• Features: for every month and stock, whether each factor holds.
• Labels: each stock's return over the next month minus the average return of all stocks (the excess return).

Turn a strategy into features and labels

from finlab import data
from finlab.backtest import sim

data.set_market('us')
with data.us_universe(index='SP500'):
    marketcap = data.get('etl:us_market_value')
    revenue = data.get('us_income_statement:revenue')
    price = data.get('price:adj_close')

cond1 = marketcap.rank(pct=True, axis=1) < 0.3
cond2 = (revenue / revenue.shift(4)).rank(pct=True, axis=1) > 0.7
cond3 = (price / price.shift(20)).rank(pct=True, axis=1) > 0.7

pos = cond1 & cond2 & cond3

report = sim(pos, resample='ME', upload=False)
report.creturn.plot()

from finlab import data
import finlab.ml.feature as feature
import finlab.ml.label as label

features = feature.combine(
    {'marketcap': cond1, 'revenue': cond2, 'momentum': cond3}, resample='ME'
)

# Compute labels for the next 1 month
labels = label.excess_over_mean(index=features.index, resample='ME')

features.dropna().head()

labels.dropna().head()

See how much return each factor brings

Factor return measures the excess return a factor brings over a period, and is commonly used to judge whether a factor works and is worth investing in. Adding up each factor's returns over time shows its long-term performance and stability, a basis for choosing and combining factors.

from finlab.tools.factor_analysis import calc_factor_return
from finlab.plot import plot_line

plot_line(
    calc_factor_return(features, labels).cumsum(),
    unit='.0%',
    title='Factor Cumulative Return',
)

Compare the curves: a factor whose cumulative return keeps rising has been adding excess return, while a flat or falling curve means the factor has not helped over that period.

Check whether a factor is crowded

Factor centrality quantifies the "commonality" of factor returns, that is, how crowded a factor is:

where $\lambda_j$ is the factor's contribution to the first principal component, and $k$ is the total number of factors.

The second argument of calc_centrality() is the rolling window length in rows; 12 here means 12 months:

from finlab.tools.factor_analysis import calc_centrality

plot_line(calc_centrality(calc_factor_return(features, labels), 12), title='Centrality')

If a factor currently has low crowding, you might switch to a different factor to make the strategy more stable, or monitor it closely and use it when crowding starts to rise (factor reversion effect).

See each factor's contribution to combinations

calc_shapley_values() enumerates every factor combination. At each point in time, it takes the equal-weight mean excess return of the stocks that satisfy every factor in the combination (0 when none) and splits it evenly among the combination's factors; a factor's contribution is the sum of its shares over all combinations that include it.

from finlab.tools.factor_analysis import calc_shapley_values

plot_line(calc_shapley_values(features, labels))

A higher value means the combinations containing the factor performed better.

Check whether a factor predicts returns

Factor IC (information coefficient) is a key measure of a factor's predictive ability: the correlation between factor scores and future returns.

where $\text{corr}$ is the correlation coefficient (usually Pearson), $\text{Factor Score}$ is the factor score, and $\text{Future Return}$ is the asset return over a future period. IC values typically range between -1 and 1:

In quantitative stock selection, IC is a key basis for judging whether a factor works; factors with higher IC are generally more worth including in a portfolio.

IC needs continuous factor values rather than True/False, so build a second feature table from the raw values. rank=True ranks both the factor values and the labels on each date before correlating them (Spearman rank IC); without it, the Pearson correlation is used.

from finlab.tools.factor_analysis import calc_ic

features_raw = feature.combine(
    {
        'marketcap': -marketcap,  # Small cap (negated)
        'revenue': revenue / revenue.shift(4),
        'momentum': price / price.shift(20),
    },
    resample='ME',
)

plot_line(calc_ic(features_raw, labels, rank=True))

Tell whether a factor is getting stronger or weaker

Centrality, contribution and IC all change over time. calc_regression_stats() fits a linear regression to each factor's series and tells you whether it is trending up, down or flat:

from finlab.tools.factor_analysis import calc_regression_stats

centrality_df = calc_centrality(calc_factor_return(features, labels), 12)
centrality_trend = calc_regression_stats(centrality_df)

The result is a DataFrame with one row per factor (marketcap, revenue, momentum) and the columns slope, p_value, r_squared, tail_estimate and trend.

A trend counts as up or down only when p_value < 0.05 and r_squared > 0.1 (the p_value_threshold and r_squared_threshold defaults); otherwise it is flat.

Reading the result

• trend = up (small p_value, high r_squared, positive slope): the factor's centrality is rising steadily. The factor is becoming popular and more capital is chasing it, which also raises the risk of a reversal.
• trend = down (small p_value, high r_squared, negative slope): the factor's centrality is falling steadily; crowding is decreasing.
• trend = flat: either the slope is not significant, or it is significant but explains too little of the variation (low r_squared), so no clear trend can be claimed.

Next steps