Understanding Regime Changes for Robustness in Backtesting

In backtests, understanding how your factor signals perform in various regimes is crucial. The reason is market conditions continually evolve, so a factor signal that appears to be robust at one point in time may prove to be less effective within a different regime. In this article, we analyze the positives and negatives of extending your backtesting period.

We selected five factor signals: value, size, market sensitivity, profitability, and momentum. Using constituents of the Russell 3000 Index, we then analyzed their performance over different periods within the past 20 years. The factor returns are represented by the spread returns of the top factor score bucket portfolio, less the bottom factor score bucket portfolio.

The backtest class offered within the FactSet Programmatic Environment (FPE) enables you to effortlessly analyze factor performance across differing time periods. You can examine and compare the effectiveness of factor signals during varying economic or market scenarios. Doing so can provide insights into potential performance improvements and strategies to fine tune your approach.

Performance Across All Periods

We start by examining the performance across all periods. The two charts below illustrate that the profitability factor has the strongest performance both in terms of cumulative returns and information coefficient.

From the data in the metrics table below, we observe that the profitability factor has the highest alpha and information ratio (IR), while the sensitivity factor generates negative alpha.

Momentum was found to have the most volatility and the highest maximum drawdown, suggesting higher overall risk despite its positive alpha.

Subperiod Analysis

Next, we compare the factor performance during specific market downturns over the past 20 years, specifically during the Financial Crisis (2007-2009) and COVID-19 pandemic (2020-2021).

In the financial crisis period, the profitability factor once again displayed the highest performance. This is evident from positive alpha, high IR, and the highest average returns. The value factor also reveals robust performance.

During the COVID period, however, the profitability factor experienced a downturn. Conversely, the market sensitivity factor outperformed the other factors. That is evident from the highest alpha, average returns, and IR along with the lowest maximum drawdown, thereby signifying the least historical decline.

Risk-Level Analysis

We then shift our focus to examine how our factor signals perform in line with the level of VIX, in turn analyzing the factor returns across different regimes of market volatility. We began by regressing the factor returns on the level of VIX over the backtest period to identify factors that are most sensitive to changes in market volatility.

From this preliminary analysis, we revealed that market sensitivity and profitability factors demonstrated statistical significance. Therefore, we delve into metrics for those factors when the VIX is low, medium, and high.

Sensitivity

Low VIX environment: During this period, the market sensitivity factor achieves a satisfactory IR of 0.633 and a low max drawdown of 0.103, suggesting strong risk-adjusted returns and minimal peak-to-trough declines. Despite the negative alpha (-0.005) signaling underperformance relative to the benchmark, its t-stat of -2.735 indicates statistical significance. 

Medium VIX environment: In medium volatility conditions, the sensitivity factor's performance falls with an IR of 0.137 and a max drawdown of 0.291. Although the alpha (-0.000) is negative, the t-stat of -0.139 reveals it is not statistically significant, implying minimal excess return generation above the benchmark. 

High VIX environment: When volatility is high, the sensitivity factor's performance worsens, as seen in the negative IR of -0.837 and higher max drawdown of 0.327. Nonetheless, the positive alpha of 0.012 suggests potential for excess returns over the benchmark, supported by a statistically significant t-stat of 1.874. 

Profitability

Low VIX environment: In periods of low volatility, the profitability factor demonstrates moderate performance with an IR of 0.217 and a max drawdown of 0.178, signaling acceptable risk-adjusted returns and manageable declines. The positive alpha (0.006) indicates outperformance against the benchmark, corroborated by a statistically significant t-stat of 3.527. 

Medium VIX environment: During medium volatility, the profitability factor enhances its performance, reflected by a robust IR of 0.520 and a lower max drawdown of 0.159, highlighting superior risk-adjusted returns. The alpha (0.005) suggests the generation of excess returns over the benchmark, although its t-stat of 1.501 shows statistical insignificance. 

High VIX environment: In high volatility periods, the profitability factor achieves an impressive IR of 1.039, accompanied by a max drawdown of 0.130, indicating strong risk-adjusted performance. However, the negative alpha (-0.005) suggests underperformance relative to the benchmark, with a t-stat of -0.940 that is not statistically significant. 

You can repeat the same exercise for other risk indicators such as interest rates, inflation rates, or oil prices.

The following charts show the performance of factor returns at different CPI levels, specifically among factors that exhibited statistical significance (value and profitability).

Conclusion

In this article we explored how factor performance can vary in different market conditions. While extending the backtesting period can potentially enhance the strategy's robustness by increasing data points, beware that it may include various market regimes. Since each investing period has unique characteristics and challenges, it is essential to understand the specific time period you’re dealing with and be mindful of the regime shifts.

This blog post is for informational purposes only. The information contained in this blog post is not legal, tax, or investment advice. FactSet does not endorse or recommend any investments and assumes no liability for any consequence relating directly or indirectly to any action or inaction taken based on the information contained in this article.