How often do you retrain the model?

Daily regime probabilities update, full model retraining monthly with 6-month rolling window.

What explainability methods did you use?

SHAP values and partial dependence plots for regulator and internal approvals.

How does this case study work?

Raise a request, talk to experts, fund the project, expert works, review and approve payment. All remote, all through our platform.

Improving Factor Model Performance

Executive Summary

A systematic fund's linear factor model was underperforming during regime shifts. Replacing linear combinations with gradient boosting and regime detection improved Sharpe ratio from 1.4 to 2.2, with significantly lower drawdowns.

Key Outcomes

▹ Sharpe ratio improved 1.4 → 2.2 (57% increase)
▹ Maximum drawdown reduced 22% → 12%
▹ 3 new factors discovered via feature importance

Client Situation

The fund's 50-factor model performed well in trending markets but broke down during volatility spikes, causing large drawdowns.

Key Challenges

⚠ Linear combination failed during regime changes
⚠ Factor crowding reduced diversification benefits
⚠ Manual factor weighting unable to adapt to market conditions

Existing Architecture

Linear multi-factor model with equal weights. Factors included value, momentum, quality, and low volatility.

Static weights ignoring market regime
No non-linear interactions captured
Factor neutralization incomplete

Solution Design

Two-layer ML model: regime classifier (HMM) + factor combination model (XGBoost) with walk-forward validation.

Key Decisions

✓ Use XGBoost for non-linear factor combinations
✓ Hidden Markov Model for regime detection (3 regimes)
✓ Walk-forward validation with 6-month retraining

XGBoostHMMlearnRayWeights & BiasesPyTorch

Implementation

Backtested 10 years of data with walk-forward validation, live paper trading for 3 months before deployment.

Phase 1: Phase 1: Feature Engineering
Created 200 features from base factors (lags, cross-products, volatility adjustments).
Phase 2: Phase 2: Regime Detection
HMM identified bull, bear, and sideways regimes with 85% accuracy.
Phase 3: Phase 3: Model Training
XGBoost with regime-specific models outperformed single model by 40%.

Technical Challenges

Overfitting during backtest

Impact: Walk-forward Sharpe 2.5 but paper trading Sharpe 1.2

Resolution: Reduced feature count from 200 to 45 via recursive elimination

Regime classification lag

Impact: Detecting regime shift 2 weeks late caused losses

Resolution: Added real-time regime probabilities with 50% threshold for early warning

Results

Sharpe ratio (live): Before1.4
After2.2
Improvement57% increase
Maximum drawdown: Before22%
After12%
Improvement45% reduction
Factor turnover: Before40% monthly
After25% monthly
Improvement38% reduction

Lessons Learned

📘 XGBoost feature importance revealed 3 novel factors the team hadn't considered
📘 Regime detection added 0.4 Sharpe alone before ML combination
📘 Walk-forward validation essential—in-sample results overfit by 60%

What We Would Do Differently

💡 Implement online learning for factor weights
💡 Add Bayesian hyperparameter tuning earlier

Role Relevance

Quant researchers with ML expertise transformed a linear factor model into a non-linear adaptive system, achieving industry-leading Sharpe ratios.

Critical Skills Demonstrated

Machine learning for financeRegime detection (HMM)Feature engineeringWalk-forward validation

Frequently Asked Questions

How often do you retrain the model?: Daily regime probabilities update, full model retraining monthly with 6-month rolling window.
What explainability methods did you use?: SHAP values and partial dependence plots for regulator and internal approvals.