Arbeitspapier
The anatomy of out-of-sample forecasting accuracy
We develop metrics based on Shapley values for interpreting time-series forecasting models, including "black-box" models from machine learning. Our metrics are model agnostic, so that they are applicable to any model (linear or nonlinear, parametric or nonparametric). Two of the metrics, iShapley-VI and oShapley-VI, measure the importance of individual predictors in fitted models for explaining the in-sample and out-of-sample predicted target values, respectively. The third metric is the performance-based Shapley value (PBSV), our main methodological contribution. PBSV measures the contributions of individual predictors in fitted models to the out-of-sample loss and thereby anatomizes out-of-sample forecasting accuracy. In an empirical application forecasting US inflation, we find important discrepancies between individual predictor relevance according to the in-sample iShapley-VI and out-ofsample PBSV. We use simulations to analyze potential sources of the discrepancies, including overfitting, structural breaks, and evolving predictor volatilities.
- Sprache
-
Englisch
- Erschienen in
-
Series: Working Paper ; No. 2022-16
- Klassifikation
-
Wirtschaft
Single Equation Models; Single Variables: Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes
Neural Networks and Related Topics
Forecasting Models; Simulation Methods
Prices, Business Fluctuations, and Cycles: Forecasting and Simulation: Models and Applications
Financial Forecasting and Simulation
- Thema
-
variable importance
out-of-sample performance
Shapley value
loss function
machine learning
inflation
- Ereignis
-
Geistige Schöpfung
- (wer)
-
Borup, Daniel
Goulet Coulombe, Philippe
Rapach, David E.
Montes Schütte, Erik Christian
Schwenk-Nebbe, Sander
- Ereignis
-
Veröffentlichung
- (wer)
-
Federal Reserve Bank of Atlanta
- (wo)
-
Atlanta, GA
- (wann)
-
2022
- DOI
-
doi:10.29338/wp2022-16
- Handle
- Letzte Aktualisierung
-
10.03.2025, 11:44 MEZ
Datenpartner
ZBW - Deutsche Zentralbibliothek für Wirtschaftswissenschaften - Leibniz-Informationszentrum Wirtschaft. Bei Fragen zum Objekt wenden Sie sich bitte an den Datenpartner.
Objekttyp
- Arbeitspapier
Beteiligte
- Borup, Daniel
- Goulet Coulombe, Philippe
- Rapach, David E.
- Montes Schütte, Erik Christian
- Schwenk-Nebbe, Sander
- Federal Reserve Bank of Atlanta
Entstanden
- 2022
Ähnliche Objekte (12)
Finite sample accuracy of integrated volatility estimators
Tourism Forecasting: Accuracy of Alternative Econometric Models Revisited
Volatility forecasting accuracy for Bitcoin
Essays on empirical asset pricing and out-of-sample forecasting
Forecasting ECB monetary policy: accuracy is (still) a matter of geography
Forecasting ECB monetary policy: accuracy is (still) a matter of geography
How good are out of sample forecasting Tests on DSGE models?
Sample-Based Forecasting Exploiting Hierarchical Time Series
The Influence of Sample Size and Selection of Financial Ratios in Bankruptcy Model Accuracy
The out-of-sample forecasting performance of variable parameter exchange rate models in South Africa
Does Complexity Matter? Methods for Improving Forecasting Accuracy in Tourism
Comparing the DSGE model with the factor model: an out-of-sample forecasting experiment
Finite sample accuracy of integrated volatility estimators
Tourism Forecasting: Accuracy of Alternative Econometric Models Revisited
Volatility forecasting accuracy for Bitcoin
Essays on empirical asset pricing and out-of-sample forecasting
Forecasting ECB monetary policy: accuracy is (still) a matter of geography
Forecasting ECB monetary policy: accuracy is (still) a matter of geography
How good are out of sample forecasting Tests on DSGE models?
Sample-Based Forecasting Exploiting Hierarchical Time Series
The Influence of Sample Size and Selection of Financial Ratios in Bankruptcy Model Accuracy
The out-of-sample forecasting performance of variable parameter exchange rate models in South Africa
Does Complexity Matter? Methods for Improving Forecasting Accuracy in Tourism
Comparing the DSGE model with the factor model: an out-of-sample forecasting experiment
Finite sample accuracy of integrated volatility estimators
Tourism Forecasting: Accuracy of Alternative Econometric Models Revisited
Volatility forecasting accuracy for Bitcoin
Essays on empirical asset pricing and out-of-sample forecasting
Forecasting ECB monetary policy: accuracy is (still) a matter of geography
Forecasting ECB monetary policy: accuracy is (still) a matter of geography
How good are out of sample forecasting Tests on DSGE models?
Sample-Based Forecasting Exploiting Hierarchical Time Series
The Influence of Sample Size and Selection of Financial Ratios in Bankruptcy Model Accuracy
The out-of-sample forecasting performance of variable parameter exchange rate models in South Africa
Does Complexity Matter? Methods for Improving Forecasting Accuracy in Tourism