The anatomy of out-of-sample forecasting accuracy

Arbeitspapier

We develop metrics based on Shapley values for interpreting time-series forecasting models, including "black-box" models from machine learning. Our metrics are model agnostic, so that they are applicable to any model (linear or nonlinear, parametric or nonparametric). Two of the metrics, iShapley-VI and oShapley-VI, measure the importance of individual predictors in fitted models for explaining the in-sample and out-of-sample predicted target values, respectively. The third metric is the performance-based Shapley value (PBSV), our main methodological contribution. PBSV measures the contributions of individual predictors in fitted models to the out-of-sample loss and thereby anatomizes out-of-sample forecasting accuracy. In an empirical application forecasting US inflation, we find important discrepancies between individual predictor relevance according to the in-sample iShapley-VI and out-ofsample PBSV. We use simulations to analyze potential sources of the discrepancies, including overfitting, structural breaks, and evolving predictor volatilities.

Object Details

• Language: Englisch

References and Relationships

• Bibliographic citation: Series: Working Paper ; No. 2022-16

Classification and Topics

• Classification: Wirtschaft ; Single Equation Models; Single Variables: Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes ; Neural Networks and Related Topics ; Forecasting Models; Simulation Methods ; Prices, Business Fluctuations, and Cycles: Forecasting and Simulation: Models and Applications ; Financial Forecasting and Simulation
• Subject: variable importance ; out-of-sample performance ; Shapley value ; loss function ; machine learning ; inflation

Contributors, Places and Time

• Event: Geistige Schöpfung
• (who): Borup, Daniel ; Goulet Coulombe, Philippe ; Rapach, David E. ; Montes Schütte, Erik Christian ; Schwenk-Nebbe, Sander
• Event: Veröffentlichung
• (who): Federal Reserve Bank of Atlanta
• (where): Atlanta, GA
• (when): 2022

Further information

• DOI: doi:10.29338/wp2022-16
• Handle: http://hdl.handle.net/10419/270459
• Last update: 10.03.2025, 11:44 AM CET