Score-based calibration testing for multivariate forecast distributions

Arbeitspapier

Multivariate distributional forecasts have become widespread in recent years. To assess the quality of such forecasts, suitable evaluation methods are needed. In the univariate case, calibration tests based on the probability integral transform (PIT) are routinely used. However, multivariate extensions of PIT-based calibration tests face various challenges. We therefore introduce a general framework for calibration testing in the multivariate case and propose two new tests that arise from it. Both approaches use proper scoring rules and are simple to implement even in large dimensions. The first employs the PIT of the score. The second is based on comparing the expected performance of the forecast distribution (i.e., the expected score) to its actual performance based on realized observations (i.e., the realized score). The tests have good size and power properties in simulations and solve various problems of existing tests. We apply the new tests to forecast distributions for macroeconomic and financial time series data.

Object Details

• ISBN: 978-3-95729-929-1
• Language: Englisch

References and Relationships

• Bibliographic citation: Series: Deutsche Bundesbank Discussion Paper ; No. 50/2022

Classification and Topics

• Classification: Wirtschaft ; Hypothesis Testing: General ; Model Evaluation, Validation, and Selection ; Forecasting Models; Simulation Methods
• Subject: Forecast Evaluation ; Density Forecasts ; Ensemble Forecasts

Contributors, Places and Time

• Event: Geistige Schöpfung
• (who): Knüppel, Malte ; Krüger, Fabian ; Pohle, Marc-Oliver
• Event: Veröffentlichung
• (who): Deutsche Bundesbank
• (where): Frankfurt a. M.
• (when): 2022

Further information

• Handle: http://hdl.handle.net/10419/268250
• Last update: 10.03.2025, 11:42 AM CET