Demand forecasting: an evaluation of DOD's accuracy metric and Navy's procedures

Abstract

In 2013, the Department of Defense (DOD) implemented an accuracy metric to monitor how well the services and Defense Logistics Agency were forecasting demand for inventory items. After three years, results were still poor. DOD uses a metric derived from the Mean of Absolute Percentage Error, yet it differs in significant ways, such as including unit cost to enable the aggregation of data pertaining to all items. In this study, we analyze how unit cost and other parameters affect the validity of DOD metric results. Our research included a review of academic literature on forecast accuracy measurement that uncovered an alternative metric, Mean of Absolute Scaled Errors (MASE), which we tested against the DOD metric. We found the DOD metric produced non-intuitive results and was adversely affected by unit cost and demand volume, while MASE avoided these errors. We utilized MASE to compare six forecasting methods and found that flexibility in choice of forecasting method produced better results than the naïve method when coefficient of variation (CV) is below 2.0. We recommend that the DOD and Navy adopt MASE for aggregation and item-level forecast accuracy evaluation. We recommend that Navy utilize flexibility in choice of forecast method for individual items with CV below 2.0