A comparison of observed global energy technology costs, with forecasts generated by models and forecasts predicted by human experts, showed that both forecasting methods underestimated cost reductions. This suggests that decisions based on forecasts may be overestimating the cost of climate mitigation and points to the need to further improve forecasting methods.

Source: PNAS press release

A team of researchers from the University of Cambridge, University College London, University of Oxford, and University of Brescia/RFF-CMCC European Institute on Economics and the Environment carried out the first systematic analysis of the relative performance of probabilistic cost forecasts from expert-based methods and model-based methods.
They specifically focused on one expert-based method — expert elicitations — and four model-based methods which model costs either as a function of cumulative installed capacity or as a function of time. The results of this comparison are published in PNAS. 

Accurately forecasting energy technology costs is a requirement for the design of robust and cost-effective decarbonization policies and business plans. The future of these and other technologies is notoriously hard to predict because the process by which technology is conceived, developed, codified, and deployed is part of a complex adaptive system and is made up of interconnected actors and institutions. 

A range of probabilistic forecasting methods have been developed and used to generate estimates of future technology costs. Two high-level types of approaches have been most often used to generate quantitative forecasts: expert-based and model-based approaches. Broadly speaking, expert-based approaches involve different ways of obtaining information from knowledgeable individuals who may have differing opinions and/or knowledge about the relative importance of various drivers of innovation and how they may evolve. Experts make implicit judgments about the underlying drivers of change when producing their forecasts and can take into account both public information about observed costs as well as information that may not yet be widely available or codified. Expert-based approaches are often the only source of information available to analysts when data, on a given technology, has not yet been collected—as is generally the case for emerging technologies. 

By contrast, model-based approaches explicitly use one or more variables from available observed data to approximate the impact of the full set of drivers of innovation on technology costs, implicitly assuming that the rate of change in the past will be the best predictor of the rate of change in the future.

“The increased availability of information on future energy technology costs allowed us to conduct the first systematic comparison of the relative performance of probabilistic technology cost forecasts generated by different expert-based and model-based methodologies with observed costs” notes senior and corresponding author Prof. Diaz Anadon, Professor of Climate Change Policy at the University of Cambridge and Director of the University’s Centre for Environment, Energy and Natural Resource Governance. “Such a comparison is essential to ensure researchers and analysts have more empirically-grounded evidence in integrated assessment models, cost benefit analyses and broader policy design efforts.”   She suggests that undertaking this type of comparison to assess and better understand different forecasting methods should become more common among modellers and forecasting practitioners, as more data is available. “Our analysis is focused on a particular period of time and on correlated energy technologies, so although our results point to current methods underestimating technological progress in this space, more research is needed”.

Prof. Anadon authored the article with Dr. Jing Meng, Lecturer at University College London at the Bartlett School, Dr. Rupert Way, postdoctoral researcher at the Oxford Martin School, and Prof. E. Verdolini from the Law Department of the University of Brescia and affiliated to the RFF-CMCC European Institute on Economics and the Environment. Prof. Anadon and Prof. Verdolini were Work Package Leaders in the EU H2020 project INNOPATHS, which funded the majority of the research work.  

A number of key results emerge from this analysis. 

As Dr. Way of the University of Oxford explains, “the comparison of expert- and model-based forecasts with observed 2019 costs over a short time frame (a maximum of 10 years) shows that model-based approaches outperformed expert elicitations. More specifically, the 5th-95th percentile range of the four model-based approaches were much more likely to contain the observed value than that of EE forecasts. Among the model-based methods, some captured 2019 observed costs more often than others”.

“In addition”, notes Dr. Meng from University College London “the 2019 medians of model-based forecasts were closer to the average observed 2019 cost for five out of the six technologies. However, this comparison was possible only for a small number of technologies; furthermore, some of the EE forecasts included the observed value”. For these reasons, the authors argue, this should not be taken as evidence that model-based approaches perform better than expert-based methods for all or most cases.  

Prof. Verdolini, from UniBrescia/EIEE points to the fact that both expert-based methods and model-based methods underestimated technological progress in most of the energy technologies analysed in this paper. “That is, in five out of six technologies analyzed, the methods produced 2019 cost forecast medians that were higher than the observed 2019 costs. This indicates that the rate of progress in cost reduction has been higher than what both historical data and expert opinions predicted. But the extent to which this faster pace of progress compared to forecasts will continue (or not) in the future remains to be seen”.

The urgency of developing policies for deep decarbonisation, as outlined in the IPCC 1.5 °C report, makes this systematic analysis timely and necessary. Taken together, results point to various worthwhile avenues for future research. Concerning  expert elicitations, this paper calls attention to  the need to continue methodological improvements to reduce overconfidence. For  model-based methods, this work highlights the challenge of finding (and collecting) data for many key energy technologies. It also calls for increased efforts in data collection and publication by international organizations and other entities. The underestimation of technological progress also points to the value of further method development to reflect structural changes and technology correlations. Lastly, given the large uncertainty ranges and major policy decisions associated with the energy transition and with addressing climate change, additional research comparing the performance of different probabilistic forecasting approaches with observed values across a wider range of technologies should be carried out as more data becomes available and more time passes.

The article is complemented with a database containing a large number of data points on the costs of 32 energy technologies relevant to support the energy transition. These data points include 25 sets of data from expert elicitations conducted between 2007 and 2016 covering a range of geographies and 25 sets of observed technology data including the evolution of cost and deployment over different periods of time. This data was made publicly available here