In April 2017 the Early Warning Project (EWP) launched a new set of questions through a public opinion pool to crowdsource questions on atrocity risk around the world. Since then, 317 participants have cast 7,025 forecasts in response to questions asking about mass killing risk in 16 countries that the project has identified as high risk. The Good Judgment Project, an online forecasting platform, aggregates the individual forecasts from both experts and amateur volunteers to alert the EWP of evolving risk of large-scale violence against civilians.
It may seem counterintuitive to ask non-experts to predict the future, but according to Good Judgment Inc. co-founder Philip E. Tetlock, experts perform worse than blind guessing when it comes to forecasting, and non-experts can be very successful at it. Based on his recent study Tetlock noted, “One could say that superforecasters were better characterized as super-generalists than as subject matter experts.” His extensive research has found that “nonexpert civilian forecasters often beat trained intelligence analysts at their own game.” EWP encourages a variety of country experts, subject-matter experts, and the general public to participate in the challenge.
We see the public opinion pool as a way to track and gauge risk in real-time, particularly since our Statistical Risk Assessment is only updated once a year due to availability of the datasets on which it is based. We especially benefit from forecaster discussion threads and comments, which give us insight into how forecasters are interpreting developments in particular countries and incorporating them into their forecasts. The Good Judgment platform also allows forecasters to share articles and links to other sources that influenced their assessments.
Today the highest risk country is Burma (Myanmar), with a 55% risk on September 18, up from 8% on August 31. This jump in forecasted risk coincided with renewed attacks on the country’s Rohingya population beginning August 25, documented in satellite imagery of burned villages and reports of an estimated 40,000 civilians fleeing into Bangladesh following the August attacks. Burma is trailed by Iraq at 10% risk and Nigeria at 10%. Unlike Burma, Iraq and Nigeria have had relatively consistent forecasted risk over the most recent 3-month challenge period.
The questions that launched in April asked, Between 1 April 2017 and 31 March 2018, will an armed group from [country] engage in a campaign that systematically kills 1,000 or more civilians in [country]?
The same question structure will be replicated each quarter to facilitate comparisons over time.
Forecasters will have to wait until the 12-month period concludes—the first will be in April 2018—for EWP to make judgments about whether or not a mass killing occurred in that period, enabling us to judge the accuracy of our forecasting pool.
Anyone can go to Good Judgment’s challenge site to see the current consensus on risk for each country. And anyone who creates a (free) login can participate and see more details about the forecasts and commentary provided by participants explaining their judgments. According to Good Judgement, “expertise is not required. In fact, as our challenge leaderboards demonstrate, thoughtful amateurs who practice the principles of Superforecasting can outperform so-called experts!”
For questions, please contact ewp@ushmm.org.