Managing Extreme Technological Risks*

In response to the issue that, so far, very limited scientific attention has been given to the topic of extreme technological risks (ETRs) - technological developments that pose new extinction-level threats - we are developing, implementing and refining an initial model of a systematic approach to addressing how this class of risks can best be identified, managed and mitigated.

In designing a science of ETR as a new sub-field and testing a structure for this, we are paying particular attention to the need for: interdisciplinarity at the theoretical level; practical links with technology developers, policy-makers and the public; and an expanded community (academic, policy and technology) aware of the contributions they can make to the field. The initial model includes sub-projects on:

Evaluation of ETRs

Understanding the limitations of standard cost-benefit analysis approaches when applied to ETRs, and developing a version  more suited to this context.

Horizon-Scanning for ETRs

Optimising the horizon-scanning techniques that could greatly assist management of ETRs by enabling early detection, bearing in mind the particularly high cost of ‘false negatives’ in this area.

Responsible Innovation and ETRs

Recognising the key role science and technology communities have in managing risks that arise from their fields, we will be exploring the potential utility of responsible innovation frameworks for linking theoretical and practical aspects of managing ETRs.

ETRs and the Culture of Science

Exploring whether some aspects of scientific culture may be ill-adapted to the successful long-term management of ETRs and investigating potential responses to this.

Decision Theory and Stability in Self-Improving AI

Exploring connections between problems central to philosophical decision theory and concerns about the stability of goals in self-improving AI to develop insights into AI risk and possible mitigation strategies.  

*A Templeton World Charity Foundation Supported Project

Figure 1: A trial model for a science of Extreme Technological Risk management

The science of existential risk is necessarily highly interdisciplinary, both ‘vertically’ in the sense that it involves considerations ranging from the highly theoretical to the immediately practical, and ‘horizontally’, in that the range of disciplines involved is very wide.