Luke Muehlhauser and Bill Hibbard have a new paper (PDF) in the September 2014 issue of Communications of the ACM, the world’s most-read peer-reviewed computer science publication. The title is “Exploratory Engineering in Artificial Intelligence.”
Excerpt:
We regularly see examples of new artificial intelligence (AI) capabilities… No doubt such automation will produce tremendous economic value, but will we be able to trust these advanced autonomous systems with so much capability?
Today, AI safety engineering mostly consists in a combination of formal methods and testing. Though powerful, these methods lack foresight: they can be applied only to particular extant systems. We describe a third, complementary approach that aims to predict the (potentially hazardous) properties and behaviors of broad classes of future AI agents, based on their mathematical structure (for example, reinforcement learning)… We call this approach “exploratory engineering in AI.”
…
In this Viewpoint, we focus on theoretical AI models inspired by Marcus Hutter’s AIXI, an optimal agent model for maximizing an environmental reward signal…
…
Autonomous intelligent machines have the potential for large impacts on our civilization. Exploratory engineering gives us the capacity to have some foresight into what these impacts might be, by analyzing the properties of agent designs based on their mathematical form. Exploratory engineering also enables us to identify lines of research — such as the study of Dewey’s value-learning agents — that may be important for anticipating and avoiding unwanted AI behaviors. This kind of foresight will be increasingly valuable as machine intelligence comes to play an ever-larger role in our world.
