Visible Thoughts Project and Bounty Announcement
| Nate Soares | News
(Update Jan. 12, 2022: We released an FAQ last month, with more details. Last updated Jan. 7.)
(Update Jan. 19, 2022: We now have an example of a successful partial run, which you can use to inform how you do your runs. Details.)
(Update Mar. 14, 2023: As of now the limited $20,000 prizes are no longer available. All the runs that start from now on will be paid at the rate of $10/step. Runs that were greenlit for full production before January 2023 and are still being produced will continue being paid at the same rate until completion.)
We at MIRI are soliciting help with an AI-alignment project centered around building a dataset, described below. We have $200,000 in prizes for building the first fragments of the dataset, plus an additional $1M prize/budget for anyone who demonstrates the ability to build a larger dataset at scale.
If this project goes well, then it may be the first of a series of prizes we offer for various projects.
Below, I’ll say more about the project, and about the payouts and interim support we’re offering.
The Project
Hypothesis: Language models can be made more understandable (and perhaps also more capable, though this is not the goal) by training them to produce visible thoughts.
We’d like to test this hypothesis by fine-tuning/retraining a language model using a dataset composed of thought-annotated dungeon runs. (In the manner of AI dungeon.)
A normal (un-annotated) dungeon run is a sequence of steps in which the player inputs text actions and the dungeon master responds with text describing what happened in the world as a result.
We’d like a collection of such runs, that are annotated with “visible thoughts” (visible to potential operators or programmers of the system, not to players) describing things like what just happened or is about to happen in the world, what sorts of things the player is probably paying attention to, where the current sources of plot tension are, and so on — the sorts of things a human author would think while acting as a dungeon master. (This is distinct from producing thoughts explaining what happened in the dungeon; “visible thoughts” are meant to play an active role in constructing the output.)
Once we have such a dataset, MIRI’s hope is that present or future technology will be able to train a model or models which iteratively produce visible thoughts along with storytelling, based on user actions plus previous history (including previous thoughts). The goal is to transition the state of AI dungeon technology from “An AI outputs story text in response to actions (and we have no idea how)” to “An AI produces thoughts as visible intermediates on the way to story text, allowing us to watch the AI think about how to design its output, and to verify that we can get different sensible outputs by intervening on the thoughts”.
Here’s an example of the first couple of steps of a thought-annotated dungeon run (or “quest”), in the format MIRI currently thinks is worth trying. Some kinds of thoughts are marked with parentheses and/or brackets; see the next section for details on this.