Why AGI could not be (just) a tool: goals, life, and general intelligence
Micah Lott and William Hasselberger
Inquiry, 2024
It is widely believed that AGI has the potential to be a wonderful tool that humans can use to meet our needs, solve our problems, and improve our lives. But any entity with truly general, human-level intelligence would have the capacity to lead its own life, with its own purposes and integrated hierarchy of goals. And thus any true AGI could not be merely a tool, even if it turned out to be extremely helpful for human beings. There is a dilemma at the heart of the ambition to build AGI as a valuable tool. On the one hand, any mere tool that we might build would lack capacities essential to the kind of general intelligence exhibited by human beings; it would not be genuine AGI. On the other hand, were we to create genuine AGI, then what we would have created would not be a mere tool, but something more. In this paper makes the case for this dilemma and in so doing, illuminates the connections between a set of core ideas: intelligence, agency, tools, and life.