Nate Soares: we are doomed (probably)

Today Razib talks to Nate Soares the President of the Machine Intelligence Research Institute (MIRI). He joined MIRI in 2014 and has since authored many of its core technical agendas, including foundational documents like Agent Foundations for Aligning Superintelligence with Human Interests. Prior to his work in AI research, Soares worked as a software engineer at Google. He holds a B.S. in computer science and economics from George Washington University.

On this episode they discuss his new book, If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, co-authored with Eliezer Yudkowsky. Soares and Yudkowsky make the stark case that the race to build superintelligent AI is a "suicide race" for humanity. Razib and Soares discuss how AI systems are "grown" rather than deliberately engineered, making them fundamentally opaque and uncontrollable. They explore a concrete extinction scenario, explain why even minimally misaligned goals could lead to human annihilation. Soares urges immediate cooperative action to prevent such a worst-case outcome.