“AlphaGo’s victory over Lee Sedol in 2016 marked a watershed moment. It demonstrated that AI could master intuition and creativity, domains previously thought to be uniquely human. This was followed by breakthroughs in protein folding with AlphaFold, showing AI’s potential to solve humanity’s greatest challenges.”

“Within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will be ended.”

“The acceleration of technological progress has been the central feature of this century. I argue in this paper that we are on the edge of a change comparable to the rise of human life on Earth. The precise cause of this change is the imminent creation by technology of entities with greater than human intelligence.”

“The three revolutions that turned insignificant apes into the masters of the world: the Cognitive Revolution (70,000 years ago), the Agricultural Revolution (12,000 years ago), and the Scientific Revolution (500 years ago). This book will argue that a fourth revolution is now beginning. Intelligent design is about to become a reality – not due to the actions of some god above the clouds, but due to the actions of clouds of data.”

“History began when humans invented gods, and will end when humans become gods.”

“We are probably one of the last generations of Homo sapiens. Within a century or two, Earth will be dominated by entities that are more different from us than we are from Neanderthals or from chimpanzees.”

“This time is different. Information technology has finally begun to advance into areas that had previously been the exclusive domain of human workers, and there is every reason to expect that this trend will accelerate.”

“The question is not whether robots and computers will take our jobs; it’s how fast they will take them. And the answer to that is: faster than we think.”

“Unlike the Industrial Revolution, the current wave of technological change is not creating new types of jobs faster than it destroys them. Instead, it appears to be systematically wiping out entire categories of employment.”

“The standard model of AI research takes it for granted that the objective is given to the machine. But what if we cannot specify the objective correctly? In that case, the machine will optimize something other than what we want.”

“A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found by the system will be highly undesirable.”

“We need to ensure that AI systems remain beneficial as they become more capable. This means we need to solve the control problem: how can we ensure that a powerful AI system will do what we want it to do?”

“We stand at the beginning of a new chapter in the history of life on Earth. Life 1.0 is biological life, which emerged about 4 billion years ago. Life 2.0 is cultural life, which emerged about 100,000 years ago when humans learned to design their software (culture). Life 3.0 is technological life, which can design both its hardware and software, and may emerge during the coming century through progress in AI.”

“The key question isn’t whether there will be a breakthrough to human-level artificial general intelligence, but when - and what happens next.”