“Fully autonomous weapons. A swarm of millions or billions of fully automated armed drones, locally controlled by powerful AI and strategically coordinated across the world by an even more powerful AI, could be an unbeatable army, capable of both defeating any military in the world and suppressing dissent within a country by following around every citizen.”

AI surveillance. Sufficiently powerful AI could likely be used to compromise any computer system in the world,30 and could also use the access obtained in this way to read and make sense of all the world’s electronic communications (or even all the world’s in-person communications, if recording devices can be built or commandeered). It might be frighteningly plausible to simply generate a complete list of anyone who disagrees with the government on any number of issues, even if such disagreement isn’t explicit in anything they say or do. A powerful AI looking across billions of conversations from millions of people could gauge public sentiment, detect pockets of disloyalty forming, and stamp them out before they grow. This could lead to the imposition of a true panopticon on a scale that we don’t see today.

For example, it would likely not be unconstitutional for the US government to conduct massively scaled recordings of all public conversations (e.g., things people say to each other on a street corner), and previously it would have been difficult to sort through this volume of information, but with AI it could all be transcribed, interpreted, and triangulated to create a picture of the attitude and loyalties of many or most citizens. I would support civil liberties-focused legislation (or maybe even a constitutional amendment) that imposes stronger guardrails against AI-powered abuses.

To give just one example, very primitive AIs, the social media algorithms have destabilized democracies all over the world. We now, in this paradoxical situation, when we have the most sophisticated information technology in history and people can’t talk to each other and certainly can’t listen, it’s becoming very difficult to hold a rational conversation. You see it now in the US between Republicans and Democrats, and you have all these explanations, “Oh, it’s because US society and economics and globalization,” whatever. But you go to almost every other democracy in the world, in my home country in Israel, you go to France, you go to Brazil, it’s the same. That it’s not the unique conditions of this or that country. It’s the underlying technology that makes it almost impossible for people to have a conversation. Democracy is a conversation and the technology is destroying the ability to have the conversation.

Now, is it worth it that we have, okay, we get these benefits, but we lose democracy all over the world. And then these technology is in the hand of authoritarian regimes that can use it to create the worst totalitarian regimes, the worst dystopias in human history. So we have to balance the potential benefits weighs the potential threats and move more carefully.

the automatic machine is the precise economic equivalent of slave labor. Any labor which competes with slave labor must accept the economic consequences of slave labor.

I think the best way to get a handle on the risks of AI is to ask the following question: suppose a literal “country of geniuses” were to materialize somewhere in the world in ~2027. Imagine, say, 50 million people, all of whom are much more capable than any Nobel Prize winner, statesman, or technologist. The analogy is not perfect, because these geniuses could have an extremely wide range of motivations and behavior, from completely pliant and obedient, to strange and alien in their motivations. But sticking with the analogy for now, suppose you were the national security advisor of a major state, responsible for assessing and responding to the situation. Imagine, further, that because AI systems can operate hundreds of times faster than humans, this “country” is operating with a time advantage relative to all other countries: for every cognitive action we can take, this country can take ten.

“thus strongly tying together the financial interests of large tech companies (which are increasingly focused on either AI or AI infrastructure) and the political interests of the government in a way that can produce perverse incentives. We already see this through the reluctance of tech companies to criticize the US government, and the government’s support for extreme anti-regulatory policies on AI.”

In Machines of Loving Grace, I discussed the possibility that authoritarian governments might use powerful AI to surveil or repress their citizens in ways that would be extremely difficult to reform or overthrow. Current autocracies are limited in how repressive they can be by the need to have humans carry out their orders, and humans often have limits in how inhumane they are willing to be. But AI-enabled autocracies would not have such limits.

“they would be outsmarted at every turn, similar to a war between humans and mice. Putting these two concerns together leads to the alarming possibility of a global totalitarian dictatorship. Obviously, it should be one of our highest priorities to prevent this outcome.”

Under the current social contract, powerful actors require people to sustain their power.

To understand how we change the incentives that lead to the intelligence curse, we should understand the current social contract, and how it is set to break.

Under the current social contract, powerful actors require people to sustain their power.

This is true for governments. We’ve seen how richest states are those that have the richest, best-educated, most-productive citizens, and that grant those citizens the most extensive freedoms to work and think as they please. This gives citizens bargaining power—if their needs aren’t met, citizens can remove their leaders from power, since after all, the leaders cannot have an economy or a military without the people.

It’s true as well for companies. We’ve seen how organizations that need high-caliber talent have to offer higher salaries and lucrative benefits. If a company fails to uphold this relationship, employees can join other ones or start their own.

It’s also true for the market as a whole. Humans are the consumers, and they buy things that either 1) they want or 2) that assist them in earning more to buy the things they want. Humans are also the producers, which means that market forces incentivize investing in humans. Humans might need to serve the market, but the market serves them back.

As AI replaces the need for humans, this could be upended. The keys to power might run entirely through non-human factors of production like land, capital, resources—and control over AI. Regular people could be economically displaced, and powerful actors could lose their incentive to invest in them.

We’ve been calling this the intelligence curse.

The intelligence curse incentivizes a breakdown in the social contract that is antithetical to democracy and human-oriented capitalism—the systems that have produced more value for mankind than any other in history. At their core, both capitalism and democracy uplift some humans as a reward for them uplifting other humans, and create incentives on institutions to care about people being well-off and capable. Everyone—including the powerful—wants regular people to do better. Both systems explicitly encourage the new to supplant the old, creating a force for progress and social mobility.

If states follow its incentives, the intelligence curse will pull them towards entrenched authoritarianism, ushering in unparalleled concentration of wealth and power and a closing of the door for regular people to get ahead. But will that actually happen?

“But another kind of disempowerment can occur if there is such a huge concentration of wealth that a small group of people effectively controls government policy with their influence, and ordinary citizens have no influence because they lack economic leverage. Democracy is ultimately backstopped by the idea that the population as a whole is necessary for the operation of the economy. If that economic leverage goes away, then the implicit social contract of democracy may stop working.”

“AI propaganda. Today’s phenomena of “AI psychosis” and “AI girlfriends” suggest that even at their current level of intelligence, AI models can have a powerful psychological influence on people. Much more powerful versions of these models, that were much more embedded in and aware of people’s daily lives and could model and influence them over months or years, would likely be capable of essentially brainwashing many (most?) people into any desired ideology or attitude, and could be employed by an unscrupulous leader to ensure loyalty and suppress dissent, even in the face of a level of repression that most populations would rebel against.”

“a personalized AI agent that gets to know you over years and uses its knowledge of you to shape all of your opinions would be dramatically more powerful than this.”