The AI apocalypse argument is hitting the mainstream. In a nutshell, AI is advancing very quickly and as we have no idea how to instill it with human values, it is likely to destroy all that we value in pursuit of some goal we have given it.
I don't want to say this is impossible. Janelle Shane has compiled a fun Twitter thread of AI agents achieving goals in unexpected ways, exploiting aspects of the environment that most humans would consider off-limits. These exploits remind me of an apocryphal test for psychopathy:
Question: A woman meets and engages in extended conversation with an attractive man at her mother's funeral. However, he leaves before she can get his contact information. A few days later, the woman kills her own sister. Why?
Answer (supposedly only deducible by psychopaths): She is hoping the attractive man will also show up at her sister's funeral.
You could imagine an AI system also coming up with that answer. "Hey, you didn't tell me I couldn't kill my own sister, you just told me to try to meet the attractive guy." The AI apocalypse argument is the idea that one day an AI system will say "hey, you didn't tell me I couldn't kill all humans, you just told me to make a bunch of paperclips".
And so I like to think of AI systems, at least the current batch, as psychopaths.
But... we already have human psychopaths. And some of them are quite intelligent. Some of the most intelligent ones are probably able to conceal their psychopathy enough to successfully further their own goals. Certain CEOs come to mind. And to a lesser extent, we're all psychopaths in that we're "unaligned" with prosocial values; most people would be willing to inflict some amount of harm on society if their own well-being or that of their loved ones were at stake.
Secondarily, the AI apocalypse argument appeals by analogy to historical examples of lesser intelligences encountering greater intelligences. Homo sapiens drove extinct earlier hominids, as well as many other mammalian species.
What's noteworthy about these examples is that the differences in intelligence are relatively small. Humans haven't driven cockroaches extinct, or ants. The intelligences of those creatures are different enough from our own that the competition for resources is limited to our pantries.
The scariest creature is a psychopathic human (or group thereof), a little bit smarter than us and competing with us for resources.
