I think the fear of AGI destroying humanity mostly stems from the suspicion that a logically thinking device would indeed come to the conclusion that we need to be exterminated.
It comes from the fact that a logically thinking device given an arbitrary goal will probably not develop "Thou shalt not kill"-type moralisms in the pursuit of that goal.
If you make an AGI to run a paperclip factory, eventually you'll come back to find most of the carbon and iron in your galaxy refactored into paperclips. The extinction of any local flora or fauna there is just a byproduct of that drive. See [2] for a brief rundown.
> If you make an AGI to run a paperclip factory, eventually you'll come back to find most of the carbon and iron in your galaxy refactored into paperclips.
Given the costs to run such an AGI, it seems unlikely one would be left unattended for long.
“A large human population isn’t the best use of the Earth’s resources” would seem to be an obvious conclusion of a superior thinking entity with no compelling reason in prioritising us over its own interests (which might not be selfish eg it’s goal might be to discover the true nature of reality.
But why exterminate humans? Why not create nature reserves for us and other life, for bioconservation and research? That’s been a feature of human societies going back thousands of years.
>Why not create nature reserves for us and other life, for bioconservation and research? That’s been a feature of human societies going back thousands of years.
Generally only for things we find cute or pretty.
Plenty of "save the whales!", not much "save this strain of this bacteria before it goes extinct!".
the risk is that we know that there is a non-zero probability mass associated with "AGI doing whatever leading to premature human extinction"
and the less serious concerns we look at the bigger the chance (of course the severity is less, so calculating the impact is not trivial), but "non-runaway AGI used for antisocial purposes" is pretty likely.
