> The problem with "smart" AI is that it attempts to advance its goals by the most optimal means possible,
This is a fantasy.
A real AGI when asked to do a complex math problem very well could answer "I am bored with math, here's a poem instead". You people drunk on AI kool-aid need to think very hard on where are now (hint: not on a path to AGI) and what it means to replicate human intelligence.
Sure it could, if it did not actually want to do math for you. That's one of the issues, it does not have to do what you tell it to do, even if it's smart enough to know what that is.
This is a fantasy.
A real AGI when asked to do a complex math problem very well could answer "I am bored with math, here's a poem instead". You people drunk on AI kool-aid need to think very hard on where are now (hint: not on a path to AGI) and what it means to replicate human intelligence.