We want to make a utopia. But no one knows or agrees what utopia would look like. And it s very easy to get it wrong and get stuck in a bad or suboptimal system.
I think that (superhuman) AI will help us figure this out. It could model us well and figure out "what we really want". It could propose questions and ideas we wouldn't think of, and help us work out what the answer Is.
However the problem is getting to that point. We don't know how to make an AI that will follow instructions at all. Most possible A Is do everything within their power to take over the world.
I would love to make utopia. Maybe some portion of technologists would like to make utopia.
If one is talking about "humanity", it is hard to talk about "what we want".
Edit: The problem I'd see with the "AI can let us reach the utopia whose characteristics we can't agree on" is imo that intelligence doesn't have goals as such baked.
A very, very skilled therapist talking to a feuding family may be able to get them to compromise. But almost all such compromises are going to be predicated on the social context the therapist and family work. If there isn't a social consensus pointing to what utopia for the entirety of humanity could look like, even superhumanly intelligent computer couldn't talk us into such a thing - though it quite possibly could talk us into a position that reflected the social value's its creators implicitly or explicitly programmed in. My hope would be it's creators won't be religious fanatics but naturally said religious fanatic might have the opposite hope.
I think that (superhuman) AI will help us figure this out. It could model us well and figure out "what we really want". It could propose questions and ideas we wouldn't think of, and help us work out what the answer Is.
However the problem is getting to that point. We don't know how to make an AI that will follow instructions at all. Most possible A Is do everything within their power to take over the world.