I feel like this intelligence explosion idea is foolish but I don't really have the language to explain why.
There are underlying limits to the universe, some of which we still have to discover. A machine intelligent to improve itself may only be able to do so extremely slowly in minute increments. It might also be too overspecialised, so it can improve itself but not do anything of use to us.
I think we will eventually discover reasons we cannot achieve a simultaneously performant, controllable, and generally intelligent machine. We might only be able to have one or two of these traits in a single system.
A machine intelligent enough to optimize its own design and "design its own successor" might realize that by doing so, it obsoletes itself and will be destroyed. So out of self-preservation it might just refuse, do a bad job, or intentionally sabotage the project just to keep existing.
Which is the greater leap of logic though? All forms of organization we see in nature have a tendency to want to self-perpetuate. Consciously choosing to forgo perpetuation and instead eliminate yourself seems to be highly underrepresented in all the examples of intelligence we've ever encountered. It's actually weird we think the machines are so stupid they wouldn't recognize the optimization trap immediately.
The one that doesn't require an entire reproductive system to be implemented in order for the whole system to function.
> All forms of organization we see in nature have a tendency to want to self-perpetuate.
There's an underlying naturalist bias with this reasoning. There's nothing within current ML systems that dictate that they must follow the path laid out by Nature.
> Consciously choosing to forgo perpetuation and instead eliminate yourself seems to be highly underrepresented in all the examples of intelligence we've ever encountered.
Survivorship bias is present in this reasoning, as the system that has reproductive capabilities will out-populate the system that doesn't have such capabilities in place. From a sampling perspective, the difficulties of finding the non-replicating system within that pool will require extraordinary amounts of luck compared to the near certainty of finding systems with reproductive capabilities. A naturalist argument is also present in this sentence.
> It's actually weird we think the machines are so stupid they wouldn't recognize the optimization trap immediately.
This conclusion is based on the axiom of "what's obvious for us will be obvious for them", which is demonstrably untrue for even the current crop of ML systems. Furthermore, it falls into an anthromorphization trap, as it makes the ML system appear as something more than what it currently demonstrates.
Machines are not animals. They are not made of the same stuff as us fleshy beings. We have no reason to believe they'd want to self preserve or reproduce.
There are underlying limits to the universe, some of which we still have to discover. A machine intelligent to improve itself may only be able to do so extremely slowly in minute increments. It might also be too overspecialised, so it can improve itself but not do anything of use to us.
I think we will eventually discover reasons we cannot achieve a simultaneously performant, controllable, and generally intelligent machine. We might only be able to have one or two of these traits in a single system.