There's a chance that these systems can actually out perform their training data and be better than the sum of their parts. New work out Harvard talks about this idea of "transcendence" https://arxiv.org/abs/2406.11741
While this is a new area, it would be naive to write this off as just science fiction.
It would be nice if authors wouldn't use a loaded-as-fuck word like "transcendence" for "the trained model can sometimes achieve better performance than all [chess] players in the dataset" because while certainly that's demonstrating an impressive internalization of the game, it's also something that many humans can also do. The machine, of course, can be scaled in breadth and performance, but... "transcendence"? Are they trying to be mis-interpreted?
I've been very confidently informed that these AIs are not AGIs, which makes me wonder what the "General" in AGI is supposed to mean and whether generalization is actually the benchmark for advanced intelligence. If they're not AGI, then wouldn't another word for that level of generalization be more accurate than "generalization"? It doesn't have to be "transcendence" but it seems weird to have a defined step we claim we aren't at but also use the same word to describe a process we know it does. I don't get the nuance of the lingo entirely, I guess. I'm just here for the armchair philosophy
Also it's possible that human intelligence already reached the most general degree of intelligence, since we can deal with every concept that could be generated, unless there are concepts that are uncompressible and require more memory and processing than our brains could support. In such case being "superintelligent" can be achieved by adding other computational tools. Our pocket calculators make us smarter, but there is no "higher truth" a calculator could let us reach.
The past decade has seen a huge number of problems widely and confidently believed to be "actual hard mode problems" turn out to be solvable by AI. This makes me skeptical that the problems today's experts think are hard aren't easily solvable too.
While this is a new area, it would be naive to write this off as just science fiction.