"Seem" is doing a lot of work here. So is the implicit claim that theory of mind in general can be demonstrated by current-gen foundation models, and only those aspects dependent on vision cannot.
I say seem but it's stronger than that. all evidence and testing points towards a general factor of intelligence. The better you perform at one "kind" of intelligence task, the better you will perform at them all. The shift in defining intelligence didn't come from nowhere. Yes, It's easy to think that there are multiple different mutually exclusive-ish kinds of intelligences and that you can excel in one and it has no bearing on performance on the other but that's not really true. all indication point otherwise. I'm not saying there aren't other factors but generally, that's what you can expect.
Yes theory of mind can be demonstrated. Make up whatever bespoke story you can with characters having varying levels if intention and knowledge. Then query GPT-4 about the state of the characters.
What i'm saying is that the vision component introduces another point of error, is it a lack of theory of mind ? or being yet unable to extract the necessary features from visual data ?
They rapidly learn to but Blind people who could recognize squares by feel do not have the ability to recognize squares by sight upon gaining vision. https://www.projectprakash.org/_files/ugd/2af8ef_5a0c6250cc3...
