>We don't fully understand how we relate words to concepts and meaning ourselves,
This is definitely true.
>but to the extent we do, LLMs are by far the closest implementation of those same ideas in a computer
Well - this is half true but meaningless. I mean - we don't understand so LLM's are as good a bet as anything.
LLMs will confidently tell you that white wine is good with fish, but they have no experience of the taste of wine, or fish, or what it means for one to compliment the other. Humans all know what it's like to have fluid in their mouths, they know the taste of food and the feel of the ground under their feet. LLMs have no experience, they exist crystalised and unchanging in an abstract eternal now, so they literally can't understand anything.
I agree with your general point: it is a mistake to say "these two things are mysterious, therefore they are the same".
That said:
> LLMs have no experience, they exist crystalised and unchanging in an abstract eternal now, so they literally can't understand anything.
Being crystalised and unchanging, doesn't tell us either way if they do or don't "understand" anything — if it did, then I could only be said to "understand" whatever I am at some moment actually experiencing, so it would not be allowed to say, for example, that I can understand "water in my mouth" because my memory of previous times I had water in my mouth seem to be like that.
They're definitely not "like us", but that's about all I can say with confidence, and it's a very vague statement.
It's incoherent to think the ability to reason requires the reasoner to be able to change permanently. You realize that LLMs do change; their context window and model weights change on every processed token. Not to mention the weights can be saved and persisted in a sense via LORAs.
The belief LLMs cannot reason maybe justifiable for other reasons, just not for reasons you've outlined.
I'm not sure you're right you know. I think that the way that an LLM maintains a conversation is to have the conversational thread fed into an instance of it at every step. You can see this if you do a conversation step by step and then take all of it (including the LLM responses) apart from the final outcome and paste that into a new thread:
if you think about how these things work as services you can see that this makes sense. The model weights are several gb, so caching the model weights for utilisation by a particular customer is impractical. So if the forward pass does update the model then that's instantly discarded, what's retained is the conversational text, and that's the bit that's uploaded to the model on each iteration for a new reply. There are hundreds of requests pinging through the data center where the models are used every second, all of these use the same models.
But if you believe that there is a reasoning process taking place in the text then fair enough.
This is definitely true.
>but to the extent we do, LLMs are by far the closest implementation of those same ideas in a computer
Well - this is half true but meaningless. I mean - we don't understand so LLM's are as good a bet as anything.
LLMs will confidently tell you that white wine is good with fish, but they have no experience of the taste of wine, or fish, or what it means for one to compliment the other. Humans all know what it's like to have fluid in their mouths, they know the taste of food and the feel of the ground under their feet. LLMs have no experience, they exist crystalised and unchanging in an abstract eternal now, so they literally can't understand anything.