My point is that a memory reference is a conceptual reference. There is no difference. Understanding references is not about building abstraction atop your understanding of them; it's about recognising that you can express natural abstractions literally through them.
I'm unclear on what it means to "understand references" in a way that doesn't involve the way they behave in code. Unless you're saying that the concept as it appears in code can be explained entirely with the tautological definition, in which case I would be inclined to disagree.
Of course, I'm also not sure that Joel's suggestion that some people have an aptitude for such understanding is "irresponsible." If you were to say he underestimates what people can pick up with practice, that would be a different story.
I just think he comes too close to saying that it's entirely the onus of the student to understand the concepts, when the other half of the problem is that they're being taught (in my humble but loud opinion) incorrectly.