Great point. There is ongoing research into whether memory is necessarily a network effect or if single neurons can hold memories. It's looking like they can.

https://www.nature.com/articles/s41598-019-51331-5