Yeah not feasible with todays methods and rag / lora shenanigans, but the way the field is moving i wouldn't be surprised if new decoder paradigms made it possible.
Saw this yesterday, 1M context window but haven't had any time to look into it, just an example new developments happening every week: