Microsoft Research published an early version of what you describe in 2004, "Image-Based Interactive Exploration of Real-World Environments":

Paper: http://research.microsoft.com/en-us/um/redmond/groups/ivm/iv...

Web Page, Video, Slides: http://research.microsoft.com/en-us/um/redmond/groups/ivm/iv...

Quoting: "Whereas many previous systems have used still pho- tography and 3D scene modeling, we avoid explicit 3D reconstruction because it tends to be brittle."