I did something like this recently. The question was more around increasing efficiency of an algorithm though, and on top of what you said, the other part of my answer was more like "I wouldn't waste my time as long as performance was acceptable until it became an actual problem at larger scale. I'd move on to shipping other features."

It felt that they were interviewing for a computer scientist despite advertising for a software engineer.

Needless to say, we were mutually not a good fit.