My wife and I moved to Colorado shortly after legalization. We hardly consume marijuana, but don't have a problem at all with people who use regularly.

Our experience has been markedly different from OP; zero downsides to legalization.

Colorado is a delightful place, though the Denver housing market is going the way of California.