I think `zsh -l` start a login shell, which does not load zshrc so oh-my-zsh don't get initialized. Try `zsh -ic exit` and it should load zshrc before executing exit.
Valid points, I learned something new today. Thanks, you were right. If using -ic flags I am getting around 300 ms... Interesting how I never noticed, guess I don't open many terminal during the day
That said, the time of `zsh -ic exit` isn't really meaningful metric for measuring the performance of an interactive shell. See https://github.com/romkatv/zsh-bench#how-not-to-benchmark for details.