Yes, this is a good point. With the Authenticator app on the phone and unguessable passwords (stored in a password manager on the phone), security all comes down to your Phone’s passcode. Know that, and you have complete access to all accounts.

Apple and Google should just short circuit this and directly be an Authenticator as well as manage the long lived token (the password) behind the scenes, which will eliminate the alt-tab dance.