He does have a point. It's not common because usually higher-ups always remind people who's in charge. It's up to you, but if you decide to stay you need to be more affirmative and build self-esteem.