The way I look at it is this: mathematics is just formalized perfect reasoning. Its job is to get you from a thing you assume to be true, through steps of formal inference, to a new thing that you can be sure is true if the original assumption holds. You then keep on chaining this, so from a small set assumptions you enumerate all the facts that follow from them.

Physics is a field of study that does experiments to identify what things are fact in our universe, and then tries to pick such a subset of math that best fits the observed data. If further experimental results deviate from what the chosen mathematical model predicts, physicists will pick a different model, up until the maths align perfectly with what they see. The reason physicists use math is thus simply because alternative to math is hand-waving and guessing, and physics is a serious field.

As a mathematician myself I'm inclined to say "everything's applied math", but it's really not the case. Physics studies and seeks after the basic laws that govern the universe. Physicists find mathematics very useful in these pursuits. But ultimately physics studies the universe and employs mathematics thusly. In theory, physicists could use other tools than math if those other tools were as useful or (preferably) more useful. I like to think of mathematics as the study of logical systems. That's how you can have mathematicians that specialize in biology, physics, music, and so on.