Students are also really fixated on being in the “right” year. I took a gap year. When I recommend other people take one, they sometimes respond that they don’t want to be behind by a year, either socially or academically. What they don’t understand is that being behind or ahead is a very arbitrary idea stemming from the structure of school. It doesn’t matter in the grand scheme of things if you graduate a year later. Or if you learn a subject a year later than your peers
You say a year but (1) colleges care how far you are in technical subjects like math, (2) you should care too because it affects what classes you take while in expensive college, (3) one year is what some students are looking at if they attempt some pedagogical plan, but others are looking at way more than one year of advancement. Without looking to extraordinary students, compare the lowest vs highest math tracks in schools, which is already an advantage in years.
Sometimes advantages accumulate until you take a different pathway in life that was otherwise not a confident bet.
I'd counter that just because colleges care doesn't mean you should. Math in particular it's quite possible to pass a class without really understanding it. I'd wager that most students who took AP Calculus could not give a good definition of a limit. Indeed many math professors would much rather have students repeat calculus in college than blindly accept AP credit. Even if this allowed you to take more math classes, that wouldn't be much help if you don't know the fundamentals.
Also gap years don't have any impact on the level you come into college. If anything I came into college far more advanced in CS than my peers because of my gap year.
Just like you say, I passed the AP calculus exam in high school and then when I arrived at college my advisor, who also happened to teach a freshman Calculus 1 section, told me, "I know you passed out of it, but why don't you sit in on a couple of my classes and see what you think?" I ended up staying the whole semester when it became clear that I didn't understand anything about Calculus.