As humans we have always tried to obtain foods that are safe and nutritious. Even though milk has potential allergens it contains no poisons, toxins or anti-nutrients unlike many plant foods. It is incredibly nutritious and contains sufficient amounts of most minerals and vitamins needed to sustain life (babies use it exclusively without developing any deficiencies after all).
Apart from this it has other qualities that make it useful in a post-agriculture world. Diets rich in grains and animal meat are too rich in iron and phosphorous and deficient in calcium. Milk is low in iron, and even binds to iron reducing its absorption. It also has higher calcium compared to phosphorous. It has been shown that the ratio of calcium to phosphorous matters more than absolute consumption of each nutrient, and should be slightly higher than one if we want to avoid leeching calcium from bones. Diets rich in leafy greens and fruits have a favorable ratio, while meats, legumes and cereals do not and this probably contributed to the skeletal deformities we see post agriculture. Dairy helps shift the ratio closer to optimal in the presence of phosphorous rich foods in the diet.
With allergies and intolerances being the only reasons not to drink it, and those being more common in poor health. The question should be, why not drink milk?
The question whether milk is beneficial for humans is far more complicated than you suggest. The only thing we know for sure is that consuming milk is not the net positive that it has been portrayed to be.
It's most likely a net zero [1] with a number of positive benefits that you describe, in particular for infants. There are however a number of risks as well, in particular when you define milk as the product that you can buy in an American/European supermarket or if you look at dairy products more generally. And that's not taking into consideration that the majority of the world's population is lactose intolerant. Here is a link to get you started (don't take this source literally for reasons of bias but treat it as a starting point) [2].
I did not even try to approach the question of whether milk is beneficial to health. First we would need to define the question better. Beneficial compared to what? To a dairy free pre agricultural diet? Then it's probably neutral at best. However compared to a calcium deficient diet it's definitely an improvement even if it didn't turn out to be absolutely optimal.
Getting enough calcium in the diet has been linked to healthier BMI and decreased risk of stroke and heart disease. You can get calcium elsewhere but most people don't, or they didn't at the time.
And if you replace milk in the diet you now need to find many other sources of nutrition to make up for lost vitamins and micronutrients which might have been challenging in cooler climates. Maybe milk isn't absolutely optimal for health, the jury is still out on that, but likely it was much better than anything they could get at the time.
Even in the context of a modern diet, replacing milk with soy milk and vegetable milks will yield too much phosphorous and leave you deficient in calcium unless it's fortified (calcium citrate, malate and other kinds of supplemental calcium are linked to worse health, in contrast to dietary calcium which seems to be protective). It can be done by eating a mostly vegetarian diet rich in fruits and leafy greens, or grinding eggshells yourself and adding it to meals but most of the population won't be doing that when eliminating dairy.
Let's put it this way: Consuming dairy products was historically seen so beneficial that we evolved the ability to digest lactose not just once, but multiple times in history.
It may not be as critical today, but at some point in time, being able to digest lactose was an evolutionary advantage.
Apart from this it has other qualities that make it useful in a post-agriculture world. Diets rich in grains and animal meat are too rich in iron and phosphorous and deficient in calcium. Milk is low in iron, and even binds to iron reducing its absorption. It also has higher calcium compared to phosphorous. It has been shown that the ratio of calcium to phosphorous matters more than absolute consumption of each nutrient, and should be slightly higher than one if we want to avoid leeching calcium from bones. Diets rich in leafy greens and fruits have a favorable ratio, while meats, legumes and cereals do not and this probably contributed to the skeletal deformities we see post agriculture. Dairy helps shift the ratio closer to optimal in the presence of phosphorous rich foods in the diet.
With allergies and intolerances being the only reasons not to drink it, and those being more common in poor health. The question should be, why not drink milk?