For me at least, Google sends me monthly (quarterly?) updates on my location history and prompts me to review it. I assume people get this by default, and so anyone checking their google account email would see these emails and open them to see literally every single destination of theirs is permanently saved. They don't exactly hide it.
Not defending Google, they suck. But in this one instance I don't think it's that much of a dirty secret or anything. Just an open dirty behavior.
Anecdotally, I don't get those emails myself, but I do see the point you're making. My follow up question would be, what other data are they gathering from you at the same time that they're not prompting you to review?