Clover invades garden beds. It will choke out an edible garden and destroy the aesthetics of an ornamental garden. It will prevent seedlings from thriving. There is absolutely nothing wrong with not wanting clover to invade your garden beds.
Beyond that, they don’t need a reason at all. Imposing one’s beliefs and opinions on a neighbour is the kind of intellectually stunted behaviour that we’re struggling with as a species.
I have 6 acres and deliberately seed all over with white and red clover as a cover crop. Never had a problem with it choking out anything, it's easily managed.
Crab grass and creeping charlie is the problem here. Chokes out everything else.
Maybe it's climate/region dependent, but clover is probably one of the most innocuous things growing here. Fixes nitrogen, is kind of pretty if left to flower, and attracts pollinators.
It’s sad that some people don’t.