Because if you've accepted patches you're not the author of the entire code base anymore. So it's not your own work, but a collaboration.

This is why I never sign CLA. Since I'm basically signing my rights away.

Right, but if you had signed a CLA then it's absolutely not violating the AGPL if the host decides to close the source. Since these organisations require that, they're not violating the AGPL.

That's not to say I agree with them and it's obviously shitty behaviour – but license violation is a specific act that they aren't guilty of.