Imagine the two sides of user behaviour, one side you can observe on the client while the other side is observed on the "server" (more like the infra where the servers are running). There are several ways of monitor traffic (that is sent by the client to the server) and identify patterns (good or bad patterns). I have seen machine learning based network intrusion detection projects that quite successfully identified bad user behaviour in HTTP traffic for example.

Riot has 100M+ users, it is probably the most statistically significant user base on Earth. You could start to map out user behaviour parameters that you need to monitor to have them as "features" in your machine learning model. There is also a ton of historical data where they identified bad behaviour somehow (reported by other players, etc.) that you can use to train the ML models.

Let's say you are trying to catch people with aim bot or in Riot's case farm bot (helping the player last hit, if you know LoL you know what I mean, if you don't nor problem, the example is probably understandable anyways).

There is the way to catch this guy by observing what is running on the client and see if there is a last hit bot process or not.

Or, you could come up with a number (or multiple numbers) that represent a typical player behaviour (CS / min adjusted by ELO, I am just making this up though) and you could try to build models around this try to see if the data to have is giving you any meaningful accuracy of predicting cheating.

I know, this is more work, more resources, probably more challenging, but it does not violate the user's privacy and it does not require the good users also having to install rootkits on their devices.

I would be a big surprise for me if it was not possible to achieve ML based user behaviour monitoring when the rest of tech companies implemented this years ago. I know that Amazon done this for sure (probably 10-15 years ago).