it is surprising, but give this question to some random people on the street without context and you would be surprised

All sorts, we go through third parties. But apps include stuff like Duo Lingo, Games, Sport Betting Apps ect. Its an optional opt in instead of watching ads or paying for the app. And obviously you are vetted that you don't spam.

We were surprise ourselfes, but if you walk around and randomly ask people in the street, I think you would be surprised what you would find. Its a trick question.

Yeah, the point of a trick question is to have fun. So the correct answer is obviously that one that makes the other laugh.

We try a bit harder than that my friend.

I actually didn't mean to criticize Rapidata. I just think that a forced-choice question like this begs for low-effort answers. At least the respondents should have had the opportunity to explain their reasoning, like the LLMs did.

All good ^^, its a fair point, we have come up with some fun ways to track peoples reliability over time. But the validation sets contain plenty of forced-choice questions, those that have an empirical true can be used directly to calculate a reliability, those that are subjective need to be re-asked after sometime to ensure consistency. People that don't pass thresholds would not be part of the 10'000 here.

But of course. If every human was told to take 3 minutes to deeply think about it and told that its a trick question, then they most likely will all get it right. But its the same with the LLMs, if you ask them like that they will get it right most of the time. The low effort is kinda the point here.