One possibly explanation for these results which would be less interesting that their suggested conclusion: bigger items with smaller margins are better than smaller icons with larger margins.
The difference on the icon results were particularly large and convincing, and if you look at their example icon stimuli, clearly the flat icons are simpler (as expected) and presented with more margins (perhaps also as expected in flat design), but that also means to get the same number of icons on screen, the icons were smaller.
I think it's at least worth considering whether that might explain a large part of the difference.
Another limitation is that tiny and non-diverse sample size, and the fact that it's a few years old. They had just 20 students, almost all male, Russian university-student-aged - it's just not a huge or diverse sample; and the experiment was run in 2014; people have a few more years of flat design expectations under their belts now.
Looks pretty decent, those results, but I still think it's a little early to draw solid conclusions based on this alone.
To back up your more skeptical points, and to hint at the linked study being a symptom of the file drawer problem, here's a study that cites the linked one and which finds no statistically significant difference using a more diverse set of subjects (although still only 21 of them): https://www.researchgate.net/publication/326069864_Affordanc... It does commit the fingernails-down-blackboard sin of talking about "trends" for non-significant results, but otherwise seems relatively sound.
Here's another that effectively finds no significant differences, while subjects perceived flat design as more usable (although full text isn't available so I'm not sure about the methods): https://www.researchgate.net/publication/325563195_A_Compara...
Caveat - this is just from a quick search through studies that cited the linked article. There may be more robust studies around.
The difference on the icon results were particularly large and convincing, and if you look at their example icon stimuli, clearly the flat icons are simpler (as expected) and presented with more margins (perhaps also as expected in flat design), but that also means to get the same number of icons on screen, the icons were smaller.
I think it's at least worth considering whether that might explain a large part of the difference.
Another limitation is that tiny and non-diverse sample size, and the fact that it's a few years old. They had just 20 students, almost all male, Russian university-student-aged - it's just not a huge or diverse sample; and the experiment was run in 2014; people have a few more years of flat design expectations under their belts now.
Looks pretty decent, those results, but I still think it's a little early to draw solid conclusions based on this alone.