The chance of falsely rejecting the null hypothesis increases as you gather more data. Put more simply, finding something that differs "significantly" from some distribution becomes easier as you gather more data. Imagine having only 3 psychology student in a study, the required effect size has to be huge for the test to say that it is significantly different.
However, the approach taken by CERN is of course right. They find a result at a certain significance level and then collect more data to verify the result. As long as there aren’t thousands of simultaneous verifications running, this approach is sound. Obviously yes, physicist’s know what they’re doing.
Having said that, please don’t read this comment as me approving of frequentists statistics. Bayesian or cross-validations are way easier to interpret where possible.
However, the approach taken by CERN is of course right. They find a result at a certain significance level and then collect more data to verify the result. As long as there aren’t thousands of simultaneous verifications running, this approach is sound. Obviously yes, physicist’s know what they’re doing.
Having said that, please don’t read this comment as me approving of frequentists statistics. Bayesian or cross-validations are way easier to interpret where possible.