- You would need some sort of barrier preventing movement of researchers between these audit teams and the institutions they are supposed to audit otherwise there would be a perverse incentive for a researcher to provide favorable treatment to certain institutions in exchange for a guaranteed position at said institutions later on. You could have an internal audit team audit the audit team, but you quickly run into an infinitely recursive structure and we'd have to question whether there would even be sufficient resources to support anything more than the initial team to begin with.
- From my admittedly limited experience as an economics research assistant in undergrad, I understood replication studies to be considered low-value projects that are barely worth listing on a CV for a tenure-track academic. That in conjunction with the aforementioned movement barrier would make such an auditing researcher position a career dead-end, which would then raise the question of which researchers would be willing to take on this role (though to be fair there would still be someone given the insane ratio of candidates in academia to available positions). The uncomfortable truth is that most researchers would likely jump at other opportunities if they are able to and this position would be a last resort for those who aren't able to land a gig elsewhere. I wouldn't doubt the ability of this pool of candidates to still perform quality work, but if some of them have an axe to grind (e.g. denied tenure, criticized in a peer review) that is another source of bias to be wary of as they are effectively being granted the leverage to cut off the lifeline for their rivals.
- You could implement a sort of academic jury duty to randomly select the members of this team to address the issues in the last point, which might be an interesting structure to consider further. I could still see conflict-of-interest issues being present especially if the panel members are actively involved in the field of research (and from what I've seen of academia, it's a bunch of high-intellect individuals playing by high school social rules lol) but it would at least address the incentive issue of self-selection. Perhaps some sort of election structure like this (https://en.wikipedia.org/wiki/Doge_of_Venice#:~:text=Thirty%....) could be used to filter out conflict of interest, but it would make selecting the panel a much more involved and time-consuming process.
The "Jury Duty" could easily be implemented in the existing grant structure - condition some new research grant on also doing an audit of some previous grant in your field (and fund it as part of the grant).
- You would need some sort of barrier preventing movement of researchers between these audit teams and the institutions they are supposed to audit otherwise there would be a perverse incentive for a researcher to provide favorable treatment to certain institutions in exchange for a guaranteed position at said institutions later on. You could have an internal audit team audit the audit team, but you quickly run into an infinitely recursive structure and we'd have to question whether there would even be sufficient resources to support anything more than the initial team to begin with.
- From my admittedly limited experience as an economics research assistant in undergrad, I understood replication studies to be considered low-value projects that are barely worth listing on a CV for a tenure-track academic. That in conjunction with the aforementioned movement barrier would make such an auditing researcher position a career dead-end, which would then raise the question of which researchers would be willing to take on this role (though to be fair there would still be someone given the insane ratio of candidates in academia to available positions). The uncomfortable truth is that most researchers would likely jump at other opportunities if they are able to and this position would be a last resort for those who aren't able to land a gig elsewhere. I wouldn't doubt the ability of this pool of candidates to still perform quality work, but if some of them have an axe to grind (e.g. denied tenure, criticized in a peer review) that is another source of bias to be wary of as they are effectively being granted the leverage to cut off the lifeline for their rivals.
- You could implement a sort of academic jury duty to randomly select the members of this team to address the issues in the last point, which might be an interesting structure to consider further. I could still see conflict-of-interest issues being present especially if the panel members are actively involved in the field of research (and from what I've seen of academia, it's a bunch of high-intellect individuals playing by high school social rules lol) but it would at least address the incentive issue of self-selection. Perhaps some sort of election structure like this (https://en.wikipedia.org/wiki/Doge_of_Venice#:~:text=Thirty%....) could be used to filter out conflict of interest, but it would make selecting the panel a much more involved and time-consuming process.