I was imagining the scenario where a person is being unfairly accused. If that's not the case my comment doesn't apply.

Because this is counter-accusation.

If judge has all reasons to believe in fairness of the algorithm, then counter-accusation is the only way through.

How can the “fairness” be determined if the specifics of the algorithm are not known to the judge, defense or jury?

We can’t allow people to be prosecuted by evidence that relies on black boxes.

The burden should be on the prosecution to prove that the algorithm is fair and reliable; the defendant is presumed innocent, ie falsely accused.