How would you learn parameters to tweak the rendered scene if the renderer is not differentiable, and you can't backpropagate through the renderer to calculate the appropriate parameter adjustments from the pixelwise errors?
I suppose you theoretically could do it with some trial and error method or grid search or something like that, but it's going to be absolutely computationally unfeasible in the general case; the pixelwise errors only become practically useful if you have an uninterrupted differentiable/'backpropagatable' path from your parameters to the pixels.
Yes, throughput would be larger and we would loose the backprop path like you said, but it seems practical in some ways and actually guiding approaches like https://nv-tlabs.github.io/meta-sim/
I suppose you theoretically could do it with some trial and error method or grid search or something like that, but it's going to be absolutely computationally unfeasible in the general case; the pixelwise errors only become practically useful if you have an uninterrupted differentiable/'backpropagatable' path from your parameters to the pixels.