No one puts raw bit source directly into private key, they always whiten it via some method (often entropy pool setup using strong hash/encryption functions).

That means that even if you "random inputs" are totally predictable, the random values which come out of whitener are completely distinct, and generated RSA keys have virtually zero chances of being similar.

That's arguing that you can just seen an RNG with the current time and use a PRNG as your randomness source - a whitener can't give you randomness out which isn't there to start with.

In the above, fairly extreme case, the risk should be obvious: if someone has a decent guess on what the uptime of your system is, and knows you're doing this, then the search space to crack certificates can be made accessibly small.

Like if you know see a certificate with a Valid From date of say, January 1, 2025 but you know the service definitely wasn't running on January 1, 2024, then by guessing what the PRNG is you've constrained your search space to 1704027600 through 1735650000. So the issue isn't whether the numbers you emit are distinct - it's that an adversary can make it suitably likely that they can produce colliding RSA keys themselves anyway (and remember, they get unlimited attempts at this - they only have to succeed once).

EDIT: And while you can certainly argue that they couldn't predict the exact noise environment of say, your server room, it's also not impossible to model which also might constrain the search space enough to accessible. It's not "haha! we know your every move" it's just making the problem space small enough to brute force.