Are you saying it is a bad test because it overestimates the difficulty? If so, I agree that it overestimates the difficulty. My argument is that nobody would dare to accept such a test, which is clearly an overestimate of difficulty, therefore nobody is even close to achieving the actual number. If you wanted a more accurate estimate of security you would probably need to bump it up by a factor of 3-5x to account for execution risk.
If you are not, I am not sure what you are arguing since all of your statements show how the test overestimates the difficulty. The test is designed to identify, with reasonably high confidence, whether any attack is profitable given a specific upside. Therefore it should be as easy as possible for a legitimate bug that could result in at least a $4.5M upside to be paid.
A 100% guaranteed legal payout clearly minimizes the risk and thus allows more attacks below a $4.5M cost to execute to be profitable. Any other form of payout means the cost to execute must be lower to be profitable. Put another way, if it is unprofitable to do it totally legally for some amount of money, it is probably even more unprofitable to do it illegally for the same amount of money (obviously this excludes cases where you might be able to gain a higher upside, but then we are not talking about mitigating attacks with a certain upside), therefore this estimate should be no lower than the true cost to execute (on average).
To use your examples:
If they want a decent ROI, the cost of attack must be significantly less than $4.5M to execute to be a good investment. ROI of illegal actions usually needs to be higher to make up for the risk since you would almost always choose a legal action with the same ROI.
If the probability of payout is less than 100%, then the cost of attack must be less than $4.5M to make up for the reduced probability of success. The probability of payout for a bug bounty is usually higher than the highly variable payout of an attack. Also since the buyer is using this to make a quality decision and has authority to force the vendor to pay out with the stated scheme, it is in the buyers best interest to pay out credible attacks.
If the non-legal nature is a serious cost, then the cost of attack must be less than $4.5M for the illegal case to make up for the extra risk and cost. Non-legal actions usually come with extra costs compared to legal actions and thus have to be even more profitable to be worth doing.
Therefore, if a $4.5M bug bounty (where payout is decided by the product buyer) is not claimed after some reasonable amount of time we can conclude, with some reasonable amount of confidence, that the lowest risk option is likely unprofitable. Therefore, higher risk illegal options with the same upside are even less likely to be profitable. Thus, the test is a relatively good lower-bound for identifying if attacks with a $4.5M upside are actually being mitigated. If you can not even institute this lower bound, then you are nowhere near the necessary level.
If you are not, I am not sure what you are arguing since all of your statements show how the test overestimates the difficulty. The test is designed to identify, with reasonably high confidence, whether any attack is profitable given a specific upside. Therefore it should be as easy as possible for a legitimate bug that could result in at least a $4.5M upside to be paid.
A 100% guaranteed legal payout clearly minimizes the risk and thus allows more attacks below a $4.5M cost to execute to be profitable. Any other form of payout means the cost to execute must be lower to be profitable. Put another way, if it is unprofitable to do it totally legally for some amount of money, it is probably even more unprofitable to do it illegally for the same amount of money (obviously this excludes cases where you might be able to gain a higher upside, but then we are not talking about mitigating attacks with a certain upside), therefore this estimate should be no lower than the true cost to execute (on average).
To use your examples:
If they want a decent ROI, the cost of attack must be significantly less than $4.5M to execute to be a good investment. ROI of illegal actions usually needs to be higher to make up for the risk since you would almost always choose a legal action with the same ROI.
If the probability of payout is less than 100%, then the cost of attack must be less than $4.5M to make up for the reduced probability of success. The probability of payout for a bug bounty is usually higher than the highly variable payout of an attack. Also since the buyer is using this to make a quality decision and has authority to force the vendor to pay out with the stated scheme, it is in the buyers best interest to pay out credible attacks.
If the non-legal nature is a serious cost, then the cost of attack must be less than $4.5M for the illegal case to make up for the extra risk and cost. Non-legal actions usually come with extra costs compared to legal actions and thus have to be even more profitable to be worth doing.
Therefore, if a $4.5M bug bounty (where payout is decided by the product buyer) is not claimed after some reasonable amount of time we can conclude, with some reasonable amount of confidence, that the lowest risk option is likely unprofitable. Therefore, higher risk illegal options with the same upside are even less likely to be profitable. Thus, the test is a relatively good lower-bound for identifying if attacks with a $4.5M upside are actually being mitigated. If you can not even institute this lower bound, then you are nowhere near the necessary level.