> A next unforeseen type of vulnerability will come, and no automated test will catch it.
But Heartbleed wasn't a new type of vulnerability. It's the same type of vulnerability we've known about at least since the dawn of C. Almost all bugs are of known types. How often do we see new types of vulnerabilities? Maybe a couple a year. If we had only a couple vulnerabilities a year across all software, we'd be many orders of magnitude better off.
Sure, I'm not saying the proposed automated test is bad. I think it might be sensible to add it. As I said, I'm 100% for automated tests (I should be, as the co-founder of a startup specializing in test automation tools). But ascribing magical capabilities to catch all future bugs to automated tests / "check lists" is IMHO detrimental to the cause because it leads to too high expectations.
A lot of stuff that gets on checklists is because not knowing about it killed someone. Other documented procedures came from time in the simulator - somewhat anagolous to manual user testing.
To be honest, the only people I know ascribing "magical capabilities" to unit testing are those disparaging it. The rest of us think it's great, and makes life a lot easier, but we've all been bitten by that unexpected bug when you see it working in the real world.
Exactly. I've always hated the "You can't test what you don't know so it's worthless." Well sorry, I can test what I do know and make sure that I don't break that. As new errors come upon, you add it to your tests (akin to checklists, essentially), and make sure you don't break that use case again.
But Heartbleed wasn't a new type of vulnerability. It's the same type of vulnerability we've known about at least since the dawn of C. Almost all bugs are of known types. How often do we see new types of vulnerabilities? Maybe a couple a year. If we had only a couple vulnerabilities a year across all software, we'd be many orders of magnitude better off.