That's a real risk. That plus incompatibilities with real-world software that ignore the specs or get creative with any gaps in it. Those have to be tested. Ideally, if it's a critical protocol, it will be formally specified and verified to ensure the code does exactly what it's supposed to. At the least, formally specified with DbyC annotations and/or tests checking each part of the specs plus restricted, coding style. That should knock out most logic errors.
Modern tooling like Haskell, Dafny and SPARK Ada make this way easier than it use to be. I've recommended doing it in (preferred language) side-by-side with one or more of those in most equivalent way possible so analysis prevents errors.
It would be wonderful if we could get some standardized language for defining test cases, and some tooling for converting to working (or mostly working) code for your language and implementation (at the API level, at least). If publicly developed per protocol/library, I think it would yield a substantial benefit for robustness of common protocols we have.
Want to develop a DNS library or server? Read the RFCs and download the public test suite (which may get regular updates as people fill out corner cases, put in tests for common bugs/exploits, etc). Occasionally run public test suite plus any of your own tests you have added to confirm things function as you expect. There's so much duplication of effort in this area currently, and i'm sure the majority of test suites for related applications have at least a few tests that would be useful to others.
I agree it would be useful to have. There's all kinds of stuff like that in academia. So many different forms and tools that the problem would likely be the time available to assess them. On top of nobody using or replicating them as usual. :)
Note: sklogic did mention doing something like that by making sure all the programs' had a scriptable API then using basic, shell input and output to test them. There were acceptance, testing tools I used a long time ago that did something similar using XML. Not sure the efficiency of this method but it should work.
Modern tooling like Haskell, Dafny and SPARK Ada make this way easier than it use to be. I've recommended doing it in (preferred language) side-by-side with one or more of those in most equivalent way possible so analysis prevents errors.