Well this is obviously a hot button topic. AFAIK I was nearby when fuzz-testing for
software was invented. I was the main advocate for hiring Andy Payne into the Digital
Cambridge Research Lab. One of his little projects was a thing that generated random but
correct C programs and fed them to different compilers or compilers with different
switches to see if they crashed or generated incorrect results. Overnight, his tester
filed 300 or so bug reports against the Digital C compiler. This was met with substantial
pushback, but it was a mostly an issue that many of the reports traced to the same
underlying bugs.
Bill McKeemon expanded the technique and published "Differential Testing of
Software"
https://www.cs.swarthmore.edu/~bylvisa1/cs97/f13/Papers/DifferentialTesting…
Andy had encountered the underlying idea while working as an intern on the Alpha processor
development team. Among many other testers, they used an architectural tester called REX
to generate more or less random sequences of instructions, which were then run through
different simulation chains (functional, RTL, cycle-accurate) to see if they did the same
thing. Finding user-accessible bugs in hardware seems like a good thing.
The point of generating correct programs (mentioned under the term LangSec here) goes a
long way to avoid irritating the maintainers. Making the test cases short is also
maintainer-friendly. The test generator is also in a position to annotate the source with
exactly what it is supposed to do, which is also helpful.
-L