My favorite noise story is from just a couple years ago. Our controller would run fine for hours or days and then reset for no apparent reason. Looking at the debug output, I could tell that it wasn't a watchdog or other internal reset (e.g., system panic) and there had been no user input. The debug log basically said that someone pushed the reset button, which clearly wasn't happening.
The EE and I were standing around the machine and he happened to be in front of the UI when it reset and I mentioned that I heard a soft click just before he said that it reset, but we had no hardware in the region where I thought the noise came from.
Finally, we put two and two together and realized that the system included a propane heater with an automatic controller and the noise I heard was probably the propane igniter. The high voltage from the igniter was wirelessly coupling into one of the I/O lines going to the controller board. The reason that the problem had suddenly started happening after months of trouble-free development was that the customer had rerouted some of the wiring when they were in the machine fixing something else and moved it closer to the heater.
In 30 years of doing this, I can count on one hand the number of times I've had to deal with noise that was coupling in through the air!
I’ve worked with electron microscopes and in silicon fabs and it’s super fun being on the teams hunting for sources of noise during construction and bringup. In fabs there are multiple teams because it’s so specialized, the HVAC team being the most interesting one because they’ve got tons of mechanical and electronic sources all over the buildings. They were also the long tail for problems with yield (which was expected and budgeted for). I think the EM startup I worked for failed in part due to not taking the issue seriously enough.
I can’t tell any specific stories but poorly shielded USB ports were the bane of our existence in the 2000s. Every motherboard would come with them and the second a random floor worker would plug something in it’d take down a section of the fab or all of the microscopes even if it were on the other side of the building. For some god forsaken reason all the SBC manufacturers used by tons of the bespoke equipment were also adding USB ports everywhere. We ended up glueing all of them shut over the course of the several months it took to discover each machine as floor workers circumvented the ban on USB devices (they had deadlines to meet so short of gluing them shut we couldn’t really enforce the ban).
The EE and I were standing around the machine and he happened to be in front of the UI when it reset and I mentioned that I heard a soft click just before he said that it reset, but we had no hardware in the region where I thought the noise came from.
Finally, we put two and two together and realized that the system included a propane heater with an automatic controller and the noise I heard was probably the propane igniter. The high voltage from the igniter was wirelessly coupling into one of the I/O lines going to the controller board. The reason that the problem had suddenly started happening after months of trouble-free development was that the customer had rerouted some of the wiring when they were in the machine fixing something else and moved it closer to the heater.
In 30 years of doing this, I can count on one hand the number of times I've had to deal with noise that was coupling in through the air!