I once interviewed a guy whose resume implied he played a large part in designing some of these redundant systems for carriers/whatever.

I asked him how he dealt with failover/etc, and said they used master election, and that it was guaranteed to work, and then proceeded to describe exactly a system guaranteed to fail. He had never heard of paxos, and when i described it to him, he was sure it wouldn't work.