Eh, seems just as likely you'd lose your keys if you lose your config. At least, I keep both in ~/.ssh