It's not just browsers. Browsers are pretty sane when it comes to charsets, because they had the time to make it right and the pressure to do so.
(It wasn't like that in times of NN4/IE4, which would interpret your text as whatever they want and won't even let you override)
Facing to something less agamant (like dreaded id3 tags), no such luck.
There's no way to avoid it unless you wrap it up and add some explicit checks and guesses.