In the US, being factually accurate is an airtight defense against libel.
Face it, you don't get to control what other people say about you, nor should you want to. (Whether that "saying" is gossip, in print, or photoshopping your face on to unsavory things.)
In Sweden, it's not. I do think that it's reasonable to have certain expectations about what can be spread about you as a private person, regardless if the things said are true or not.