I don't really buy the comparison. If you're really unlucky, you can get cancer from a "safe dose" of radiation.
Low exposures of both things are statistically less likely to hurt you than large doses. We pick a line to call "safe", but completely safety in either case is not guaranteed.
There is a natural level of radioactivity which the body is used to compensate. Small additional doses of radioactivity can therefor be neglected. This is not true for stuff like abestos.
Low exposures of both things are statistically less likely to hurt you than large doses. We pick a line to call "safe", but completely safety in either case is not guaranteed.