The status quo is you either have to pay someone ~$10/hr to drive for you, or you have to focus and keep both hands on the wheel and actually drive yourself, and it's still pretty dangerous. Self-driving cars could improve on the status quo significantly without actually being less dangerous.
Driving is dangerous in generally predictable ways, and people accept that. People also accept that there is usually someone to blame for most car accidents. Driving is dangerous but it is under human control.
Autonomous vehicles are dangerous in unpredictable ways. People might be injured or die in accidents that would not have happened with human drivers, as the result of software bugs rather than human decisions. When things like that happen, statistics about accident and death rates being lower with self-driving cars are beside the point.
Hopefully our regulators can regulate rationally, even if our click-bait-emotionally driven general public cannot. If they can manage it, thousands of lives will be saved.
If you're concerned about safety, driving aids like drowsiness/attention warnings, lane keeping assist, automatic emergency braking, etc are the way to make things a bit safer. In combination with the continuing march on collision safety for when they do happen.
If you're concerned about the costs of a driver, I grant that the marginal costs per hour of driving is low, but the material costs seem high, unless a significant cost breakthrough is found in lidar production, or a technique breakthrough using multiple cameras; and the R&D costs seem pretty enormous -- people have been seriously working on this since the 80s and it's clearly closer than fusion, but it seems perenially 15 years away.
