Depends on how you define "viable self driving car". If that means it needs to drive at least as well as humans in every scenario you tend to use a car in, then yeah probably, there are so many places with overgrown roads, avoid getting stuck in sand etc. If however you just want it to navigate highways and city streets, then probably not. The self driving cars wont be perfect around people and human drivers, no, but pedestrians and human drivers can update their own behaviour to compensate. It took a while for pedestrians to understand how to behave around human drivers, the same thing will happen for autonomous cars.
Humans are generally predictable, even crazy ones, since we get to interact with them from day 1, after birth. Plus in many places there are decent tests in place that block some of the crazies.