You can partially infer something from the lack of ratings. People usually only do them to give good or bad ones, so you can balance it out a bit with non-voters as "average" scores.
Not voting is nearly impossible on Uber though. As soon as you open up the app it prompts you. Often I forgot the ride and just press 5 stars unless something was exceptionally bad.
It's still goofy, though.