"On October 28, 2021, after turning right onto Fremont Blvd from Cushing Pkwy, the Pony.ai Autonomous Vehicle ("Pony.ai AV") performed a left lane change maneuver in autonomous mode. While performing the lane change, the Pony.ai AV came into contact with a center divider on Fremont Blvd. and the traffic sign that was posted on the divider. The Pony.ai AV suffered moderate damage to the front of the vehicle and the undercarriage. There were no injuries and no other vehicles involved. Fremont Police Department were called to report the incident and the damaged street sign. Pony.ai has subsequently worked with local authorities to resolve all issues related to the damaged sign."
It's not a serious incident, but it is a fully autonomous vehicle under good conditions going somewhere it was absolutely not supposed to go. So DMV revoked their license for full autonomous operation without a safety driver. They have to go back to testing with a safety driver.
Could Tesla have put up adversarial triggers nearby their own facilities to test, and it overlapped on whatever opencv or vision package pony.ai is using?
Using invisible-to-humans paint on signs for adversarial recognition attacks on autonomous driving systems would also be an anti-Tesla tactic not entirely unreasonable to consider.
"Tesla autopilot crashes right outside Tesla factory!" would be a desired headline for a certain crowd.
It'll be interesting to see what caused this, at any rate, probably less than 1% chance the above is what happened.
This one gives a better idea of why this might have happened, in the direction of travel, the white back of the sign is easy to miss with a white-sided box truck behind it:
The pole at the end of the divider with the sign facing away, stretched-arrow street paint left-turn symbol, and the enormous expanse of intersection pavement are all a little disorienting.
It’s not a serious incident for a human, there are lots of explanations for such a thing which are innocent. It is a serious incident for a robot for which running into a street sign in the middle of the road is a “never” kind of problem and indicative of a significant issue.
And perhaps that's part of the problem. "Oh, this person crashed his heavy machine into something...it happens" seems to significantly undervalue the responsibility and risks of driving. It should really be a "never" problem for people, too.
FSD isn't even close to human capability yet so its a moot point. When FSD is as good as a human driver we can then think about quantifying how seriously to take its occasional malfunctions / mistakes.
“Do not drive into stationary objects” should be one of the more basic requirements for a self driving car, any company that messes that up makes me nervous about how they might handle more complicated situations.
That was part of spec of the traction/break of a vehicle I worked on.
Passagers life was prioritized as the premise behind the decision was : "Outside the vehicle is an unknown, we don't know if it's a human, an animal, an object ; we know that inside we have human life (at least the driver, at most the hundred of passengers), so we give priority to what we know)
That sounds bad. Yes, you know that there’s a person in the car, but you also know that if the object in front of you is a person then they are multiple orders of magnitude more likely to be killed than the occupants. You have to weigh the uncertainty and the disparity of outcomes.
Sounds like a continuation of the automotive trend that pretends like the only person whose life matters is the occupant.
"On October 28, 2021, after turning right onto Fremont Blvd from Cushing Pkwy, the Pony.ai Autonomous Vehicle ("Pony.ai AV") performed a left lane change maneuver in autonomous mode. While performing the lane change, the Pony.ai AV came into contact with a center divider on Fremont Blvd. and the traffic sign that was posted on the divider. The Pony.ai AV suffered moderate damage to the front of the vehicle and the undercarriage. There were no injuries and no other vehicles involved. Fremont Police Department were called to report the incident and the damaged street sign. Pony.ai has subsequently worked with local authorities to resolve all issues related to the damaged sign."
It's not a serious incident, but it is a fully autonomous vehicle under good conditions going somewhere it was absolutely not supposed to go. So DMV revoked their license for full autonomous operation without a safety driver. They have to go back to testing with a safety driver.
[1] https://www.dmv.ca.gov/portal/file/pony-ai_102821-pdf/