California DMV disengagements reports are out for 2019, and it is time to plot some data.
As usual, these number are not really measuring reliably the safety of AV's and there are plenty ways to game them, or overreport. Please refer to my last years post for a deeper discussion (and 2017 post here, 2018 post here) on why these numbers are essentially flawed. Nevertheless these are the only official numbers we get, the only glimpse of transparency into this giant corporate endeavor called the "self driving car".
First the disclaimer - this data came from
- California DMV disengagement reports for years 2019, 2018, 2017, 2016 and 2015
- Insurance Institute for Highway Safety fatality data.
- RAND driving to safety report.
- Bureau of Transportation Statistics
all which is easily verifiable. And so here comes the plot everyone is waiting for (click to enlarge):
And as usual a quick commentary:
First of all, the only players who really have a number anywhere in the vicinity of interesting are Waymo, Cruise and Baidu. I'll discuss Baidu later, since their sudden jump in performance seems a bit extraordinary. Nevertheless even Waymo and Cruise disengagements are still approximately an order of magnitude from the upper bound of human crash rate. Would every disengagement result in a crash? Likely not, but looking at how California DMV defines these events and how Waymo and Cruise are reporting them, likely a large number of these could result with a pretty bad outcome. Again please refer to my longer discussion in 2019 post where I cite the definitions. If the trend lines on the log-plot were to hold, we should not expect a safely deployable AV sooner than in 10 years. My personal bet is that unless there is a real breakthrough in AI, these curves will asymptote near to where they are now for a long time (that asymptote is already quite visible with Waymo).
Baidu seems to have made a great progress, so much so, that their numbers look cooked to me. Last year I did not even include them in the plot since they only drove 20k miles, this year on the other hand they drove more but their disengagement rate seems unbelievably low. Either they advanced the tech beyond what anyone had ever seen, or they've been simply spinning their cars in circles on some well marked freeway on a sunny day for a few weeks (a simple way to game these numbers).
Last but not least is Tesla. They finally have recorded some autonomous testing miles with the DMV, all 12.2 of them. Yes, Tesla had 12 autonomous testing miles recorded last year. This is really suspicious since as many may remember Tesla held an "autonomy day" last year where they've demoed their capabilities and even apparently gave autonomous rides to some investors. Now given the DMV number, it looks like most of these demos and drives were indeed not even considered autonomous, hence they were simply a hoax to dupe investors (or alternatively Tesla "forgot" to report them to the DMV). Given Musks famous integrity and honesty, either would not be too surprising.
Since last year some voices started to publicly acknowledge what everyone with half a brain and any knowledge in the matter knew a long time ago - that these disengagement numbers are really very close to meaningless.
In fact the whole idea that we could infer safety if these vehicles via statistical terms is rather misguided, since all the risk is in the tail of the distribution here and it looks to be a rather long tail.
Nevertheless, I recently went to Phoenix and wanted to allow Waymo to convince me that I'm wrong. In a response to my tweet Waymo claimed they'd be holding a fest on the day I was in town and that rides would be available. So I showed up at the fest.
Turned out that two hours into the fest they were out of rides... They allocated 4 cars for that event (they have hundreds in the area as far as I know). The logical conclusion to me was that it is just a dog and pony show - if they indeed had the technology working (yes at least on a sunny day in Phoenix suburb), there would be nothing preventing them from running these rides all day long, or a least for the duration of a 3 hour fest. But apparently this is not the case.
Anyway, that's it for now, I have a longer post on the AI landscape in general in preparation, but might need some more time to finish it.
If you found an error, highlight it and press Shift + Enter or click here to inform us.
