It's time for another post in the Tesla FSD series, which is a part of a general self driving car debacle discussed in this blog since 2016 [1,2,3,4,5,6,7]. In summary, the thesis of this blog is that AI hasn't reached the necessary understanding of physical reality to become truly autonomous and hence the contemporary AI contraptions cannot be trusted with important decisions such as those risking human life in cars. In various posts I go into detail of why I think that is the case [1,2,3,4,5,6] and in others I propose some approaches to get out if this pickle [1,2,3]. In short my claim is that our current AI approach is at the core statistical and effectively "short tailed" in nature, i.e. the core assumption of our models is that there exist distributions representing certain semantical categories of the world and that those distributions are compact and can be efficiently approximated with a set of rather "static" models. I claim this assumption is wrong at the foundation, the semantic distributions, although technically do exist, are complex in nature (as in fractal type complex, or in other words fat tailed), and hence cannot be effectively approximated using the limited projections we try to feed into AI and consequently everything built on those shaky foundations is a house of cards. To solve autonomy we rather need models that embrace the full physical dynamics responsible for generating the complexity we need to deal with. Models whose understanding of the world ultimately becomes "non-statistical". I don't think we currently know how to build those types of models and hence we just try to brute-force break into autonomy using the methods we have and contrary to popular belief that is not going very well. And the best place to see just how hopeless those efforts are, is in the broad "autonomous vehicle" industry and Tesla in particular.
Russian roulette
Let me begin this post by discussing why "probabilistic" approach is inadequate for mission critical applications. The crux of the discussion that probabilities are deceptive when what really matters is not the random variable itself, but a certain function of the random variable, often times called payoff function in the context of economical discussions. To illustrate this, imagine you are playing a simplified Russian roulette with a toy gun. The gun has got six chambers, if you hit any of the 5 empty ones, you win a dollar, if you hit the one with a bullet, the revolver will make a fart sound and you lose two dollars. Would you play this game? Obviously the probability of winning is 5/6 and losing only 1/6, the mean gain from a six shot round of this game is $3 (every pull of the trigger gets you $0.5 on average), so it's a no brainer. Everybody would play. Now imagine you play that same game with a real gun and a real bullet. Unless you are suicidal, you would stay away from that kind of entertainment. Why? Neither of the probabilities have changed? Of course what really changed is the payoff function. When you lose, you don't only lose 2 bucks, but also your life. What if the gun had 100 chambers? Would you play? I know I wouldn't. What if it had a thousand chambers? Most people wouldn't have touched that game even if the revolver had a million chambers. That is if they knew with certainty that one of them has a bullet and will cause an instant death. Things are a bit different if players didn't know about the deadly load. In such case, observing one player pulling the trigger hundreds of times and getting a dollar each time would attract many players. Until one time the gun fires. But if the game is played in such a way that there are multiple independent revolvers and when one goes off, players triggering other guns don't know about it, you could probably have a large group of players constantly try their luck. And that is exactly what is going on with Tesla FSD. If people knew the real danger the FSD game poses, nobody sane would have attempted it. But because incidents are rare and so far were't disastrous (in case of FSD, but at least 12 people lost their lives in autopilot related crashes), there is no shortage of volunteers to try. But that is where the government safety agencies need to step in. And just like the government wouldn't allow people to offer the game of Russian Roulette to uninformed public (even with a revolver with a million chambers), based on the expert knowledge and risk assessment, the FSD experiment needs to be curbed ASAP.
How did we get here?
Many automakers have been putting perception and intelligence in their cars for many years. Cruise control, lane following, even street sign detection have been around for more than a decade. All these contraptions rely on a fundamental assumption that the driver is in control of the car. The tech features are there to assist the driver but ultimately he is constantly in control of the vehicle. Systems such as automatic emergency breaking only take over the control very rarely, only when a crash is imminent. For the most part people learned how to live with this separation of responsibilities. In the meanwhile the ongoing development of fully autonomous vehicle has been slowly taking off, with projects such as Ernst Dickmanns Mercedes in 1986, Carnegie Mellon University’s NavLab 1, Alberto Broggi ARGO project. These efforts were rather low key until 2005 DARPA Grand Challenge in Mojave desert, where finally five contestants managed to complete 132 mile off-road course. The excitement and media coverage of this event gathered attention of Silicon Valley and gave rise to a self driving car project at Google in January of 2009. The appearance of Google "self driving" prototypes in the streets of San Francisco in the early part of 2010's ignited a hysteria in the Bay Area, where the "self driving car" became the next big thing. By 2015, aside from Google, Uber started its own self driving project in collaboration with CMU, Apple secretly started their project Titan, and various startups were popping out and raising money like crazy. While almost everybody in this field would make use of relatively recent invention of a LIDAR (light based "radar"), which allows for very accurate measurement of distance to obstacles, Tesla decided to use a different path, ignoring LIDAR altogether and relying purely on vision based approach. In March 2015 Tesla introduced their "autopilot" software. The name being controversial, in essence autopilot was a set of features known from other vehicles, such as lane following, adaptive cruise control and a set of what could be described as relatively useless gadgets such as summon, which allowed a car to drive out of a garage towards an owner with a cell phone (often scratching garage door or hitting fences). However, whatever the features were, marketing for them was a completely different story. Elon Musk in his typical style of overpromising essentially stated that this is the bold birth to autonomous Tesla cars and that while not yet ready, the software would be getting better and within few years tops no Tesla owner will have to worry about driving. At that time, Tesla was relying on off the shelf system delivered by Israeli company Mobileye and really aside from enabling features that no responsible car company would have enabled in a consumer grade product (effectively abusing a system designed as a driver assist), there was nothing proprietary in their solution. But that was about to change in 2016 when unfortunate Joshua Brown decapitated himself while driving Tesla on autopilot and watching a movie instead of paying attention to the road.
Soon after Brown crash, Mobileye decided to sever their dealings with Tesla in an effort to distance themselves from the irresponsible use of technology. Since apparently no other tech provider wanted to have anything to do with them at that time, Musk announced that they will be rolling out their own solution, based entirely on neural nets (Mobileye was using their own proprietary chip with a set of sophisticated algorithms, few if any based on deep learning). In addition in a bold statement Musk announced that from now on, every Tesla will have hardware ready to support Full Self Driving which will be coming soon via over the air software update. In fact people could even order the software package for a mere additional $10k. As it should be apparent by now, it was all a gigantic bluff which has been becoming more and more farcical with every passing minute. Tesla even showed a clip in 2016 of a car completing a trip without intervention [I wrote my comments about it here], but it later turned out to be a hoax, the drive was selected from multiple drives on that day. In fact only just recently additional color was added to the story behind that video in a set of interviews from ex team members. In short the 2016 video was Theranos level fake.
Next few years were littered with various missed promises, while Tesla struggled to get Autopilot 2 to the same level of reliability as their Mobileye system and even today some people prefer the Mobileye vintage solution. Tesla was supposed to demo a coast to coast autonomous drive in 2017, which hasn't happened. Later Musk stated that they could have done it but it just would have been too specifically optimized for that task and hence would not be all that useful for the development. Which of course sounds like BS particularly now when we know their 2016 video was a hoax, in fact a rumor was circulating that there were many attempts they tried and simply never could get it to work over the entire road trip.
But all shame was gone in 2019, when Musk presented at an "autonomy day". Soon it turned out that it was just a pretext to raise additional round of financing backed by a load of wishful thinking about fleets of self driving Teslas roaming around the streets making money for their owners. Coincidentally it was around the same time when Uber went public on Nasdaq, so in his typical fashion Musk took a ride on that wave of investor enthusiasm, selling a fairy tale about how Tesla will be like Uber only better, cheaper and autonomous. Back in those days Uber still had a self driving car program, which subsequently got abandoned in 2020, after a complete disaster of acquisition of fraudulent Otto company (Anthony Levandovsky who once a hero of autonomy eventually got a jail sentence, but was pardoned by Trump) but that is a whole other story which I touched on in my other posts. Guests of the Autonomy day were demoed "autonomous" driving on a set of pre-scripted roads, while Musk promised that by the end of 2020 there would be a million Tesla robo-taxis on the road, to great fanfares of the fanboys. Then end of 2020 came and nothing happened. With increasing scrutiny and doubts of even the most devoted followers, Musk had to deliver something, so he delivered FSD beta. Which is perhaps an autopilot on steroids, but is frankly a giant farce.
FSD Whack a mole
First FSD was released to "testers" in May of 2021, "testers" are put in quotes, because these people are not testers by a "strict" engineering sense. In fact not even by a "loose" engineering sense. These are mostly the devoted fanboys, preferably with media influence, ready to hide the inconvenient and blow the trumpet about how great that stuff is. But even from that highly biased source, the information available shows that FSD is comically far from being usable. Since then every new version released was a little game of whack a mole, in which the fanboys would report various dangerous situations, Tesla would (most likely) retrain their models to include those cases, only for the fanboys to find out that the problems were either still unsolved, or new problems showed up. In either case it is clear beyond any certainty to anybody who knows even an iota about AI, autonomy or robotics, that this approach is essentially doomed.
The above clip shows just 20 of the most recent FSD fails circulating on social media while this post was written, and given the bias of the "testers" it's likely just a tip of the iceberg and there would be many more by the time you read this. There is an important idea in safety critical systems, that at some point it is more important how the system fails than how often the system fails. Notably neither of these situations are even what would be considered challenging or tricky. None of these is even a failure of reasoning. These are pretty much all basic errors of perception and inadequate scene representation. The cars are turning into oncoming traffic, plowing into a barrier, endanger pedestrians, going into a divider or train tracks. These types of mistakes would be very concerning even if they happened extremely rarely, but in this case even those types of silly mistakes seem to be disturbingly frequent. Any one of such failures could result in a fatal accident. The stuff that Tesla tries to solve with great difficulty using vision is the stuff that every other serious player had long solved using a combination of LIDAR and vision (and that is a big deal because having a reliable distance information allows to completely rebalance the confidence in visual data too). Every other player in the field has a much better "situational awareness" and scene representation than Tesla (and consequently different type of "problems"), and yet not even any of these more advanced companies is ready to roll out their solution as ready for autonomy in a broad market. The most technically advanced developments such as Waymo are still operating in geofenced areas, with good weather, under strict supervision, and even those projects continuously find situations with which the cars can't deal. It's hard to express just how far behind Tesla is, and it becomes even more pathetic when one realizes how far even the Waymo's of the world are from seriously deploying anything in the wild. It's really climbing a ladder to the Moon.
While discussing other AV players, the LIDAR non LIDAR discussion needs a comment here as usual, since the argument from Elon Musk is that LIDAR is unnecessary because humans can drive with a pair of eyes. This is true on the surface, but there are a bunch of subtle details missing here:
- Humans also use ears and vestibular sense, hell even the sense of smell when driving
- Human eyes are still vastly superior to any existing camera, especially with regards to dynamic range
- Humans can articulate eyes to where they are needed, avoid obstructions
- Humans also totally can use LIDAR/Radar or any other fancy set of sensors such as night vision camera to improve the safety of their driving.
- Human can act to clear up windshield, roll down side windows to get a better look e.g. when strong sunlight is causing even those amazing eyes to have problems
- Humans have brains that can understand reality and are especially good to spacial navigation
So yes, LIDAR is not a silver bullet and in fact it is a crutch. But it's a crutch that allows other companies to get to where real problems with autonomy begin, make their cars very safe while they work to make them practical. Tesla isn't even there. Personally I don't think even Waymo is anywhere close to deploying their cars beyond just the minimal geofenced setting they are in right now and until I see a real breakthrough in AI, it's not even possible to put a date on when that might become a reality. So far the AI research isn't even asking the right questions IMHO, not to mention the answers. The way I see Waymo and other such approaches get stuck, is not with their solution being unsafe, but with their solution being too "safe" to the point of being completely impractical. These cars will be stopping and getting stuck in front of any "challenging" situation and much like even today is the case in Phoenix, their predictable safe behavior will be used by clever humans to get an advantage, in essence rendering the service impractical and unattractive. Tesla doesn't care about safety. They just want to hit a silver bullet with some magical neural net.
Comedy of mistakes
Every next version of FSD that gets rolled out to the "selected" "testers" is causing a buzz on social media, initial burst of enthusiasm is quickly followed by "it still tried to kill me here" admissions. Any time this software gets into the hands on non-fanboys, it becomes even more apparent just how ridiculous Elon Musk claims are. Recently e.g. the feature was tested by a CNN reporter (he was given access to the car by an owner who probably now regrets his decision, since it didn't turn out very well.
Tesla approach relies on a bunch of hidden assumptions and there is no evidence whatsoever that these assumptions will ever be satisfied. Let's list some of those assumptions and comment briefly:
- Driving can be solved by a deep learning network - although many have tried and perhaps in some ways deep learning is the best we have right now, nevertheless this set of techniques is far from being easy and reliable in robotic control. Imitation learning only works in simplest of conditions, perception systems are noisy and susceptible to catastrophic mistakes and there is no feedback to allow "higher reasoning" to modulate "lower perception", visual perception seems to rely on spurious correlations rather than legitimate semantic understanding. The idea that deep learning can deliver such levels of control is at best a bold hypothesis and nowhere near being proven even in much simpler robotic control settings.
- Even if deep learning was indeed sufficient to deliver the level of control necessary, it is completely unclear whether the kind of computer system that Teslas were equipped over the last several years is even anywhere close to being sufficient to run the proverbial silver bullet deep network. What if the network needs 10x neurons? What if it needs 1000x?
- Even if the network existed and the computer was sufficient it is very likely that the several cameras placed along the car might have dangerous blindspots or insufficient dynamic range etc. or that the car will still need extra sensors such as a good stereo microphone or even an artificial "nose" to detect potentially dangerous substances or malfunctions.
- It is not clear if a control system for a complex environment can exist in a form that does not continuously learn online. The system that currently Tesla has, does not learn on the car i.e. is static. If at all, it uses the fleet of cars to collect data used to train a new version of the model.
- It is not even clear if the "ultimate driver" for all conditions exists. Humans are incredibly good at driving, but they very much trade efficiency for safety on roads they explore for the first time. Particularly driving in other countries and new geographic conditions is rather difficult for us, i.e. we tend to be a lot more cautious and self aware. In regions explored and memorized we become on the other hand extremely efficient. This apparent dichotomy may not have a meaningful "average". I.e. even if a car is able to drive everywhere but not being able learn and adjust to the most frequent paths, it might be always behind humans in terms of efficiency or safety or both.
- Even if the car could drive and learn it is not clear if it would not need additional ways to actuate to be practical. I.e. much like driver can get out of the car and clear the frost from the windshield, a car might need to be able to unblock its sensors or do other tasks. What if the hood isn't shut? Will the car ask the passenger to walk outside and close it shut? Will the car know if the conditions are safe to ask the passenger for that favor? What if the passenger doesn't know how to operate the hood? What if there is a leaf stuck and obscuring the camera? What if a splash of mud covered the side cams? Will the car ask the passenger to clean those up? This is really an unexplored area of user interaction with these anticipated new devices, where for now we brush off any such issues under a carpet.
It's easy to see a massive set of assumptions which are far from being proven either way, in fact rarely even discussed. And frankly, it's not just Tesla, but pretty much everyone else in this business is staring into these and similar questions like a deer into the headlight. But Tesla unlike others is trying to claim they've got something people can use today, and that is just egregious lie that needs to be exposed and stopped.
Conclusion
Six years after bold promises were made, the evidence is overwhelming they were all a giant bluff. What we have instead of a safe self driving car, is a farcical comedy of silly mistakes, even in the best of conditions. Tesla fanboys need to enter new heights of mental gymnastics to defend this spectacle, but I don't think it will be very long until everybody realizes the emperor wears no clothes. And much like I predicted 3 years ago, I think that this ultimate realization will be the final blow to the current wave of AI enthusiasm, and will potentially cause a rather chilly AI winter.
If you found an error, highlight it and press Shift + Enter or click here to inform us.
