We Were Passengers in a Las Vegas ‘Self-Driving’ Rideshare

Autonomous vehicles have been a part of tech culture for so long that it’s hard to realize that only a handful of people have actually ridden in one. So it was with great surprise that our very first Lyft ride out of our Las Vegas hotel on Sunday night was in a “self-driving” vehicle. Lyft partnered with Irish auto-parts-company-turned-autonomous-vehicle-startup Aptiv (formerly known as Delphi) to offer CES attendees and Vegas commuters the option to ride in one of their 30 “self-driving” BMW 5 Series.

Lyft has been offering the rides since CES 2018, and basically kept the service going, logging as many as 5,000 autonomous rides in Las Vegas in the first half of 2018. A similar pilot between the two companies in Boston ended last year without fanfare (or details for why).

In the app, the experience is exactly the same as booking a regular ride, as is the price. One of the most striking initial impressions when the car pulls up is that its sensor set is extremely small, almost invisible: just two small Lidar devices on the roof, and an onboard camera facing forward … that’s it.

“No Pictures, No Video Please”

Two operators are sitting in the front seat of the nicely upholstered BMW 540i: one to drive, and one to monitor the data and send bug reports. Neither are engineers. In the middle of the console, an iPad-sized screen shows in real-time “what the car sees.” Geometric shapes of various colors seem to categorize specific objects detected by the sensors, although none of them reflects the geometry of the object that has been detected outside, save for pedestrians and other cars. At the top of the screen, the rider can see if the car is in manual or auto mode. And of course, no photography is allowed.

The driver pulls out of the hotel in manual mode, since autonomous driving is only possible on the Vegas Strip. This is likely because Aptiv wasn’t able to convince municipal authorities of the accuracy of its self-driving system on other roadways.

According to one of the operators, Lyft and Aptiv worked very closely with the city of Las Vegas, not only to create a special regulatory framework, but also to deploy a network of sensors embedded in the city infrastructure. For example, devices (he didn’t specify which kind) were installed on all traffic lights along the Strip, to allow autonomous vehicles a perfect — and perfectly safe — input.

This makes sense, considering that traffic light decisions represent the most important stakes of a ride, and that the Las Vegas Strip is basically all LED, which makes it extremely challenging for onboard cameras to read the status of traffic lights. But it also suggests that, despite its years of research in the field, Aptiv hasn’t been able to reliably solve some serious computer vision problems, and has to resort to infrastructure-embedded sensors.

Needless to say, this important caveat was left out of official marketing. It’s easy to see why, but the company being a fairly credible technology player, this is another indication of a trend we’ve seen slowly emerge last year.

Despite billions of dollars of investment, and NFL quarterback-sized salaries for senior machine learning researchers, autonomous car companies have hit a serious wall in their “all deep learning all the time” machine learning strategies, and are now starting to experiment with more hybrid models incorporating environmental feedback from an array of sensors, as well as good-old-fashioned rules-based reasoning, knowledge bases, and even probabilistic graph models. Or as the whiz kids of Silicon Valley call it: “grandpa’s AI.”

This is the right thing to do. The domain of application (or as systems theorists call it, the “state space” of all combinations and permutations of actions) is simply too large to be tackled solely by supervised machine learning, so a more contextual and probabilistic approach is required. After all, it worked pretty well for the human mind, which is able to reason on unstructured, ambiguous and incomplete information.

AI Grows Up and Comes Up

Indeed, one of the most striking things during the 5-minute trip (we since did two more trips, with the same observation) was how often the system misclassified objects near it, especially when traveling on tighter roads with a wider set of diverse objects on the outside, like hotel driveways. On several occasions (and always in hotel driveways) signs were misinterpreted as pedestrians, which would have been a problem had the car not been driven in manual mode.

At times, it seemed that adjacent cars were misplaced, and — per one of the operators, who again wasn’t an engineer — sensor input wasn’t entirely hierarchical, meaning that the system had trouble telling which objects detected — outside of cars and pedestrians — was more relevant than others.

It would be difficult to determine how different the Aptiv self-driving system is from others such as Waymo or Zoox, the two giants in the field, but reading between the lines of press releases and reports of the pilots that all companies have conducted for the past year or so, it’s clear that, while self-driving companies have accumulated hundreds of millions of miles of training data (including on “Grand Theft Auto,” since a lot of training now happens in that game environment), there’s still not enough training data to represent all possible combinations of road and weather conditions, maneuvers, and incidents, for a strict supervised machine learning approach, even using deep neural networks.

As seen above, the state space of roadways and weather conditions is way too large for a strict supervised learning approach. This also explains why both Aptiv and Waymo are testing their systems in seriously bounded environments (thereby decreasing the complexity of the state space): Aptiv only drives autonomously on the Strip, and Waymo in just a handful of neighborhoods in Phoenix.

The technology is almost there (the autonomous part of our ride was incredibly smooth, literally identical to a safe and confident human driver), but that likely won’t be enough to convince in-house counsels and municipalities to roll out the services to the public. There have already been serious accidents tied to self-driving cars (Uber and Tesla had their first deaths last year), so everybody is — rightly — very nervous.

Not to mention that it’s already infuriating parents, pedestrians, and Lyft drivers. In its Phoenix pilot, Waymo had to contend with angry pedestrians throwing stones at their vehicles, and drivers deliberately throwing the vehicles into ditches.

In Vegas, one of our operators confirmed that Lyft and Uber drivers would deliberately swerve aggressively in front of their vehicle, or go in front of it and break suddenly, either to make a point or cause an accident.

In an informal poll to other Lyft drivers over the past few days, it’s clear that autonomous vehicles are threatening some livelihoods, although most drivers have another, more meaningful career, and just drive on the side.