February 3, 2017
The California Department of Motor Vehicles released its annual report from the 11 companies with state permits to test autonomous vehicles as of end of 2015, and they have all made rapid progress. The report, which covers December 2015 to November 2016, recounts how many times humans had to take over driving tasks. Google and General Motors in particular have excelled, with cars that can drive hundreds of miles at a stretch without a hitch. Nissan has gone from needing intervention every 14 miles — to assistance needed after 247 miles.
Wired reports that Google’s Waymo, which outpaced its rivals, drove “636,000 miles with just 124 disengagements, a 19 percent drop from 2015.” Hardware or software discrepancies triggered the need for intervention.
GM’s Cruise improved from five miles in June 2015 to 400 in June 2016. Meanwhile, Delphi’s car had trouble changing lanes in heavy traffic, and Ford’s two self-driving cars “only drive on the highway, during the day, with fine weather and road conditions,” explaining why humans only assisted three times in 590 miles.
According to Bloomberg, Tesla, whose cars have a rate of 330 disengagements per 1,000 miles, says it’s gathered “more than 300 million miles of data from cars driven by customers with Autopilot engaged, giving it a treasure trove of real world data.” BMW’s self-driving car had one disengagement in 638 miles driven, on U.S. Highway 101 with dry roads and clear weather; Mercedes-Benz reported 336 disengagements in 673 miles, entirely on urban streets.
Still, not everyone thinks rates of “disengagement” are a valuable metric. “[They] are not a scientific measure of the complexity and operating characteristics of these vehicles,” said MIT’s Bryan Reimer. “They’re just one very interesting data point.” Each disengagement involves “all sorts of variables,” but, still, provides “a picture of how each autonomous program is developing.”
Missing factors include weather, information about human drivers and where the disengagement occurred. Raj Rajkumar, who studies autonomous technology at Carnegie Mellon University, believes “these reports would more helpful if they were more rigorous,” including qualitative comments, consistent formatting and defined terms.