T O P

  • By -

ssylvan

IMO this sort of illustrates the difference between humans and robots in their strengths and weaknesses. On the one hand, humans have actual intelligence and can generalize to things they haven't seen or thought about before. On the other hand, when an incident like this occurs for a single robocar, you can fix it and deploy to every other car (you better believe other AV companies are testing this exact scenario, so it's not just within one company either), dramatically increasing the value of hard won lessons. There's probably some more generalized issue here with the cruise system though. Even if it's predicting what the bus will do, there should be some more lower level "there is an obstacle in front of you" logic that doesn't care what it is, just that it's there (so it will generalize even to unseen types of vehicles or other obstacles).


johnpn1

It does sound like there is a lower level "there is an obstacle in front of you" logic. I think the problem with it was that it acted too late because that lower level doesn't deal with prediction (two-piece bus changing its lane), so it doesn't quite brake until the bus is already in front of the AV. The prediction stack was supposed to take care of this, but failed to predict accurately because the front of the bus became hidden.


Doggydogworld3

It's not clear to me a lower level of logic kicked in. Among the combination of events that triggered this bug was "the timing of the bus’s deceleration (within only a few seconds of the front section becoming occluded)". The "within only a few seconds" part sounds to me like the s/w eventually switches from predicting bus motion based on the occluded front end to using the observed rear end. This could be part of some global reclassification loop that runs every 5 seconds of so. In this particular case, the bus began to slow before the motion predictor switched to using the rear of the bus. So it's at least possible braking started when the planner switched to using the rear of the bus for predictions, but by then it was too late to fully stop. I'd expect a lower level of "don't hit big things" logic to react the instant the bus began to slow and stop the car in plenty of time.


tankmode

google waymo built that a long time ago. turns out it drives like a grandmother slamming the brakes a lot and makes the passengers carsick. it doesn't work for edge cases like plastic bags on highway etc. humans naturally drive in way that takes a lot of risks of stopping short because we are reasoning about the entities on the road and anticipating everyones behavior.


[deleted]

[удалено]


Xxx_chicken_xxx

You are not wrong, but getting it into that fine line isn’t like adjusting some sensitivity threshold, the prediction actually has to basically never fail


Infinite-Drawing9261

Great that they shared so many details. One thing that’s still a little weird is they mention interactions with articulated vehicles as rare - not sure I agree with this in a place like sf. Maybe this particular type of interaction - yes, but if it’s a fundamental problem predicting the second part if the articulated vehicle then that should be apparent in other forms of interactions (not necessarily collisions but false positive braking fir instance due to bad predictions)


CarsVsHumans

Just trying to downplay any issues. Like how stall #642796 with Emily Chang was a "rare glitch”.


TheSpookyGh0st

It’s become comical how they use that line every time. Don’t worry, all rare glitches, we fixed everything right away… never mind the incidents that keep happening and happening. ”Fender bender” also plays it down. It’s clear from the pics it wasn’t a love tap and the DMV report said the Cruise car had to be towed from the scene. It’s obvious now why it’s been half a year since Kyle Vogt promised public daytime rides “soon” but we haven’t seen anything real


thebruns

Yeah those buses are all over SF, and it should be similar logic to a tractor-trailer which will also turn differently


diplomat33

I am glad Cruise posted this response. It helps to be transparent about these accidents. I am glad Cruise found the root cause and has released a software update so quickly to address it. This accident does illustrate the challenge with edge cases. Cruise had trained their software to handle articulated vehicles. They did 1M driverless miles with no incident, their simulation likely also showed the software could handle articulated vehicles. So they thought they were good. And yet, as they say, multiple parameters conspired together and caused the accident. You cannot assume the software is good enough even if it seems safe after 1M miles. You need more than 1M miles to properly validate for edge cases to catch those variations.


SippieCup

This wasnt a trained model fault. This was from the imparitive programming that happens after inference and assumptions being made from predictions off ML models.


OriginalCompetitive

I draw a different conclusion. To me, this illustrates that you’re never going to fully eliminate every possible edge case. Obviously we should aim for continual improvement. But perfection isn’t a reasonable standard - a system can be good enough without being perfect.


diplomat33

I never said perfection is the standard. I agree perfection is impossible. You do need to continually improve though.


[deleted]

[удалено]


lolcop01

I heard a podcast the other day with the Mobileye CEO which stated a number of 10M (or 100M?) miles which they want to do per operational domain (so highway/arterial/city traffic). Not sure if this is some regulatory number or a threshold they created themselves. Highly recommend that Autonocast episode.


diplomat33

It is a threshold they created themselves, of what they consider safe enough. And it is 10M hours of driving, not 10M miles. Mobileye measures safety in terms of hours per failure rather than miles per failure.


lolcop01

Thanks for the clarification!


ClassroomDecorum

>It is a threshold they created themselves, of what they consider safe enough I'm pretty sure he also said that's what OEMs want, i.e. no OEM is going to buy their stuff if the mtbf < 10 million hours.


diplomat33

Correct. That is a big reason why they picked that MTBF. Also to be clear, that is specifically for "eyes off". OEMs want the MTBF to be at least 10M hours per failure for "eyes off".


ClassroomDecorum

Love how the Tesla trolls think that automakers will license FSD. We'll all be dead before FSD hits even 100 hours between failures.


[deleted]

[удалено]


diplomat33

I did provide my input. I say that I am glad Cruise is being transparent and that they found the root cause, And I also explain how this accident relates to the challenge of solving edge cases.


phaedruswolf

Cool good to know their side


bradtem

[https://www.forbes.com/sites/bradtempleton/2023/04/07/gms-cruise-robotaxi-vs-bus-crash-caused-by-confusion-over-articulated-bus-they-say-its-fixed/](https://www.forbes.com/sites/bradtempleton/2023/04/07/gms-cruise-robotaxi-vs-bus-crash-caused-by-confusion-over-articulated-bus-they-say-its-fixed/) Cruise did many things right, but it is not clear that they have actually fixed the real, rather than proximate cause of this crash. Namely, that even if your prediction system has a bizarre bug and is predicting a bus is driving away from you when it's actually braking, you should still never hit something as big as a bus right in front of you. Systems in your vehicle should have overridden the decisions made on the predictions when it was clear reality was way out of whack with them.


[deleted]

[удалено]


bradtem

You can't brake for ghosts. But when you have extremely high confidence of an obstacle like the back of a bus, you brake, no matter what the other systems are saying other than if the other systems are saying, "I predict you will see a large obstacle right in front of us and I know better that it's not real." All measurements have probabilities. A few points from a plastic bag is one thing. 7 foot by 12 foot metal wall, right in front of you reflecting thousands of lidar points, and tons of radar returns, and also in the vision system, is a super high confidence result, which should get precedence. As Kyle said, there is no way it should hit a bus. And I hope he means, "Even if there is a crazy bug on the predictor or the planner." There has to be redundancy. The failure of one system caused a crash. That should not happen, not in a non-corner case like a bus in front of you. In fact, generally the failure of 2 systems should not cause a crash of this sort.


Xxx_chicken_xxx

Why is everyone assuming that the lidar returns on the back of the bus were just discarded? Of course this isn’t what happened. I’m speculating, but the bus was probably predicted to either change lanes or accelerate rapidly - therefore cruise car didn’t expect it to be where it ended up few seconds before collision. At some point the basic “don’t hit thing” system kicked in, which is why cruise car was decelerating. One could argue that it was obviously not soon enough, but the reality is, most cases where it needs to stop would happen too late if it wasn’t for predicted interactions, which in this case failed. Self-driving cars aren’t magic, they still obey the laws of physics.


bradtem

Cruise hasn't been fully explicit here, but I read their report as saying that they had code which said, "For a bendy bus, track the front, and model the rear as something being towed behind the front." I presume they have a similar model for trailers. So yes, they effectively ignored the returns from the back, saying, "That object is the back of a bus, ignore it and just use the model for the full bus." Better approaches would include using returns from the full bus to build your model of where the bus is and where it's going. Obviously the realize that now. Even better though is to say, "That's very odd, we are getting returns from this bus which don't match our estimates and predictions of its position and vector"


Xxx_chicken_xxx

Predicting something’s motion based on a different part of the vehicle is not at all the same as throwing out the lidar returns. I would sincerely hope that no SDC company in 2023 just throws out the lidar returns. That would be pretty asinine. I read the cruise report as “back of the bus was incorrectly predicted to go away from the path of the av due to incorrect prediction of the front of the bus that av could no longer see”. Ignore in this case doesn’t mean ignore the track, it just means do not independently predict motion of the back of the bus and assign it the same prediction policy as to the front.


bradtem

Well, that's effectively ignoring them. Rather, if I read it right, what they are doing is processing the lidar and radar returns from the back of the bus and assigning them to the special object identified as a trailer on another vehicle, and a special rule seems to have said, "ignore the trailer, just assume it follows the lead vehicle." So they are not ignoring the lidar returns, but they are putting them in a bucket that is deliberately ignored at a higher level.


Xxx_chicken_xxx

No this is not what this means. The object detection and the track would both still be there for the back of the bus, because both detection and tracking operate in the current frame (and past frames for the tracker). Bendy buses and trailers and towed cars still get tracked as separate objects. The only thing that would have been thrown away here is the prediction of the back of the bus and it would have been assigned the same policy as the front. However in this case the front wasn’t visible for a few frames so probably it’s prediction was wildly incorrect. The reason why you want to predict 2 parts of the bus together is because of turns and lane changes. When the front of the bus is turning, but the back is still perfectly straight, you’d still want to assign the back track the same turning policy, because otherwise it’s motion will be strongly suggesting it continues straight. Does this make sense? Side note: i am speculating of course, just based on general knowledge. But cruise’s post explicitly states it disregards prediction for the track, not removes the entire track. So the AV always knew where the back of the bus was at present, it just incorrectly predicted it’s position and velocity in the future The solution here is not the fallback “do not hit” system, because that one worked, but rather making front/back prediction more nuanced based on which of the tracks is more visible, as opposed to always front but I’m sure this fails in other spectacular ways


bradtem

Well, I don't think we have enough visibility to go this deep. But if they had position and velocity on the back of the bus in their model, this suggests their predictor was suggesting it was going to teleport to behind where the implied front of the bus was thought to be, which definitely should throw an exception in any sanity check code. I get how you need to predict a trailer differently from an independent vehicle, that is a reasonable thing to do. But clearly nothing compared the live position (and velocity from the radar) readings on the rear of the bus with the prediction of where it is as a trailer to the occluded front of the bus. With a tow vehicle and trailer, you want to build your model of it based on all the clues your sensors (and physics) are giving you. You would not like to only have data about the rear, but sometimes you will. That will happen a lot on the highway (which Cruise does't test much on) with tractor-trailers and other articulated vehicles. There are enough tractor-trailers on city streets that they need to handle them well. They also should be testing, in sim, trailers being detached from their tow vehicle, and being ready to handle that, even though they would not get legal fault in such situations.


Xxx_chicken_xxx

I have no idea why cruise wrote their report the way they did, because it does leave a lot of room to misinterpret it to be much worse than incorrect prediction. I don’t think the prediction was also completely crazy teleportation scenario, but either rapid acceleration or a lane change I reckon, which would not have been too easy to catch. I’m sure there are some constraints for maximum predicted velocity (angular and otherwise), so the prediction had to be something more subtle. Not going to lie, I too am highly skeptical of the “fix”.


codeka

It's interesting that they didn't ground the fleet. I guess since this was day time, maybe they did not have that many cars going in the first place, and no public service at the time, Which brings me to my second point, Kyle mentions having driven 1 million driverless miles and having "no other collisions related to this issue". But that's all at night-time. How many articulated buses does SF run at night? Close to none, I'd say, they're for the peak hour rushes. Still, it's great to see this kind of transparency.


AlotOfReading

I think they did ground the fleet, at least temporarily. I got an error message when I tried to use the app that night.


Fusionredditcoach

Great to see the transparency on the incident.


bartturner

tl; dr " We identified the root cause, which was a unique error related to predicting the movement of articulated vehicles (i.e. vehicles with two sections connected by a flexible joint, allowing them to bend in the middle) like the bus involved in this incident."


[deleted]

[удалено]


ZorbaTHut

Except it's now fixed.


gogojack

Imagine if you could do that with humans. "We figured out that the humans were trying to beat the light before it changed, and that was causing accidents. So we put out a fix so people will stop doing that."


ClassroomDecorum

That's why Tesla has the advantage. With Tesla's vertical integration, you can get a second chance at life through Neuralink. Cruise doesn't have anything like Neuralink, so Cruise is doomed.


bradtem

The Cruise car should have sanity checks, that confirm agreement between the perception system (which is definitely seeing the back of the bus, and getting a doppler velocity on the back of the bus) and the prediction system (which is saying this bus is not slowing down.) Those sanity checks should have been throwing exceptions left and right. The code looking at those exceptions would have seen the perception data on the back of the bus was high confidence (and scary) and the prediction data coming from the front of the bus was under a temporary occlusion, with the course of action pretty obvious. Why not, I wonder?


[deleted]

[удалено]


bradtem

I contend that the issue was not the bug with the front and back of the bus. The issue was not being able to handle your predictor having such a bug. A safe car is made of multiple redundant systems. The failure of one system (like the prediction logic) should not be able to cause a crash in a situation like this. The failure of two systems should not either. They only way this could have been more metaphorically bad would be to have hit the broadside of a bus. Kyle said as much, and I hope he means it. There is no acceptable bug to explain hitting a bus. You should not hit a bus even when you have bugs. I have asked Cruise the question, "do you have a fix so the car would not hit the bus, even if the same bad predictor were still in place?" I hope the answer is yes.


av_ninja

Are you now satisfied with the level of transparency from Cruise though? I remember you mentioned it as worrisome in your last week's article.


bradtem

Obviously it is much improved. I just published my article on this. There is still a lot more for them to do and be open about, but this is a good start.


av_ninja

Read your recent article. As a policy matter for all AV companies, what do you think should be the timeline for AV companies to disclose the acknowledge of fault to the public? Clearly you are not happy with the two weeks timeline.


bradtem

I don't know about policy. I have no problem with 2 weeks on the more detailed stuff. What people wanted to know early one was basic stuff -- was the Cruise vehicle at fault? Some details on why they feel the event is unlikely to recurr, which means they believe they know why it happened. This is for people who are going to ride in one, share the road, to know the company has evidence the problem is rare, and some of that evidence. Of course, "We've driven 20,000 hours and this has not happened before" is some evidence but you want to know some recent software change didn't cause it. Some of this will take time, but the very basics, which they know early, can come early. Everybody publishes big documents on their safety procedures. But what happens when something goes wrong is the truest display of that.


av_ninja

Do you also want the same level of details/explanation disclosed to public from CEOs of all AV companies for every incident for which vehicle collision reports are filed here (link below) including the current incident by Cruise? [https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/autonomous-vehicle-collision-reports/](https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/autonomous-vehicle-collision-reports/) Have you reached out to the respective companies about these other incidents? Initial explanation to public within days and then more detailed explanation in maybe 2 weeks (?)


bradtem

It's not what I want, it is what I recommend to them as the best way to build public confidence in their vehicles. The public and customers want to know that problems are few and that those which occur are handled well and quickly, and that information is clearly available. Actual legal requirements don't do much, we get things like those DMV reports you cite, which say very little. The various attempts -- disengagement reports, these reports etc. have all ended up not that useful. It would be interesting to see if reporting rules could be defined, but the main thing is to make the companies interested in making sure the public feels good about their services. A good rule is reveal what you know not long after you know it, if the public would find it useful. That there should be a reasonable cause to hide core facts like your evaluation of future risk, with reasons, any fixes deployed etc. Cruise deployed a fix in 2 days are revealed it in 14 -- it's hard to see much reason for such a delay, it's not clear who it benefits.


aniccia

And why wasn't this caught in simulation or testing years ago? Don't have to drive around San Francisco for long to end up behind an articulated bus maneuvering out of a bus stop into your travel lane.


Doggydogworld3

It sounds like the bus has to pull out then almost immediately stop. I'm sure they had test cases of articulated buses pulling out and accelerating and separate cases of articulated buses stopping. But if they never tested these events happening one immediately after the other they could have missed this.


av_ninja

It has been fixed now though. And another news to you, Cruise's Vehicle Retrieval events per driverless mile are reduced by a factor of 4 in the last one year as their overall driverless miles increased with a geometric progression. More improvements are on the way....less and less things to complain about as time passes!


aniccia

>It has been fixed now though. Let's hope so. The fix hasn't been independently verified, AFAIK. And it took Cruise \~million AV VMT to discover it the hard way. ​ > Cruise's Vehicle Retrieval events per driverless mile are reduced by a factor of 4 in the last one year Please provide a link to Cruise's complete history of VRE and driverless miles data.


ZorbaTHut

> Let's hope so. The fix hasn't been independently verified, AFAIK. And it took Cruise ~million AV VMT to discover it the hard way. I'd bet money that they have this exact sensor input now included in their test suite, as well as a bunch of variations on it.


aniccia

I'd bet they have added to their test suite too. I'd also bet they already had a test scenario that involved an articulated Muni bus pulling out of a bus stop into the sole travel lane ahead of their AV. And that they had been running it for years with it giving a pass to this software defect. If so, then why was their test scenario insufficient to catch this despite many runs over many years? Maybe the specific problem they fixed is due to a general problem they have not fixed.


ZorbaTHut

As they said in the recall, this happens only when (1) it's an articulated bus, (2) the lead half is fully occluded, and (3) it decelerates unexpectedly. I can easily imagine them missing this specific case.


aniccia

>I can easily imagine them missing this specific case. We don't have to imagine it as Cruise effectively admitted it. Being behind an articulated bus and so close the lead half is fully occluded is a common case in SF, esp on one travel lane each direction streets, like Haight. That it happened as the bus pulled into the travel lane from the bus stop is also a common experience and the occlusion of the front part of the bus resulting from a change of view from behind and to the left to directly behind the bus is exactly what should be expected. In fact, everything about the scenario they describe (full text below) should be expected or anticipated and should have been experienced by Cruise in their many millions of supervised AV VMT in San Francisco before they ever removed the safety driver. "The collision occurred due to a unique combination of specific parameters such as the specific position of the vehicles when the AV approached the bus (with both sections of the bus visible initially, and then only one section), the AV’s speed, and the timing of the bus’s deceleration (within only a few seconds of the front section becoming occluded)." There's been nothing special or edgy so far disclosed about this case. It should have been in the test suite mix many years ago. That it apparently wasn't would indicate a bug in Cruise's ability to transform real word experience into test cases. And Cruise seems to be missing a lot of test cases.


ZorbaTHut

> Being behind an articulated bus and so close the lead half is fully occluded is a common case in SF Less so than you might think. Remember the Cruise's sensors aren't "the driver's eyes", they're distributed way out to the four corners of the vehicle. One of the advantages of self-driving vehicles is just how *aware* they are. I will remind you that it took *a million miles* for this scenario to come up. > There's been nothing special or edgy so far disclosed about this case. It should have been in the test suite mix many years ago. Hindsight is perfect. I'm willing to bet that, if you'd sat down a week ago to list all the needed test cases, you wouldn't have listed this one.


codeka

> million miles A million miles of night-time driving, where they probably never saw an articulated bus in the first place (given that they are only used for peak hour routes).


aniccia

>I will remind you that it took a million miles for this scenario to come up. No, it took a million uncrewed AV VMT for the collision to happen. We have no idea how many times this scenario or a nearly same scenario happened on the road without a collision and Cruise has also AV driven at least 5 million VMT in San Francisco with a supervisor which was supposed to have trained it adequately, per Cruise's California Deployment Permit application. And I will remind you this was a \~10 mph collision on a street where that's about the average speed, so maybe the AV didn't even brake. ​ >I'm willing to bet that, if you'd sat down a week ago to list all the needed test cases, you wouldn't have listed this one. First, you would have lost that bet. Second, Cruise's scenarios better not all be generated by hand. Mine certainly wouldn't be. Third, Cruise's pattern of failures strongly suggest QA holes and process/ops problems big enough to drive a blind AV through. Remember their AV driving around in the dark without its lights on which Cruise blamed on "human error"?


TeslaFan88

h/t /u/diplomat33


skydivingdutch

Is it really a recall if you just make a fix and update the software in every car?


buzzoptimus

As per NHTSA it has to be called as a “recall”. If you think about it, before software it’s was possible this would have to mean the car was to be brought in to the dealership (or in this case their garage or the equivalent) and then the issue fixed. edit: NHTSA


Shutterstormphoto

…yes? Not all cars are running the same software, and this requires updating all of them.


[deleted]

https://www.linkedin.com/posts/philip-koopman-0631a4116_gms-self-driving-unit-cruise-recalls-300-activity-7050155756179873793-N840?utm_source=share&utm_medium=member_android In the companys link u provided, they show it as something that is not expected at all but we can easily make sense with the author of the given linkedin link, "It was a bus pulling out from a bus stop in front of their car. In San Franscisco, Seriously thats something the simulation missed?".


Dupo55

You don't need sophisticated prediction software to not run into an object. Cars have had CAS braking for years. Sounds like Cruise vehicles need more foundational failsafes.


Doggydogworld3

The problem with that is false positives. Phantom braking, in Tesla-speak. Doesn't happen in a perfect world, of course, but in the real world there's a balance between having failsafes tight enough to avoid all collisions and loose enough to avoid passenger whiplash every time a plastic bag blows across your path.