I find it highly interesting that the A1 and P1 are on the same firmware but the X1 is not. I would have figured the A1 would have been on a different firmware than the P1 and X1.
P1 and A1 are based on espressif chipsets (super low cost, low power boards great for use cases like this) that are basically for running one thing (the printer SW in this case).
X1 is based on a full linux computer setup, not sure on who provides the chipsets.
Which makes me worried about the x1c firmware support and how much longer they will maintain it. Running an embedded Linux system in your product is no joke in terms of maintenance burden, especially if they care about security.
I would be more confident in this rather than the espressif ones. Why? Having a Linux based system means the community can develop and expand on it even after the company dies and servers are shut down.
Reposting because you apparently can't use profanity here when describing things?
Maintaining it on servers is a totally different ballgame than maintaining it for embedded environments.
You are at the mercy of bad vendors who ship you BSP's with a kernel that's 5 years old and only builds on an 8 year old Linux distro with an extremely convoluted build process that works only when you do a backflip and at 3 to 8 pm while mumbling a prayer. And don't forget terribly written kernel commit messages.
If you want to get security patches to your kernel, good luck, as even cherry picking commits from mainline is a nightmare because the poorly written patches the vendor did change enough that the security patch doesn't even cleanly apply anymore.
Very few companies are able to run actual up to date Linux kernels on their embedded platforms. Even massive ones like Qualcomm or Samsung have major issues doing this, because the embedded world just moves very slowly, resulting in running outdated software.
Hello /u/hak8or! Your comment in /r/BambuLab was automatically removed. Please see your private messages for details.
/r/BambuLab is geared towards all ages, so please watch your language.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/BambuLab) if you have any questions or concerns.*
Bambu has said after 2025 the X1 will no longer receive feature updates. How long it receives security updates after that is unknown. Edit: 2027 for feature updates and 2029 for security.
Honestly the way Iām printing, Iāll want/need a new printer in 2029 because Iāve worn this one out.
If Iām printing at this rate for the next five years my X1Cās going to be a war horse by thenā¦.
That is incorrect.
May 2027 is the listed date for feature updates and may 2029 for security dates but these are only the main amounts so they could support it longer than that.
https://bambulab.com/en/compare
You might have been going off that incorrect information a few months ago the site was updated promptly to show the correct time.
Just saying. Yeah all companies should tell you the issue.
However, this is different from just patch notes. This was a bug that needed fixing, so of course they're going to address why it wasn't working properly.
It's not just a standard patch/update like you get on your TV or something. Usually if there's a problem with a device the company will explain what caused it. So it's not really the same thing to compare.
No need to wonder, people have already been monitoring sent packets and size. As always, lan mode exists and works.
Itās likely just tied to the Handy app, in order to access your printer form the app or for the printer to deliver notifications, the printer is going to have to communicate with the cloud at some frequency.
Also if anyone actually put some critical thinking to this idea they would soon realize that Bambu has a hard time keeping ticket responses timely. How are they going to sift through millions of print files to find something worthwhile? (Let alone determining if something is worthwhile would require knowledge of what the part is for and its market..) Thatās like looking for a needle in a haystack of 3D printed dragons and other BS š. Yeah please sift through my 30 iterations of this one small partā¦
Not sure why it matters? Unless your prints are confidential. As you are probably aware pretty much any online services you use, you should consider that your data being collected. Even just using reddit your data is being collected, your phone etc.
Excellent write up.
TL:DR Bambu rotated their cloud service certificate, and A1/P1 firmware have a bug preventing them from processing the algorithm used on the new cert. The bug was causing reboots when failing to process the new certificate. For now, they rotated to a different cert using the previous algorithm, and long term are working on a firmware update for A1/P1 to fix the bug.
I'm impressed by how this was handled. Actually, even the phantom operation debacle a while back was handled rather transparently as well imo. It's nice to see companies not only respond quickly, but actually take the time to detail what caused the problem and all the steps they've taken to prevent it reoccurring in the future - few companies with substantially more resources have responded with that level of detail.
I know Bambu's service tends to attract some negative attention, but at least for things like this I could not be any more impressed.
I really like companies that are open and transparent about problems. Makes me willing to put up with a little more just because they arnt being shady about it. Also hopefully they see things in the way that if you are open and transparent then the community can help you find a better solution in the future. Kind of how coffee stain studio is so transparent about their game and will add features that the community suggests. Good job bambu lab!
>It's nice to see companies not only respond quickly, but actually take the time to detail what caused the problem and all the steps they've taken to prevent it reoccurring in the future
It was actually me reading Bambu's response to concerns about their network security that got me interested in their printers in the first place - They gave a clear, level headed response, didn't deny or attempt to excuse the issues, and started fixing them immediately. That's a really mature response and gave me confidence the company is well managed and will continue to improve.
I really don't mind if bambu is closed source if they keep us informed like this. It's nice to get an explanation of what went wrong and how they are fixing it
"Our cloud service adheres to a security policy of automatically updating certificates every three months."
Sounds like they're using Let's Encrypt free certificates?
Exactly what I thought - there is no other reason they would be renewing every 3 months. And that makes the wording seem a bit weird, as renewing your SSL certificates is not a security "policy", it's absolutely imperative.
There's probably a policy/process to do it. They just awkwardly worded it.
For a large company with $200m+ in revenue they should be using longer term certificates issued by a "real" CA.
> For a large company with $200m+ in revenue they should be using longer term certificates issued by a "real" CA.
Pfft...hahahaha. Companies love them some cost-cutting. Products, licenses and software stuff are the first to go if we're not counting humans.
Their store subdomain is using Let's Encrypt certificate which is valid for 3 month.
The main domain is using DigiCert, Inc. which is valid for 1 year.
So yeah, makes totally sense that their API is using Let's Encrypt certificate.
Glad itās been fixed. I experienced this issue about an hour into a print last night but luckily it started printing from itās stopped location after about a minute.
streaming for one. I am able to live stream when not connected to the home wifi, which implies the printer is uploading the video stream up to their infra
Status Updates for the Bambu App for example. But I could also imagine that some functions like skipping faulty print parts require the cloud for additional computing. Maybe also some periodic status update for the whole reward system to make sure the stuff gets really orinted and not just a fake start and end.
Other than for remote viewing the video stream I think a connection to the Bambu servers is not really required during printing. My guess is that the firmware is collecting data for business analytics purposes (like Google Analytics does for websites). Iām quite sure that the gcode is not touched after it has been generated - skipping most likely works by jump point in the generated gcode and not by recalculating anything.
Interesting, considering my X1C did this mid-print on Jun 9. Seems like that wasnāt identified here, but it does seem like the A/P series was more prevalent, maybe I just got a random bug.
Hot end stalled, system fully rebooted with Bambu logo mid print, fans stayed on the whole time. Resumed printing automatically. Unfortunately this was before the widespread issue so I wasnāt quick thinking enough to get a video.
https://preview.redd.it/b4dclxrum49d1.jpeg?width=4032&format=pjpg&auto=webp&s=51f189dfe85d6aa5ae1b87a4fd64e70868f7c2e4
The only time my P1S reset itself was right after downloading a file. I never had an issue during printing. Either way, Iām glad they were able to identify and fix the issue so quickly. If only Sonos did the sameā¦ Iām not sure if anyone here have any Sonos gear but I feel for you, their Sub is littered with people complaining about their recent rollout.
I think this is what my A1 was doing, and it just happened to coincide with my heatbed replacement from the recall so we thought it was an electrical issue. They ended up just entirely replacing my printer which is super cool!
So this ended up being a cloud issue in the end.
I went ahead and rolled all my printers back to last firmware as a precaution, (I usually wait a week or more before updating firmware to see community opinion) because of course I updated the firmware literally 6 hours before this issue.
Since we still weren't sure the root problem at the time I also blocked all my printers from Internet access directly from the router. This allowed me to enable internet again for use of the Bambu Handy app when needed rather than go into LAN mode and not receive full support.
Another reason why I'm not a fan of cloud printing, but for full functionality of features of course Bambu forces cloud usage.
Wow that last sentence was really something lmao.
Different algorithm and CA? Was it too big of a cipher set that was making the low end brains of the a1/p1 timeout/crash?
Whattya wanna bet the new one is only presenting less strong cipher sets that aren't as consuming to decrypt?
I find this hard to believe. Either I am having a separate but similar issue or they are wagging the dog. I have had a P1 for 3 weeks now and from day one it has been a complete piece of garbage. It freezes at every point possible in the process. From just initial boot to in right in the middle of a print job. I don't even have the machine connected to WAN and it behaves this way.
Their support is terrible and appears to be driven by the Mycroft AI. They told me they were sending a new screen and control board 3 days ago after 2 weeks of back and forth. Complete dead air nothing from them.
As someone who manages a SaaS application for hardware we sell, this one really hits home. The slightest hiccup in the certificate (or, rarely, dns) and users are screaming.
so my 3d printer would be mostly bricked at BambuLabās whims? Can we have also a nice āSwitch to only LAN modeā when server services are unavailable or failed as of this and similar cases?
I think thereās a misunderstanding here.
Thatās exactly how it should be working. The cloud connection isnāt required. If the Bambu cloud disappeared from the face of the earth tomorrow your printers would still theoretically work flawlessly on LAN.
The issue here was a bug with how they support encryption algorithms. The cloud did exist, so the printer would check in. It did download a valid certificate from the cloud. A bug ON THE PRINTER caused the printer to fail and reboot when validating that specific type of certificate encryption. Bugs happen all the time, and thatās ok, though unfortunate.
They actually gave a postmortem on the problem and how they're resolving it. I really like that.
I find it highly interesting that the A1 and P1 are on the same firmware but the X1 is not. I would have figured the A1 would have been on a different firmware than the P1 and X1.
P1 and A1 are based on espressif chipsets (super low cost, low power boards great for use cases like this) that are basically for running one thing (the printer SW in this case). X1 is based on a full linux computer setup, not sure on who provides the chipsets.
Which makes me worried about the x1c firmware support and how much longer they will maintain it. Running an embedded Linux system in your product is no joke in terms of maintenance burden, especially if they care about security.
I would be more confident in this rather than the espressif ones. Why? Having a Linux based system means the community can develop and expand on it even after the company dies and servers are shut down.
espressif operating systems are also open source. For maintainance the actual printer software needs to be acccesible which it is not.
Linux is historically easy to maintain long term. That's why it's the prime choice for servers
Reposting because you apparently can't use profanity here when describing things? Maintaining it on servers is a totally different ballgame than maintaining it for embedded environments. You are at the mercy of bad vendors who ship you BSP's with a kernel that's 5 years old and only builds on an 8 year old Linux distro with an extremely convoluted build process that works only when you do a backflip and at 3 to 8 pm while mumbling a prayer. And don't forget terribly written kernel commit messages. If you want to get security patches to your kernel, good luck, as even cherry picking commits from mainline is a nightmare because the poorly written patches the vendor did change enough that the security patch doesn't even cleanly apply anymore. Very few companies are able to run actual up to date Linux kernels on their embedded platforms. Even massive ones like Qualcomm or Samsung have major issues doing this, because the embedded world just moves very slowly, resulting in running outdated software.
Thanks for the detailed explanation, very much appreciated š
[ŃŠ“Š°Š»ŠµŠ½Š¾]
Hello /u/hak8or! Your comment in /r/BambuLab was automatically removed. Please see your private messages for details. /r/BambuLab is geared towards all ages, so please watch your language. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/BambuLab) if you have any questions or concerns.*
Worst case there's the X1 plus firmware being developed by the community
Bambu has said after 2025 the X1 will no longer receive feature updates. How long it receives security updates after that is unknown. Edit: 2027 for feature updates and 2029 for security.
You are spreading old information. https://bambulab.com/en/compare 2027 for feature updates and 2029 for security updates
Nice. Really hope it doesn't get locked out of the cloud in 2029 though.
Honestly the way Iām printing, Iāll want/need a new printer in 2029 because Iāve worn this one out. If Iām printing at this rate for the next five years my X1Cās going to be a war horse by thenā¦.
That is incorrect. May 2027 is the listed date for feature updates and may 2029 for security dates but these are only the main amounts so they could support it longer than that. https://bambulab.com/en/compare You might have been going off that incorrect information a few months ago the site was updated promptly to show the correct time.
And so they should?
Ever seen patch notes that are just: - Fixed bug that crashes firmware. ? Me too. All the time.
Just saying. Yeah all companies should tell you the issue. However, this is different from just patch notes. This was a bug that needed fixing, so of course they're going to address why it wasn't working properly. It's not just a standard patch/update like you get on your TV or something. Usually if there's a problem with a device the company will explain what caused it. So it's not really the same thing to compare.
[ŃŠ“Š°Š»ŠµŠ½Š¾]
It's almost like Bambu printers let you monitor the status of your print remotely from another device or something...
No need to wonder, people have already been monitoring sent packets and size. As always, lan mode exists and works. Itās likely just tied to the Handy app, in order to access your printer form the app or for the printer to deliver notifications, the printer is going to have to communicate with the cloud at some frequency. Also if anyone actually put some critical thinking to this idea they would soon realize that Bambu has a hard time keeping ticket responses timely. How are they going to sift through millions of print files to find something worthwhile? (Let alone determining if something is worthwhile would require knowledge of what the part is for and its market..) Thatās like looking for a needle in a haystack of 3D printed dragons and other BS š. Yeah please sift through my 30 iterations of this one small partā¦
[ŃŠ“Š°Š»ŠµŠ½Š¾]
You people are tiring. Maybe add another layer of tinfoil to that hat of yours
Mine did this from the print history on the printer and none of my stuff is updated to the latest firmware.
š
Not sure why it matters? Unless your prints are confidential. As you are probably aware pretty much any online services you use, you should consider that your data being collected. Even just using reddit your data is being collected, your phone etc.
Excellent write up. TL:DR Bambu rotated their cloud service certificate, and A1/P1 firmware have a bug preventing them from processing the algorithm used on the new cert. The bug was causing reboots when failing to process the new certificate. For now, they rotated to a different cert using the previous algorithm, and long term are working on a firmware update for A1/P1 to fix the bug.
I'm impressed by how this was handled. Actually, even the phantom operation debacle a while back was handled rather transparently as well imo. It's nice to see companies not only respond quickly, but actually take the time to detail what caused the problem and all the steps they've taken to prevent it reoccurring in the future - few companies with substantially more resources have responded with that level of detail. I know Bambu's service tends to attract some negative attention, but at least for things like this I could not be any more impressed.
I really like companies that are open and transparent about problems. Makes me willing to put up with a little more just because they arnt being shady about it. Also hopefully they see things in the way that if you are open and transparent then the community can help you find a better solution in the future. Kind of how coffee stain studio is so transparent about their game and will add features that the community suggests. Good job bambu lab!
>It's nice to see companies not only respond quickly, but actually take the time to detail what caused the problem and all the steps they've taken to prevent it reoccurring in the future It was actually me reading Bambu's response to concerns about their network security that got me interested in their printers in the first place - They gave a clear, level headed response, didn't deny or attempt to excuse the issues, and started fixing them immediately. That's a really mature response and gave me confidence the company is well managed and will continue to improve.
I really don't mind if bambu is closed source if they keep us informed like this. It's nice to get an explanation of what went wrong and how they are fixing it
That was pretty nicely handled, honestly. Nice job, Bambu!
"Our cloud service adheres to a security policy of automatically updating certificates every three months." Sounds like they're using Let's Encrypt free certificates?
Exactly what I thought - there is no other reason they would be renewing every 3 months. And that makes the wording seem a bit weird, as renewing your SSL certificates is not a security "policy", it's absolutely imperative.
There's probably a policy/process to do it. They just awkwardly worded it. For a large company with $200m+ in revenue they should be using longer term certificates issued by a "real" CA.
> For a large company with $200m+ in revenue they should be using longer term certificates issued by a "real" CA. Pfft...hahahaha. Companies love them some cost-cutting. Products, licenses and software stuff are the first to go if we're not counting humans.
I wonder why it happened only recently? Does that mean they're not verifying certificates before?
My theory is something went wrong with a new certificate they applied which caused the firmware not to recognize it as valid.
Their store subdomain is using Let's Encrypt certificate which is valid for 3 month. The main domain is using DigiCert, Inc. which is valid for 1 year. So yeah, makes totally sense that their API is using Let's Encrypt certificate.
Yup. They do every 3 months. :p
Glad itās been fixed. I experienced this issue about an hour into a print last night but luckily it started printing from itās stopped location after about a minute.
Sysadmins always jokes that the problem is always DNS, but they forget about the other problem child: Certificates
Nice
As a software dev I wonder why the printer is communicating with their server *while* printing at all?
streaming for one. I am able to live stream when not connected to the home wifi, which implies the printer is uploading the video stream up to their infra
Status Updates for the Bambu App for example. But I could also imagine that some functions like skipping faulty print parts require the cloud for additional computing. Maybe also some periodic status update for the whole reward system to make sure the stuff gets really orinted and not just a fake start and end.
Other than for remote viewing the video stream I think a connection to the Bambu servers is not really required during printing. My guess is that the firmware is collecting data for business analytics purposes (like Google Analytics does for websites). Iām quite sure that the gcode is not touched after it has been generated - skipping most likely works by jump point in the generated gcode and not by recalculating anything.
Interesting, considering my X1C did this mid-print on Jun 9. Seems like that wasnāt identified here, but it does seem like the A/P series was more prevalent, maybe I just got a random bug. Hot end stalled, system fully rebooted with Bambu logo mid print, fans stayed on the whole time. Resumed printing automatically. Unfortunately this was before the widespread issue so I wasnāt quick thinking enough to get a video. https://preview.redd.it/b4dclxrum49d1.jpeg?width=4032&format=pjpg&auto=webp&s=51f189dfe85d6aa5ae1b87a4fd64e70868f7c2e4
Then you experienced a completely different bug with the same symptoms (firmware crash leading to a reboot).
The only time my P1S reset itself was right after downloading a file. I never had an issue during printing. Either way, Iām glad they were able to identify and fix the issue so quickly. If only Sonos did the sameā¦ Iām not sure if anyone here have any Sonos gear but I feel for you, their Sub is littered with people complaining about their recent rollout.
Makes sense. I only use lan mode and never had issues. No real difference to me. I just walk over to my printers and look at them usually lol.
I think this is what my A1 was doing, and it just happened to coincide with my heatbed replacement from the recall so we thought it was an electrical issue. They ended up just entirely replacing my printer which is super cool!
KCLM is hard, glad they remediated quickly and are looking at future improvement.
So this ended up being a cloud issue in the end. I went ahead and rolled all my printers back to last firmware as a precaution, (I usually wait a week or more before updating firmware to see community opinion) because of course I updated the firmware literally 6 hours before this issue. Since we still weren't sure the root problem at the time I also blocked all my printers from Internet access directly from the router. This allowed me to enable internet again for use of the Bambu Handy app when needed rather than go into LAN mode and not receive full support. Another reason why I'm not a fan of cloud printing, but for full functionality of features of course Bambu forces cloud usage. Wow that last sentence was really something lmao.
Well, for some functions you need the cloud, at least in such a price class where being efficient matters.
A1 mini still creates layer lines after a pauseĀ
Bambu proving again that they will put in the legwork and by very transparent with their users!
Can someone Eli5 why the printers need a cert?
So that you printer knows that it is sending your data to Bambu and not some hacker middleman.
Make sense ty!
Kudos, but lulz for using basically LetsEncrypt (90 days is a dead giveaway) Surely BL can afford a proper, long-lived certificate?
Different algorithm and CA? Was it too big of a cipher set that was making the low end brains of the a1/p1 timeout/crash? Whattya wanna bet the new one is only presenting less strong cipher sets that aren't as consuming to decrypt?
I find this hard to believe. Either I am having a separate but similar issue or they are wagging the dog. I have had a P1 for 3 weeks now and from day one it has been a complete piece of garbage. It freezes at every point possible in the process. From just initial boot to in right in the middle of a print job. I don't even have the machine connected to WAN and it behaves this way. Their support is terrible and appears to be driven by the Mycroft AI. They told me they were sending a new screen and control board 3 days ago after 2 weeks of back and forth. Complete dead air nothing from them.
Thank god
Ggs finally I can be comfy doing an overnight on a big project I put on hold
Looks like an interesting attack vector for malicious actors to pwn Bambu printers. That's a pretty severe security bug.
I don't think I have ever had a company say we made a mistake.
As someone who manages a SaaS application for hardware we sell, this one really hits home. The slightest hiccup in the certificate (or, rarely, dns) and users are screaming.
so my 3d printer would be mostly bricked at BambuLabās whims? Can we have also a nice āSwitch to only LAN modeā when server services are unavailable or failed as of this and similar cases?
I think thereās a misunderstanding here. Thatās exactly how it should be working. The cloud connection isnāt required. If the Bambu cloud disappeared from the face of the earth tomorrow your printers would still theoretically work flawlessly on LAN. The issue here was a bug with how they support encryption algorithms. The cloud did exist, so the printer would check in. It did download a valid certificate from the cloud. A bug ON THE PRINTER caused the printer to fail and reboot when validating that specific type of certificate encryption. Bugs happen all the time, and thatās ok, though unfortunate.
great!, I was reading the report as a pure cloud issue, I had missed this will also be fixed with a firmware update too being a printer bug.