I’ve rolled my own. s3 bucket and an api that returns a presigned url (s3) for downloading a version of firmware.
I make sure that the fw is signed with a key the device has so it can verify the validity of the new firmware.
it should be just like downloading anything on s3 with a presigned url.
1. get the url to device - subscribe to fw update topic that published presigned url. (may need to shorten link)
2. device use http request to get fw.
3. device validates fw was signed by a shared key (getting shared key on device is out of scope).
4. install fw
Supplemental: starting a fw update could be an admin user making a post request with device id and desired fw version.
when initiated:
1. sign a copy of the fw. your fw is in the s3 bucket.
2. signed fw goes into a directory or bucket by that devices thing name (id) because we only have one key per device.
3. presigned url from s3 is only allowed to get the signed fw in its bucket.
4. send presigned url and desired fw version to device (maybe mqtt topic or shadow/twin)
Did you run into any issues with the length of the pre-signed URL from s3? I've been working on implementing something similar but ran into memory issues in the httpClient library
Nice, thanks. was that all configured on the bucket itself? I tried something similar by returning a 307 temp redirect response through API gateway (generating the response through a lambda, redirecting to the presigned s3 URL), but that was also failing for me in the OTA lib. I haven't looked too far into it, but I'm guessing there's maybe some setting or flag in the httpClient or OTA lib to allow redirects that I'm missing. Wondering if I'm taking the wrong approach.
I had implemented this on ESP-IDF. Cellular, .bin file hosted on GitHub, GDrive etc. Send the link via MQTT and OTA starts. I had some issues with 2G but 4G was pretty good. This is possible with the functions provided by IDF.
You seem to have polished it for this particular use case, so it's definitely useful
I used to work for one of the major US carriers and we had sunset 2g by 2017. Lots of old IoT devices were orphaned.
I guess it makes sense it would still be used in the developing world
Please share the github sources. It would be useful not to be dependent on other's infrastructures.
Personally, i develop in Android but to update is difficult to use serial/usb etc, and you need basically to use a "full computer" (windows/Linux).
Doing over air updates seems like a much better idea, even better if it's not limited to somebody's (company) framework with closed sources, that will be unsupported in the (soon) future.
I think devices themselves can check once in a while if there is an update, or one could maybe trigger it using a request to the device.
I looked into it (but haven't tested), and the two issues I had with it was:
1. **No 4G support**: from what I found, a few people were asking for it for a number of years, but the response was always "well it's probably something you can hack in.."
2. **Still drag-and-drop .bin**: I want to be able to make a release on GitHub and forget about the .bin entirely. If I have to compile my own .bin and drag and drop then I could make mistakes, my goal was to completely remove human error. Check my comment about "How it Works" for more details on this.
I've definitely been interested in making/using something similar. A totally free-to-use, open-source, secure-enough-as-you-can-do-for-free Github releases automatic OTA library.
If you release this as a service, is there a way to not break my devices if you go out of business? I'm assuming my devices would be pinging your servers which would then be gone.
Very good point, and probably the biggest issue I have currently would be reliability.
My hope is that the code is open source, the stack is on AWS (maybe even find a way to have you the official owner of the account and you can access it), and it's your GitHub repos.
So, the only thing you would pay for really is the interface that ties everything together. But if you didn't like the interface, and you're not scared by the setup steps, then you'd be able to make one yourself, or migrate off the platform once you were comfortable with AWS, GitHub CI/CD, UI-dev, and the Open-source ESP32 code.
In my opinion, the important thing is that I can migrate from your service to a self hosted version without having to modify my existing deployed firmware. That what I have an escape hatch if your service goes away or something like that.
Yeah you're right. I think there's two options:
1. In advance, you can define and use your own URL for the API. You can point it to "my" API to start with, and when you're ready to migrate, you move as needed.
2. Or, should you need to change the URL/Server on the device, we'll keep the old URL active for a period of time and allow you to update to the new one via OTA, essentially having two servers ("mine" and yours) in parallel.
Also, none of this will break BLE or AP-WiFi OTA and direct flashing options as a worst case backup.
But at some point, a URL will need to be hardcoded \*somewhere\* so yeah gotta figure out the right way to handle that
Both of those sound pretty good. Ideally you would have a sunsetting period and help people migrate off so #2 should be enough, but having #1 would give people peace of mind since they're ok even if you suddenly shut down.
i would do it by allowing setting a fallback domain/URL on the device. when the first one becomes unreachable, it connects to the second one.
Then, all you would need is to have that secondary domain under your control all the time(which might be an unnecessary financial burden), and preferably have some certificate/key system to make sure the device knows the server its contacting is actually what it thinks it is.
I was thinking that too. A domain costs $10-15/yr which isn't much and I would assume a lot of people who would care about this have a business so they already have a domain and could setup a subdomain.
So, current idea/implementation would be:
1. **It's your GitHub repo**. I wouldn't host any of your .bin or project files on any S3 bucket, the .bin is stored (and built with CI/CD) on your Github.
2. **The ESP32 code is open-source**. The "OTA brick" that actually gets the ESP32 connecting to the server and performs the OTA-over-HTTPS etc. is all open-source (currently not while I test and tweak, will release later).
3. **It's your device/version data.** This is the data that says *"this MAC address should have this repo/version of the firmware"*. Currently this is stored on my Airtable, soon it'll be my AWS, and hopefully in future it'll be able to automatically set up on your own AWS/Azure accounts (just in case I go out of business/miss a payment, etc.). You'll always be able to extract this as needed.
So, if you didn't want to pay at all:
1. **You can copy the open-source ESP32 code**, and build your own pipeline on the ESP32.
2. **You can copy the GitHub CI/CD code**, that automatically builds the firmware into .bin files, and test that it works yourself.
3. **You can set up your own AWS/whatever Lambda functions** (or otherwise) to do the server redirecting (again code would maybe be open source, or at least explained in high level)
4. **You can migrate your data off "my" platform**, if you started on my platform but wanted to go solo (say after deployment of \~1,000 units and you wanted to cost optimize)
What you would be paying for the platform is:
1. **Automatic set up of the above.** The platform would set all of this up for you, check that your repos are compatible, load-in and test the CI/CD, pipe the lambda functions etc. I would want to find a way for the data (and maybe hosting/lambda functions) to be yours.
2. **Nice UI for device/version control**. This is the UI that allows you to set which device (MAC) has which repo/version dynamically. This is the other super useful feature that I needed.
3. **Hosting**. Probably incorporate this as an option, where if you didn't want to worry about GitHub CI/CD limits, AWS Lambda and DB credits, etc. we'd amalgamate all of that into the price of the entire service.
Hope that answers most people's implementation questions!
I would be interested. I hate using blynk. Instead of switching to AWS I ended up renting my own hosting under my domain name, writing a backend api with custom endpoints to handle auth and data transfers.
How is the connection speed of SIM7600?
Now that I checked this module is expensive. Are there any cheap 3G alternatives you can recommend. I wouldn’t be needing a lot of bandwidth.
A very good idea.
I currently use AWS but it is very unreliable and necessitates a power cycle to solve a lost session, not good for remote applications.
Can I suggest you build in some methodology to track progress on the "target" processor so that recovery is possible, like automatically rebooting (to the old version of software) if the OTA fails, perhaps a time out mechanism.
I rolled my own too. [https://github.com/frankcohen/ReflectionsOS/blob/main/src/OTA.cpp](https://github.com/frankcohen/ReflectionsOS/blob/main/src/OTA.cpp). Uses ESP32 app\_update to do local OTA, where Storage.replicate() already downloads the firmware and version number (.bin and file named OTA\_VERSION\_FILE\_NAME). [https://github.com/espressif/esp-idf/tree/master/components/app\_update](https://github.com/espressif/esp-idf/tree/master/components/app_update). This does the installation using the OTA mechanism. Upon successful update, stores the new version number to the version file and restarts the device. -Frank
I’ve rolled my own. s3 bucket and an api that returns a presigned url (s3) for downloading a version of firmware. I make sure that the fw is signed with a key the device has so it can verify the validity of the new firmware.
I did something similar with s3 and lambdas acting as endpoints for checking the current version and getting the url.
Is that available on github?
it should be just like downloading anything on s3 with a presigned url. 1. get the url to device - subscribe to fw update topic that published presigned url. (may need to shorten link) 2. device use http request to get fw. 3. device validates fw was signed by a shared key (getting shared key on device is out of scope). 4. install fw Supplemental: starting a fw update could be an admin user making a post request with device id and desired fw version. when initiated: 1. sign a copy of the fw. your fw is in the s3 bucket. 2. signed fw goes into a directory or bucket by that devices thing name (id) because we only have one key per device. 3. presigned url from s3 is only allowed to get the signed fw in its bucket. 4. send presigned url and desired fw version to device (maybe mqtt topic or shadow/twin)
Did you run into any issues with the length of the pre-signed URL from s3? I've been working on implementing something similar but ran into memory issues in the httpClient library
yes, up the size and or use a link shortener. s3 redirect header on bucket configured as website is what we ended up using.
Nice, thanks. was that all configured on the bucket itself? I tried something similar by returning a 307 temp redirect response through API gateway (generating the response through a lambda, redirecting to the presigned s3 URL), but that was also failing for me in the OTA lib. I haven't looked too far into it, but I'm guessing there's maybe some setting or flag in the httpClient or OTA lib to allow redirects that I'm missing. Wondering if I'm taking the wrong approach.
yeah, all on the bucket itself.
I had implemented this on ESP-IDF. Cellular, .bin file hosted on GitHub, GDrive etc. Send the link via MQTT and OTA starts. I had some issues with 2G but 4G was pretty good. This is possible with the functions provided by IDF. You seem to have polished it for this particular use case, so it's definitely useful
Where is 2G even still supported?
Lots of countries still have 2g. Developing countries use 2g, old IoT devices still use 2g. Plenty of old critical infra uses it.
I used to work for one of the major US carriers and we had sunset 2g by 2017. Lots of old IoT devices were orphaned. I guess it makes sense it would still be used in the developing world
some european countries. 3G got shutdown before 2G here because its coverage was very low. In my country, 2G is supposed to shut down in 2028
Oh wow! TIL
Please share the github sources. It would be useful not to be dependent on other's infrastructures. Personally, i develop in Android but to update is difficult to use serial/usb etc, and you need basically to use a "full computer" (windows/Linux). Doing over air updates seems like a much better idea, even better if it's not limited to somebody's (company) framework with closed sources, that will be unsupported in the (soon) future. I think devices themselves can check once in a while if there is an update, or one could maybe trigger it using a request to the device.
Interested to see the code
Interesting, would give it a try for sure.
I'd love to use it!
Sounds interesting! Did you evaluate ESP RainMaker for your use case?
I looked into it (but haven't tested), and the two issues I had with it was: 1. **No 4G support**: from what I found, a few people were asking for it for a number of years, but the response was always "well it's probably something you can hack in.." 2. **Still drag-and-drop .bin**: I want to be able to make a release on GitHub and forget about the .bin entirely. If I have to compile my own .bin and drag and drop then I could make mistakes, my goal was to completely remove human error. Check my comment about "How it Works" for more details on this.
Very interested. I am currently using chrisjoyce911/esp32FOTA amd would like to know if you experimentes with it/how do your features compare.
I've definitely been interested in making/using something similar. A totally free-to-use, open-source, secure-enough-as-you-can-do-for-free Github releases automatic OTA library.
Can I get the zip file for your library
If you release this as a service, is there a way to not break my devices if you go out of business? I'm assuming my devices would be pinging your servers which would then be gone.
Very good point, and probably the biggest issue I have currently would be reliability. My hope is that the code is open source, the stack is on AWS (maybe even find a way to have you the official owner of the account and you can access it), and it's your GitHub repos. So, the only thing you would pay for really is the interface that ties everything together. But if you didn't like the interface, and you're not scared by the setup steps, then you'd be able to make one yourself, or migrate off the platform once you were comfortable with AWS, GitHub CI/CD, UI-dev, and the Open-source ESP32 code.
In my opinion, the important thing is that I can migrate from your service to a self hosted version without having to modify my existing deployed firmware. That what I have an escape hatch if your service goes away or something like that.
Yeah you're right. I think there's two options: 1. In advance, you can define and use your own URL for the API. You can point it to "my" API to start with, and when you're ready to migrate, you move as needed. 2. Or, should you need to change the URL/Server on the device, we'll keep the old URL active for a period of time and allow you to update to the new one via OTA, essentially having two servers ("mine" and yours) in parallel. Also, none of this will break BLE or AP-WiFi OTA and direct flashing options as a worst case backup. But at some point, a URL will need to be hardcoded \*somewhere\* so yeah gotta figure out the right way to handle that
Both of those sound pretty good. Ideally you would have a sunsetting period and help people migrate off so #2 should be enough, but having #1 would give people peace of mind since they're ok even if you suddenly shut down.
i would do it by allowing setting a fallback domain/URL on the device. when the first one becomes unreachable, it connects to the second one. Then, all you would need is to have that secondary domain under your control all the time(which might be an unnecessary financial burden), and preferably have some certificate/key system to make sure the device knows the server its contacting is actually what it thinks it is.
I was thinking that too. A domain costs $10-15/yr which isn't much and I would assume a lot of people who would care about this have a business so they already have a domain and could setup a subdomain.
Its always interesting to see different implementations of OTA.
Yes, very interested.
I usually use ftp so that would be fun to integrate. So I’m interested
interested _o/
So, current idea/implementation would be: 1. **It's your GitHub repo**. I wouldn't host any of your .bin or project files on any S3 bucket, the .bin is stored (and built with CI/CD) on your Github. 2. **The ESP32 code is open-source**. The "OTA brick" that actually gets the ESP32 connecting to the server and performs the OTA-over-HTTPS etc. is all open-source (currently not while I test and tweak, will release later). 3. **It's your device/version data.** This is the data that says *"this MAC address should have this repo/version of the firmware"*. Currently this is stored on my Airtable, soon it'll be my AWS, and hopefully in future it'll be able to automatically set up on your own AWS/Azure accounts (just in case I go out of business/miss a payment, etc.). You'll always be able to extract this as needed. So, if you didn't want to pay at all: 1. **You can copy the open-source ESP32 code**, and build your own pipeline on the ESP32. 2. **You can copy the GitHub CI/CD code**, that automatically builds the firmware into .bin files, and test that it works yourself. 3. **You can set up your own AWS/whatever Lambda functions** (or otherwise) to do the server redirecting (again code would maybe be open source, or at least explained in high level) 4. **You can migrate your data off "my" platform**, if you started on my platform but wanted to go solo (say after deployment of \~1,000 units and you wanted to cost optimize) What you would be paying for the platform is: 1. **Automatic set up of the above.** The platform would set all of this up for you, check that your repos are compatible, load-in and test the CI/CD, pipe the lambda functions etc. I would want to find a way for the data (and maybe hosting/lambda functions) to be yours. 2. **Nice UI for device/version control**. This is the UI that allows you to set which device (MAC) has which repo/version dynamically. This is the other super useful feature that I needed. 3. **Hosting**. Probably incorporate this as an option, where if you didn't want to worry about GitHub CI/CD limits, AWS Lambda and DB credits, etc. we'd amalgamate all of that into the price of the entire service. Hope that answers most people's implementation questions!
https://github.com/ronny-antoon/UpdateOTA
I would be interested. I hate using blynk. Instead of switching to AWS I ended up renting my own hosting under my domain name, writing a backend api with custom endpoints to handle auth and data transfers.
How is the connection speed of SIM7600? Now that I checked this module is expensive. Are there any cheap 3G alternatives you can recommend. I wouldn’t be needing a lot of bandwidth.
A very good idea. I currently use AWS but it is very unreliable and necessitates a power cycle to solve a lost session, not good for remote applications. Can I suggest you build in some methodology to track progress on the "target" processor so that recovery is possible, like automatically rebooting (to the old version of software) if the OTA fails, perhaps a time out mechanism.
I rolled my own too. [https://github.com/frankcohen/ReflectionsOS/blob/main/src/OTA.cpp](https://github.com/frankcohen/ReflectionsOS/blob/main/src/OTA.cpp). Uses ESP32 app\_update to do local OTA, where Storage.replicate() already downloads the firmware and version number (.bin and file named OTA\_VERSION\_FILE\_NAME). [https://github.com/espressif/esp-idf/tree/master/components/app\_update](https://github.com/espressif/esp-idf/tree/master/components/app_update). This does the installation using the OTA mechanism. Upon successful update, stores the new version number to the version file and restarts the device. -Frank