Seeking Feedback & Contributions: Merlin Auto Update Solutions

[RMerlin]

hnd-write is an rc applet. It sets an nvram variable, then calls bca_sys_upgrade(), which is in rc/mtd.c. At first glance it does not appear to be doing any validation, which is handled by httpd when the file is uploaded. The validation code lies in shared/* and that part is closed source. That's why directly using hnd-write is dangerous, as it will be skipping all the image validation steps from libshared calls.

 

[ExtremeFiretop]

Something that should have been clearly stated up front, not three pages later. "People fail to understand" (in your words) because you haven't indicated or clearly articulated until that post on page 3 that you didn't have plans for anyone else using it. Your OP post and many subsequent posts/replies seems to imply it was intended for a wider audience because you were seeking input from a wide audience.

I think this is the fundamental misunderstanding of your intention then. There is nothing in your initial post to suggest this. In fact from the subsequent posts it sounds like you're trying to create a publicly available script for any user and any model of router. You've now limited the project to HND routers only but there's still nothing to suggest that this is for your own use only using only the models that you have access to. If you had said from the outset that this was for your own personal use and you only wanted code improvement suggestions you probably wouldn't have received such negative feedback.

I was seeking input from a wide audience of experienced router scripters on how to improve my script, I made it clear many times it wasn't for anyone to go and use, as is:

Use it with caution, and manual supervision is advised initially for the first few runs. Report any issues on Github or here...

still in preview (only started working on this may 12 hour ago).

I'm exploring further enhancements, and would appreciate any help, feedback, or suggestions:

• Log analysis to determine the necessity of a factory reset within a specified date range... requiring a notification system.
• Install and Uninstall features for AMTM... needing further research and assistance.

Let's discuss any potential concerns, suggestions, or contributions regarding these solutions, especially regarding compatibility...

If your only value is "just be happy the way it is", please go elsewhere, we have 2 scripts that can do the job here in this thread, it's just about compatibility and cleaning it up now as much as possible..
Naturally If your not happy with something, you fix it! Looking forwards to feedback and concerns!

Thank you in advance!

like I said above this is mostly as the challenge, I do see the need for the desktop version for myself as I mentioned above for my SSL certs being updated on NGINX when they are refreshed on the router so I don't get the "this page isn't secure" message in the browser for my visitors.

It's still early days, I wouldn't want anything officially offered on AMTM until it's been vetted through such scenarios,

For example the script for the router implementation was only just started a few hours ago, and while it creates a cron job, I don't think that cron job actually runs the update right now. I needed some sleep so I took a break but I'll get back into it later this evening.

I also still need to find the best way to implement the notification system Incase a factory default is recommended on that firmware release. That code exists in the script but is commented out until I either implement my own notification system or install rmerlins as a requirement.

I agree I can make the original post more clear that this is a preview of a project for me and my friend alone, but the truth is, if someone does want to contribute and confirms it works on more models, I don't want to limit it to just me, if someone else puts time into reviewing the script, asking questions on why it's done this way, how it can be improved, how we can secure the payload up to the end etc, and then even goes above and beyond and starts testing it like my friend, than by all means, have the code and use it, grow it, make it better, otherwise, be aware this is naturally something extremely new in the world of and flashing and scripting here, people are cautious and i understand that, please treat it as such. Don't do anything your not comfortable doing, the code exists to be reviewed and picked at and improved, not to be run blindly.

I didn't think I needed to say that, but maybe I do and your right on that front.

 

[ExtremeFiretop]

hnd-write is an rc applet. It sets an nvram variable, then calls bca_sys_upgrade(), which is in rc/mtd.c. At first glance it does not appear to be doing any validation, which is handled by httpd when the file is uploaded. The validation code lies in shared/* and that part is closed source. That's why directly using hnd-write is dangerous, as it will be skipping all the image validation steps from libshared calls.

So I can start poking at libshared to see what validation it does and we would be skipping, but any quick lists you can run me through?

Is it just the RSA signature ASUS uses? Or also the checksum for the firmware flash?
Because we can use checksum ourselves, right? and the script can be scripted in such a way that it always gets the right firmware (tbh it never has got the wrong one ever, so i don't think really much has to change except a few extra catchs maybe)

So assuming we always loaded the right firmware to the router, lets say, even with a pause for approvel/deny before upload/download to confirm the firmware model, and then checksumed it all the way to the router itself, would we be missing any other validations that libshared handles?

 

[ExtremeFiretop]

hnd-write is an rc applet. It sets an nvram variable, then calls bca_sys_upgrade(), which is in rc/mtd.c. At first glance it does not appear to be doing any validation, which is handled by httpd when the file is uploaded. The validation code lies in shared/* and that part is closed source. That's why directly using hnd-write is dangerous, as it will be skipping all the image validation steps from libshared calls.

Btw thank you Rmlerin for just sticking to it, this is extactly the type of disscussions I wanted to have.

 

[ExtremeFiretop]

Don't do anything your not comfortable doing, the code exists to be reviewed and picked at and improved, not to be run blindly.

I didn't think I needed to say that, but maybe I do and your right on that front.

There i modified the original OP to be more clear, so I don't have to keep repeating myself (hopefully).

 

[RMerlin]

So I can start poking at libshared to see what validation it does and we would be skipping, but any quick lists you can run me through?

Asus has made change to that code over the years, so I can`t comment on its current state. Historically, the libshared validation function was checking the model in the header, checking the checksum to ensure the file wasn't incomplete or corrupted,, and also (for some models, like the RT-AC68U) checking for the version to prevent flashing firmwares older than a minimal version. That was done to prevent RT-AC68U V3 owners from flashing a very old version that didn`t support the newer hardware revisions. They've done it for other models, but I don't know the up-to-date list of these checks as last time I decompiled that code was years ago.

I have no idea what other checks are done these days.

Is it just the RSA signature ASUS uses?

The RSA signature is only used (AFAIK) for liveupdates. Manual updates aren`t RSA-signed. Again, this is something they changed these past few years as they completely revised their liveupdate implementation, so I don`t know the details on how it now works beyond the RSA signature check, and also that they use a different manifest format which now allows them to check for a minimal version being present. That process used to be in a shell script, they moved it to a binary process now, which is in part closed source.

would we be missing any other validations that libshared handles?

Not knowing the extent of what they do, it`s impossible to tell. And it`s also impossible to tell if this won`t change in the future, as it has already changed many times over the 10+ years of Asuswrt`s existence.

Another recent change to take into account: are you able to handle firmware files that contain two separate images for separate hardware revisions? Asus uses this for the RT-AC68U and RT-AX58U so far, possibly other models as well. I don`t know how these are being handled.

I think Dave had the right idea there - it would be much safer to trigger an update from the firmware image present on the router rather than manually writing it yourself. You will have to review Asus' live update implementation to see if/how it's possible.

 

[Tech9]

Do you even know how to script?

I know what I used your method of flashing firmware for and it wasn’t something approved by Asus. It may kill the router. Use whatever you want on your router, but don’t post it on Internet.

 

[ExtremeFiretop]

Asus has made change to that code over the years, so I can`t comment on its current state. Historically, the libshared validation function was checking the model in the header, checking the checksum to ensure the file wasn't incomplete or corrupted,, and also (for some models, like the RT-AC68U) checking for the version to prevent flashing firmwares older than a minimal version. That was done to prevent RT-AC68U V3 owners from flashing a very old version that didn`t support the newer hardware revisions. They've done it for other models, but I don't know the up-to-date list of these checks as last time I decompiled that code was years ago.

I have no idea what other checks are done these days.


The RSA signature is only used (AFAIK) for liveupdates. Manual updates aren`t RSA-signed. Again, this is something they changed these past few years as they completely revised their liveupdate implementation, so I don`t know the details on how it now works beyond the RSA signature check, and also that they use a different manifest format which now allows them to check for a minimal version being present. That process used to be in a shell script, they moved it to a binary process now, which is in part closed source.


Not knowing the extent of what they do, it`s impossible to tell. And it`s also impossible to tell if this won`t change in the future, as it has already changed many times over the 10+ years of Asuswrt`s existence.

Another recent change to take into account: are you able to handle firmware files that contain two separate images for separate hardware revisions? Asus uses this for the RT-AC68U and RT-AX58U so far, possibly other models as well. I don`t know how these are being handled.

I think Dave had the right idea there - it would be much safer to trigger an update from the firmware image present on the router rather than manually writing it yourself. You will have to review Asus' live update implementation to see if/how it's possible.

Thank you RMerlin this is actually something I didn't know about which I should note for, the RT-AC68U is currently listed as as unsupported model and wouldn't even run, now I can quickly put the same kind of block in for the RT-AX58U, but my instant thought is handle it the same way I handle the GT-AXE11000 which has a rog and a pure build in the zip, and give the user the promp/choice of which default build to use on initial setup. So basically if it's a RT-AX58U that it prompts for hardware revision and uses the file accordingly.

I agree the unknowing changes of the feature can cause issues, but in it's current state I can probably handle at least most of those concerns via script I would think? Except for maybe the minor revision checks, I'd have to think more on that one..
Would any requirements be listed in the log file? If so I can handle it similar to how I handle factory default recommendations and simply look between the last 2 entries for the recommendation/requirement.

As for daves idea, can you explain in what you mean by trigger the update from the firmware image present on the router rather than witting it myself?

If the firmware update file is in home/root and it runs the command to hnd-write that file, is that not triggering the update from the firmware image present on the router?
The active partition/firmware image is the one writing that update to the inactive partition is it not? Maybe this is a barrier I'm not understanding.

 

[ExtremeFiretop]

I know what I used your method of flashing firmware for and it wasn’t something approved by Asus. It may kill the router. Use whatever you want on your router, but don’t post it on Internet.

Really critical way of thinking, not sure how you plan on growing knowledge on a subject or sharing knowledge on a subject then, are you planning to write a book on hnd-write and put it in the library?
Didn't you say you were done already, I said I was almost done, RMerlin is on the topic at hand, please refrain unless you have something on topic to add.

 

[RMerlin]

I agree the unknowing changes of the feature can cause issues, but in it's current state I can probably handle at least most of those concerns via script I would think

What`s worrying there is the unknown, and any potential future changes. Asus makes changes to the upgrade process a number of times, and someone having to discover a new change through a bricked router might be a high price to pay.

As for daves idea, can you explain in what you mean by trigger the update from the firmware image present on the router rather than witting it myself?

Review either the httpd code we both pointed at, or analyse Asus' liveupdate process. The idea would be to deposit the new firmware in the exact location expected by the router when uploaded by the user or downloaded by the existing liveupdate process, and trigger the upgrade process from rc. I don't know however if it`s technically doable, that`s something you will have to research on your own.

If the firmware update file is in home/root and it runs the command to hnd-write that file, is that not triggering the update from the firmware image present on the router?

hnd-write is only the writing portion. Triggering the update mans having the router go through all its usual pre-process/pre-write actions and validation.

 

[ExtremeFiretop]

What`s worrying there is the unknown, and any potential future changes. Asus makes changes to the upgrade process a number of times, and someone having to discover a new change through a bricked router might be a high price to pay.

Agreed, I totally and completely understand that, right now it's only me and my friend and we understand the risks involved. But I am interested in this other method now than you explain it.

The idea would be to deposit the new firmware in the exact location expected by the router when uploaded by the user or downloaded by the existing liveupdate process, and trigger the upgrade process from rc. I don't know however if it`s technically doable, that`s something you will have to research on your own.

This helps clarify everything much better! Thank you! Will start to search down this path.
Are we aware if I would be starting from zero or if someone has already investigated this a bit?

 

[Yota]

I strongly support the OP and other developers actively developing this project, and look forward to an automatic update script that can run completely on the router without a desktop environment.

Of course, there are many difficulties here, but please let us try to solve them and not give up when we encounter difficulties.

Asus stock firmware already implements this feature, there is no reason why we can't implement it, instead reverse engineering it will make it easier for us to implement it.

For those who oppose this project, you have the right not to run such scripts on your routers.



For solutions that rely on a desktop environment, you can use curl to simulate a browser for firmware upload through the router's GUI, which is a safer way. But I'd like to see beyond that and do everything directly inside the router.

 

[ExtremeFiretop]

I strongly support the OP and other developers actively developing this project, and look forward to an automatic update script that can run completely on the router without a desktop environment.

For those who oppose this project, you have the right not to run such scripts on your routers.

Thank you, this is what I've been saying, looking for suggestions and solutions to problems, not haters that just want to say they don't want it or how the current solution is too dangerous, as I said in my original post, if we don't like something, we fix it.

For solutions that rely on a desktop environment, you can use curl to simulate a browser for firmware upload through the router's GUI, which is a safer way. But I'd like to see beyond that and do everything directly inside the router.

I thought about that as well after the fact lol, but because I was already working on those other steps for a desktop environment (such as backing up configs, ssl certs, and downloading the firmware to the device and installing the certs installs NGINX service (stopping it, copy certs, start service, etc)) it's just the path I took. to integrate it all into one.
Going your route would still work and actually would be safer for the update process, but just simply means I'd have to break up the other steps into their own separate scripts.

But for the purpose of actually flashing the firmware, your right, it is safer and probably smarter to split that off from the rest of the desktop script.

 

[Viktor Jaep]

For solutions that rely on a desktop environment, you can use curl to simulate a browser for firmware upload through the router's GUI, which is a safer way. But I'd like to see beyond that and do everything directly inside the router.

But for the purpose of actually flashing the firmware, your right, it is safer and probably smarter to split that off from the rest of the desktop script.

It would certainly be really interesting to find out/reverse-engineer the way stock Asus does their auto-updates, and whether they just curl an update or if there's closed source magic happening on the backend.

 

[Yota]

Thank you, this is what I've been saying, looking for suggestions and solutions to problems, not haters that just want to say they don't want it or how the current solution is too dangerous, as I said in my original post, if we don't like something, we fix it.

The reasons given by those opposed to this project in this post might have made sense when it was still an idea.

But what you're doing now is beyond the idea that you've created it, don't listen to them when they say "you're going to destroy a million routers", these off topic discussions are now detrimental to the project.

What you have to do now is make a to-do list and a list of problems/questions, and then wait for those who are interested in the project and capable of solving them to join you in growing the project.

I have a feeling that as the leader and developer of this project you're going to screw up your own router a few times, but if you're not afraid of that then there's nothing you should worry about and just move forward.
If you don't want to hear their complaints, select your team to start a private conversation.

I firmly believe that the first person who invented an airplane or a rocket was going through what you are going through right now, like, "You're going to kill people," "I'll never go up in that thing, that's crazy," but if they stopped because of that, Now, maybe we are still smashing rocks in the cave.

 

[Yota]

It would certainly be really interesting to find out/reverse-engineer the way stock Asus does their auto-updates, and whether they just curl an update or if there's closed source magic happening on the backend.

Your reply made me think that we can completely call curl in the router and then simulate the browser upload through the GUI. This can take advantage of all the protection measures built into the router when uploading firmware, and can be applied to all routers (including AC series, AX Series, and AiMesh) because they all have firmware upload buttons in the GUI.

The only drawback is that the clear text router password must be stored. Because Asus has introduced quite a few security measures to avoid storing clear text passwords in the router.

So this solution is a compromise, without the tedium of reverse engineering, and will quickly silence most of the naysayers in this thread. Since the router logs into the GUI and uploads the firmware itself, just like a user would, there's nothing wrong with it, and if it messes up somewhere, it's not your bug.

 

[ExtremeFiretop]

Your reply made me think that we can completely call curl in the router and then simulate the browser upload through the GUI. This can take advantage of all the protection measures built into the router when uploading firmware, and can be applied to all routers (including AC series, AX Series, and AiMesh) because they all have firmware upload buttons in the GUI.

The only drawback is that the clear text router password must be stored. Because Asus has introduced quite a few security measures to avoid storing clear text passwords in the router.

So this solution is a compromise, without the tedium of reverse engineering, and will quickly silence most of the naysayers in this thread. Since the router logs into the GUI and uploads the firmware itself, just like a user would, there's nothing wrong with it, and if it messes up somewhere, it's not your bug.

I like this alot, it's making me think of redesigning the approach for sure, I'm still currently looking into Rmlerins and daves idea for now, but In summary achieving a programmable firmware upgrade as described would likely require:

1. A method to deposit the new firmware file in the location expected by the router, possibly using a network file transfer protocol or other mechanism supported by the router firmware.
2. A method to programmatically trigger the upgrade process, possibly by invoking system utilities, scripts, or other mechanisms provided by the router firmware for this purpose.

The specifics of these methods would likely be highly dependent on the firmware and the underlying operating system, it seems to require a pretty deep understanding of these systems as well as possibly privileged access to the router's operating environment. *Cough* *Cough*

 

[Yota]

I like this alot, it's making me think of redesigning the approach for sure, I'm still currently looking into Rmlerins and daves idea for now, but In summary achieving a programmable firmware upgrade as described would likely require:

1. A method to deposit the new firmware file in the location expected by the router, possibly using a network file transfer protocol or other mechanism supported by the router firmware.
2. A method to programmatically trigger the upgrade process, possibly by invoking system utilities, scripts, or other mechanisms provided by the router firmware for this purpose.

The specifics of these methods would likely be highly dependent on the firmware and the underlying operating system, it seems to require a pretty deep understanding of these systems as well as possibly privileged access to the router's operating environment. *Cough* *Cough*

I think the idea of reverse engineering makes a lot of sense because it's a really elegant way to automatically upgrade the firmware, whereas using curl to emulate a browser is not elegan.

But considering reducing the difficulty of the project and maintenance costs (when Asus changes the upgrade process, it will need to be reverse-engineered again and push the new code to all routers in a timely manner), the method of curl simulating the browser will be simpler, because it will support all routers, no firmware will be flashed when the code does not work, and as long as there is a firmware upgrade button, all you need to do is tell the script to click that button, and Asus's closed source code takes care of the rest.


Another potential thing to add to your to-do list is that you have to assume that the firmware downloaded from SourceForge or uploaded by RMerlin is not 100% working.

In fact, there have been two cases recently. The first is that SourceForge failed not long ago and the CDN was unable to distribute files, so SourceForge is not 100% reliable.

and RMerlin once provided firmware in alpha testing that could cause the router to malfunction, so we must assume that even the final release is not 100% safe.

So how to ensure that the router can be restored to the previous firmware in this case?


A suggested idea is: prerequisite the router must have a USB drive connected, and then save the configuration files, jffs backup and current and newer versions of firmware in the USB drive before each update, because this is the only persistent storage location that has enough storage space and will work when router jffs doesn't work. (Yes, we have to assume that in some cases the jffs partition does not work after upgrading, because in history, they have adjusted the jffs partition by upgrading the firmware, and this kind of thing may cause jffs to not be mounted or need to be initialized).

When the firmware upgrade is completed, a simple check is required, including wifi and internet checks. If the check does not pass, we must assume that there is a problem with the firmware. we must be able to downgrade to the previous firmware (even without internet connection, in which case we will use the older firmware stored in the USB drive).


P.S. new routers no longer have JFFS2 as they use UBIFS.

Anyway, we do need to assume how to run the script in case jffs is not working, and revert to the old firmware or fix it. (Possibly, this will be the $1 million question. Hahaha)


EDIT:
I do have some thoughts on the $1 million question: the old way, reverse engineer the Download Master.

 

[ExtremeFiretop]

I think the idea of reverse engineering makes a lot of sense because it's a really elegant way to automatically upgrade the firmware, whereas using curl to emulate a browser is not elegan.

But considering reducing the difficulty of the project and maintenance costs (when Asus changes the upgrade process, it will need to be reverse-engineered again and push the new code to all routers in a timely manner), the method of curl simulating the browser will be simpler, because it will support all routers, no firmware will be flashed when the code does not work, and as long as there is a firmware upgrade button, all you need to do is tell the script to click that button, and Asus's closed source code takes care of the rest.


Another potential thing to add to your to-do list is that you have to assume that the firmware downloaded from SourceForge or uploaded by RMerlin is not 100% working.

In fact, there have been two cases recently. The first is that SourceForge failed not long ago and the CDN was unable to distribute files, so SourceForge is not 100% reliable.

and RMerlin once provided firmware in alpha testing that could cause the router to malfunction, so we must assume that even the final release is not 100% safe.

So how to ensure that the router can be restored to the previous firmware in this case?


A suggested idea is: prerequisite the router must have a USB drive connected, and then save the configuration files, jffs backup and current and newer versions of firmware in the USB drive before each update, because this is the only persistent storage location that has enough storage space and will work when router jffs doesn't work. (Yes, we have to assume that in some cases the jffs partition does not work after upgrading, because in history, they have adjusted the jffs partition by upgrading the firmware, and this kind of thing may cause jffs to not be mounted or need to be initialized).

When the firmware upgrade is completed, a simple check is required, including wifi and internet checks. If the check does not pass, we must assume that there is a problem with the firmware. we must be able to downgrade to the previous firmware (even without internet connection, in which case we will use the older firmware stored in the USB drive).


P.S. new routers no longer have JFFS2 as they use UBIFS.

Anyway, we do need to assume how to run the script in case jffs is not working, and revert to the old firmware or fix it. (Possibly, this will be the $1 million question. Hahaha)


EDIT:
I do have some thoughts on the $1 million question: the old way, reverse engineer the Download Master.

Lots and lots of valuable take away here. Thank you very much!!! Especially the SourceForge stuff and the idea on the backup to the USB.
You've given both valuable points and ideas to alternative solutions and I like that, it's really what the discussion was aiming to achieve. @RMerlin as well, you've just stayed on track the entire time and never did anything but just provide straight up information to the best of your knowledge. It's much appreciated.

Like I mentioned I had actually though about the browser emulation (login and upload) before posting about this really when starting the project, but because I had already worked so far down this path, hnd-write was the easier choice to include in this all-in-one type solution.
However discussing openly the restrains around hnd-writre and the hesitation on actually relying on it (which I completely understand, it is unknowable what ASUS may change, just because it works today and in the last 8 months doesn't mean it will work forever).

These other alternatives of triggering the systems natural update process once you dump the file in the right location feels like the natural next step, however requires a bit more digging, I won't lie, I didn't click these links the first time they were posted and I'm a big enough man to admit when I was wrong.

For the desktop app, I would probably work as you said on splitting it from the rest of the script I have now, and really focus on the browser emulation part, find a way to prompt for credentials, cache those somewhere locally and login and do the update, if I can do that on the desktop (where I have extra tools and knowledge) then naturally I can move over lots of the logic to Shell and have the router try the steps itself of logging into it's own UI and doing the update that way.

I'll probably be going down a rabbit hole for a while, like Merlin said some of this stuff is fairly complex and closed source, but the whole point was the challenge of finding a reliable way to accomplish this update/flashing step, however possible.

 

[Yota]

This project actually reminded me of an exploration I attempted a few years ago: automatically factory-resetting and then importing user-defined changed configurations after every router update.

So there are never dirty updates, it's brand new every time, but no need to spend 2 hours manually resetting everything.

But these are several paradoxes

The first is because jffs will not have scripts after restoring factory settings. the solution is to store the scripts on the USB drive.

The second is that scripts cannot be run after factory reset. I can't resolve this paradox. So the project was shelved.

I don't think your project will encounter the second paradox unless you really intend to do an automatic factory reset after a router upgrade.