MerlinAU MerlinAU v1.1.1 - The Ultimate Firmware Auto-Updater (Now available in AMTM)

[Yaffle]

We would need more info to make a proper assessment, just because it didn't have enough RAM on one router once, does not mean it would be the case on every router of that model.
Can you send me your log file for MerlinAU please? It doesn't look like you did the reboot? Why not?

How many add-ons do you have installed? Also did it work when you rebooted? This is important to test.
Did that successfully clear enough RAM to do the update?

If it does, then it's just an example how I can't limit the model based on the amount of add-ins you installed/using at the time.

When it says "Attempting to free up memory" it runs a: sync; echo 1 > /proc/sys/vm/drop_caches command which should give back the router some memory, it then rechecks, and if that still wasn't enough then it prompts for a reboot as you see above.
This tells me you had enough "things" running that even when doing sync; echo 1 > /proc/sys/vm/drop_caches it still was not sufficient to continue.

Please try to reboot the router, send me the logs when you can, and try to list all your other add-ins or scripts running.
Thanks!

Hi,

Thanks for that. I did do the reboot (I'd just done the screenshot before I pressed Yes) and still get the same. I've removed a couple of add-ons (including Backupmon) for testing. I've now got Yazfi, MerlinAU, scMerlin, WAN IP Notification and Disk check script. Still getting the same problem. It seems to download and extract the file and then not free up an awful lot. It's not long since I last did a full reset but when I get chance I'll do a clean install and try just MerlinAU as it must be something I have installed..

 

[ExtremeFiretop]

Hi,

Thanks for that. I did do the reboot (I'd just done the screenshot before I pressed Yes) and still get the same. I've removed a couple of add-ons (including Backupmon) for testing. I've now got Yazfi, MerlinAU, scMerlin, WAN IP Notification and Disk check script. Still getting the same problem. It seems to download and extract the file and then not free up an awful lot. It's not long since I last did a full reset but when I get chance I'll do a clean install and try just MerlinAU as it must be something I have installed..

I see your log file attached, my apologies.
Seems to me that it's unable to make the required memory, I'll send you a quick message shortly for something to try

 

[ExtremeFiretop]

Just based on the log file alone, you started with only 43MB of available memory, the script managed to increase that to 63MB.
But still did not get nearly close enough to the zip size (73MB in your case) + 50% overhead which is what we required in the script. (Total of 109MB in your case).

The overhead was set to 50% to be safe, we can experiment with lowering it on the sidelines by PM, or shutting stuff down before hand, but there's limited support I can offer.
This would need some troubleshooting work to identify what is hogging all the memory.

 

[Viktor Jaep]

Just based on the log file alone, you started with only 43MB of available memory, the script managed to increase that to 63MB.
But still did not get nearly close enough to the zip size (73MB in your case) + 50% overhead which is what we required in the script. (Total of 109MB in your case).

The overhead was set to 50% to be safe, we can experiment with lowering it on the sidelines by PM, or shutting stuff down before hand, but there's limited support I can offer.
This would need some troubleshooting work to identify what is hogging all the memory.

So off-the-wall question about this... this piqued my interest... If he would have gone the regular router of using Administration -> Firmware Update -> Upload... what would the difference be? He would have still started with 43MB of available memory. Would the FW have done something different to accommodate the needed size of available RAM to kick off a successful upgrade... or would it just wing it?

 

[ExtremeFiretop]

So off-the-wall question about this... this piqued my interest... If he would have gone the regular router of using Administration -> Firmware Update -> Upload... what would the difference be? He would have still started with 43MB of available memory. Would the FW have done something different to accommodate the needed size of available RAM to kick off a successful upgrade... or would it just wing it?

The difference is he is doing the upload, not the router.
The router has a script running on itself doing the upload, this is why it's important it has the required resources, not to freeze.

This is why we implemented an overhead to make sure the script has the requires resources to run until the end.
If it doesn't have the overhead then it won't try to update. We implemented more memory requirements than just the WebGUI alone since it does more steps.

When he uploads the firmware himself, he is not using more of the routers resources to download, extract, unzip and upload the firmware, all that is done on his own machine with it's own memory.
We need to make sure all these steps do not push the router over the edge, else the script can freeze at a undesirable step.

 

[Viktor Jaep]

The difference is he is doing the upload, not the router.
The router has a script running on itself doing the upload, this is why it's important it has the required resources, not to freeze.

This is why we implemented an overhead to make sure the script has the requires resources to run until the end.
If it doesn't have the overhead then it won't try to update. We implemented more memory requirements than just the WebGUI alone since it does more steps.

When he uploads the firmware himself, he is not using more of the routers resources to download, extract, unzip and upload the firmware, all that is done on his own machine with it's own memory.
We need to make sure all these steps do not push the router over the edge, else the script can freeze at a undesirable step.

Thanks for taking such care with all this!

 

[Yaffle]

Just based on the log file alone, you started with only 43MB of available memory, the script managed to increase that to 63MB.
But still did not get nearly close enough to the zip size (73MB in your case) + 50% overhead which is what we required in the script. (Total of 109MB in your case).

The overhead was set to 50% to be safe, we can experiment with lowering it on the sidelines by PM, or shutting stuff down before hand, but there's limited support I can offer.
This would need some troubleshooting work to identify what is hogging all the memory.

After the downloading yes, but before everything started it had:

2024-03-18 17:20:29 Required RAM: 112207 KB - RAM Free: 148748 KB - RAM Available: 139436 KB

 

[ExtremeFiretop]

Thanks for taking such care with all this!

It's a fair question, why are we making it require more memory than usual.. The short answer is because we are using more resources than usual to automate the steps usually done off the router, and we don't want resource starvation to be the cause of a failure.

 

[ExtremeFiretop]

After the downloading yes, but before everything started it had:

2024-03-18 17:20:29 Required RAM: 112207 KB - RAM Free: 148748 KB - RAM Available: 139436 KB

Your PM and your latest message has given me a lightbulb moment.

I'll get back to you after I test some stuff out.

 

[ExtremeFiretop]

For anyone curious, @Yaffle has been updated successfully using MerlinAU in a "low memory" state today. All went successfully and no errors were found in his logs.
We are still testing and touching up the PR with the changes. So far... Things look positive but not promises yet, not until results are confirmed and validated on multiple routers. I will do some more testing this weekend.

For people interested in the progress, the general idea is to set a bare minimum just above the system kernel watermarks of low memory:

Min:   5120 pages ~= 20 MB
Low:   6400 pages ~= 25 MB
High:  7680 pages ~= 30 MB

MerlinAU will set it's own limit right above this "High" value.
Somewhere around ~35MB free as the bare minimum of entry to avoid swapping by the kernel.

Doing this change will loosen the restrictions around memory requirements, but only once the firmware file has been downloaded and extracted into the HOME folder successfully. (Which is a virtual RAM disk and takes a big chunk of the "Available Memory" for routers with limited resources.)
While keeping a bare minimum above the swapping point for RAM. If it falls below the 35MB minimum required from that point onwards, the original logic to try and free memory or reboot will trigger.

 

[vlord]

For some reason, on all three of my Asus devices, the upgrade didn't occur as scheduled:

GT-AX6000 Log
admin@GT-AX6000:/tmp/mnt/sda1/MerlinAU.d/logs# more GT-AX6000_FW_Update_2024-03-17_03_40_01.log
2024-03-17 03:40:01 The firmware update to 3004.388.6.2 version is currently postponed for 15 day(s).
2024-03-17 03:40:01 The firmware update is expected to occur on or after 2024-Mar-17 depending on when your cron job is scheduled to check again.


RT-AX86U #1 Log
admin@RT-AX86U-AP2:/jffs/addons/MerlinAU.d/logs# more RT-AX86U_FW_Update_2024-03-17_03_20_00.log
2024-03-17 03:20:01 The firmware update to 3004.388.6.2 version is currently postponed for 15 day(s).
2024-03-17 03:20:01 The firmware update is expected to occur on or after 2024-Mar-17 depending on when your cron job is scheduled to check again.
admin@RT-AX86U-AP2:/jffs/addons/MerlinAU.d/logs# date
Wed Mar 20 17:12:49 EDT 2024

RT-AX86U #2 Log
admin@RT-AX86U:/jffs/addons/MerlinAU.d/logs# more RT-AX86U_FW_Update_2024-03-17_03_00_00.log
2024-03-17 03:00:01 The firmware update to 3004.388.6.2 version is currently postponed for 15 day(s
2024-03-17 03:00:01 The firmware update is expected to occur on or after 2024-Mar-18 depending on when your cron job is scheduled to check again.
admin@RT-AX86U:/jffs/addons/MerlinAU.d/logs# date
Wed Mar 20 17:03:00 EDT 2024
admin@RT-AX86U:/jffs/addons/MerlinAU.d/logs#

Any suggestions on where/what to check next?

 

[ExtremeFiretop]

Any suggestions on where/what to check next?

Sure, it would be helpful to know what the cron schedule is set as currently and when you received first notification as per MerlinAU!

 

[vlord]

Sure, it would be helpful to know what the cron schedule is set as currently and when you received first notification as per MerlinAU!

Here is the schedule for each of my devices:

GT-AX6000
Current Schedule: 40 3 * * 0
admin@GT-AX6000:/tmp/mnt/sda1/MerlinAU.d/logs# more GT-AX6000_FW_Update_2024-03-03_03_40_01.log
2024-03-03 03:40:02 The firmware update to 3004.388.6.2 version is currently postponed for 15 day(s).
2024-03-03 03:40:02 The firmware update is expected to occur on or after 2024-Mar-17 depending on when your cron job is scheduled to check again.

RT-AX86U #1
Current Schedule: 0 3 * * 0
admin@RT-AX86U:/jffs/addons/MerlinAU.d/logs# more RT-AX86U_FW_Update_2024-03-03_03_00_00.log
2024-03-03 03:00:02 The firmware update to 3004.388.6.2 version is currently postponed for 15 day(s).
2024-03-03 03:00:02 The firmware update is expected to occur on or after 2024-Mar-18 depending on when your cron job is scheduled to check again.


RT-AX86U #2
Current Schedule: 20 3 * * 0
admin@RT-AX86U-AP2:/jffs/addons/MerlinAU.d/logs# more RT-AX86U_FW_Update_2024-03-03_03_20_00.log
2024-03-03 03:20:01 The firmware update to 3004.388.6.2 version is currently postponed for 15 day(s).
2024-03-03 03:20:01 The firmware update is expected to occur on or after 2024-Mar-17 depending on when your cron job is scheduled to check again.

 

[ExtremeFiretop]

Here is the schedule for each of my devices:

GT-AX6000
Current Schedule: 40 3 * * 0
admin@GT-AX6000:/tmp/mnt/sda1/MerlinAU.d/logs# more GT-AX6000_FW_Update_2024-03-03_03_40_01.log
2024-03-03 03:40:02 The firmware update to 3004.388.6.2 version is currently postponed for 15 day(s).
2024-03-03 03:40:02 The firmware update is expected to occur on or after 2024-Mar-17 depending on when your cron job is scheduled to check again.

RT-AX86U #1
Current Schedule: 0 3 * * 0
admin@RT-AX86U:/jffs/addons/MerlinAU.d/logs# more RT-AX86U_FW_Update_2024-03-03_03_00_00.log
2024-03-03 03:00:02 The firmware update to 3004.388.6.2 version is currently postponed for 15 day(s).
2024-03-03 03:00:02 The firmware update is expected to occur on or after 2024-Mar-18 depending on when your cron job is scheduled to check again.


RT-AX86U #2
Current Schedule: 20 3 * * 0
admin@RT-AX86U-AP2:/jffs/addons/MerlinAU.d/logs# more RT-AX86U_FW_Update_2024-03-03_03_20_00.log
2024-03-03 03:20:01 The firmware update to 3004.388.6.2 version is currently postponed for 15 day(s).
2024-03-03 03:20:01 The firmware update is expected to occur on or after 2024-Mar-17 depending on when your cron job is scheduled to check again.

Hi there, thanks for that.

When was the last time you started the MerlinAU update manually by the Addon GUI over SSH?

Thanks!

 

[vlord]

Hi there, thanks for that.

When was the last time you started the MerlinAU update manually by the Addon GUI over SSH?

Thanks!

I don’t think I’ve ever done that. I assumed that after installing it, other than the version updates, it was more of a set it and forget it.

 

[ExtremeFiretop]

I don’t think I’ve ever done that. I assumed that after installing it, other than the version updates, it was more of a set it and forget it.

It is, but considering your having issues, please try it as a test.

I'd like you to try on one of your 3 routers please, and validate the issue is only with the cron job.
A cron job postpone period can be reset, so ideally if you can validate the issue is only with the crons on one your devices, and also validate that the notification date did not reset from the GUI. Not sure why it would, but it can.

Can you send me a screenshot of your actual "latest" MerlinAU notification in the GUI please?
Seems what your sending above is from the log files. (Could be potential updated with a newer entry)

It would be much appreciated

Thanks,
Joel

 

[vlord]

Updated one of the APs without any issue. Worked great.

I am seeing this message Cron job hook already exists in ‘/jffs/scripts/services-start’ script repeatedly on both the AP devices. On my main router, I’m not seeing any MerlinAU notices.

 

[ExtremeFiretop]

Updated one of the APs without any issue. Worked great.

I am seeing this message Cron job hook already exists in ‘/jffs/scripts/services-start’ script repeatedly on both the AP devices. On my main router, I’m not seeing any MerlinAU notices.

View attachment 57310
View attachment 57311

Perfect, so we know the postpone period does not cause any issues when running interactively. This cron job hook already exists message is normal. You will see it everytime the script runs, both interactively in the SSH session or by cron job.

The last one seems to be happening pretty often, did you turn up the cron job schedule? Or run MerlinAU multiple times by chance over SSH?

Can I please have a screenshot of the MerlinAU notification date, in the Addon GUI over SSH please as requested above? Actually can I have a screenshot of all your settings to replicate.

Also, if you haven't tried, can you try to turn the cron schedule up to something soon? Like every 10 minutes, and monitor to see if the cron has the issue again?

 

[TonyK132]

I just installed this script via amtm, then tried to change the update schedule. It failed. See the attached screenshot.

 

[ExtremeFiretop]

I just installed this script via amtm, then tried to change the update schedule. It failed. See the attached screenshot.

Just uninstalled and reinstalled and have no issues here.

Can you send me a photo of your settings and a copy of your logs via PM?

Thanks