Entware Entware Mystery

[CaptnDanLKW]

This is now the 2nd time this has happened.

Setup - 2x RT-AC86U in AI Mesh. Scribe, ScribeUI, scMerlin and connmon are the only addons running.

I don't stare at my logs all day, but I did log in to check today and noticed by uptime was only 1ish days... and scribe and scribeUI was not running.

going into amtm and trying to check a few things and i get the message entware not installed

However, it is. I can go to my mount point /tmp/mnt/8GBUSB and entware is there, along with all the binaries, libs, shared, etc. its just that the symlink to /opt is missing.

I didnt try to recreate the link, (in retrospect, I should have). Instead I just reran entware-setup.sh and it blew away everything in the entware dir structure and back to basic. It also nuked all my old syslog-ng.logs so I lost the data to see if there was a crash event in there.

I'm trying to understand why / what / how in the sequence entware is started and what is casuing this to break-itself. Ive made no config changes since 384_19 when I reconfigured everything manually. The uptime would have been in the 40 day range if not for whatever happened. Coincidentally, it was about 40ish days last time this happened.

 

[elorimer]

First, don't use the entware-setup.sh script, ever again. I'm surprised it's still there in a way. Use amtm.

Second, that message is a wait message that I think is part of the way @Jack Yaz had his scripts waiting until entware was up. He is in the process of changing all that.

 

[Mutzli]

The cause is probably a failing USB drive. Try to activate disk check in AMTM to diagnose possible problems.

 

[dave14305]

First, don't use the entware-setup.sh script, ever again. I'm surprised it's still there in a way. Use amtm.

entware-setup.sh is now a soft link to amtm.

 

[CaptnDanLKW]

entware-setup.sh is now a soft link to amtm.

Yea, I knew that - so clarification - i used 'ep' and that's what reinstalled entware.

 

[CaptnDanLKW]

Thanks for the replies. I still am looking to understand the under the hood startup sequence and what and where entware is first started. (like the symlink I mentioned)

 

[CaptnDanLKW]

The cause is probably a failing USB drive. Try to activate disk check in AMTM to diagnose possible problems.

Drive is fine. Checked OK. I thought this might be the reason last time, so I used a new drive. Since it happened again, now Im looking to dig deeper.

 

[Clark Griswald]

@CaptnDanLKW
AMTM has made script installation effortless.
Wipe the drive and reinstall your scripts.
I have found that faster, than chasing gremlins.

 

[CaptnDanLKW]

@CaptnDanLKW
AMTM has made script installation effortless.
Wipe the drive and reinstall your scripts.
I have found that faster, than chasing gremlins.

As I mentioned above, this is the 2nd time I've wiped and started over. So now im interested in learning more about how things work. That will help me longer term.

So, question again.... how does entware startup? I see rc.unslung is executed in /jffs/scripts but that assume /opt/etc/init.d exists. Where does the entware 'detection' start? i.e. i assume somewhere check for the presences of a /tmp/mnt/(whatever)/entware directory, and if it exists, creates the symlink.

but this is all conjecture, which is why i was asking.

 

[dave14305]

My Entware is installed via amtm and runs /jffs/scripts/post-mount when the USB is detected. That script contains /jffs/addons/diversion/mount-entware.div which detects if the currently mounting USB contains opkg, and if so it creates the /opt link.

 

[kidd232]

dude, is it possible you got second failing USB drive?

 

[SomeWhereOverTheRainBow]

Drive is fine. Checked OK. I thought this might be the reason last time, so I used a new drive. Since it happened again, now Im looking to dig deeper.

Make sure entware and swap file are on same partition.

 

[thelonelycoder]

I didnt try to recreate the link, (in retrospect, I should have). Instead I just reran entware-setup.sh and it blew away everything in the entware dir structure and back to basic. It also nuked all my old syslog-ng.logs so I lost the data to see if there was a crash event in there.

On Asuswrt-Merlin firmware 384.15 and newer, the file entware-setup.sh is symlinked to amtm. So amtm should have given you the option to install Entware.

I'm trying to understand why / what / how in the sequence entware is started and what is casuing this to break-itself. Ive made no config changes since 384_19 when I reconfigured everything manually. The uptime would have been in the 40 day range if not for whatever happened. Coincidentally, it was about 40ish days last time this happened.

Entware detection and mounting:
1. A plugged in device gets found by the firmware which subsequently runs /jffs/scripts/post-mount
2. /jffs/scripts/post-mount sources script /jffs/addons/diversion/mount-entware.div
3. /jffs/addons/diversion/mount-entware.div looks for a folder named entware (if not found it looks for an entware* folder and would rename it to entware if valid).
4. /jffs/addons/diversion/mount-entware.div looks for the file entware/bin/opkg in the entware folder.
5. If entware/bin/opkg is found, the symlink /tmp/opt is created which links to the entware folder.
6. /opt/etc/init.d/rc.unslung is then run. It starts all active Entware services.

If all this runs without a hiccup, then the only syslog entry "Entware Starting Entware services on <device>" is written.

Should the above fail, it also writes to syslog:
1. When entware/bin/opkg is found and directory /opt/bin is present it logs "Not starting Entware services on <device>, Entware is already started"
2. When a folder entware* is found and it is valid, it is renamed to entware with the log entry "(Alert) Entware folder <folder name> renamed to <device>/entware"
3. When no entware folder is found it logs "(Notice) <device> does not contain Entware, skipping device"

That's all there is to it. Just look for the log entries to figure out why.

 

[CaptnDanLKW]

Thanks @thelonelycoder this is what I needed, and working backwards I realized the source of the issues.
*Self-Inflicted*

In summary, I had tried diversion some time back. For reasons outside this reply, I removed it. In removing it, I also (IIRC) found some cleanup steps, including the removal of addons/diversion, and it's call from post-mount and renamed the 'obsolete_services-start' back to normal. (Since it references changes made by diversion).
I suppose at some point in my 2nd event of this type, I found these entries again, and removed them again. (I restored my jffs structure at some point, thinking that's how they got back in there). Now that my 3rd reinstall, without any presence of Diversion, has the same diversion entries it all makes sense.

In summary, I now realize that diversion entries and references are there to accommodate its future presences or removal.

One idea / recommendation for the entware install process is that if (mountpoint)/entware/bin exists, an option to try to use in-place as is... with the warning that the assumed structure and libraries are all present and valid, for a prompt to re-use vs. clobber. Just a thought.

Thanks for the details, which now seem obvious since they are in your script I can now see, since I didn't delete it this time.

 

[thelonelycoder]

In summary, I now realize that diversion entries and references are there to accommodate its future presences or removal.

Both the amtm and the Diversion Entware installer use the same file paths and names to mount Entware - for simplicity. Diversion uses an extended file which it replaces if the simpler amtm file is found at /jffs/addons/diversion/mount-entware.div.
Both leave a comment in the line they add to /jffs/scripts/post-mount', either # Added by amtm or # Added by Diversion.
The Diversion uninstaller works flawless and leaves no traces of itself on the router, except when one choses to keep the Entware installation - as you found out.

One idea / recommendation for the entware install process is that if (mountpoint)/entware/bin exists, an option to try to use in-place as is... with the warning that the assumed structure and libraries are all present and valid, for a prompt to re-use vs. clobber. Just a thought.

You make a good point. My reasoning at the time of coding for Diversion three years ago was based on the Entware mess that existed at the time.
Now that Diversion or amtm are the official and only way to install Entware on our routers, I can reevaluate the situation.

My local amtm copy now gives the choice to re-use an existing proper Entware installation when found.
This extra code will be copied over to the Diversion installer should testing prove successful.