AC88U 384.18 USB mounted quite late during boot - Entware stuck waiting

  • ATTENTION! As of November 1, 2020, you are not able to reply to threads 6 months after the thread is opened if there are more than 500 posts in the thread.
    Threads will not be locked, so posts may still be edited by their authors.
    Just start a new thread on the topic to post if you get an error message when trying to reply to a thread.

Phantomski

Occasional Visitor
Hi,

this question might be maybe obvious for some of you seasoned Entware / USB drive pros, I’ve just only noticed yesterday, so wanted to clarify if the following is a normal / designed behaviour, if it poses any potential risks and if there are any ways how to improve it.

Setup
I’m using RT-AC88U with Merlin’s 384.18 firmware. I have an USB 2.0 8GB thumb drive plugged in, formatted with ext4 with Journalling / 2GB swap file and used for nothing else but few amtm installed and regularly updated / Entware addons (Skynet, connmon, spdMerlin). I am sending the System log to my own external logging environment (Graylog / ElasticSearch) for analysis.

Behaviour
- I have noticed the USB drive comes online quite late in the boot process, after all (or most of) the other services get started
- One particular made me worried a little - Skynet. For significant amount of time, before and after the USB gets mounted, NTP synchronised, Entware started and Skynet initialised, I’m not 100% about the Router’s Firewall status and protection level. I have noticed the System Log / General shows kernel: DROP IN= messages in this interim period, which are common for the built in Firewall (i.e. no Skynet installed) dropped inbounds. After some time (after USB mounts and Skynet starts), those disappear(!) from the Router’s log (even the already generated ones) and Skynet’s logs start appearing as kernel: [BLOCKED - INBOUND] IN=. I noticed that only because the logs already sent to Graylog are saved this way, the Router’s webUI just magically deletes all those entries.
- I can see from the log, all three addons are patiently waiting for USB / Entware with the 1/10 10sec attempts.
Rich (BB code):
May  5 06:05:29 kernel: scsi 0:0:0:0: Direct-Access     SMI      USB DISK         1100 PQ: 0 ANSI: 0 CCS
May  5 06:05:29 kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
May  5 06:05:29 kernel: sd 0:0:0:0: [sda] 15974400 512-byte logical blocks: (8.17 GB/7.61 GiB)
May  5 06:05:29 kernel: sd 0:0:0:0: [sda] Write Protect is off
May  5 06:05:29 kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
May  5 06:05:29 kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
May  5 06:05:29 kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
May  5 06:05:29 kernel: sd 0:0:0:0: [sda] Attached SCSI removable disk
Dec  3 21:35:04 spdMerlin: NTP synced, spdMerlin will now continue
Dec  3 21:35:04 connmon: NTP synced, connmon will now continue
Dec  3 21:35:04 spdMerlin: Entware not found, sleeping for 10s (attempt 1 of 10)
Dec  3 21:35:04 connmon: Entware not found, sleeping for 10s (attempt 1 of 10)
Dec  3 21:35:04 Skynet: [*] USB Not Found - Sleeping For 10 Seconds ( Attempt 1 Of 10 )
Dec  3 21:35:07 Entware: Starting Entware services on /tmp/mnt/ASUS_Router
Dec  3 21:35:14 connmon: Entware found, connmon will now continue
Dec  3 21:35:14 spdMerlin: Entware found, spdMerlin will now continue
Dec  3 21:35:24 connmon: Mounting connmon WebUI page as user1.asp
Dec  3 21:35:29 spdMerlin: Mounting spdMerlin WebUI page as user2.asp
- Sometimes the whole process is little bit more complicated - USB takes much longer, Entware addons start only on 2nd or 3rd attempt
- Skynet’s eventual successful start usually doesn’t show in the log until the next hourly statistics, so I could only see it’s running via SSH/webUI. What is less ideal, kernel: [BLOCKED - INBOUND] IN= messages sometimes don’t appear in the log either, even though number of blocked IPs appear in the hourly Skynet’s statistics. /tmp/mnt/ASUS_Router/skynet/skynet.log has the entries correctly from 21:27:18 (strangely even before USB comes up?), so most likely it’s running, just not writing to System Log.

Is this all normal / expected / not to worry or am I missing something? Can it be optimised? Am I protected in the “uncertainty period” ? Is there a way how to check the behaviour in the meantime, for example via SSH?

Thanks everyone.
 
Last edited:

Mutzli

Very Senior Member
I had similar issues before with USB drives that are not fully supported under a Linux environment or are about to die. I used to have the best success with a SanDisk 32GB Ultra Fit USB 3.1 Flash Drive - SDCZ430-032G-G46. Now I have changed to an external SSD drive since I had several SATA III drives laying around from computers that have been retired but the drives were still like new. This is the USB 3.1 gen. 2 enclosure I have connected to my AX88U routers = UGREEN 2.5" Hard Drive Enclosure, USB C 3.1 Gen 2 to SATA III 6Gbps. Make sure that your USB enclosure comes with an ASMedia chipset if you go this route. It works best under Linux
 

Phantomski

Occasional Visitor
I had similar issues before with USB drives that are not fully supported under a Linux environment or are about to die.

Thanks Mutzli.

I’ve spent some time dealing with similar problems trying to use NVMe SSDs with Raspberry Pi 4, which magically disappeared after I’ve plugged them via powered USB HUB - in other words power issues. I somehow didn’t expect small thumb drive would pose any problems, but truth is as you said - there are many quite bad Linux compatibility issues with some manufacturers, with anything - SD cards, USB drives, USB/SATA or USB/PCIe adapters.

Btw I’m happy with UGREEN as well, using the PCIe/NVMe version, but also on ASMedia chipset. Another one from Startech (the cable version) works great too, also ASMedia.

Thanks for the tips, will look into it.
 

thecheapseats

Regular Contributor
<snip>...on ASMedia chipset. Another one from Startech (the cable version) works great too, also ASMedia.
just saw your post re: nvme on Rpi... Yes, go asmedia chipsets or go home for Rpi... have you seen james chamber's blog regarding usb-storage quirks and UAS mode issues for Rpi?... a good hardware reference...
 

Phantomski

Occasional Visitor
just saw your post re: nvme on Rpi... Yes, go asmedia chipsets or go home for Rpi... have you seen james chamber's blog regarding usb-storage quirks and UAS mode issues for Rpi?... a good hardware reference...

Yep, seen that one, that’s how I started, all hopeful and optimistic ;). But despite best efforts, even with a good chipset, I was still getting data errors. The problem was the NVMe drive + enclosure, they tend to use a LOT of power. While the USB chain on rPi4 should be capable of 1.2A, even if I had this device connected as the only USB device, it still wasn’t enough. The highest I’ve measured on the cable was about 1.05A, which should be enough, but either there were short spikes that have been higher, but not measured, or my rPi4 doesn’t have the voltage regulator up to standard (not unheard of).

All the problems disappeared after I’ve used the external powered USB hub. With the added benefit of SSD enclosure being noticeably cooler (NVMe drives can get ridiculously hot) and more ports available.

This is eventually running like clockwork for me:
- Samsung 970 EVO Plus 500 GB PCIe NVMe M.2 (2280)
- UGREEN M.2 NVMe SSD Enclosure USB C External PCIe M2 SSD Caddy Reader Type C 3.1 Gen2 Thunderbolt 3 M+B M-Key NGFF 2280 2260
- Anker 10 Port 60W Data Hub with 7 USB 3.0 Ports and 3 PowerIQ Charging Ports

For SATA version, I have good experience with these:
- StarTech.com USB 3.1 to 2.5" SATA Hard Drive Adapter - USB 3.1 Gen 2 10Gbps with UASP External HDD/SSD Storage Converter (USB312SAT3CB)
- v Crucial MX500 500 GB CT500MX500SSD1(Z)
In my opinion best price/performance/reliability combination.

Yes, both the NMVe drive and enclosure are bit over the top for rPi4’s USB 3.1 Gen1 5Gbps ports, but I had my reasons ;) (other computers I use it with and higher Samsung’s TBW for my server needs).

The UGREEN NVMe enclosure is running on ASMedia ASM2362 and StarTech SATA is running on ASMedia ASM1351 - both very reliable options and crucially capable of both UASP and TRIM, without the need for quirks. For some reason both Realtek and JMicron are not doing well in the USB / SATA / NVMe space on Linux.

I’ve meant to update James on the findings in more detail, but other stuff came along ;)
 

thecheapseats

Regular Contributor
Yep, seen that one, that’s how I started, all hopeful and optimistic...<snip>... All the problems disappeared after I’ve used the external powered USB hub.

good - you've been there/done that - I love these goofy little Rpi4 boards... and yes, I found for nvme, injecting seperate power was absolutely necessary for that ugreen m.2 nvme enclosure - although I did a physical soldered breakout-board solution to seperate the usb-c data and power lines to avoid dealing with a hub - but that's me... I hate usb-c for a lot of reasons - mostly bad power implementation (pull up and sense resistors) by mfgs....

new pi4 compute boards with pcie slot will change-up everything with m.2 nvme...

agree that m.2 nvme is over the top for an Rpi4 (I use m.2 sata(s) on each of several pi4 boards) but like yourself, it's convenient to have that flexible physical interface capability functioning long before you 'need' it...

just a heads-up - after confirming one of the (pricey) startech m.2 ssd cables worked, I switched to those absurdly cheap (appx $7) off brands with the same asmedia chipsets - they give the same performance... I assumed they would be junk but was pleasantly surprised.
 
Last edited:

Phantomski

Occasional Visitor
good - you've been there/done that - I love these goofy little Rpi4 boards...
Oh big time. I’m running Docker on Ubuntu server now, with tons of containers for all the home network needs and it performs really well. Home Assistant, NextCloud, influxDB, Grafana, Graylog, ElasticSearch, MongoDB, MariaDB, Node-RED, you name it. Perfect little thing hums nicely still with enough CPU and memory to spare.. The only problem in the current Docker world is that sometimes Arm64v8 builds are not available...yet. But that will change, a lot can be re-built as well.

and yes, I found for nvme, injecting seperate power was absolutely necessary for that ugreen m.2 nvme enclosure - although I did a physical soldered breakout-board solution to seperate the usb-c data and power lines to avoid dealing with a hub - but that's me... I hate usb-c for a lot of reasons - mostly bad power implementation (pull up and sense resistors) by mfgs....
Well done for making it yourself. For a moment I was thinking about just ripping the cable apart, but then couldn’t be asked ;) The USB-C is a real nightmare. Everyone sells crap cables (finding good one able to reach 40Gbit/100W is sometimes detective work) and what gets sometimes negotiated as PD doesn’t make any sense. Right when I’d thing the USB world is getting consolidated. lol.

new pi4 compute boards with pcie slot will change-up everything with m.2 nvme...
Can’t wait. So easy to chuck them anywhere with that size and still able to run desktop/server class things on them.

just a heads-up - after confirming one of the (pricey) startech m.2 ssd cables worked, I switched to those absurdly cheap (appx $7) off brands with the same asmedia chipsets - they give the same performance... I assumed they would be junk but was pleasantly surprised.
I had some ridiculous experiences with China’s produce recently (fake CPUs and the lot), so I’m a bit picky to be honest. But thanks for the tip, I’ll have a look. Plenty of old SATA drives lying around. It’s great someone‘s testing them and keeping the list!
 

thecheapseats

Regular Contributor
Docker world/containers - agree... time will fix.... arm stuff has a big horizon...

usb-c/still hate it... but love 'real' thunderbolt3 on big motherbd builds - because it just plain screams...

compute boards/pi4 and other sbc(s) - great, these goofy gadgets... small investment - big return and damn near disposable...

let me look -> 'eluteng' ssd cables on amazon... be sure to run lsusb to confirn asmedia chipset outta' the box-o-china lottery... cheap, runs cool - need 1 piece? - buy 2... two is one, one is none... later...
 

Similar threads

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top