ASUS RT-N66U USB HDD cross-linking and master file table errors

[Guz]

Hate to say it, ASUS really needs to find someone that knows Samba, ext, NTFS really well. I ran into this with the RT-N56U also (and other issues of not handling hidden and system files properly).

The way I handled it, is I basically use the file sharing as read only. I unmount the drive, take to my computer, copy the files to it (or move them around), then take it back to the router (keeping the drive as NTFS). A royal pain, but at least the files remain healthy.

From observing everyone's issues, I've come to the conclusion that ASUS makes a great hardware kit, but generally drops the ball with their software. The open source firmware guys generally do a better job.

 

[RMerlin]

A few comments based on my own experience.

NTFS access on a Linux-based device is always iffy, because NTFS drivers are based on reverse-engineering and guessing work. There are basically three available solutions for Linux that I am aware of:

-NTFS-3G. Considered fairly reliable, but very slow. This is what Linux distro uses, because it's open sourced
- A commercial solution from Tuexera, the makers of NTFS-3G. I haven't seen anyone use it yet, so I can't really comment on it
- Paragon. Their solution is quite fast, and maybe it's reliable on an x86, but my experiences while developing on the WDLXTV project showed me the same problem as Asus: it's unreliable, and prone to cause filesystem corruption. It could be a problem specific to their Mipsel version, who knows.

Another problem (which might explain why ext2/ext3 users also experience issues) is the fact that this router can crash quite often when running Asus's release firmwares. Combined with Paragon's lack of robustness, this probably lead to a good portion of those corruption issues.

Also, a lot of users will plainly plug/unplug, without properly unmounting (ejecting) the disk first. While this is probably not the case with people reporting issues in this forum (as it's one thing they have looked at), for some people the problem starts right there.

The file corruption probably has nothing to do with Samba. Samba merely interfaces with the filesystem drivers to do its read/write operations.

Finally, it's also possible there is a bug with the USB driver or the hardware implementation. It's unlikely, but nonetheless a possibility.

My recommendations:

1) Use ext2 or ext3. You will probably gain in performance, and definitely gain in reliability.
2) Pay attention to whether your router has a tendency to reboot itself randomly (i.e. crash). If you telnet in, run "uptime" to see how long it's been up and running. It will be obvious if your router reboots a few times per day. If that's the case, either revert to build 112, or use one of my builds (130 RM9 or newer).
3) Always use the eject function from the web interface before unplugging a disk.

For the records, I haven't encountered any corruption issue with the 4 GB flash drive I keep plugged in. But, it's formatted in ext3, and I only use the SMB part for test purposes. It's mostly used for Optware and storing development files. So if that issue only occurs in specific scenarios, that might explain why Asus hasn't run into it yet. Making them aware of the issues might get them to devote more time in testing this specific area tho.

 

[piotrpiano]

Ok, so I recreated the test:
in a folder on NTFS partition (250GB hd) I created two subfolders and in each of them another subfolder. Then I copied there bunch of jpg's and moved around up and down the directory a few times. Then I unmounted the drive and connected to my Win7 laptop.
NO problems whatsoever. Neither did chkdsk find any errors nor did I have any problem accessing the jpg files. Now the disk is attached to rt-n66 again and working as is supposed to.

Sorry I can't confirm your problem.
One important thing to note though: I am currently running Tomato Shibby 095 AIO.

 

[piotrpiano]

Thanks for this... (just been told by ASUS to reflash my unit in recovery mode and if that doesn't help, I can return the unit to the retailer. Well, that is how I got my _112 firmware on after having Tomato Toastman installed which corrupted my drives as well).... How do I now format my hard drive in ext3 format from a Win pc? I see quite a few people had issues with the Ext3 File system driver for Windows 7 and even XP.

There is a free version of EaseUS Partition Master that allows you -amongst other things - to format to ext2 and ext3. Doing it's job pretty well.

 

[Nix]

Long post but ends with question

@ last 3 replies, RMerlin and piotrpiano.... It was coincidence that I went back to the store yesterday to return my unit for a new one thinking that my problem may be unit specific. Jeremy from ASUS actually emailed late yesterday suggesting same thing. So I stood there in the store testing the folder creation thing and moving .jpg files around... and to my delight no problems whatsoever!! I kept re-checking with small and large external drives, FAT32 and NTFS... no cross-linking errors. Then I came home and did one last test ... I copied a large 450 MB .avi file wirelessly, created a subfolder and moved the file into it.... I tried creating a new subfolder and suddenly I get a message I do not have permission to do that, after I had just created a subfolder before. I then unmounted and ran chkdsk.... and there was the cross-linking error... again. Only lucky this time the file does not get corrupted... chkdsk /f corrects the problem without me having to reformat the whole damn drive. This was on .90 firmware... I flashed to .112, did the whole NVRAM clear and factory reboot... and I managed to repeat the problem.... with the same file. Today I tested on a different external HDD.... using a different .avi file..... same problem occured. With .jpg's.. no problems at all.

I tested the previous unit using ext3 and it didn't make a difference. I haven't tested this one though. Thanks piotrpiano for the tip on EaseUS but I already installed Ubuntu VMware, so I can do the whole ext3 thing.

I took home with me a Netgear WNDR4500 to test concurrently. The range and speed does not compare to the RT-N66U and the Netgear DLNA server sux big time. It never finishes indexing... even before and after I tested the same drive on the RT-N66U which indexed like a dream.... HOWEVER, the WNDR4500 file sharing via SAMBA is absolutely rock solid. I threw everything at it and the chkdsk never found a single thing. I especially want the DLNA capability so I guess the WNDR4500 isn't for me either. Life sux in the router world right now. I also thought of doing what 'Guz' does... use the RT-N66U in read-only mode, seeing as the write mode is what causes the problems... maybe ASUS get their stuff together and actually come up with something. I didn't buy the RT-N56U because it also had file sharing problems. I THINK RMerlin IS ONTO SOMETHING THAT THE PROBLEM IS MAYBE THE USB DRIVER... WHY IS THE NETGEAR SAMBA SERVER SO SOLID?????

 

[RMerlin]

Just a thought: before unmounting the disk, try telneting in, and running the following:

sync && sync && sync

Then try unmounting. Just wondering if maybe Asus isn't cleanly unmounting the disk.

 

[Nix]

@RMerlin...thanks, I'll give it a try... can I use the http://192.168.1.1/Main_AdmStatus_Content.asp route instead of finding some telnet agent?

Just a comment... the whole idea of unmounting your drive before switching off the router doesn't hold when you live in an area where we experience regular power cuts.... unannounced. I haven't bought a UPS yet... the WNDR4500 doesn't have that problem.... wouldn't the Shibby, Toastman or your own builds address the unmounting issue? What controls that?

 

[RMerlin]

@RMerlin...thanks, I'll give it a try... can I use the http://192.168.1.1/Main_AdmStatus_Content.asp route instead of finding some telnet agent?

Just a comment... the whole idea of unmounting your drive before switching off the router doesn't hold when you live in an area where we experience regular power cuts.... unannounced. I haven't bought a UPS yet... the WNDR4500 doesn't have that problem.... wouldn't the Shibby, Toastman or your own builds address the unmounting issue? What controls that?

Yes, the web page would work as well.

Having to unmount a drive before switching off any host device (not just a router) is just a fact of life. Hard drive (especially slow devices such as USB flash drives) will rely on buffers to store unwritten data, because writing is slower than the speed at which the CPU can push data. Unwritten data + power loss = certain data loss, and potential filesystem corruption. You haven't encountered such issue with the Netgear device because you never unplugged it while it still had unwritten data in buffer, but that device isn't immune to that either: you must unmount a disk before unplugging it (or turning off the router). Same applies to any third party firmware.

The "sync" command I gave you forces the filesystem to flush out any unwritten data to disk before returning. People typically queue two or three calls for historical reasons I won't delve into.

Using a journaling filesystem such as ext3 will greatly reduce the risks of corruption, as the filesystem uses a journal to know which operation has been queued and potentially not completed yet, so it can reverse incomplete write requests. Instead of data loss + corruption, it will mitigate damage to just data loss.

This is the same reason why you should always use "Safely remove" before unplugging a USB disk in Windows. I have a customer who almost lost all her files as she unplugged her USB disk without safely removing it first - the filesystem got corrupted and unreadable. Fortunately for her I was able to recover the content...

 

[RMerlin]

Another thing I just thought of: is your hard drive using regular 512 bytes sectors, or the newer 4K large sector format? I don't know if Asus supports 4KB sector-sized drives.

 

[Nix]

RMerlin for knighthood

@RMerlin.... firstly, I did check the whole default 512 vs 1k sector size thing last week. It made no difference.

Secondly, I did check the ext3 format as well, using Ubuntu and it made no difference either. I prefer to stay with NTFS or FAT32 because my home theatre has a USB port that only reads those.... Another thing I found strange was I formatted a flash disk to FAT 32 using Ubuntu and the router wouldn't read it... only after I re-formatted it in Win7.

Thirdly.... crescendo.... I think we should rename this thread S&S&S because your tip 4 posts ago works!! I.e.:
Type following URL in new tab/window in Chrome/IE/etc....(depending on gateway IP used for the router) (... or Telnet of course):
http://192.168.1.1/Main_AdmStatus_Content.asp

In the script box type:
sync && sync && sync
end press [Enter]

I stopped just short of throwing the kitchen sink at this one and NO CROSS-LINKING WHATSOEVER! I even executed the script just before simply switching off the router without a dismount. I took out the HDD, plugged it into my pc, ran a chkdsk ... all clear and plugged it back into the router after switching the router back on already. No problems!! If this script can just be run in the background after each write event the file sharing experience would be rock solid (I actually checked if you can continue accessing files after running the script/code...no prob)..... If ASUS write this into their next fw upgrade they should do themselves a big favour.... if you write it into your next custom fw I think you'll have a new customer.... If I could I'd change your status from 'very senior member' to 'Knighted Member'.... Sir, thank you very much. Perhaps the Shibby guys are onto something if Shibby addressed this issue through coding, but I know Toastman didn't. I hope Guz and 2000wolf are following this still.....

 

[RMerlin]

Interesting result. Please do some additional tests when you can, in case the issue is still random (as it seemed to be when you did your first tests with the replacement unit).

When I get the time, I'll check to see if Asus's unmount function on the webui actually takes care of syncing the filesystem. Could be they don't, and if the unmount fails for some reason, you end up unplugging a drive that still has unflushed buffers. If that's the case, inserting a sync call just before the unmounting procedure might help greatly mitigate the issue.

 

[MoogLe]

Interesting result. Please do some additional tests when you can, in case the issue is still random (as it seemed to be when you did your first tests with the replacement unit).

When I get the time, I'll check to see if Asus's unmount function on the webui actually takes care of syncing the filesystem. Could be they don't, and if the unmount fails for some reason, you end up unplugging a drive that still has unflushed buffers. If that's the case, inserting a sync call just before the unmounting procedure might help greatly mitigate the issue.

I'd help with the testing as well but i wont be home for another week or two.

Nix, did you only get problems AFTER unmounting the drive though? Cause I would get cross linking problems even before unmounting the drive.

And in my case my router NEVER crashes! I've had uptimes of months on end! My only problem is power outages that do happen once in a while, but I'm pretty sure those were not the issue in this case.

 

[Nix]

I tested again today on NTFS..... I always make sure I go in with a clean HDD, i.e. I chkdsk it and make sure there are no errors on my pc. If I S&S&S and do an unmount I get no errors (chkdsk detects no 'problems') after subfolder and copy write events.... 3 times in a row. I tested to see what happens if I do just S&S&S without an 'unmount'... I picked up a security descriptor error which chkdsk made a quick meal of. I also tested a 'power cut' after doing no write or subfolder editing... i.e. read-only access only... and chkdsk mumbled something about a security descriptor but didn't call it an 'error'. I tested again and even got a 'no problems' feedback.....

I wish I could switch off the LAN broadcasting whilst the router is still switched on like the WNDR4500 (yes, I know one can schedule it but what if I quickly want to access because I forgot something?). I guess the problems arise when one wants to switch off the router (partially due to security and power consumption) and when one unplugs the USB HDD to be used somewhere else. I now know to S&S&S and unmount before switching off the router or removing HDD and..... always make sure you S&S&S after a write event (copy to, subfolder creation/move, etc...). Because if you can at least save the data via chkdsk after a power outage it's a great deal better than dealing with corrupted files.

Next test: to see how ext3 responds under power outage events....

@MoogLe: The previous router that I exchanged for this one did have the problem of cross-linking without unmounting.... this one I have now behaves much better. I put it down to a 'faulty unit'.... my current unit is pretty predictable regarding the S&S&S issue. I suggest you run S&S&S every time after folder management and write.... I'm a nube here but I wonder if you do a fw upgrade at any stage without doing a NVRAM and/or factory reboot if you don't start picking up problems down the line.... ??

 

[RMerlin]

I'll add a sync() call before the unmount code in the firmware for the next version. Certainly can't hurt, and might improve reliability.

 

[Nix]

... tested with ext3... I was put off with the previous unit (which I returned because it was deemed faulty) but I must say I can confirm on this unit that ext3 is more stable. I powered off and back on the router (simulating a power cut).... and the file system remained 'clean'. I even did some file management.... without S&S&S and it remained stable. The management was jerky though.... it seems ext3 simply doesn't allow certain actions unless it is ready... there seems to be a lag... and you have to be patient. For example, if I try move an .avi file it sometimes first tells me win explorer is busy playing it and can't access, which is not the case... but if you wait a while you can continue with the action. With NTFS it is less jerky ... actually seamless but then you have to S&S&S otherwise you get cross-linking.

 

[MoogLe]

... tested with ext3... I was put off with the previous unit (which I returned because it was deemed faulty) but I must say I can confirm on this unit that ext3 is more stable. I powered off and back on the router (simulating a power cut).... and the file system remained 'clean'. I even did some file management.... without S&S&S and it remained stable. The management was jerky though.... it seems ext3 simply doesn't allow certain actions unless it is ready... there seems to be a lag... and you have to be patient. For example, if I try move an .avi file it sometimes first tells me win explorer is busy playing it and can't access, which is not the case... but if you wait a while you can continue with the action. With NTFS it is less jerky ... actually seamless but then you have to S&S&S otherwise you get cross-linking.

sooo its looking like i need to RMA my unit then

Nix, what kinda speed are you getting with file transfers? Cause i couldnt manage more than 6-7 mb/sec.

which is terrible compared to 20+mb/sec with my NAS

 

[Nix]

I get 1.5-2 Mbytes/s write and 5-6 Mbytes/s read... slightly slower than reported by piotrpiano . I'm on ASUS .112 fw though. The Tomato and RMerlin fw are quicker... on fw .90 I was a whole lot quicker .. I once managed a write up to 3.8 Mbytes/s.

 

[MoogLe]

I get 1.5-2 Mbytes/s write and 5-6 Mbytes/s read... slightly slower than reported by piotrpiano . I'm on ASUS .112 fw though. The Tomato and RMerlin fw are quicker... on fw .90 I was a whole lot quicker .. I once managed a write up to 3.8 Mbytes/s.

yea.... that's pretty freakin terrible dont know if its even worth bothering for.

 

[thewul]

I know it seems weird that I'm replying to my own thread but there really are issues I'm experiencing and no love from ASUS support. If the SAMBA server managed to corrupt a file or two due to the problem mentioned in this thread relating to cross-linking and MFT in FAT32 and NTFS external USB HDD's the DLNA media server gets stuck at that corrupted file, i.e. it stops indexing right there and goes into some kind of a 'scanning' loop as its staus gets locked there.

Late response, I agree. That is because I have my router just a week now. Same problem here as well. When a USB thumb drive of say 8GB or so, FAT formatted, no problem.
I went on to connect a 1TB NTFS drive - copied a bunch of file to that drive.
Asus went on scanning.. scanning.. and I could not assign R/W authorization to a user: it constantly jumped back to 'No'.
Gave it a few tries, resets, but gave up.
Using latest firmware 178.

Maybe create a separate fat32-partition?

 

[RMerlin]

What was scanning, the DLNA server, or a filesystem check?

Try copying your files with DLNA and iTunes servers disabled at first.

And always use the safely remove option from the webui before turning off or unplugging the hard disk, otherwise you will cause corruption - not the router's fault, just how external disks work in general.