Why isn't /jffs cleaned out after formatting?

[M@rco]

I'm preparing to do a completely clean install of 384.7 final. I disconnected my USB drives after unmounting them through the UI and choose to 'Format/jffs at next reboot' through the UI as well before installing 384.7. I made sure I applied the setting to format /jffs at next reboot. Before rebooting this is the output of du and df:

# du /jffs
363     /jffs/bin
6       /jffs/ssl
326     /jffs/.sys/nc
326     /jffs/.sys
3       /jffs/configs
192     /jffs/scripts
14      /jffs/openvpn
0       /jffs/usericon
1190    /jffs

# df /jffs
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mtdblock4           64256      2200     62056   3% /jffs

Then I reboot to actually format /jffs. After the router is back up again, this is the output of du and df:

# du /jffs
0       /jffs

# df /jffs
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                34816     34816         0 100% /

I believe this is the same issue I've encountered before (and I think @martinr as well, see this thread) where writing to /jffs fails after first reboot, which makes sense as it looks it isn't properly mounted, I guess? There's no space left to write anything anyway.

So I reboot again and to my surprise, it looks like everything is back again/still there:

# du /jffs
363     /jffs/bin
6       /jffs/ssl
327     /jffs/.sys/nc
327     /jffs/.sys
3       /jffs/configs
192     /jffs/scripts
14      /jffs/openvpn
0       /jffs/usericon
1200    /jffs

# df /jffs
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mtdblock4           64256      2216     62040   3% /jffs

Looking at the contents of /jffs, it's been either repopulated or was never wiped in the first place:

# ls
bin                         shared-Diversion-whitelist
cfg.json                    shared-Skynet-whitelist
configs                     shared-Skynet2-whitelist
nmp_cl_json.js              ssl
nmp_client_list             syslog.log
openvpn                     syslog.log-1
scripts                     usericon

There are no USB devices attached, I'm (still) on 384.7 beta 3 - but this has been the case on earlier builds as well. I'd like to know whether this is a bug and whether there's a way to manually format /jffs without messing things up?

Your assistance is highly appreciated.

 

[M@rco]

Just went through the steps above again twice and the result is the same. Don't know if this helps but I noticed that on first reboot (when /jffs is supposed to be formatted) the UI won't refresh automatically, it stays at 100% and I get a certificate error when refreshing manually, because apparently the certificate can't be loaded. Also, when trying to SSH in, Putty gives me the 'Potential security breach' warning, saying that the host's fingerprint has changed. When rebooting though (when /jffs gets repopulated), no security or fingerprint errors anymore.

After first reboot, /jffs appears empty but it's mounted read-only:

# cd /jffs

# l s -alF
drwxr-xr-x    2 marco    root             3 Oct  1 00:11 ./
drwxr-xr-x   17 marco    root           320 Oct  1 00:11 ../

# touch /jffs/caniwritehere.txt
touch: /jffs/caniwritehere.txt: Read-only file system

(extra spaces added because of CF apparently trippin' over l+s)

However, when rebooting another time, it's writable again but as mentioned above, it's repopulated.

 

[M@rco]

I tried formatting /jffs twice in a row with reboots in between, to see if that would make a difference, but to no avail. It stays read only after the first and second reboot with the format on next boot option active and after the third reboot everything is back again or still there. Can I just rm -rf /jffs to wipe it completely? Or is there a more proper way to really format it manually?

 

[Xentrk]

Not sure if this is exactly the same issue. Try rebooting twice in a row according to this post:

https://github.com/RMerl/asuswrt-merlin.ng/issues/95

<snip>
In the past some users reported that they sometime needed to reboot a second time after formatting, for the partition to get remounted correctly. That's probably why the reflash fixed it for you - it generated a second reboot. There's nothing in the flash process that should affect the jjffs partition.

 

[Xentrk]

This is what I have for nvram values with an active jffs:

 nvram show | grep jffs
size: 67898 bytes (63174 left)
jffs2_on=1
jffs2_exec=
jffs2_enable=1
jffs2_size=67108864
jffs2_format=0
jffs2_scripts=1

Look at the settings after you Applied the setting on the web page where you instruct the firmware to format at next boot.

 

[M@rco]

Not sure if this is exactly the same issue. Try rebooting twice in a row according to this post:

Yes, I read that on the wiki as well. That's most likely the first issue I run into. After formatting it by changing the option 'Format /jffs at next reboot', the first time /jffs is read-only and looking at its size (see second output in first post) not mounted correctly.

I even tried setting 'Format /jffs at next reboot' twice in a row, with reboots in between, but it won't change a thing. It's still being repopulated at the next reboot.

 

[M@rco]

This is what I have for nvram values with an active jffs:

 nvram show | grep jffs
size: 67898 bytes (63174 left)
jffs2_on=1
jffs2_exec=
jffs2_enable=1
jffs2_size=67108864
jffs2_format=0
jffs2_scripts=1

Look at the settings after you Applied the setting on the web page where you instruct the firmware to format at next boot.

When executing that, it outputs:

#  nvram show | grep jffs
jffs2_on=1
jffs2_exec=
jffs2_enable=1
size: 55421 bytes (10115 left)
jffs2_format=0
jffs2_scripts=1

That size doesn't look right, I guess?

Edit: above results are from /jffs after being repopulated and being writable.

 

[Xentrk]

When executing that, it outputs:

#  nvram show | grep jffs
jffs2_on=1
jffs2_exec=
jffs2_enable=1
size: 55421 bytes (10115 left)
jffs2_format=0
jffs2_scripts=1

That size doesn't look right, I guess?

Edit: above results are from /jffs after being repopulated and being writable.

I suspect the size difference may be due to the router model and the AC88U having more disk space than the AC68U.

 

[M@rco]

I suspect the size difference may be due to the router model and the AC88U having more disk space than the AC68U.

Yes, but still... ~10KB left doesn't seem much? Not sure whether this is related though.

 

[skeal]

Stupid question. Have you tried enabling jffs support and format jffs at the same time, then reboot?

 

[skeal]

The reason I ask is I did a reset. Not initiate, but reset in the webui and lost jffs.

 

[M@rco]

Stupid question. Have you tried enabling jffs support and format jffs at the same time, then reboot?

Support for custom scripts was (and still is) enabled during toggling the option to format /jffs on next reboot. If I recall correctly it's now on by default.

 

[ColinTaylor]

M@rco Can you post the output from the following command:

# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00080000 00020000 "boot"
mtd1: 00180000 00020000 "nvram"
mtd2: 03e00000 00020000 "linux"
mtd3: 03c5cf28 00020000 "rootfs"
mtd4: 03ec0000 00020000 "brcmnand"
mtd5: 00140000 00020000 "asus"

You could try this:

# nvram set jffs2_enable=1
# nvram set jffs2_format=1
# nvram unset jffs2_size
# nvram commit
# reboot

And then reboot again for a second time after about 5 minutes.

But I suspect this might be a bug in jffs2.c.

 

[M@rco]

The reason I ask is I did a reset. Not initiate, but reset in the webui and lost jffs.

I can't do a reset right now (kid's home) but I will try later today. However, as even 'Initialize' doesn't fully wipe /jffs, I have a feeling it will still have data after resetting. You say you 'lost /jffs'... was it erased? Or did it mount only after another reboot?

 

[M@rco]

@M@rco Can you post the output from the following command:

Here's the output:

# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00080000 00020000 "boot"
mtd1: 00180000 00020000 "nvram"
mtd2: 03e00000 00020000 "linux"
mtd3: 03c6240c 00020000 "rootfs"
mtd4: 03ec0000 00020000 "brcmnand"
mtd5: 00140000 00020000 "asus"

You could try this:

# nvram set jffs2_enable=1
# nvram set jffs2_format=1
# nvram unset jffs2_size
# nvram commit
# reboot

And then reboot again for a second time after about 5 minutes.

I will try your suggestion later today when junior hits the sack and report back.

 

[ColinTaylor]

I will try your suggestion later today when junior hits the sack and report back.

Thanks for the output, I just wanted to check that the partition labels were correct (which they are).

The difference between formatting the jffs and erasing it is that the first case uses "mtd_erase" while the second merely uses "rm". The end result should effectively be the same, but the first option ought to work.

 

[skeal]

I know this sucks but when you get time a reset to defaults by using reset cleared everything I had in /jffs. Wait until you have the most recent screen shots and such saved. Then when time permits take the plunge.

 

[M@rco]

I know this sucks but when you get time a reset to defaults by using reset cleared everything I had in /jffs. Wait until you have the most recent screen shots and such saved. Then when time permits take the plunge.

I already backup'd everything and made screenshots with FF this morning, so it's no issue. I'm wondering though whether a reset would solve this, as resetting only clears nvram if I'm not mistaking? Initialize does the same and wipes some files of /jffs, but not format / wipe it entirely, if I'm correct?

 

[skeal]

I already backup'd everything and made screenshots with FF this morning, so it's no issue. I'm wondering though whether a reset would solve this, as resetting only clears nvram if I'm not mistaking? Initialize does the same and wipes some files of /jffs, but not format / wipe it entirely, if I'm correct?

I was surprised to lose everything in /jffs, when choosing reset. It did inspire me to reinstall all my scripts, which was likely a good idea.

 

[M@rco]

You could try this:

# nvram set jffs2_enable=1
# nvram set jffs2_format=1
# nvram unset jffs2_size
# nvram commit
# reboot

And then reboot again for a second time after about 5 minutes.

I followed your instructions, but after the second reboot, everything is back again:

# df /jffs
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mtdblock4           64256      2236     62020   3% /jffs

# du /jffs
363     /jffs/bin
6       /jffs/ssl
337     /jffs/.sys/nc
337     /jffs/.sys
3       /jffs/configs
192     /jffs/scripts
14      /jffs/openvpn
0       /jffs/usericon
1157    /jffs

# l s -alF /jffs
drwxr-xr-x   10 admin    root             0 Oct 11 20:34 ./
drwxr-xr-x   17 admin    root           320 Oct  7 19:33 ../
-rw-rw-rw-    1 admin    root          6368 Oct 11 20:33 .ash_history
drwxrwxrwx    3 admin    root             0 May  5 07:05 .sys/
drwxrwxrwx    2 admin    root             0 Aug 31 23:03 bin/
-rw-r--r--    1 admin    root            54 Sep 18 13:29 cfg.json
drwxr-xr-x    2 admin    root             0 Oct  6 08:04 configs/
-rw-r--r--    1 admin    root          2108 Oct 11 20:32 nmp_cl_json.js
-rw-rw-rw-    1 admin    root           504 Oct 11 20:32 nmp_client_list
drwx------    2 admin    root             0 Oct  1 22:32 openvpn/
drwxr-xr-x    2 admin    root             0 Oct  6 08:02 scripts/
-rw-rw-rw-    1 admin    root           230 Oct  8 01:00 shared-Diversion-whitelist
-rw-rw-rw-    1 admin    root          1623 Oct 11 02:25 shared-Skynet-whitelist
-rw-rw-rw-    1 admin    root           204 Oct 11 20:31 shared-Skynet2-whitelist
drw-------    2 admin    root             0 Sep 30 17:00 ssl/
-rw-rw-rw-    1 admin    root         58784 Oct 11 20:34 syslog.log
-rw-rw-rw-    1 admin    root        177961 Oct 11 20:34 syslog.log-1
drwxr-xr-x    2 admin    root             0 Oct 11 20:31 usericon/

Any other suggestions? Can the /jffs content listed above be removed safely (perhaps not syslog, but's that's no problem)?

But I suspect this might be a bug in jffs2.c

If this is indeed a bug in jffs2.c, hopefully @RMerlin is willing to look into this.