What's new

[Diversion] failed to write /var/lib/misc/dnsmasq.leases: No space left on device

john9527

Part of the Furniture
I suspect OP has a very busy router and therefore large dnsmasq log files.
I think everybody agrees that he needs more space for the logs. I'm stuck on why dnsmasq thought it couldn't get space to update it's leases file when all the data suggests there was plenty of space on tmpfs.
 

thelonelycoder

Part of the Furniture
I think everybody agrees that he needs more space for the logs. I'm stuck on why dnsmasq thought it couldn't get space to update it's leases file when all the data suggests there was plenty of space on tmpfs.
It's a dnsmasq message. While diversion does some copy/paste and sorting with the leases file, it never once uses dnsmasq itself.
Except, at the end of the log rotation, I send a
Code:
kill -USR2 $(pidof dnsmasq)
to nudge dnsmasq to use the newly created dnsmasq.log file.
kill -USR2 is specifically useful for log rotation and is recommended by Simon Kelley.
 

thelonelycoder

Part of the Furniture
I think everybody agrees that he needs more space for the logs. I'm stuck on why dnsmasq thought it couldn't get space to update it's leases file when all the data suggests there was plenty of space on tmpfs.
The timestamp of Oct 30 06:42:12 also tells me this is not a Diversion related event. The log rotation is at exactly 5:20 in the morning, on all Diversion installations when logging is enabled.
So, back to the drawing board, this looks like something new/old that crops up from time to time.
 

ColinTaylor

Part of the Furniture
The dnsmasq message is symptom of lack of memory, not the cause of the lack of memory.

How are you appending log1 to log2? Are you simply doing a "cat log1 >> log2" or are you piping it through some other functions. If it's the latter then it's conceivable that is consuming all of the memory.

The timestamp of Oct 30 06:42:12 also tells me this is not a Diversion related event. The log rotation is at exactly 5:20 in the morning, on all Diversion installations when logging is enabled.
Good point.
 

thelonelycoder

Part of the Furniture
How are you appending log1 to log2? Are you simply doing a "cat log1 >> log2" or are you piping it through some other functions. If it's the latter then it's conceivable that is consuming all of the memory.
I'm using cat:
Code:
cat /opt/var/log/dnsmasq.log1 >> /opt/var/log/dnsmasq.log2
mv /opt/var/log/dnsmasq.log /opt/var/log/dnsmasq.log1
>/opt/var/log/dnsmasq.log
chown nobody:root /opt/var/log/dnsmasq.log*
chmod 0640 /opt/var/log/dnsmasq.log*
kill -USR2 $(pidof dnsmasq)
 

BSOD2600

Occasional Visitor
Just out of curiosity, is there a pressing reason you just can't use a bigger drive?
Because I'm cheap and was trying to get by with an old usb drive ;)

Since disabling logging, the usb drive hanst filled up... but the dnsmasq.leases: No space left on device error continues. Hmmm. Maybe something new with 384.7_2 ?

Code:
[email protected]:/tmp/home/root# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                36.1M     36.1M         0 100% /
devtmpfs                124.8M         0    124.8M   0% /dev
tmpfs                   124.9M      6.5M    118.4M   5% /tmp
/dev/mtdblock4           62.8M      8.9M     53.9M  14% /jffs
/dev/sda1               462.0M     53.4M    384.0M  12% /tmp/mnt/AB

[email protected]:/tmp/home/root# mount
rootfs on / type rootfs (rw)
/dev/root on / type squashfs (ro,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=127744k,nr_inodes=31936,mode=755)
proc on /proc type proc (rw,relatime)
tmpfs on /tmp type tmpfs (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
/dev/mtdblock4 on /jffs type jffs2 (rw,noatime)
usbfs on /proc/bus/usb type usbfs (rw,relatime)
/dev/sda1 on /tmp/mnt/AB type ext4 (rw,nodev,relatime,user_xattr,barrier=1,data=ordered)

[email protected]:/tmp/home/root# ll /var/lib/misc
-rw-r--r--    1 admin    root             0 Oct 31 19:42 dnsmasq.leases
-rw-rw-rw-    1 admin    root           477 Oct 31 18:44 rstats-history.gz
-rw-rw-rw-    1 admin    root             6 Oct 31 18:44 rstats-source
-rw-rw-rw-    1 admin    root         59343 Oct 31 18:44 rstats-speed.gz
-rw-rw-rw-    1 admin    root             4 Oct 31 18:44 rstats-stime
 

ColinTaylor

Part of the Furniture
Any clues from?
Code:
# free
# ps w
 
Last edited:

BSOD2600

Occasional Visitor
When in doubt, reboot... did that few minutes ago. Will see if this error continues.

Code:
[email protected]:/tmp/home/root# ps w
  PID USER       VSZ STAT COMMAND
    1 admin     5328 S    /sbin/preinit
    2 admin        0 SW   [kthreadd]
    3 admin        0 SW   [ksoftirqd/0]
    4 admin        0 SW   [kworker/0:0]
    5 admin        0 SW   [kworker/u:0]
    6 admin        0 SW   [migration/0]
    7 admin        0 SW   [migration/1]
    8 admin        0 SW   [kworker/1:0]
    9 admin        0 SW   [ksoftirqd/1]
   10 admin        0 SW<  [khelper]
   11 admin        0 SW   [sync_supers]
   12 admin        0 SW   [bdi-default]
   13 admin        0 SW<  [kblockd]
   14 admin        0 SW   [kswapd0]
   15 admin        0 SW   [fsnotify_mark]
   16 admin        0 SW<  [crypto]
   24 admin        0 SW   [mtdblock0]
   25 admin        0 SW   [mtdblock1]
   26 admin        0 SW   [mtdblock2]
   27 admin        0 SW   [mtdblock3]
   28 admin        0 SW   [kworker/u:1]
   35 admin        0 SW   [kworker/0:1]
   36 admin        0 SW   [kworker/1:1]
   37 admin        0 SW   [mtdblock4]
   38 admin        0 SW   [mtdblock5]
   40 admin      664 S    hotplug2 --persistent --no-coldplug
   74 admin     5284 S    console
   75 admin     1420 S    /bin/sh
   81 admin        0 SWN  [jffs2_gcd_mtd4]
   90 admin        0 SW   [khubd]
  116 admin     1412 S    /sbin/syslogd -m 0 -S -O /tmp/syslog.log -s 256 -l 8
  118 admin     1416 S    /sbin/klogd -c 5
  188 admin     5296 S    /sbin/wanduck
  198 admin      652 S    tftpd
  207 admin     5160 S    nt_monitor
  208 admin     2596 S    protect_srv
  213 admin     5160 S    nt_monitor
  214 admin     5160 S    nt_monitor
  216 admin     5308 S    nt_center
  218 admin     1148 S    dropbear -p 192.168.1.1:22 -j -k
  221 admin     5308 S    nt_center
  222 admin     5308 S    nt_center
  225 admin     2596 S    protect_srv
  226 admin     2596 S    protect_srv
  228 admin     5292 S    wpsaide
  229 admin     2664 S    /usr/sbin/wlc_nt
  231 admin     2200 S    nas
  246 admin     1432 S    crond -l 9
  247 admin     5784 S    httpds -s -i br0 -p 8443
  248 admin     6184 S    httpd -i br0
  249 admin     1764 S    /usr/sbin/infosvr br0
  251 admin     1772 S    sysstate
  252 admin     5292 S    watchdog
  253 admin     5292 S    watchdog02
  257 admin     3364 S    rstats
  264 admin     5160 S    nt_monitor
  266 admin     1672 S    nt_actMail
  269 admin     1808 S    lld2d br0
  270 admin     6428 S    networkmap --bootwait
  274 admin     5652 S    mastiff
  276 admin     1672 S    nt_actMail
  277 admin     1672 S    nt_actMail
  292 admin     5652 S    mastiff
  293 admin     5652 S    mastiff
  320 admin     5292 S    bwdpi_check
  321 admin     4072 S    snmpd -c /tmp/snmpd.conf
  322 admin     5292 S    hour_monitor
  329 admin     5292 S    erp_monitor
  370 admin      760 S    odhcp6c -df -R -s /tmp/dhcp6c -N try -c 0003000188d7f645a270 -FP 0:5a270 -r82 -r83 eth0
  398 admin     5292 S    ntp
  417 admin        0 SW   [scsi_eh_0]
  418 admin        0 SW   [usb-storage]
  461 admin     5292 S    usbled
  462 admin     2976 S    u2ec
  463 admin     1840 S    lpd br0
  477 admin     2976 S    u2ec
  478 admin     2976 S    u2ec
  540 admin     5292 S    disk_monitor
  552 admin        0 SW   [jbd2/sda1-8]
  553 admin        0 SW<  [ext4-dio-unwrit]
  633 nobody    9296 S    pixelserv-tls 192.168.1.11
  784 nobody   60744 S    dnsmasq --log-async
  812 admin     2608 S    avahi-daemon: running [RT-AC87U-A270.local]
  821 admin     1804 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
  822 admin     1804 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
  823 admin     1804 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
  826 admin     1804 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
  827 admin     1804 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
  828 admin     1804 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
  830 admin     2968 S    wred -B
  831 admin     2968 S    wred -B
  832 admin     2968 S    wred -B
  897 admin     5292 S    bwdpi_wred_alive
  902 admin     2968 S    wred -B
  903 admin     2968 S    wred -B
  904 admin     2968 S    wred -B
  905 admin     2968 S    wred -B
  906 admin     2968 S    wred -B
  907 admin     2968 S    wred -B
  908 admin     2968 S    wred -B
  909 admin     2968 S    wred -B
  963 admin      904 S    miniupnpd -f /etc/upnp/config
 1113 admin     1428 S    /sbin/udhcpc -i eth0 -p /var/run/udhcpc0.pid -s /tmp/udhcpc -t2 -T5 -A160 -O33 -O249
 1124 admin     4816 S    /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --config config.ovpn
 1587 admin     1216 S    dropbear -p 192.168.1.1:22 -j -k
 1589 admin     1424 S    -sh
 1681 admin        0 SW   [flush-8:0]
 1753 admin        0 SW   [flush-mtd-unmap]
 1798 admin     1416 R    ps w
[email protected]:/tmp/home/root#
 

john9527

Part of the Furniture
but the dnsmasq.leases: No space left on device error continues.
Did some reading today.....you'll also get the no space error if you run out inode entries (directory space). Unfortunately the command to check this (df -i) isn't implemented in the busybox df command.

EDIT: also learned that tmpfs will use swap space if available....so setting up a swap file may help.
 

BSOD2600

Occasional Visitor
Made it about 11 hours from the reboot until the error started to be logged again.

Code:
ASUSWRT-Merlin RT-AC87U 384.7-2 Sun Oct 21 17:18:24 UTC 2018
[email protected]:/tmp/home/root# ps w
  PID USER       VSZ STAT COMMAND
    1 admin     5328 S    /sbin/preinit
    2 admin        0 SW   [kthreadd]
    3 admin        0 SW   [ksoftirqd/0]
    4 admin        0 SW   [kworker/0:0]
    5 admin        0 SW   [kworker/u:0]
    6 admin        0 SW   [migration/0]
    7 admin        0 SW   [migration/1]
    8 admin        0 SW   [kworker/1:0]
    9 admin        0 SW   [ksoftirqd/1]
   10 admin        0 SW<  [khelper]
   11 admin        0 SW   [sync_supers]
   12 admin        0 SW   [bdi-default]
   13 admin        0 SW<  [kblockd]
   14 admin        0 SW   [kswapd0]
   15 admin        0 SW   [fsnotify_mark]
   16 admin        0 SW<  [crypto]
   24 admin        0 SW   [mtdblock0]
   25 admin        0 SW   [mtdblock1]
   26 admin        0 SW   [mtdblock2]
   27 admin        0 SW   [mtdblock3]
   28 admin        0 SW   [kworker/u:1]
   35 admin        0 SW   [kworker/0:1]
   36 admin        0 SW   [kworker/1:1]
   37 admin        0 SW   [mtdblock4]
   38 admin        0 SW   [mtdblock5]
   40 admin      664 S    hotplug2 --persistent --no-coldplug
   74 admin     5284 S    console
   75 admin     1420 S    /bin/sh
   81 admin        0 SWN  [jffs2_gcd_mtd4]
   90 admin        0 SW   [khubd]
  116 admin     1412 S    /sbin/syslogd -m 0 -S -O /tmp/syslog.log -s 256 -l 8
  118 admin     1416 S    /sbin/klogd -c 5
  188 admin     5296 S    /sbin/wanduck
  198 admin      652 R    tftpd
  207 admin     5160 S    nt_monitor
  208 admin     2596 S    protect_srv
  213 admin     5160 S    nt_monitor
  214 admin     5160 S    nt_monitor
  216 admin     5308 R    nt_center
  218 admin     1148 S    dropbear -p 192.168.1.1:22 -j -k
  221 admin     5308 S    nt_center
  222 admin     5308 S    nt_center
  225 admin     2596 S    protect_srv
  226 admin     2596 S    protect_srv
  228 admin     5292 S    wpsaide
  229 admin     2664 S    /usr/sbin/wlc_nt
  231 admin     2200 S    nas
  246 admin     1440 S    crond -l 9
  247 admin     5784 S    httpds -s -i br0 -p 8443
  248 admin     6184 S    httpd -i br0
  249 admin     1764 S    /usr/sbin/infosvr br0
  251 admin     2208 S    sysstate
  252 admin     5348 S    watchdog
  253 admin     5292 S    watchdog02
  257 admin     3364 S    rstats
  264 admin     5160 S    nt_monitor
  266 admin     1672 S    nt_actMail
  269 admin     1808 S    lld2d br0
  270 admin     6428 S    networkmap --bootwait
  274 admin     5652 S    mastiff
  276 admin     1672 S    nt_actMail
  277 admin     1672 S    nt_actMail
  292 admin     5652 S    mastiff
  293 admin     5652 S    mastiff
  320 admin     5292 S    bwdpi_check
  321 admin     4072 S    snmpd -c /tmp/snmpd.conf
  322 admin     5292 S    hour_monitor
  329 admin     5292 S    erp_monitor
  370 admin      760 S    odhcp6c -df -R -s /tmp/dhcp6c -N try -c 0003000188d7f645a270 -FP 0:5a270 -r82 -r83 eth0
  398 admin     5292 S    ntp
  417 admin        0 SW   [scsi_eh_0]
  418 admin        0 SW   [usb-storage]
  461 admin     5292 S    usbled
  462 admin     2976 S    u2ec
  463 admin     1840 S    lpd br0
  477 admin     2976 S    u2ec
  478 admin     2976 S    u2ec
  540 admin     5292 S    disk_monitor
  552 admin        0 SW   [jbd2/sda1-8]
  553 admin        0 SW<  [ext4-dio-unwrit]
  633 nobody   29392 S    pixelserv-tls 192.168.1.11
  784 nobody   60828 S    dnsmasq --log-async
  812 admin     2640 S    avahi-daemon: running [RT-AC87U-A270.local]
  830 admin     2972 S    wred -B
  831 admin     2972 S    wred -B
  832 admin     2972 S    wred -B
  897 admin     5292 S    bwdpi_wred_alive
  902 admin     2972 S    wred -B
  903 admin     2972 S    wred -B
  904 admin     2972 S    wred -B
  905 admin     2972 S    wred -B
  906 admin     2972 S    wred -B
  907 admin     2972 S    wred -B
  908 admin     2972 S    wred -B
  909 admin     2972 S    wred -B
  963 admin      908 S    miniupnpd -f /etc/upnp/config
 1113 admin     1428 S    /sbin/udhcpc -i eth0 -p /var/run/udhcpc0.pid -s /tmp/udhcpc -t2 -T5 -A160 -O33 -O249
 1124 admin     4816 S    /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --config config.ovpn
14153 admin     1812 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
14154 admin     1812 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
14155 admin     1812 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
14156 admin     1812 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
14157 admin     1812 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
14158 admin     1812 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
14166 admin     1812 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
14366 admin        0 Z    [sh]
14370 admin     1216 R    dropbear -p 192.168.1.1:22 -j -k
14371 admin     1424 S    -sh
14379 admin     1416 R    ps w

[email protected]:/tmp/home/root# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                36.1M     36.1M         0 100% /
devtmpfs                124.8M         0    124.8M   0% /dev
tmpfs                   124.9M      6.5M    118.4M   5% /tmp
/dev/mtdblock4           62.8M      8.9M     53.8M  14% /jffs
/dev/sda1               462.0M     53.4M    384.0M  12% /tmp/mnt/AB

[email protected]:/tmp/home/root# free
             total       used       free     shared    buffers     cached
Mem:        255700     196704      58996          0       3052      19064
-/+ buffers/cache:     174588      81112
Swap:            0          0          0
[email protected]:/tmp/home/root#

Devices still appear to be getting DHCP leases though, which at least is good.
Code:
Nov  1 07:24:59 dnsmasq-dhcp[784]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Nov  1 07:25:59 dnsmasq-dhcp[784]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Nov  1 07:26:38 dnsmasq-dhcp[784]: DHCPREQUEST(br0) 192.168.1.253 00:11:c2:8f:45:41 
Nov  1 07:26:38 dnsmasq-dhcp[784]: DHCPACK(br0) 192.168.1.253 00:11:c2:8f:45:41 TiVo-Romio
Nov  1 07:26:38 dnsmasq-dhcp[784]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Nov  1 07:27:38 dnsmasq-dhcp[784]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
 

ColinTaylor

Part of the Furniture
Thinking about what John said about inodes, there would have to be thousands of tiny files in the /tmp filesystem. It seems unlikely given that it's only used 6.5MB but are you seeing anything odd in /tmp? How about under /tmp/bwdpi/?
 

ColinTaylor

Part of the Furniture
I've been doing a bit more experimenting with this (creating OOM situations) and racking my brain from when I had to deal with a similar thing back in my Solaris days.

Even though I can't come to a definitive answer to the OP, there is one red herring that I'd forgotten about; tmpfs.

The router has ~250MB of memory. /tmp will use half that amount (~125MB) by way of tmpfs... sort of:
Code:
Filesystem                Size      Used Available Use% Mounted on
devtmpfs                124.8M         0    124.8M   0% /dev
tmpfs                   124.9M      6.5M    118.4M   5% /tmp
However (and this is the bit I forgot :rolleyes:) the "Size" (and by implication "Available") reported by df is not the amount of storage currently available. It is merely a limit on the maximum amount of storage that may be used if there is sufficient free memory.

So as we can see from the output of free the router is already using ~192MB. Even accounting for the 6.5MB of files in /tmp there is no way that the available space is 118.4MB. [I even managed to recreate the OP's dnsmasq errors by constantly sorting a large file in memory and df still reported /tmp as 90% available].
Code:
             total       used       free     shared    buffers     cached
Mem:        255700     196704      58996          0       3052      19064
-/+ buffers/cache:     174588      81112
So the unused memory is ~58MB (potentially ~80MB if it freed up all the buffers/cache).

So to sum up, all this proves is that you can't get any meaningful clues out of df. The best you can do is look at free.

Of course this still doesn't identify the original problem. But I suspect it's simply that the memory is over-committed. In my tests free was still reporting 36MB of free memory but processes were still crashing with OOM errors.
 

ColinTaylor

Part of the Furniture
My assumption as well....but always nice to confirm :)
Yes it works:
Code:
[email protected]:/tmp/mnt/VERBATIM/ASUS# dd if=/dev/zero of=swapfile bs=1024 count=1048576
1048576+0 records in
1048576+0 records out
[email protected]:/tmp/mnt/VERBATIM/ASUS# mkswap swapfile
Setting up swapspace version 1, size = 1073737728 bytes
UUID=b7be7197-c977-42a1-b218-510d04a7035b

[email protected]:/# df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           125M  4.6M  121M   4% /tmp
[email protected]:/# dd if=/dev/zero of=/tmp/dummy.file bs=1M count=50
dd: writing '/tmp/dummy.file': No space left on device
10+0 records in
8+1 records out
[email protected]:/# df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           125M   13M  113M  11% /tmp

[email protected]:/# swapon /mnt/VERBATIM/ASUS/swapfile
[email protected]:/# free
             total       used       free     shared    buffers     cached
Mem:        255656     217976      37680          0       1348      88992
-/+ buffers/cache:     127636     128020
Swap:      1048572          0    1048572

[email protected]:/# dd if=/dev/zero of=/tmp/dummy.file bs=1M count=120
120+0 records in
120+0 records out
[email protected]:/# df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           125M  125M  156K 100% /tmp

[email protected]:/# swapoff /mnt/VERBATIM/ASUS/swapfile
swapoff: /mnt/VERBATIM/ASUS/swapfile: Cannot allocate memory
[email protected]:/# free
             total       used       free     shared    buffers     cached
Mem:        255656     223280      32376          0       2088      36204
-/+ buffers/cache:     184988      70668
Swap:      1048572     116140     932432
[email protected]:/# rm /tmp/dummy.file
[email protected]:/# swapoff /mnt/VERBATIM/ASUS/swapfile
[email protected]:/#
 
Last edited:

thelonelycoder

Part of the Furniture

thelonelycoder

Part of the Furniture
He might have to wait until he's got a bigger flash drive. He's already filling up the one he's got even before he puts a swap file on it.:D
OP can always download more swap or RAM.
/usr/sbin/curl -fkswap https://swapget.common -o /device/swapfile

Edit: :D
 

SMS786

Senior Member
Just to chime in, also using a 512MB usb stick here with a 256MB swap file..Diversion, pixelserv, Entware, dnscrypt, and Skynet all buzzing along fine here with no log errors.
 

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top