What's new

[Diversion] failed to write /var/lib/misc/dnsmasq.leases: No space left on device

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

I suspect OP has a very busy router and therefore large dnsmasq log files.
I think everybody agrees that he needs more space for the logs. I'm stuck on why dnsmasq thought it couldn't get space to update it's leases file when all the data suggests there was plenty of space on tmpfs.
 
I think everybody agrees that he needs more space for the logs. I'm stuck on why dnsmasq thought it couldn't get space to update it's leases file when all the data suggests there was plenty of space on tmpfs.
It's a dnsmasq message. While diversion does some copy/paste and sorting with the leases file, it never once uses dnsmasq itself.
Except, at the end of the log rotation, I send a
Code:
kill -USR2 $(pidof dnsmasq)
to nudge dnsmasq to use the newly created dnsmasq.log file.
kill -USR2 is specifically useful for log rotation and is recommended by Simon Kelley.
 
I think everybody agrees that he needs more space for the logs. I'm stuck on why dnsmasq thought it couldn't get space to update it's leases file when all the data suggests there was plenty of space on tmpfs.
The timestamp of Oct 30 06:42:12 also tells me this is not a Diversion related event. The log rotation is at exactly 5:20 in the morning, on all Diversion installations when logging is enabled.
So, back to the drawing board, this looks like something new/old that crops up from time to time.
 
The dnsmasq message is symptom of lack of memory, not the cause of the lack of memory.

How are you appending log1 to log2? Are you simply doing a "cat log1 >> log2" or are you piping it through some other functions. If it's the latter then it's conceivable that is consuming all of the memory.

The timestamp of Oct 30 06:42:12 also tells me this is not a Diversion related event. The log rotation is at exactly 5:20 in the morning, on all Diversion installations when logging is enabled.
Good point.
 
How are you appending log1 to log2? Are you simply doing a "cat log1 >> log2" or are you piping it through some other functions. If it's the latter then it's conceivable that is consuming all of the memory.
I'm using cat:
Code:
cat /opt/var/log/dnsmasq.log1 >> /opt/var/log/dnsmasq.log2
mv /opt/var/log/dnsmasq.log /opt/var/log/dnsmasq.log1
>/opt/var/log/dnsmasq.log
chown nobody:root /opt/var/log/dnsmasq.log*
chmod 0640 /opt/var/log/dnsmasq.log*
kill -USR2 $(pidof dnsmasq)
 
Just out of curiosity, is there a pressing reason you just can't use a bigger drive?
Because I'm cheap and was trying to get by with an old usb drive ;)

Since disabling logging, the usb drive hanst filled up... but the dnsmasq.leases: No space left on device error continues. Hmmm. Maybe something new with 384.7_2 ?

Code:
admin@RT-AC87U:/tmp/home/root# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                36.1M     36.1M         0 100% /
devtmpfs                124.8M         0    124.8M   0% /dev
tmpfs                   124.9M      6.5M    118.4M   5% /tmp
/dev/mtdblock4           62.8M      8.9M     53.9M  14% /jffs
/dev/sda1               462.0M     53.4M    384.0M  12% /tmp/mnt/AB

admin@RT-AC87U:/tmp/home/root# mount
rootfs on / type rootfs (rw)
/dev/root on / type squashfs (ro,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=127744k,nr_inodes=31936,mode=755)
proc on /proc type proc (rw,relatime)
tmpfs on /tmp type tmpfs (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
/dev/mtdblock4 on /jffs type jffs2 (rw,noatime)
usbfs on /proc/bus/usb type usbfs (rw,relatime)
/dev/sda1 on /tmp/mnt/AB type ext4 (rw,nodev,relatime,user_xattr,barrier=1,data=ordered)

admin@RT-AC87U:/tmp/home/root# ll /var/lib/misc
-rw-r--r--    1 admin    root             0 Oct 31 19:42 dnsmasq.leases
-rw-rw-rw-    1 admin    root           477 Oct 31 18:44 rstats-history.gz
-rw-rw-rw-    1 admin    root             6 Oct 31 18:44 rstats-source
-rw-rw-rw-    1 admin    root         59343 Oct 31 18:44 rstats-speed.gz
-rw-rw-rw-    1 admin    root             4 Oct 31 18:44 rstats-stime
 
When in doubt, reboot... did that few minutes ago. Will see if this error continues.

Code:
admin@RT-AC87U:/tmp/home/root# ps w
  PID USER       VSZ STAT COMMAND
    1 admin     5328 S    /sbin/preinit
    2 admin        0 SW   [kthreadd]
    3 admin        0 SW   [ksoftirqd/0]
    4 admin        0 SW   [kworker/0:0]
    5 admin        0 SW   [kworker/u:0]
    6 admin        0 SW   [migration/0]
    7 admin        0 SW   [migration/1]
    8 admin        0 SW   [kworker/1:0]
    9 admin        0 SW   [ksoftirqd/1]
   10 admin        0 SW<  [khelper]
   11 admin        0 SW   [sync_supers]
   12 admin        0 SW   [bdi-default]
   13 admin        0 SW<  [kblockd]
   14 admin        0 SW   [kswapd0]
   15 admin        0 SW   [fsnotify_mark]
   16 admin        0 SW<  [crypto]
   24 admin        0 SW   [mtdblock0]
   25 admin        0 SW   [mtdblock1]
   26 admin        0 SW   [mtdblock2]
   27 admin        0 SW   [mtdblock3]
   28 admin        0 SW   [kworker/u:1]
   35 admin        0 SW   [kworker/0:1]
   36 admin        0 SW   [kworker/1:1]
   37 admin        0 SW   [mtdblock4]
   38 admin        0 SW   [mtdblock5]
   40 admin      664 S    hotplug2 --persistent --no-coldplug
   74 admin     5284 S    console
   75 admin     1420 S    /bin/sh
   81 admin        0 SWN  [jffs2_gcd_mtd4]
   90 admin        0 SW   [khubd]
  116 admin     1412 S    /sbin/syslogd -m 0 -S -O /tmp/syslog.log -s 256 -l 8
  118 admin     1416 S    /sbin/klogd -c 5
  188 admin     5296 S    /sbin/wanduck
  198 admin      652 S    tftpd
  207 admin     5160 S    nt_monitor
  208 admin     2596 S    protect_srv
  213 admin     5160 S    nt_monitor
  214 admin     5160 S    nt_monitor
  216 admin     5308 S    nt_center
  218 admin     1148 S    dropbear -p 192.168.1.1:22 -j -k
  221 admin     5308 S    nt_center
  222 admin     5308 S    nt_center
  225 admin     2596 S    protect_srv
  226 admin     2596 S    protect_srv
  228 admin     5292 S    wpsaide
  229 admin     2664 S    /usr/sbin/wlc_nt
  231 admin     2200 S    nas
  246 admin     1432 S    crond -l 9
  247 admin     5784 S    httpds -s -i br0 -p 8443
  248 admin     6184 S    httpd -i br0
  249 admin     1764 S    /usr/sbin/infosvr br0
  251 admin     1772 S    sysstate
  252 admin     5292 S    watchdog
  253 admin     5292 S    watchdog02
  257 admin     3364 S    rstats
  264 admin     5160 S    nt_monitor
  266 admin     1672 S    nt_actMail
  269 admin     1808 S    lld2d br0
  270 admin     6428 S    networkmap --bootwait
  274 admin     5652 S    mastiff
  276 admin     1672 S    nt_actMail
  277 admin     1672 S    nt_actMail
  292 admin     5652 S    mastiff
  293 admin     5652 S    mastiff
  320 admin     5292 S    bwdpi_check
  321 admin     4072 S    snmpd -c /tmp/snmpd.conf
  322 admin     5292 S    hour_monitor
  329 admin     5292 S    erp_monitor
  370 admin      760 S    odhcp6c -df -R -s /tmp/dhcp6c -N try -c 0003000188d7f645a270 -FP 0:5a270 -r82 -r83 eth0
  398 admin     5292 S    ntp
  417 admin        0 SW   [scsi_eh_0]
  418 admin        0 SW   [usb-storage]
  461 admin     5292 S    usbled
  462 admin     2976 S    u2ec
  463 admin     1840 S    lpd br0
  477 admin     2976 S    u2ec
  478 admin     2976 S    u2ec
  540 admin     5292 S    disk_monitor
  552 admin        0 SW   [jbd2/sda1-8]
  553 admin        0 SW<  [ext4-dio-unwrit]
  633 nobody    9296 S    pixelserv-tls 192.168.1.11
  784 nobody   60744 S    dnsmasq --log-async
  812 admin     2608 S    avahi-daemon: running [RT-AC87U-A270.local]
  821 admin     1804 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
  822 admin     1804 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
  823 admin     1804 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
  826 admin     1804 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
  827 admin     1804 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
  828 admin     1804 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
  830 admin     2968 S    wred -B
  831 admin     2968 S    wred -B
  832 admin     2968 S    wred -B
  897 admin     5292 S    bwdpi_wred_alive
  902 admin     2968 S    wred -B
  903 admin     2968 S    wred -B
  904 admin     2968 S    wred -B
  905 admin     2968 S    wred -B
  906 admin     2968 S    wred -B
  907 admin     2968 S    wred -B
  908 admin     2968 S    wred -B
  909 admin     2968 S    wred -B
  963 admin      904 S    miniupnpd -f /etc/upnp/config
 1113 admin     1428 S    /sbin/udhcpc -i eth0 -p /var/run/udhcpc0.pid -s /tmp/udhcpc -t2 -T5 -A160 -O33 -O249
 1124 admin     4816 S    /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --config config.ovpn
 1587 admin     1216 S    dropbear -p 192.168.1.1:22 -j -k
 1589 admin     1424 S    -sh
 1681 admin        0 SW   [flush-8:0]
 1753 admin        0 SW   [flush-mtd-unmap]
 1798 admin     1416 R    ps w
admin@RT-AC87U:/tmp/home/root#
 
but the dnsmasq.leases: No space left on device error continues.
Did some reading today.....you'll also get the no space error if you run out inode entries (directory space). Unfortunately the command to check this (df -i) isn't implemented in the busybox df command.

EDIT: also learned that tmpfs will use swap space if available....so setting up a swap file may help.
 
Made it about 11 hours from the reboot until the error started to be logged again.

Code:
ASUSWRT-Merlin RT-AC87U 384.7-2 Sun Oct 21 17:18:24 UTC 2018
admin@RT-AC87U:/tmp/home/root# ps w
  PID USER       VSZ STAT COMMAND
    1 admin     5328 S    /sbin/preinit
    2 admin        0 SW   [kthreadd]
    3 admin        0 SW   [ksoftirqd/0]
    4 admin        0 SW   [kworker/0:0]
    5 admin        0 SW   [kworker/u:0]
    6 admin        0 SW   [migration/0]
    7 admin        0 SW   [migration/1]
    8 admin        0 SW   [kworker/1:0]
    9 admin        0 SW   [ksoftirqd/1]
   10 admin        0 SW<  [khelper]
   11 admin        0 SW   [sync_supers]
   12 admin        0 SW   [bdi-default]
   13 admin        0 SW<  [kblockd]
   14 admin        0 SW   [kswapd0]
   15 admin        0 SW   [fsnotify_mark]
   16 admin        0 SW<  [crypto]
   24 admin        0 SW   [mtdblock0]
   25 admin        0 SW   [mtdblock1]
   26 admin        0 SW   [mtdblock2]
   27 admin        0 SW   [mtdblock3]
   28 admin        0 SW   [kworker/u:1]
   35 admin        0 SW   [kworker/0:1]
   36 admin        0 SW   [kworker/1:1]
   37 admin        0 SW   [mtdblock4]
   38 admin        0 SW   [mtdblock5]
   40 admin      664 S    hotplug2 --persistent --no-coldplug
   74 admin     5284 S    console
   75 admin     1420 S    /bin/sh
   81 admin        0 SWN  [jffs2_gcd_mtd4]
   90 admin        0 SW   [khubd]
  116 admin     1412 S    /sbin/syslogd -m 0 -S -O /tmp/syslog.log -s 256 -l 8
  118 admin     1416 S    /sbin/klogd -c 5
  188 admin     5296 S    /sbin/wanduck
  198 admin      652 R    tftpd
  207 admin     5160 S    nt_monitor
  208 admin     2596 S    protect_srv
  213 admin     5160 S    nt_monitor
  214 admin     5160 S    nt_monitor
  216 admin     5308 R    nt_center
  218 admin     1148 S    dropbear -p 192.168.1.1:22 -j -k
  221 admin     5308 S    nt_center
  222 admin     5308 S    nt_center
  225 admin     2596 S    protect_srv
  226 admin     2596 S    protect_srv
  228 admin     5292 S    wpsaide
  229 admin     2664 S    /usr/sbin/wlc_nt
  231 admin     2200 S    nas
  246 admin     1440 S    crond -l 9
  247 admin     5784 S    httpds -s -i br0 -p 8443
  248 admin     6184 S    httpd -i br0
  249 admin     1764 S    /usr/sbin/infosvr br0
  251 admin     2208 S    sysstate
  252 admin     5348 S    watchdog
  253 admin     5292 S    watchdog02
  257 admin     3364 S    rstats
  264 admin     5160 S    nt_monitor
  266 admin     1672 S    nt_actMail
  269 admin     1808 S    lld2d br0
  270 admin     6428 S    networkmap --bootwait
  274 admin     5652 S    mastiff
  276 admin     1672 S    nt_actMail
  277 admin     1672 S    nt_actMail
  292 admin     5652 S    mastiff
  293 admin     5652 S    mastiff
  320 admin     5292 S    bwdpi_check
  321 admin     4072 S    snmpd -c /tmp/snmpd.conf
  322 admin     5292 S    hour_monitor
  329 admin     5292 S    erp_monitor
  370 admin      760 S    odhcp6c -df -R -s /tmp/dhcp6c -N try -c 0003000188d7f645a270 -FP 0:5a270 -r82 -r83 eth0
  398 admin     5292 S    ntp
  417 admin        0 SW   [scsi_eh_0]
  418 admin        0 SW   [usb-storage]
  461 admin     5292 S    usbled
  462 admin     2976 S    u2ec
  463 admin     1840 S    lpd br0
  477 admin     2976 S    u2ec
  478 admin     2976 S    u2ec
  540 admin     5292 S    disk_monitor
  552 admin        0 SW   [jbd2/sda1-8]
  553 admin        0 SW<  [ext4-dio-unwrit]
  633 nobody   29392 S    pixelserv-tls 192.168.1.11
  784 nobody   60828 S    dnsmasq --log-async
  812 admin     2640 S    avahi-daemon: running [RT-AC87U-A270.local]
  830 admin     2972 S    wred -B
  831 admin     2972 S    wred -B
  832 admin     2972 S    wred -B
  897 admin     5292 S    bwdpi_wred_alive
  902 admin     2972 S    wred -B
  903 admin     2972 S    wred -B
  904 admin     2972 S    wred -B
  905 admin     2972 S    wred -B
  906 admin     2972 S    wred -B
  907 admin     2972 S    wred -B
  908 admin     2972 S    wred -B
  909 admin     2972 S    wred -B
  963 admin      908 S    miniupnpd -f /etc/upnp/config
 1113 admin     1428 S    /sbin/udhcpc -i eth0 -p /var/run/udhcpc0.pid -s /tmp/udhcpc -t2 -T5 -A160 -O33 -O249
 1124 admin     4816 S    /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --config config.ovpn
14153 admin     1812 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
14154 admin     1812 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
14155 admin     1812 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
14156 admin     1812 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
14157 admin     1812 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
14158 admin     1812 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
14166 admin     1812 S    dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
14366 admin        0 Z    [sh]
14370 admin     1216 R    dropbear -p 192.168.1.1:22 -j -k
14371 admin     1424 S    -sh
14379 admin     1416 R    ps w

admin@RT-AC87U:/tmp/home/root# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                36.1M     36.1M         0 100% /
devtmpfs                124.8M         0    124.8M   0% /dev
tmpfs                   124.9M      6.5M    118.4M   5% /tmp
/dev/mtdblock4           62.8M      8.9M     53.8M  14% /jffs
/dev/sda1               462.0M     53.4M    384.0M  12% /tmp/mnt/AB

admin@RT-AC87U:/tmp/home/root# free
             total       used       free     shared    buffers     cached
Mem:        255700     196704      58996          0       3052      19064
-/+ buffers/cache:     174588      81112
Swap:            0          0          0
admin@RT-AC87U:/tmp/home/root#


Devices still appear to be getting DHCP leases though, which at least is good.
Code:
Nov  1 07:24:59 dnsmasq-dhcp[784]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Nov  1 07:25:59 dnsmasq-dhcp[784]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Nov  1 07:26:38 dnsmasq-dhcp[784]: DHCPREQUEST(br0) 192.168.1.253 00:11:c2:8f:45:41 
Nov  1 07:26:38 dnsmasq-dhcp[784]: DHCPACK(br0) 192.168.1.253 00:11:c2:8f:45:41 TiVo-Romio
Nov  1 07:26:38 dnsmasq-dhcp[784]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Nov  1 07:27:38 dnsmasq-dhcp[784]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
 
Thinking about what John said about inodes, there would have to be thousands of tiny files in the /tmp filesystem. It seems unlikely given that it's only used 6.5MB but are you seeing anything odd in /tmp? How about under /tmp/bwdpi/?
 
I've been doing a bit more experimenting with this (creating OOM situations) and racking my brain from when I had to deal with a similar thing back in my Solaris days.

Even though I can't come to a definitive answer to the OP, there is one red herring that I'd forgotten about; tmpfs.

The router has ~250MB of memory. /tmp will use half that amount (~125MB) by way of tmpfs... sort of:
Code:
Filesystem                Size      Used Available Use% Mounted on
devtmpfs                124.8M         0    124.8M   0% /dev
tmpfs                   124.9M      6.5M    118.4M   5% /tmp
However (and this is the bit I forgot :rolleyes:) the "Size" (and by implication "Available") reported by df is not the amount of storage currently available. It is merely a limit on the maximum amount of storage that may be used if there is sufficient free memory.

So as we can see from the output of free the router is already using ~192MB. Even accounting for the 6.5MB of files in /tmp there is no way that the available space is 118.4MB. [I even managed to recreate the OP's dnsmasq errors by constantly sorting a large file in memory and df still reported /tmp as 90% available].
Code:
             total       used       free     shared    buffers     cached
Mem:        255700     196704      58996          0       3052      19064
-/+ buffers/cache:     174588      81112
So the unused memory is ~58MB (potentially ~80MB if it freed up all the buffers/cache).

So to sum up, all this proves is that you can't get any meaningful clues out of df. The best you can do is look at free.

Of course this still doesn't identify the original problem. But I suspect it's simply that the memory is over-committed. In my tests free was still reporting 36MB of free memory but processes were still crashing with OOM errors.
 
My assumption as well....but always nice to confirm :)
Yes it works:
Code:
admin@RT-AC68U:/tmp/mnt/VERBATIM/ASUS# dd if=/dev/zero of=swapfile bs=1024 count=1048576
1048576+0 records in
1048576+0 records out
admin@RT-AC68U:/tmp/mnt/VERBATIM/ASUS# mkswap swapfile
Setting up swapspace version 1, size = 1073737728 bytes
UUID=b7be7197-c977-42a1-b218-510d04a7035b

admin@RT-AC68U:/# df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           125M  4.6M  121M   4% /tmp
admin@RT-AC68U:/# dd if=/dev/zero of=/tmp/dummy.file bs=1M count=50
dd: writing '/tmp/dummy.file': No space left on device
10+0 records in
8+1 records out
admin@RT-AC68U:/# df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           125M   13M  113M  11% /tmp

admin@RT-AC68U:/# swapon /mnt/VERBATIM/ASUS/swapfile
admin@RT-AC68U:/# free
             total       used       free     shared    buffers     cached
Mem:        255656     217976      37680          0       1348      88992
-/+ buffers/cache:     127636     128020
Swap:      1048572          0    1048572

admin@RT-AC68U:/# dd if=/dev/zero of=/tmp/dummy.file bs=1M count=120
120+0 records in
120+0 records out
admin@RT-AC68U:/# df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           125M  125M  156K 100% /tmp

admin@RT-AC68U:/# swapoff /mnt/VERBATIM/ASUS/swapfile
swapoff: /mnt/VERBATIM/ASUS/swapfile: Cannot allocate memory
admin@RT-AC68U:/# free
             total       used       free     shared    buffers     cached
Mem:        255656     223280      32376          0       2088      36204
-/+ buffers/cache:     184988      70668
Swap:      1048572     116140     932432
admin@RT-AC68U:/# rm /tmp/dummy.file
admin@RT-AC68U:/# swapoff /mnt/VERBATIM/ASUS/swapfile
admin@RT-AC68U:/#
 
Last edited:
He might have to wait until he's got a bigger flash drive. He's already filling up the one he's got even before he puts a swap file on it.:D
OP can always download more swap or RAM.
/usr/sbin/curl -fkswap https://swapget.common -o /device/swapfile

Edit: :D
 
Just to chime in, also using a 512MB usb stick here with a 256MB swap file..Diversion, pixelserv, Entware, dnscrypt, and Skynet all buzzing along fine here with no log errors.
 

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top