DNScrypt Help debugging dnscrypt-proxy2 crashes

  • ATTENTION! You'll notice a Prefix dropdown when you create a thread. If your post applies to one of the topics listed, please use that Prefix for your post. When browsing the thread list you can use the Prefix to filter the view.
  • ATTENTION! As of November 1, 2020, you are not able to reply to threads 6 months after the thread is opened if there are more than 500 posts in the thread.
    Threads will not be locked, so posts may still be edited by their authors.
    Just start a new thread on the topic to post if you get an error message when trying to reply to a thread.

sbsnb

Very Senior Member
Just got a new RT-AX86U yesterday. Flashed 386.2_6 and configured all manually. Having trouble with dnscrypt_proxy2 crashing every couple of hours with no log entries. Ran the same version on my RT-AC88U for months at a time with no crashes. Same version, same config file. I don't even know how to start debugging the cause. I have 14 days left to figure out if this router has to go back.
 

sbsnb

Very Senior Member
The only thing from the logs around the general time of the last crash that even looks suspicious is:

Code:
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 HWA3b TxFIFO: TX FIFO-69 shadow is empty
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 wl1: wlc_bmac_dma_getnexttxp Could not reclaim data dd(hwa) fifo=5
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 HWA4a TxSTAT: hwa_txstat_reclaim: reinit<1> stall<0>
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 HWA1b RxFILL: mac_counter_status sat<0> need_post<0> aval<0>
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 HWA1b RxFILL: reclaim core<0> 0 rxbuffers RD<0> WR<0>
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 HWA1a RxPOST: wi_cnt<0> in hwa internal memory<RD:0 WR:0>
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 wl1: wlc_bmac_wowlucode_start mctrl=0x4020402  0x4000400
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 wl1: wlc_bmac_wowlucode_start (mctrl | MCTL_PSM_JMP_0)=0x4020406 0x4020406
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 wl1: wlc_bmac_wowlucode_start mctrl=0x4020402 0x4020402
Jul 22 16:57:27 kernel: CONSOLE: 198694.655 wl1: CORE INIT : mode 4 pktclassify 1 rxsplit 1  hdr conversion 1 DMA_CT Enabled
Jul 22 16:57:27 kernel: CONSOLE: 198694.674 wlc_mutx_active_update vasip mu_supported_Ntx 4
Jul 22 16:57:27 kernel: CONSOLE: 198694.674 HWA1a RxPOST: H2D RxPost ring: id<1> type<0> item_type<1> max_items<1024> len_item<8>
Jul 22 16:57:27 kernel: CONSOLE: 198694.674 HWA1a RxPOST: item_size 8 CWI32 config parser<00000000> format<0> size<8>
Jul 22 16:57:27 kernel: CONSOLE: 198694.674 HWA1a RxPOST: rxpost_data_buf_len<1836>
Jul 22 16:57:27 kernel: CONSOLE: 198694.676 wl1: wlc_enable_probe_req: state down, deferring setting of host flags
Jul 22 16:57:27 kernel: CONSOLE: 198694.676 CSIMON: already initialized ...
Jul 22 16:57:27 kernel: CONSOLE: 198694.677 wl1 enable 1: q0 frmcnt 0, wrdcnt 0, q1 frmcnt 0, wrdcnt 0
Jul 22 16:57:27 kernel: CONSOLE: 198694.677 wl1: wlc_phy_cal_cache_restore_acphy: Chanspec 0xe09b found, but not valid in phycal cache
Jul 22 16:57:27 kernel: CONSOLE: 198694.759 wl1: link up (wl1)
Jul 22 16:57:27 kernel: CONSOLE: 198694.759 wl1: link up (wl1.3)
 

SomeWhereOverTheRainBow

Part of the Furniture
Just got a new RT-AX86U yesterday. Flashed 386.2_6 and configured all manually. Having trouble with dnscrypt_proxy2 crashing every couple of hours with no log entries. Ran the same version on my RT-AC88U for months at a time with no crashes. Same version, same config file. I don't even know how to start debugging the cause. I have 14 days left to figure out if this router has to go back.
are you using entwares dnscrypt proxy ? or the one installed with AMTM.
 

SomeWhereOverTheRainBow

Part of the Furniture
The only thing from the logs around the general time of the last crash that even looks suspicious is:

Code:
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 HWA3b TxFIFO: TX FIFO-69 shadow is empty
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 wl1: wlc_bmac_dma_getnexttxp Could not reclaim data dd(hwa) fifo=5
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 HWA4a TxSTAT: hwa_txstat_reclaim: reinit<1> stall<0>
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 HWA1b RxFILL: mac_counter_status sat<0> need_post<0> aval<0>
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 HWA1b RxFILL: reclaim core<0> 0 rxbuffers RD<0> WR<0>
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 HWA1a RxPOST: wi_cnt<0> in hwa internal memory<RD:0 WR:0>
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 wl1: wlc_bmac_wowlucode_start mctrl=0x4020402  0x4000400
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 wl1: wlc_bmac_wowlucode_start (mctrl | MCTL_PSM_JMP_0)=0x4020406 0x4020406
Jul 22 16:57:27 kernel: CONSOLE: 198694.653 wl1: wlc_bmac_wowlucode_start mctrl=0x4020402 0x4020402
Jul 22 16:57:27 kernel: CONSOLE: 198694.655 wl1: CORE INIT : mode 4 pktclassify 1 rxsplit 1  hdr conversion 1 DMA_CT Enabled
Jul 22 16:57:27 kernel: CONSOLE: 198694.674 wlc_mutx_active_update vasip mu_supported_Ntx 4
Jul 22 16:57:27 kernel: CONSOLE: 198694.674 HWA1a RxPOST: H2D RxPost ring: id<1> type<0> item_type<1> max_items<1024> len_item<8>
Jul 22 16:57:27 kernel: CONSOLE: 198694.674 HWA1a RxPOST: item_size 8 CWI32 config parser<00000000> format<0> size<8>
Jul 22 16:57:27 kernel: CONSOLE: 198694.674 HWA1a RxPOST: rxpost_data_buf_len<1836>
Jul 22 16:57:27 kernel: CONSOLE: 198694.676 wl1: wlc_enable_probe_req: state down, deferring setting of host flags
Jul 22 16:57:27 kernel: CONSOLE: 198694.676 CSIMON: already initialized ...
Jul 22 16:57:27 kernel: CONSOLE: 198694.677 wl1 enable 1: q0 frmcnt 0, wrdcnt 0, q1 frmcnt 0, wrdcnt 0
Jul 22 16:57:27 kernel: CONSOLE: 198694.677 wl1: wlc_phy_cal_cache_restore_acphy: Chanspec 0xe09b found, but not valid in phycal cache
Jul 22 16:57:27 kernel: CONSOLE: 198694.759 wl1: link up (wl1)
Jul 22 16:57:27 kernel: CONSOLE: 198694.759 wl1: link up (wl1.3)
nothing in this log has any indication of what is going on with your dnscrypt-proxy2.
 

SomeWhereOverTheRainBow

Part of the Furniture
Just got a new RT-AX86U yesterday. Flashed 386.2_6 and configured all manually. Having trouble with dnscrypt_proxy2 crashing every couple of hours with no log entries. Ran the same version on my RT-AC88U for months at a time with no crashes. Same version, same config file. I don't even know how to start debugging the cause. I have 14 days left to figure out if this router has to go back.
also did you factory reset before configuring manually, because it may clear out any issues brought over from stock firmware to Asuswrt-Merlin.
 

sbsnb

Very Senior Member
also did you factory reset before configuring manually, because it may clear out any issues brought over from stock firmware to Asuswrt-Merlin.
Yes. Factory reset after flashing Merlin.

I started with Entware's and experienced the crashes. So I replaced their binary with one downloaded directly from https://github.com/DNSCrypt/dnscrypt-proxy/releases/tag/2.0.45, but still the same issue. I'm wondering if it's an issue with the arm64 version as the AC88U ran the 32-bit version.
 

SomeWhereOverTheRainBow

Part of the Furniture
Yes. Factory reset after flashing Merlin.

I started with Entware's and experienced the crashes. So I replaced their binary with one downloaded directly from https://github.com/DNSCrypt/dnscrypt-proxy/releases/tag/2.0.45, but still the same issue. I'm wondering if it's an issue with the arm64 version as the AC88U ran the 32-bit version.
What I recommend doing is completely ditch the entware setup one.
with

opkg remove dnscrypt-proxy2

use the one installed by the AMTM, it already configures dnsmasq settings for you and does everything you need to work correctly. This will give you some kind of idea if it is an issue with entwares package setup, or if it is dnscrypt proxy 2.
 

sbsnb

Very Senior Member
I can try that, but unless there's something behind the scenes I can't see there's not much to go wrong. The config file is the same one I've been using for years. MD5 to make sure it's identical. Entware just drops a binary in /opt/sbin and an startup script in /opt/etc/init.d. I don't think there's anything else. The dnsmasq config has nothing other than server=127.0.0.1#65053.
 

SomeWhereOverTheRainBow

Part of the Furniture
I can try that, but unless there's something behind the scenes I can't see there's not much to go wrong. The config file is the same one I've been using for years. MD5 to make sure it's identical. Entware just drops a binary in /opt/sbin and an startup script in /opt/etc/init.d. I don't think there's anything else. The dnsmasq config has nothing other than server=127.0.0.1#65053.
They have added a lot of new requirements to the .toml file, and a lot of name changes. you are better to append your settings to a new .toml file to prevent conflict.
For example, there are instances where dnscrypt proxy 2 will not start if legacy options are used in the .toml file.
 

sbsnb

Very Senior Member
I always use the example toml file that comes with the release and edit. I only change four things:

Code:
listen_addresses = ['127.0.0.1:65053']

use_syslog = true

tls_cipher_suite = [52392, 49199]

  ## OpenNIC
   [sources.'opennic']
   urls = ['https://raw.githubusercontent.com/DNSCrypt/dnscrypt-resolvers/master/v2/opennic.md', 'https://download.dnscrypt.info/resolvers-list/v2/opennic.md']
   minisign_key = 'RWQf6LRCGA9i53mlYecO4IzT51TGPpvWucNSCh1CBM0QTaLn73Y7GFO3'
   refresh_delay = 72
   cache_file = 'opennic.md'

This one has been the same since the 2.0.45 release, which has run on the AC88U for up to 4 months straight without a crash or reboot. I'm always up for testing, though. I'll try the amtm version.
 

SomeWhereOverTheRainBow

Part of the Furniture
I always use the example toml file that comes with the release and edit. I only change four things:

Code:
listen_addresses = ['127.0.0.1:65053']

use_syslog = true

tls_cipher_suite = [52392, 49199]

  ## OpenNIC
   [sources.'opennic']
   urls = ['https://raw.githubusercontent.com/DNSCrypt/dnscrypt-resolvers/master/v2/opennic.md', 'https://download.dnscrypt.info/resolvers-list/v2/opennic.md']
   minisign_key = 'RWQf6LRCGA9i53mlYecO4IzT51TGPpvWucNSCh1CBM0QTaLn73Y7GFO3'
   refresh_delay = 72
   cache_file = 'opennic.md'

This one has been the same since the 2.0.45 release, which has run on the AC88U for up to 4 months straight without a crash or reboot. I'm always up for testing, though. I'll try the amtm version.
did you use the same entware installation you used from the AC88U? or did you start out fresh with that too?
 

sbsnb

Very Senior Member
Fresh. Not even the same USB.

I did notice that the 'opennic.md' file has this gem:

Code:
# *** THIS LIST IS FOR OLD DNSCRYPT-PROXY VERSIONS ***

Version 2 of the list is for dnscrypt-proxy <= 2.0.42 users.

If you are running up-to-date software, replace `/v2/` with `/v3/` in the sources URLs
of the `dnscrypt-proxy.toml` file (relevant lines start with `urls = ['https://...']`
and are present in the `[sources]` section).

THIS LIST IS AUTOMATICALLY GENERATED AS A SUBSET OF THE V3 LIST. DO NOT EDIT IT MANUALLY.

Which is a bad place to put such a notice since it's normally only read by automated processes. Still, I doubt it's the issue since it's strictly a subset of the V3 file. I'll try that first.

I'm also enabling log level 0 in dnscrypt-proxy2. We'll see if that makes it spit out something before it dies.
 

SomeWhereOverTheRainBow

Part of the Furniture
I always use the example toml file that comes with the release and edit. I only change four things:

Code:
listen_addresses = ['127.0.0.1:65053']

use_syslog = true

tls_cipher_suite = [52392, 49199]

  ## OpenNIC
   [sources.'opennic']
   urls = ['https://raw.githubusercontent.com/DNSCrypt/dnscrypt-resolvers/master/v2/opennic.md', 'https://download.dnscrypt.info/resolvers-list/v2/opennic.md']
   minisign_key = 'RWQf6LRCGA9i53mlYecO4IzT51TGPpvWucNSCh1CBM0QTaLn73Y7GFO3'
   refresh_delay = 72
   cache_file = 'opennic.md'

This one has been the same since the 2.0.45 release, which has run on the AC88U for up to 4 months straight without a crash or reboot. I'm always up for testing, though. I'll try the amtm version.
keep in mind the listen address setup by the installer is different. you may have to adapt your dnsmasq setting temporarily until you see if it will run.
 

SomeWhereOverTheRainBow

Part of the Furniture
Fresh. Not even the same USB.

I did notice that the 'opennic.md' file has this gem:

Code:
# *** THIS LIST IS FOR OLD DNSCRYPT-PROXY VERSIONS ***

Version 2 of the list is for dnscrypt-proxy <= 2.0.42 users.

If you are running up-to-date software, replace `/v2/` with `/v3/` in the sources URLs
of the `dnscrypt-proxy.toml` file (relevant lines start with `urls = ['https://...']`
and are present in the `[sources]` section).

THIS LIST IS AUTOMATICALLY GENERATED AS A SUBSET OF THE V3 LIST. DO NOT EDIT IT MANUALLY.

Which is a bad place to put such a notice since it's normally only read by automated processes. Still, I doubt it's the issue since it's strictly a subset of the V3 file. I'll try that first.

I'm also enabling log level 0 in dnscrypt-proxy2. We'll see if that makes it spit out something before it dies.
That is what I mean, random changes that sometimes break things. You are right though, it could very well not be the issue.

Here is what you would want to use if it were the case.

Code:
[sources.'opennic']
urls = ['https://raw.githubusercontent.com/DNSCrypt/dnscrypt-resolvers/master/v3/opennic.md', 'https://download.dnscrypt.info/resolvers-list/v3/opennic.md']
minisign_key = 'RWQf6LRCGA9i53mlYecO4IzT51TGPpvWucNSCh1CBM0QTaLn73Y7GFO3'
cache_file = 'opennic.md'
 

sbsnb

Very Senior Member
I think changing it to use user 'nobody' has fixed the crashes (or being force closed). I hope I'm not jinxing it.
 

sbsnb

Very Senior Member
I spoke too soon. It crashed about 10 minutes ago. I noticed when my phone alerted me that my WiFi had no internet access. I connected by SSH and did ./S09dnscrypt-proxy2 start and it didn't even get all the way through initializing before it crashed again. The second time was the charm and it's running again. Nothing in the logs. This is what it looked like when it crashed mid-start:

Code:
Jul 23 20:59:47 admin: Started dnscrypt-proxy from .
Jul 23 20:59:47 dnscrypt-proxy[21889]: dnscrypt-proxy 2.0.45
Jul 23 20:59:47 dnscrypt-proxy[21889]: Network connectivity detected
Jul 23 20:59:47 dnscrypt-proxy[21889]: Dropping privileges
Jul 23 20:59:47 dnscrypt-proxy[21889]: Network connectivity detected
Jul 23 20:59:47 dnscrypt-proxy[21889]: Now listening to 127.0.0.1:65053 [UDP]
Jul 23 20:59:47 dnscrypt-proxy[21889]: Now listening to 127.0.0.1:65053 [TCP]
Jul 23 20:59:47 dnscrypt-proxy[21889]: Source [opennic] loading from URL [https://raw.githubusercontent.com/DNSCrypt/dnscrypt-resolvers/master/v3/opennic.md]
Jul 23 20:59:48 dnscrypt-proxy[21889]: /tmp/mnt/sda1/entware/etc/opennic.md: open sf-lhcjlmg4uvqkxjsl.tmp: permission denied
Jul 23 20:59:48 dnscrypt-proxy[21889]: Source [opennic] loaded
Jul 23 20:59:48 dnscrypt-proxy[21889]: Firefox workaround initialized
Jul 23 20:59:48 dnscrypt-proxy[21889]: [publicarray-au-doh] TLS version: 304 - Protocol: h2 - Cipher suite: 4867
Jul 23 20:59:48 dnscrypt-proxy[21889]: [publicarray-au-doh] OK (DoH) - rtt: 173ms
Jul 23 20:59:49 dnscrypt-proxy[21889]: [opennic-luggs2] the key validity period for this server is excessively long (3650 days), significantly reducing reliability and forward security.
Jul 23 20:59:49 dnscrypt-proxy[21889]: [opennic-luggs2] OK (DNSCrypt) - rtt: 87ms

And nothing after that. It just died mid-start. I'm suspicious of the issue it had trying to open that tmp file, but there is nothing similar in the logs for the prior crash. The prior crash did NOT occur near the time of the 4-hour renewal that dnscrypt-proxy does. That was not due until 22:45.
 
Last edited:

SomeWhereOverTheRainBow

Part of the Furniture
I spoke too soon. It crashed about 10 minutes ago. I noticed when my phone alerted me that my WiFi had no internet access. I connected by SSH and did ./S09dnscrypt-proxy2 start and it didn't even get all the way through initializing before it crashed again. The second time was the charm and it's running again. Nothing in the logs. This is what it looked like when it crashed mid-start:

Code:
Jul 23 20:59:47 admin: Started dnscrypt-proxy from .
Jul 23 20:59:47 dnscrypt-proxy[21889]: dnscrypt-proxy 2.0.45
Jul 23 20:59:47 dnscrypt-proxy[21889]: Network connectivity detected
Jul 23 20:59:47 dnscrypt-proxy[21889]: Dropping privileges
Jul 23 20:59:47 dnscrypt-proxy[21889]: Network connectivity detected
Jul 23 20:59:47 dnscrypt-proxy[21889]: Now listening to 127.0.0.1:65053 [UDP]
Jul 23 20:59:47 dnscrypt-proxy[21889]: Now listening to 127.0.0.1:65053 [TCP]
Jul 23 20:59:47 dnscrypt-proxy[21889]: Source [opennic] loading from URL [https://raw.githubusercontent.com/DNSCrypt/dnscrypt-resolvers/master/v3/opennic.md]
Jul 23 20:59:48 dnscrypt-proxy[21889]: /tmp/mnt/sda1/entware/etc/opennic.md: open sf-lhcjlmg4uvqkxjsl.tmp: permission denied
Jul 23 20:59:48 dnscrypt-proxy[21889]: Source [opennic] loaded
Jul 23 20:59:48 dnscrypt-proxy[21889]: Firefox workaround initialized
Jul 23 20:59:48 dnscrypt-proxy[21889]: [publicarray-au-doh] TLS version: 304 - Protocol: h2 - Cipher suite: 4867
Jul 23 20:59:48 dnscrypt-proxy[21889]: [publicarray-au-doh] OK (DoH) - rtt: 173ms
Jul 23 20:59:49 dnscrypt-proxy[21889]: [opennic-luggs2] the key validity period for this server is excessively long (3650 days), significantly reducing reliability and forward security.
Jul 23 20:59:49 dnscrypt-proxy[21889]: [opennic-luggs2] OK (DNSCrypt) - rtt: 87ms

And nothing after that. It just died mid-start. I'm suspicious of the issue it had trying to open that tmp file, but there is nothing similar in the logs for the prior crash. The prior crash did NOT occur near the time of the 4-hour renewal that dnscrypt-proxy does. That was not due until 22:45.
You could just make a cron job that restarts dnscrypt proxy every 3 hours. It looks like since dnscrypt is manually downloading the server files for opennic, it is not setting the right router permissions for dnscrypt proxy and the router.
 

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top