What's new

FlexQoS FlexQoS issues with 388.4 HND5.04 models

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

I’m waiting for someone to be able to post a complete syslog covering the reboot time and subsequent boot sequence. Surprisingly, this has been hard to come by.

Any extra iptables logging would be more likely to overload the router than to provide any clues.
This challenge becoming like peeling an onion to find the source(s) of problem(s). I have now removed scribe from my AXE16000 installation. Debug set and your two IP table logging set. Anything else? Logging panel now appears as per picture. Now we await with a large bag of popcorn, w/soda...

Update Note: with the iptable logging, only six minutes in syslog from oldest to newest record. Will have to be at my laptop when a reboot occurs to grab the log before the event gets flushed/deleted.

1694738132939.png
 
What is the MARK on these log entries near the end of the line?
Scrolling through syslog the MARK is mostly 0X405400b9 but others appear at times. On the "Blocked - Inbound", MARK = 0x8000000. Syslog attached.

Two examples of each MARK I saw...
Sep 14 20:33:00 kernel: [FlexQoS PRE Debug] IN= OUT=eth0 SRC=192.168.50.186 DST=34.120.208.123 LEN=71 TOS=0x00 PREC=0x00 TTL=127 ID=24627 DF PROTO=TCP SPT=58858 DPT=443 WINDOW=1020 RES=0x00 ACK PSH URGP=0 MARK=0x405400b9
Sep 14 20:33:00 kernel: [FlexQoS POST Debug] IN= OUT=eth0 SRC=192.168.50.186 DST=34.120.208.123 LEN=71 TOS=0x00 PREC=0x00 TTL=127 ID=24627 DF PROTO=TCP SPT=58858 DPT=443 WINDOW=1020 RES=0x00 ACK PSH URGP=0 MARK=0x405400b9

Sep 14 20:33:10 kernel: [FlexQoS PRE Debug] IN= OUT=eth0 SRC=192.168.50.186 DST=24.226.33.33 LEN=328 TOS=0x00 PREC=0x00 TTL=127 ID=46092 DF PROTO=TCP SPT=58870 DPT=80 WINDOW=1026 RES=0x00 ACK PSH URGP=0 MARK=0x404e0009
Sep 14 20:33:10 kernel: [FlexQoS POST Debug] IN= OUT=eth0 SRC=192.168.50.186 DST=24.226.33.33 LEN=328 TOS=0x00 PREC=0x00 TTL=127 ID=46092 DF PROTO=TCP SPT=58870 DPT=80 WINDOW=1026 RES=0x00 ACK PSH URGP=0 MARK=0x404e0009

Sep 14 20:33:36 kernel: [FlexQoS PRE Debug] IN= OUT=eth0 SRC=192.168.50.186 DST=151.101.127.10 LEN=41 TOS=0x00 PREC=0x00 TTL=127 ID=39327 DF PROTO=TCP SPT=58357 DPT=443 WINDOW=1025 RES=0x00 ACK URGP=0 MARK=0x404d00d1
Sep 14 20:33:36 kernel: [FlexQoS POST Debug] IN= OUT=eth0 SRC=192.168.50.186 DST=151.101.127.10 LEN=41 TOS=0x00 PREC=0x00 TTL=127 ID=39327 DF PROTO=TCP SPT=58357 DPT=443 WINDOW=1025 RES=0x00 ACK URGP=0 MARK=0x404d00d1

Sep 14 20:33:10 kernel: [BLOCKED - INBOUND] IN=eth0 OUT= MAC=c8:7f:54:48:df:44:00:c1:b1:60:e0:19:08:00 SRC=194.26.135.128 DST=24.150.132.8 LEN=40 TOS=0x00 PREC=0x00 TTL=243 ID=18708 PROTO=TCP SPT=57294 DPT=10050 SEQ=817818206 ACK=0 WINDOW=1024 RES=0x00 SYN URGP=0 MARK=0x8000000
Sep 14 20:33:10 kernel: [BLOCKED - INBOUND] IN=eth0 OUT= MAC=c8:7f:54:48:df:44:00:c1:b1:60:e0:19:08:00 SRC=162.216.149.55 DST=24.150.132.8 LEN=44 TOS=0x00 PREC=0x00 TTL=250 ID=32057 PROTO=TCP SPT=56371 DPT=34129 SEQ=378567776 ACK=0 WINDOW=1024 RES=0x00 SYN URGP=0 OPT (020405B4) MARK=0x8000000
 

Attachments

  • syslog.txt
    427.8 KB · Views: 21
If you are having problems getting logs due to reboots then set up a syslog server at home and forward the logs to it. Kiwi syslog is free. It's just a small app that runs on any Windows machine that listens on udp 514. The free version can listen to 5 hosts.
 
Last edited:
If you are having problems getting logs due to reboots then set up a syslog server at home and forward the logs to it. Kiwi syslog is free. It's just a small app that runs on any Windows machine that listens on udp 514. The free version can listen to 5 hosts.
Thanks for tip. Requested the free version via SolarWinds website GUI process. Can take two days for approval.
 
Thanks for tip. Requested the free version via SolarWinds website GUI process. Can take two days for approval.
@nzwayne thanks for taking the time providing these logs. Hopefully it's something that can be fixed for these routers!!!
 
@nzwayne thanks for taking the time providing these logs. Hopefully it's something that can be fixed for these routers!!!
Anything to assist @dave14305 !! I have noted on my AXE11000 & AXE16000 that the web performance when the FlexQoS fq-codel option is enabled can become slow, long delays and even hang (especially on the AXE16000). Where as when the ASUS option enabled no reponse time or performance issues. Panel reference:
1694907583628.png
 
@dave14305 , today this morning I installed again FlexQoS. I got about 2-3 reboots whithin a 5 to 10 minutes period.
I attach here the logs after the reboots. SysLog (2) was the log generated right after installation and first reboot, and subsequently SysLog (3) was captured right after second reboot. Let me know if I can do something more to help.
I have also set Asus routers to send Syslog messages to Synology Log , let me know if the attached files are not very useful so I can export from Synology.
 

Attachments

  • syslog (3).txt
    488.7 KB · Views: 24
  • syslog (2).txt
    476.1 KB · Views: 17
@dave14305 , today this morning I installed again FlexQoS. I got about 2-3 reboots whithin a 5 to 10 minutes period.
I attach here the logs after the reboots. SysLog (2) was the log generated right after installation and first reboot, and subsequently SysLog (3) was captured right after second reboot. Let me know if I can do something more to help.
I have also set Asus routers to send Syslog messages to Synology Log , let me know if the attached files are not very useful so I can export from Synology.
We at least can see the kernel panic details now. Thanks for that. bcmsw_rx seems to crash with a very flowcache-y call stack. I’m not an expert at reading these, but not surprising that a lot of closed-source modules are mentioned.
Code:
May  5 08:05:09 crashlog: <4> [last unloaded: tdts]
May  5 08:05:09 crashlog: <0>Process bcmsw_rx (pid: 409, stack limit = 0x000000007db6c041)
May  5 08:05:09 crashlog: <4>CPU: 0 PID: 409 Comm: bcmsw_rx Tainted: P           O      4.19.183 #1
May  5 08:05:09 crashlog: <4>Hardware name: RTAX86U_PRO (DT)
May  5 08:05:09 crashlog: <4>pstate: 20000005 (nzCv daif -PAN -UAO)
May  5 08:05:09 crashlog: <4>pc : fc_hook_cb+0x108/0x180 [tdts_udbfw]
May  5 08:05:09 crashlog: <4>lr : fc_hook_cb+0x4c/0x180 [tdts_udbfw]
May  5 08:05:09 crashlog: <4>sp : ffffff801770b530
May  5 08:05:09 crashlog: <4>x29: ffffff801770b530 x28: ffffffc02d14b98e
May  5 08:05:09 crashlog: <4>x27: 0000000000000000 x26: 0000000000000000
May  5 08:05:09 crashlog: <4>x25: 000000402c57d000 x24: ffffff8008a90000
May  5 08:05:09 crashlog: <4>x23: ffffffc026987840 x22: ffffffc026987000
May  5 08:05:09 crashlog: <4>x21: ffffff80089c9948 x20: ffffffc01ddae800
May  5 08:05:09 crashlog: <4>x19: ffffff80089c9948 x18: 0000000000000001
May  5 08:05:09 crashlog: <4>x17: 0000000000000000 x16: 000000000102340f
May  5 08:05:09 crashlog: <4>x15: 0000000000000000 x14: 0000000000004900
May  5 08:05:09 crashlog: <4>x13: 0000000000000001 x12: 000000000000008c
May  5 08:05:09 crashlog: <4>x11: 0000000000000000 x10: 0000000000000001
May  5 08:05:09 crashlog: <4>x9 : 0000000000000000 x8 : ffffffc01ddae9e8
May  5 08:05:09 crashlog: <4>x7 : 0000000000000000 x6 : 00000000000000b2
May  5 08:05:09 crashlog: <4>x5 : 0000000000000040 x4 : ffffffc026987000
May  5 08:05:09 crashlog: <4>x3 : ffffff8000ff2660 x2 : ffffffc01ddae800
May  5 08:05:09 crashlog: <4>x1 : ffffffc094c6ffff x0 : ffffff8000000000
May  5 08:05:09 crashlog: <4>Call trace:
May  5 08:05:09 crashlog: <4> fc_hook_cb+0x108/0x180 [tdts_udbfw]
May  5 08:05:09 crashlog: <4> br_dev_xmit+0x1c0/0x3c0
May  5 08:05:09 crashlog: <4> fc_dev_xmit+0x88/0x100 [pktflow]
May  5 08:05:09 crashlog: <4> fc_queue_fkb+0x2c/0x40 [pktflow]
May  5 08:05:09 crashlog: <4> fc_stack_pkt+0x290/0x8500 [pktflow]
May  5 08:05:09 crashlog: <4> fc_stack_ucast+0xd4/0x150 [pktflow]
May  5 08:05:09 crashlog: <4> fc_stack+0x6c/0x80 [pktflow]
May  5 08:05:09 crashlog: <4> fc_receive+0x2c24/0x53c0 [pktflow]
May  5 08:05:09 crashlog: <4> _blog_finit+0x130/0x2a0
May  5 08:05:09 crashlog: <4> chan_thread_handler+0x274/0x790 [bcm_enet]
May  5 08:05:09 crashlog: <4> kthread+0x118/0x150
May  5 08:05:09 crashlog: <4> ret_from_fork+0x10/0x24
May  5 08:05:09 crashlog: <0>Code: b5000341 f9400bf3 a8d17bfd d65f03c0 (b9406c20)
May  5 08:05:09 crashlog: <4>---[ end trace 35c408fe1097d635 ]---
May  5 08:05:09 crashlog: <0>Kernel panic - not syncing: Fatal exception
May  5 08:05:09 crashlog: <2>SMP: stopping secondary CPUs
May  5 08:05:09 crashlog: <0>Kernel Offset: disabled
May  5 08:05:09 crashlog: <0>CPU features: 0x0,24802004
May  5 08:05:09 crashlog: <0>Memory Limit: none
May  5 08:05:09 crashlog: ?????????????????
 
Let me know if I can do something more to help.
Not sure what’s going on with these messages. Check your /jffs/scripts files for the permissions.
Code:
Sep 17 12:16:47 custom_script: Found service-event-end, but script is not set executable!
Sep 17 12:16:48 custom_script: Found service-event-end, but script is not set executable!
Sep 17 12:16:48 custom_script: Found service-event-end, but script is not set executable!
Sep 17 12:16:56 custom_script: Found service-event-end, but script is not set executable!
Sep 17 12:16:59 custom_script: Found firewall-start, but script is not set executable!
Sep 17 12:16:59 custom_script: Found service-event-end, but script is not set executable!
Sep 17 12:17:00 custom_script: Found service-event-end, but script is not set executable!
Sep 17 12:17:01 custom_script: Found service-event-end, but script is not set executable!
 
Not sure what’s going on with these messages. Check your /jffs/scripts files for the permissions.
Code:
Sep 17 12:16:47 custom_script: Found service-event-end, but script is not set executable!
Sep 17 12:16:48 custom_script: Found service-event-end, but script is not set executable!
Sep 17 12:16:48 custom_script: Found service-event-end, but script is not set executable!
Sep 17 12:16:56 custom_script: Found service-event-end, but script is not set executable!
Sep 17 12:16:59 custom_script: Found firewall-start, but script is not set executable!
Sep 17 12:16:59 custom_script: Found service-event-end, but script is not set executable!
Sep 17 12:17:00 custom_script: Found service-event-end, but script is not set executable!
Sep 17 12:17:01 custom_script: Found service-event-end, but script is not set executable!
I haven't setup any special script on my own. Let me know if something is unusual there:
1694948577796.png
 
I haven't setup any special script on my own. Let me know if something is unusual there:
The zero-byte files are unusual, unless you had already uninstalled FlexQoS and nothing else was using them, but the permissions as they look now would never have worked. You can remove those 2 files for now, just to stop the log messages.
 
Thanks, I 've removed those files.
Back to my log files showing the kernel panic, maybe we could ask for @RMerlin to rescue?
 
Back to my log files showing the kernel panic, maybe we could ask for @RMerlin to rescue?
I wouldn’t expect much from him. He generally steers clear of add-on issues.
 
RT-AX88U Pro crashes after I receive a wifi calling call.
I use FlexQoS,

When I receive a call on my mobile phone using WiFi Calling, the router immediately restart.

A temporary solution? Eliminating the WiFi Calling entry from iptables Rules. It probably doesn't work out when you have to tag packets while talking over WiFi on the phone. As soon as I establish a connection, the router CRASHes.


Maybe the problem will be solved in the next update?

Best Regards, Marcin.
 
When I receive a call on my mobile phone using WiFi Calling, the router immediately restart.

A temporary solution? Eliminating the WiFi Calling entry from iptables Rules. It probably doesn't work out when you have to tag packets while talking over WiFi on the phone. As soon as I establish a connection, the router CRASHes.
That’s a nice, specific reproducer of the problem. Was it just the default WiFi Calling rule, or had you customized it at all?
 
I think the new 1.3.3 update messed up something with upload speeds, also more traffic is being clasified as net control packets. Uploads speeds are under .7 mbps while I usually get 1.4-5
This is with a RT-AC88U.
 
I think the new 1.3.3 update messed up something with upload speeds, also more traffic is being clasified as net control packets. Uploads speeds are under .7 mbps while I usually get 1.4-5
This is with a RT-AC88U.
What firmware are you running? Very little has changed in FlexQoS in the last year. 1.3.3 fixed some rare situations with newer routers (not your older AC88U).

Make sure you’re not trying to use the Gear Accelerator feature that moves Gaming to Net Control.
 
bcmsw_rx seems to crash with a very flowcache-y call stack
Any old timers know what this means, especially at 5108?
Code:
0000000000005000 <fc_hook_cb>:
    5000:       a9af7bfd        stp     x29, x30, [sp, #-272]!
    5004:       910003fd        mov     x29, sp
    5008:       f9000bf3        str     x19, [sp, #16]
    500c:       90000013        adrp    x19, 0 <__stack_chk_guard>
    5010:       91000273        add     x19, x19, #0x0
    5014:       b90027ff        str     wzr, [sp, #36]
    5018:       f9400262        ldr     x2, [x19]
    501c:       f90087e2        str     x2, [sp, #264]
    5020:       d2800002        mov     x2, #0x0                        // #0
    5024:       b4000a00        cbz     x0, 5164 <fc_hook_cb+0x164>
    5028:       90000003        adrp    x3, 100 <forward_find_ct_by_tuples+0x100>
    502c:       91000063        add     x3, x3, #0x0
    5030:       aa0003e2        mov     x2, x0
    5034:       aa0103e4        mov     x4, x1
    5038:       91006063        add     x3, x3, #0x18
    503c:       90000001        adrp    x1, 100 <forward_find_ct_by_tuples+0x100>
    5040:       91000021        add     x1, x1, #0x0
    5044:       52800020        mov     w0, #0x1                        // #1
    5048:       94000000        bl      0 <__ll_sc_atomic_add>
    504c:       a902ffff        stp     xzr, xzr, [sp, #40]
    5050:       92400440        and     x0, x2, #0x3
    5054:       f1000c1f        cmp     x0, #0x3
    5058:       a903ffff        stp     xzr, xzr, [sp, #56]
    505c:       a904ffff        stp     xzr, xzr, [sp, #72]
    5060:       a905ffff        stp     xzr, xzr, [sp, #88]
    5064:       a906ffff        stp     xzr, xzr, [sp, #104]
    5068:       a907ffff        stp     xzr, xzr, [sp, #120]
    506c:       a908ffff        stp     xzr, xzr, [sp, #136]
    5070:       a909ffff        stp     xzr, xzr, [sp, #152]
    5074:       a90affff        stp     xzr, xzr, [sp, #168]
    5078:       a90bffff        stp     xzr, xzr, [sp, #184]
    507c:       a90cffff        stp     xzr, xzr, [sp, #200]
    5080:       a90dffff        stp     xzr, xzr, [sp, #216]
    5084:       a90effff        stp     xzr, xzr, [sp, #232]
    5088:       a90fffff        stp     xzr, xzr, [sp, #248]
    508c:       540004e0        b.eq    5128 <fc_hook_cb+0x128>  // b.none
    5090:       f9403041        ldr     x1, [x2, #96]
    5094:       b25963e0        mov     x0, #0xffffff8000000000         // #-549755813888
    5098:       eb00003f        cmp     x1, x0
    509c:       54000368        b.hi    5108 <fc_hook_cb+0x108>  // b.pmore
    50a0:       52800000        mov     w0, #0x0                        // #0
    50a4:       d2800005        mov     x5, #0x0                        // #0
    50a8:       52800002        mov     w2, #0x0                        // #0
    50ac:       b941f887        ldr     w7, [x4, #504]
    50b0:       32000006        orr     w6, w0, #0x1
    50b4:       121f7801        and     w1, w0, #0xfffffffe
    50b8:       910093e4        add     x4, sp, #0x24
    50bc:       528000a3        mov     w3, #0x5                        // #5
    50c0:       9100a3e0        add     x0, sp, #0x28
    50c4:       f24000ff        tst     x7, #0x1
    50c8:       f9003fe5        str     x5, [sp, #120]
    50cc:       1a860021        csel    w1, w1, w6, eq  // eq = none
    50d0:       290a0be1        stp     w1, w2, [sp, #80]
    50d4:       f9004be4        str     x4, [sp, #144]
    50d8:       b900bbe3        str     w3, [sp, #184]
    50dc:       94000000        bl      0 <udb_shell_do_fastpath_action>
    50e0:       7100001f        cmp     w0, #0x0
    50e4:       52800061        mov     w1, #0x3                        // #3
    50e8:       1a9f0020        csel    w0, w1, wzr, eq  // eq = none
    50ec:       f94087e2        ldr     x2, [sp, #264]
    50f0:       f9400261        ldr     x1, [x19]
    50f4:       ca010041        eor     x1, x2, x1
    50f8:       b5000341        cbnz    x1, 5160 <fc_hook_cb+0x160>
    50fc:       f9400bf3        ldr     x19, [sp, #16]
    5100:       a8d17bfd        ldp     x29, x30, [sp], #272
    5104:       d65f03c0        ret
    5108:       b9406c20        ldr     w0, [x1, #108]
    510c:       91018045        add     x5, x2, #0x60
    5110:       b9006040        str     w0, [x2, #96]
    5114:       b9405842        ldr     w2, [x2, #88]
    5118:       52800000        mov     w0, #0x0                        // #0
    511c:       b9410821        ldr     w1, [x1, #264]
    5120:       b90027e1        str     w1, [sp, #36]
    5124:       17ffffe2        b       50ac <fc_hook_cb+0xac>
    5128:       927ef442        and     x2, x2, #0xfffffffffffffffc
    512c:       b25963e1        mov     x1, #0xffffff8000000000         // #-549755813888
    5130:       f9401040        ldr     x0, [x2, #32]
    5134:       eb01001f        cmp     x0, x1
    5138:       54fffb49        b.ls    50a0 <fc_hook_cb+0xa0>  // b.plast
    513c:       b9406c01        ldr     w1, [x0, #108]
    5140:       91008045        add     x5, x2, #0x20
    5144:       f9001041        str     x1, [x2, #32]
    5148:       b9401842        ldr     w2, [x2, #24]
    514c:       b9410800        ldr     w0, [x0, #264]
    5150:       b90027e0        str     w0, [sp, #36]
    5154:       d3405c42        ubfx    x2, x2, #0, #24
    5158:       b94053e0        ldr     w0, [sp, #80]
    515c:       17ffffd4        b       50ac <fc_hook_cb+0xac>
    5160:       94000000        bl      0 <__stack_chk_fail>
    5164:       90000000        adrp    x0, 0 <forward_find_ct_by_tuples>
    5168:       91000000        add     x0, x0, #0x0
    516c:       94000000        bl      0 <printk>
    5170:       52800000        mov     w0, #0x0                        // #0
    5174:       17ffffde        b       50ec <fc_hook_cb+0xec>
    5178:       d503201f        nop
    517c:       d503201f        nop
 

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top