I am on 3006.102.5 (so a version prior to the latest GPL updates). I had tried 3006.102.6 when it came out but it was *so* unstable for me that I went back to 102.5.
On that version it is *mostly* stable. I still have a programmed reboot every night and I have a cron job that monitors for clock discrepancies and reboots if detected and *mostly*, it is stable.
Sometimes, however, it does crash. I finally got around to setting up remote logging and here are the excerpts of the latest two crashes. They are typically similar like this. I have not figured out what triggers them. Like I said, it can be stable for weeks and then, in a day, it just reboots a few times like this.
I can try 102.7 at some point if that is helpful. 102.6 was just TOO unstable and it would crash multiple times a day and since 102.7 seems to use the same kernel, I didn't both with it yet.
```
Feb 26 12:55:19 router kernel: [BLOCKED - INBOUND] IN=vlan4094 OUT= MAC=XXX SRC=XXX DST=XXX LEN=60 TOS=0x00 PREC=0x00 TTL=43 ID=31098 DF PROTO=TCP SPT=53570 DPT=7015 SEQ=37579354
9 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B4080A4445414400000000030301040200)
Sep 12 14:57:21 router chronyd[10486]: Forward time jump detected!
Aug 7 08:29:05 router kernel: rcu: INFO: rcu_preempt self-detected stall on CPU
Aug 7 08:29:05 router kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
Aug 7 08:29:05 router kernel: rcu: INFO: rcu_bh detected stalls on CPUs/tasks:
Aug 7 08:29:05 router kernel: rcu: 1-...!: (16 ticks this GP) idle=0da/0/0x1 softirq=5235988/5235996 fqs=0
Aug 7 08:29:05 router kernel: rcu: (detected by 3, t=900726558046 jiffies, g=2448077, q=1360)
Aug 7 08:29:05 router kernel: Task dump for CPU 1:
Aug 7 08:29:05 router kernel: swapper/1 R running task 0 0 1 0x0000000a
Aug 7 08:29:05 router kernel: Call trace:
Aug 7 08:29:05 router kernel: __switch_to+0xe8/0x170
Aug 7 08:29:05 router kernel: 0xffffff800809bef0
Aug 7 08:29:05 router kernel: rcu: rcu_sched kthread starved for 900726558046 jiffies! g2448077 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
Aug 7 08:29:05 router kernel: rcu: RCU grace-period kthread stack dump:
Aug 7 08:29:05 router kernel: rcu_sched I 0 11 2 0x00000008
Aug 7 08:29:05 router kernel: Call trace:
Aug 7 08:29:05 router kernel: __switch_to+0xe8/0x170
Aug 7 08:29:05 router kernel: __schedule+0x214/0x5a0
Aug 7 08:29:05 router kernel: schedule+0x38/0xa0
Aug 7 08:29:05 router kernel: schedule_timeout+0x15c/0x2a0
Aug 7 08:29:05 router kernel: rcu_gp_kthread+0x488/0x900
Aug 7 08:29:05 router kernel: kthread+0x118/0x150
Aug 7 08:29:05 router kernel: ret_from_fork+0x10/0x24
Aug 7 08:29:05 router kernel: rcu: 1-...!: (2 ticks this GP) idle=0da/0/0x1 softirq=5235996/5235996 fqs=0
Aug 7 08:29:05 router kernel: rcu: 1-...!: (7822 ticks this GP) idle=0da/0/0x1 softirq=5228421/5235996 fqs=1
Aug 7 08:29:05 router kernel: rcu: (t=900726558053 jiffies g=6702293 q=32270)
Aug 7 08:29:05 router kernel: rcu: (detected by 0, t=900726558058 jiffies, g=-255, q=1)
Aug 7 08:29:05 router kernel: Task dump for CPU 1:
Aug 7 08:29:05 router kernel: Task dump for CPU 1:
Aug 7 08:29:05 router kernel: swapper/1 R running task 0 0 1 0x0000000a
Aug 7 08:29:05 router kernel: swapper/1 R running task 0 0 1 0x0000000a
Aug 7 08:29:05 router kernel: Call trace:
Aug 7 08:29:05 router kernel: Call trace:
Aug 7 08:29:05 router kernel: dump_backtrace+0x0/0x150
Aug 7 08:29:05 router kernel: __switch_to+0xe8/0x170
Aug 7 08:29:05 router kernel: show_stack+0x14/0x20
Aug 7 08:29:05 router kernel: 0xffffff800809bef0
Aug 7 08:29:05 router kernel: sched_show_task+0x100/0x120
Aug 7 08:29:05 router kernel: dump_cpu_task+0x40/0x4c
Aug 7 08:29:05 router kernel: rcu_dump_cpu_stacks+0x90/0xcc
Aug 7 08:29:05 router kernel: rcu_check_callbacks+0x67c/0x810
Aug 7 08:29:05 router kernel: update_process_times+0x2c/0x70
Aug 7 08:29:05 router kernel: tick_sched_timer+0x54/0xd0
Aug 7 08:29:05 router kernel: __hrtimer_run_queues+0x13c/0x1d0
Aug 7 08:29:05 router kernel: hrtimer_interrupt+0xe4/0x2b0
Aug 7 08:29:05 router kernel: arch_timer_handler_phys+0x30/0x40
Aug 7 08:29:05 router kernel: handle_percpu_devid_irq+0x80/0x140
Aug 7 08:29:05 router kernel: __handle_domain_irq+0x70/0xd0
Aug 7 08:29:05 router kernel: gic_handle_irq+0x5c/0xc0
Aug 7 08:29:05 router kernel: el1_irq+0xe8/0x190
Aug 7 08:29:05 router kernel: cpuidle_enter_state+0x80/0x220
Aug 7 08:29:05 router kernel: cpuidle_enter+0x18/0x20
Aug 7 08:29:05 router kernel: do_idle+0x1d8/0x260
Aug 7 08:29:05 router kernel: cpu_startup_entry+0x20/0x40
Aug 7 08:29:05 router kernel: secondary_start_kernel+0x13c/0x170
Aug 7 08:29:05 router crond[4736]: time disparity of 15012122 minutes detected
Aug 7 08:29:05 router kernel: tdts_core_ioctl_udb_op_prog_ctrl() fail!
```
I attached another one in a file. FOr that one there are two stalls very close to one another. The first one seems to be OK and then the second one kills it.
On reboot, it does seem to print out information too (that it doesn't print on a regular reboot). I attached one such output.
Thanks!
Romain
On that version it is *mostly* stable. I still have a programmed reboot every night and I have a cron job that monitors for clock discrepancies and reboots if detected and *mostly*, it is stable.
Sometimes, however, it does crash. I finally got around to setting up remote logging and here are the excerpts of the latest two crashes. They are typically similar like this. I have not figured out what triggers them. Like I said, it can be stable for weeks and then, in a day, it just reboots a few times like this.
I can try 102.7 at some point if that is helpful. 102.6 was just TOO unstable and it would crash multiple times a day and since 102.7 seems to use the same kernel, I didn't both with it yet.
```
Feb 26 12:55:19 router kernel: [BLOCKED - INBOUND] IN=vlan4094 OUT= MAC=XXX SRC=XXX DST=XXX LEN=60 TOS=0x00 PREC=0x00 TTL=43 ID=31098 DF PROTO=TCP SPT=53570 DPT=7015 SEQ=37579354
9 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B4080A4445414400000000030301040200)
Sep 12 14:57:21 router chronyd[10486]: Forward time jump detected!
Aug 7 08:29:05 router kernel: rcu: INFO: rcu_preempt self-detected stall on CPU
Aug 7 08:29:05 router kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
Aug 7 08:29:05 router kernel: rcu: INFO: rcu_bh detected stalls on CPUs/tasks:
Aug 7 08:29:05 router kernel: rcu: 1-...!: (16 ticks this GP) idle=0da/0/0x1 softirq=5235988/5235996 fqs=0
Aug 7 08:29:05 router kernel: rcu: (detected by 3, t=900726558046 jiffies, g=2448077, q=1360)
Aug 7 08:29:05 router kernel: Task dump for CPU 1:
Aug 7 08:29:05 router kernel: swapper/1 R running task 0 0 1 0x0000000a
Aug 7 08:29:05 router kernel: Call trace:
Aug 7 08:29:05 router kernel: __switch_to+0xe8/0x170
Aug 7 08:29:05 router kernel: 0xffffff800809bef0
Aug 7 08:29:05 router kernel: rcu: rcu_sched kthread starved for 900726558046 jiffies! g2448077 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
Aug 7 08:29:05 router kernel: rcu: RCU grace-period kthread stack dump:
Aug 7 08:29:05 router kernel: rcu_sched I 0 11 2 0x00000008
Aug 7 08:29:05 router kernel: Call trace:
Aug 7 08:29:05 router kernel: __switch_to+0xe8/0x170
Aug 7 08:29:05 router kernel: __schedule+0x214/0x5a0
Aug 7 08:29:05 router kernel: schedule+0x38/0xa0
Aug 7 08:29:05 router kernel: schedule_timeout+0x15c/0x2a0
Aug 7 08:29:05 router kernel: rcu_gp_kthread+0x488/0x900
Aug 7 08:29:05 router kernel: kthread+0x118/0x150
Aug 7 08:29:05 router kernel: ret_from_fork+0x10/0x24
Aug 7 08:29:05 router kernel: rcu: 1-...!: (2 ticks this GP) idle=0da/0/0x1 softirq=5235996/5235996 fqs=0
Aug 7 08:29:05 router kernel: rcu: 1-...!: (7822 ticks this GP) idle=0da/0/0x1 softirq=5228421/5235996 fqs=1
Aug 7 08:29:05 router kernel: rcu: (t=900726558053 jiffies g=6702293 q=32270)
Aug 7 08:29:05 router kernel: rcu: (detected by 0, t=900726558058 jiffies, g=-255, q=1)
Aug 7 08:29:05 router kernel: Task dump for CPU 1:
Aug 7 08:29:05 router kernel: Task dump for CPU 1:
Aug 7 08:29:05 router kernel: swapper/1 R running task 0 0 1 0x0000000a
Aug 7 08:29:05 router kernel: swapper/1 R running task 0 0 1 0x0000000a
Aug 7 08:29:05 router kernel: Call trace:
Aug 7 08:29:05 router kernel: Call trace:
Aug 7 08:29:05 router kernel: dump_backtrace+0x0/0x150
Aug 7 08:29:05 router kernel: __switch_to+0xe8/0x170
Aug 7 08:29:05 router kernel: show_stack+0x14/0x20
Aug 7 08:29:05 router kernel: 0xffffff800809bef0
Aug 7 08:29:05 router kernel: sched_show_task+0x100/0x120
Aug 7 08:29:05 router kernel: dump_cpu_task+0x40/0x4c
Aug 7 08:29:05 router kernel: rcu_dump_cpu_stacks+0x90/0xcc
Aug 7 08:29:05 router kernel: rcu_check_callbacks+0x67c/0x810
Aug 7 08:29:05 router kernel: update_process_times+0x2c/0x70
Aug 7 08:29:05 router kernel: tick_sched_timer+0x54/0xd0
Aug 7 08:29:05 router kernel: __hrtimer_run_queues+0x13c/0x1d0
Aug 7 08:29:05 router kernel: hrtimer_interrupt+0xe4/0x2b0
Aug 7 08:29:05 router kernel: arch_timer_handler_phys+0x30/0x40
Aug 7 08:29:05 router kernel: handle_percpu_devid_irq+0x80/0x140
Aug 7 08:29:05 router kernel: __handle_domain_irq+0x70/0xd0
Aug 7 08:29:05 router kernel: gic_handle_irq+0x5c/0xc0
Aug 7 08:29:05 router kernel: el1_irq+0xe8/0x190
Aug 7 08:29:05 router kernel: cpuidle_enter_state+0x80/0x220
Aug 7 08:29:05 router kernel: cpuidle_enter+0x18/0x20
Aug 7 08:29:05 router kernel: do_idle+0x1d8/0x260
Aug 7 08:29:05 router kernel: cpu_startup_entry+0x20/0x40
Aug 7 08:29:05 router kernel: secondary_start_kernel+0x13c/0x170
Aug 7 08:29:05 router crond[4736]: time disparity of 15012122 minutes detected
Aug 7 08:29:05 router kernel: tdts_core_ioctl_udb_op_prog_ctrl() fail!
```
I attached another one in a file. FOr that one there are two stalls very close to one another. The first one seems to be OK and then the second one kills it.
On reboot, it does seem to print out information too (that it doesn't print on a regular reboot). I attached one such output.
Thanks!
Romain
