[SOLVED-FOR REAL] rc_service: skip the event: start_firewall.

[unsynaps]

This is an issue that cropped up recently and I am at a loss for why it is happening.

When the router is restarted (method does not matter) there is something in the syslog "rc_service: skip the event: start_firewall.". Basically the firewall does not start.

When this happens I can only access things via IP and can not get to the internet (duh, MASQ isnt running). To fix it I have to log in via SSH and run 'service start_firewall' then re-run my tunnel broker script in /jffs/scripts/wan_start (FiOS likes to give me a new IP every single time I restart the router) and sometimes 'service restart_upnp' as well. Once I do this everything is back to normal.

I can post a full clean syslog in a little while. Need to restart to get it but room mate is gaming at the moment.

 

[Zirescu]

What router and firmware version?

 

[unsynaps]

Gah, dummy me.

RT-AC87R
380.64_2

 

[Zirescu]

Will need to see the full log files to see what might be going on. Looks like you're doing some custom configuration scripting any chance one of those is mangling the firewall start? Most of the log files I've seen when that messages comes through the WAN connection wasn't up when the router attempted to get an IP.

 

[unsynaps]

Still does it with all custom scripts disabled. Even shut off the option in the settings.

Oh? The WAN is a bit weird on my FiOS.

On a reboot after a while the red light comes on meaning no internet. Then it goes off. Then it turns back on again. Then it goes off and the boot is complete.

 

[Zirescu]

Might need to do a factory reset and minimal config to see if it still happens. I know a few other users reported issues with FIOS and the WAN, I think the solution for most was to change the DHCP Query frequency from Aggressive mode to Normal mode under WAN->Internet Connection.

 

[unsynaps]

Just found a line in the log.

WAN Connection: ISP's DHCP did not function properly.

http://pastebin.com/WxmxBMFT

 

[Zirescu]

Yeah, try changing the DHCP query frequency and see if that fixes the issue - you may need to power down the FIOS modem for 10 minutes and then reconnect. YMMV

 

[unsynaps]

Humm set it to 'Normal Mode' and still getting the 'WAN Connection: ISP's DHCP did not function properly.' error in syslog.

I wonder if it is somehow related to me getting a new IP every single time I restart the router.

 

[john9527]

Humm set it to 'Normal Mode' and still getting the 'WAN Connection: ISP's DHCP did not function properly.' error in syslog.

That has shown up in every syslog I've ever looked at (the first attempt to bring up the WAN occurs before everything is ready). As long as it's the very first entry relating to the WAN and doesn't occur multiple times it's 'normal'.

Back to the firewall.....do you see any 'waiting for start_firewall' messages prior to the 'skip the event' message.

 

[unsynaps]

Searched the log and the only two lines with 'firewall' in them are as follows.

Jul 31 20:00:36 rc_service: udhcpc 820:notify_rc start_firewall
Jul 31 20:00:36 rc_service: waitting "restart_nasapps" via hotplug ...
Jul 31 20:00:36 TunnelBroker: Not Connected. Sleeping 15 seconds then retrying.
Jul 31 20:00:51 TunnelBroker: Not Connected. Sleeping 15 seconds then retrying.
Jul 31 20:00:56 qtn: bootcfg.tgz exists
Jul 31 20:01:02 dfs: start dfs scan
Jul 31 20:01:03 nodfs_scan: complete
Jul 31 20:01:06 rc_service: skip the event: start_firewall.

 

[john9527]

Ahh...good info. It looks like it's your TunnelBroker connection causing things to hang up for 30 secs. If the scheduler (rc) can't start the process in 30 sec it gives up and 'skips' it.

 

[unsynaps]

Even if that script is disabled I still have the issue of the firewall not starting. Ill disable it once more to verify.

 

[john9527]

Even if that script is disabled I still have the issue of the firewall not starting. Ill disable it once more to verify.

If it still fails grab the same part of the syslog between the start and skip.....

 

[unsynaps]

Jul 31 20:00:37 rc_service: udhcpc 772:notify_rc start_firewall
Jul 31 20:00:37 rc_service: waitting "restart_nasapps" via hotplug ...
Jul 31 20:00:56 qtn: bootcfg.tgz exists
Jul 31 20:01:03 dfs: start dfs scan
Jul 31 20:01:03 nodfs_scan: complete
Jul 31 20:01:07 rc_service: skip the event: start_firewall.

 

[john9527]

OK....it's still timing out getting started. Two things to try....
- it's also waiting on nasapps/hotplug....try unplugging any USB devices in case there's a problem there
- I also see it logging something about bootcfg on the Quantenna chipset. Try powering down the router. Remove the power plug. Then turn it on and off a couple of times. Replug the power and power up. (this can sometime free things up if there is a circuit latched up).

 

[unsynaps]

Looking like its the hanging on the 'rc_service: waitting "restart_nasapps" via hotplug ...' line.

Pulled the USB drive and everything came up as it should.

Jul 31 20:00:34 custom script: Found wan-start, but custom script execution is disabled!
Jul 31 20:00:34 rc_service: udhcpc 722:notify_rc start_firewall
Jul 31 20:00:34 rc_service: udhcpc 722:notify_rc stop_dnsmasq
Jul 31 20:00:34 rc_service: waitting "start_firewall" via udhcpc ...
Jul 31 20:00:35 start_nat_rules: apply the nat_rules(/tmp/nat_rules_vlan2_vlan2)!
Jul 31 20:00:35 kernel: nf_conntrack_rtsp v0.6.21 loading
Jul 31 20:00:35 kernel: nf_nat_rtsp v0.6.21 loading
Jul 31 20:00:37 rc_service: udhcpc 722:notify_rc start_dnsmasq
Jul 31 20:00:37 custom config: Found /jffs/configs/hosts.add, but custom configs are disabled!

 

[john9527]

Progress......next step is to hook up the drive to a PC and run chkdsk or fsck as appropriate.

 

[unsynaps]

Yeah turned all my boot scripts back on. Had to add a 5s wait on the tunnel broker script to wait for iptables to be up but everything is coming up milhouse now.

Now to find out whats being stupid about this flash drive. Its like brand new. Partitioned and formatted it (ext4) on the router itself.

Thanks for the help narrowing it down.

 

[Zirescu]

I've had issues with flash drives and EXT4 due to the journaling. I've stuck with EXT2 for the USB and SD cards and not had any issues since.