Restart VPN server when it breaks?

[Frejoh4666]

So I run an VPN server on the router. I access to it with OpenVPN for android. But sometimes it just breaks (around once a month) and I'm unable to connect and need to restart the service.
When it works I get,

ovpn-server1[25979]: read UDPv4 [CMSG=8|ECONNREFUSED|CMSG=8|ECONNREFUSED]: Connection refused (fd=8,code=111)
ovpn-server1[25979]: read UDPv4 [CMSG=8|ECONNREFUSED]: Connection refused (fd=8,code=111)

I guess that is normal messages, about 1 message each minute when I use the phone. But today it broke, and I manage to find this,

Jul  4 06:52:14 kernel: vpnserver1/25981: potentially unexpected fatal signal 11.
Jul  4 06:52:14 kernel: Pid: 25981, comm:           vpnserver1
Jul  4 06:52:14 kernel: CPU: 1    Tainted: P             (2.6.36.4brcmarm #1)
Jul  4 06:52:14 kernel: PC is at 0x40422d8c
Jul  4 06:52:14 kernel: LR is at 0x0
Jul  4 06:52:14 kernel: pc : [<40422d8c>]    lr : [<00000000>]    psr: 20000010
Jul  4 06:52:14 kernel: sp : beda7e00  ip : 000009e0  fp : 000009e0
Jul  4 06:52:14 kernel: r10: 4044334c  r9 : 40443180  r8 : 000c0e38
Jul  4 06:52:14 kernel: r7 : 404431b4  r6 : 000bca08  r5 : 4043d4f8  r4 : 00000088
Jul  4 06:52:14 kernel: r3 : 00002268  r2 : 404431b4  r1 : 000021e0  r0 : 00000000
Jul  4 06:52:14 kernel: Flags: nzCv  IRQs on  FIQs on  Mode USER_32  ISA ARM  Segment user
Jul  4 06:52:14 kernel: Control: 10c53c7d  Table: 9a6bc04a  DAC: 00000015

Then I'm unable to connect the server and I don't get the normal "Connection refused" messages (I'm not sure I always get that message when it breaks). Is there a way to notice that the VPN server breaks and restart it? As I'm unable to reach the router when I'm not at home and the VPN server is dead. Running it one a RT-AC66U B1.

 

[ColinTaylor]

There should already be a mechanism in place that restarts the OpenVPN server if it fails.

# cru l
:
:
*/2 * * * * /etc/openvpn/server1/vpn-watchdog1.sh #CheckVPNServer1#

# cat /etc/openvpn/server1/vpn-watchdog1.sh
#!/bin/sh
if [ -z "$(pidof vpnserver1)" ]
then
   service restart_vpnserver1
fi

 

[Frejoh4666]

There should already be a mechanism in place that restarts the OpenVPN server if it fails.

# cru l
:
:
*/2 * * * * /etc/openvpn/server1/vpn-watchdog1.sh #CheckVPNServer1#

# cat /etc/openvpn/server1/vpn-watchdog1.sh
#!/bin/sh
if [ -z "$(pidof vpnserver1)" ]
then
   service restart_vpnserver1
fi

I have

*/2 * * * * /etc/openvpn/server1/vpn-watchdog1.sh #CheckVPNServer1#

in cru l

 

[ColinTaylor]

Same as me then. So there's nothing you need to do as the server will be restarted within two minutes.

 

[Frejoh4666]

Same as me then. So there's nothing you need to do as the server will be restarted within two minutes.

But it doesn't. It has been 16 hours, and it still doesn't work. It will start working once I restart the server.

 

[ColinTaylor]

Can you confirm the output of this command:

cat /etc/openvpn/server1/vpn-watchdog1.sh

Also, what firmware version are you using?

 

[Frejoh4666]

Can you confirm the output of this command:

cat /etc/openvpn/server1/vpn-watchdog1.sh

Also, what firmware version are you using?

#!/bin/sh
if [ -z "$(pidof vpnserver1)" ]
then
   service restart_vpnserver1
fi

386.14

 

[ColinTaylor]

Thanks. That all looks in order so I don't know why you can't connect now.

I think we need to look at the bigger picture to see if there's something else that we're missing. Can you post the output of ps w as well as the complete, unedited System Log.

P.S. Also make sure that System Log - General Log > Log only messages more urgent than is set to "all".

P.P.S. The final firmware release for your router was 386.14_2 so it's probably worth installing that anyway as it has an updated version of OpenVPN.

 

[Frejoh4666]

I can update 386.14_2 tomorrow, once I do the server will start working again as the vpn server will restart.

I don't like that "unbound_rpz.sh" runs every 15min, but that what the unbound creator has set it to, so...
I rather have it run once a day, but I don't think modifying the script is a good idea.

cru l if you want it

12 4 * * * curl -o \/opt\/var\/lib\/unbound\/root\.hints https://www.internic.net/domain/named.cache #root_servers#
59 * * * * /jffs/addons/unbound/unbound_stats.sh generate #Unbound_Stats.sh#
57 * * * * /jffs/addons/unbound/unbound_log.sh #Unbound_Log.sh#
*/10 * * * * /jffs/scripts/ntpmerlin generate #ntpMerlin#
*/2 * * * * /etc/openvpn/server1/vpn-watchdog1.sh #CheckVPNServer1#
1 0 * * * /opt/bin/find /opt/var/lib/unbound/unbound.log -size +10M -exec rm -f {} \; #unboundLOG#
25 23 * * Mon sh /jffs/scripts/firewall banmalware #Skynet_banmalware#
21 1 * * Mon sh /jffs/scripts/firewall update #Skynet_autoupdate#
0 * * * * sh /jffs/scripts/firewall save #Skynet_save#
00 2 * * Thu /bin/sh /opt/share/diversion/file/update-bl.div reset #Diversion_UpdateBL#

 

[ColinTaylor]

Thanks for the files. It looks like when the VPN server crashed the process didn't die but became a zombie, which is... unusual. It's also the reason why the cron job doesn't attempt to restart the server.

I think the best course of action would be to install the newer firmware and see if that fixes the problem. Otherwise you're left with ugly workarounds, like restarting the server periodically (e.g. twice a day) irrespective of whether it's crashed or not.

If you run the service restart_vpnserver1 command does the server come back to life? I'm assuming it does.

 

[Frejoh4666]

Yes, "service restart_vpnserver1" is what I do to fix the server (I used to turn it on/off in the GUI or restart the router before I found the command). And I thought about adding a cron job to restart it every week, but I thought there must be a better way. But I guess not.

Thanks for the help.

 

[elorimer]

Consider setting up the second server, so you can connect to that and restart the first.