Solved Failed to start OPENVPN Server

[joe scian]

Hi Merlin
my router rebooted overnight. When I looked at it this morning both vpn servers disabled. I tried to restart both - both will not start. Rebooted at least twice - still wont start. These are error messages in log- any ideas or is anyone else seeing this. Seems to be an issue with updown.sh?

The only way to restart both VPN's was to do a full factory default restore.

May 14 09:52:06 kernel: ADDRCONF(NETDEV_CHANGE): tun21: link becomes ready
May 14 09:52:06 ovpn-server1[20906]: /usr/sbin/ip addr add dev tun21 10.8.0.1/24 broadcast 10.8.0.255
May 14 09:52:06 ovpn-server1[20906]: updown.sh tun21 1500 1623 10.8.0.1 255.255.255.0 init
May 14 09:52:06 ovpn-server1[20906]: WARNING: Failed running command (--up/--down): external program exited with error status: 2
May 14 09:52:06 ovpn-server1[20906]: Exiting due to fatal error

May 14 09:52:03 kernel: ADDRCONF(NETDEV_CHANGE): tun22: link becomes ready
May 14 09:52:03 ovpn-server2[20785]: /usr/sbin/ip addr add dev tun22 10.16.0.1/24 broadcast 10.16.0.255
May 14 09:52:03 ovpn-server2[20785]: updown.sh tun22 1500 1621 10.16.0.1 255.255.255.0 init
May 14 09:52:03 ovpn-server2[20785]: WARNING: Failed running command (--up/--down): external program exited with error status: 2
May 14 09:52:03 ovpn-server2[20785]: Exiting due to fatal error

 

[joe scian]

OPENVPN server is failing a reboot - after doing a factory default reflash - it worked fine - until I did a further reboot and now the same problem exists as above.

 

[L&LD]

What other scripts are you running? Did you manually and minimally configure the router afterward? Did you use a saved config file?

Your words above are not clear what you did exactly 'factory default reflash'?

 

[Butterfly Bones]

There is a discussion between Martineau and RMerlin in this thread I started. There is a bug in the use of "sh" symlinked to BCM's HND SDK. See this post and follow the discussion between them to see if this relates to your problem. I think it might, but I am certainly no pro, but it did bite me hard in my testing.
https://www.snbforums.com/threads/v...pn-ac86u-help-needed.56500/page-4#post-489924

You may have to go back and forth in their posts to follow along. Maybe tagging Martineau and RMerlin will shed some light on this?

 

[joe scian]

There is a discussion between Martineau and RMerlin in this thread I started. There is a bug in the use of "sh" symlinked to BCM's HND SDK. See this post and follow the discussion between them to see if this relates to your problem. I think it might, but I am certainly no pro, but it did bite me hard in my testing.
https://www.snbforums.com/threads/v...pn-ac86u-help-needed.56500/page-4#post-489924

You may have to go back and forth in their posts to follow along. Maybe tagging Martineau and RMerlin will shed some light on this?

Ok will review thanks

 

[joe scian]

What other scripts are you running? Did you manually and minimally configure the router afterward? Did you use a saved config file?

Your words above are not clear what you did exactly 'factory default reflash'?

Hi
I am running all scripts as per my signature however you are right my factory default reflash was from a backed up confit file when I updated to 384.11 final less than 2 weeks ago. At that time I did do a manual config from the top. Since that was less than 2 weeks ago I thought I could get away with a backed up config. The problem may stem from a thread replied to by butterflybones above. However for peace of. Mind I will do a manual config after my factory default flash this time to see if any different. It does seem such a wierd issue though tied to updown.sh that’s what’s intriguing.

 

[joe scian]

Both OPENVPN Servers will not start after doing a M&M config. It works initially but does not survive a reboot. same error messages as per my original post. Is anyone else experiencing this catastrophic error.
So has this bug been introduced in the release version og 384.11 - i did not have this issue whatsoever in the betas or in any other release prior to 384.11.
Appreciate if merlin can provide his expert opinion as to whats going on ?

 

[Martineau]

Both OPENVPN Servers will not start after doing a M&M config. It works initially but does not survive a reboot. same error messages as per my original post

If you still have the "up/down" error, can you perform another M&M? - sorry

Then when presumably the OpenVPN Server 1 initialises correctly and is UP, can you issue the following

 grep -E "^up|^down" /etc/openvpn/server1/config.ovpn

 ls -lah /etc/openvpn/server1

If you then reboot, and the OpenVPN Server 1 fails, you should check the status of the symlinked file.

i.e. if it previously showed (when it initialised) something like this

lrwxrwxrwx    1 admin    root          27 May 14 08:47 updown.sh -> /jffs/scripts/openvpn-event

I would rename '/jffs/scripts/openvpn-event', and create a new '/jffs/scripts/openvpn-event', simply containing

#!/bin/sh
logger -t "($(basename $0))" "Executed"

then try to start OpenVPN Server 1 again.

 

[joe scian]

ok - will try try this - thanks - so it seems from your post and butterfly bones this is a well documented issue - im surprised noone else seems to be complaining

 

[Martineau]

ok - will try try this - thanks - so it seems from your post and butterfly bones this is a well documented issue - im surprised noone else seems to be complaining

I believe the BCM SDK bug only affects HND models.

If the following produces output, then you're affected

ls -lahF /usr/sbin | grep memaccess

 

[joe scian]

I believe the BCM SDK bug only affects HND models.

If the following produces output, then you're affected

ls -lahF /usr/sbin | grep memaccess

so I assume the ac5300 is not an HND model - so why am i getting this issue

 

[Martineau]

so I assume the ac5300 is not an HND model - so why am i getting this issue

If there is no 'memaccess' utility installed on your AC5300 then you probably have a different problem.

 

[joe scian]

so I assume the ac5300 is not an HND model - so why am i getting this issue

no I do not get any output from that command

 

[joe scian]

Martineau

If there is no 'memaccess' utility installed on your AC5300 then you probably have a different problem.

Martineau - I noticed the \usr\sbin\updown.sh has octal 0700 NOT 0755 - is this an issue - i guess not its a read only system file

 

[Martineau]

I noticed the \usr\sbin\updown.sh has octal 0700 NOT 0755 - is this an issue

If I said YES,

Q. What would you do to fix it?
A. Nowt, You can't chmod the file as it is on a READ-ONLY file system.

 

[joe scian]

If you still have the "up/down" error, can you perform another M&M? - sorry

Then when presumably the OpenVPN Server 1 initialises correctly and is UP, can you issue the following

 grep -E "^up|^down" /etc/openvpn/server1/config.ovpn

 ls -lah /etc/openvpn/server1

If you then reboot, and the OpenVPN Server 1 fails, you should check the status of the symlinked file.

i.e. if it previously showed (when it initialised) something like this

lrwxrwxrwx    1 admin    root          27 May 14 08:47 updown.sh -> /jffs/scripts/openvpn-event

I would rename '/jffs/scripts/openvpn-event', and create a new '/jffs/scripts/openvpn-event', simply containing

#!/bin/sh
logger -t "($(basename $0))" "Executed"

then try to start OpenVPN Server 1 again.

Hi Martineau

I was too lazy to do another M&M config. What I did was rename Openvpn-event to 1openvpn-event and created a new openvpn-event file containing the 1 line you mentioned. Immediately I got OPENVPN Server 1 up and i restarted OPENVPN Server 2 and that remained up as well. This was a major win since with the original openvpn-event file I couldnt get either server up on a restart or reboot. When I rebooted again OPENVPN Server 1 remained UP - and OPENVPN Server 2 - "Initialinzing the settings of OpenVPN server now, please wait a few minutes to let the server to setup completed before VPN clients establish the connection. " gets this message on OVPN configuration page. However all I did was stop it and restart it and now I have both servers up and available.

I dont get that pesky failure message like I did before below -

Code:
May 14 09:52:03 kernel: ADDRCONF(NETDEV_CHANGE): tun22: link becomes ready
May 14 09:52:03 ovpn-server2[20785]: /usr/sbin/ip addr add dev tun22 10.16.0.1/24 broadcast 10.16.0.255
May 14 09:52:03 ovpn-server2[20785]: updown.sh tun22 1500 1621 10.16.0.1 255.255.255.0 init
May 14 09:52:03 ovpn-server2[20785]: WARNING: Failed running command (--up/--down): external program exited with error status: 2
May 14 09:52:03 ovpn-server2[20785]: Exiting due to fatal error

Now I get

May 14 20:39:28 kernel: ADDRCONF(NETDEV_CHANGE): tun22: link becomes ready
May 14 20:39:28 ovpn-server2[9031]: /usr/sbin/ip addr add dev tun22 10.16.0.1/24 broadcast 10.16.0.255
May 14 20:39:28 ovpn-server2[9031]: updown.sh tun22 1500 1621 10.16.0.1 255.255.255.0 init
May 14 20:39:28 (updown.sh): Executed

IM very happy - so obviously this openvpn-event file is causing issues - Can I leave that 1 liner in the openvpn-event file going forward?

 

[Martineau]

Hi Martineau

I was too lazy to do another M&M config. What I did was rename Openvpn-event to 1openvpn-event and created a new openvpn-event file containing the 1 line you mentioned. Immediately I got OPENVPN Server 1 up and i restarted OPENVPN Server 2 and that remained up as well. This was a major win since with the original openvpn-event file I couldnt get either server up on a restart or reboot. When I rebooted again OPENVPN Server 1 remained UP - and OPENVPN Server 2 - "Initialinzing the settings of OpenVPN server now, please wait a few minutes to let the server to setup completed before VPN clients establish the connection. " gets this message on OVPN configuration page. However all I did was stop it and restart it and now I have both servers up and available.

I dont get that pesky failure message like I did before below -

Code:
May 14 09:52:03 kernel: ADDRCONF(NETDEV_CHANGE): tun22: link becomes ready
May 14 09:52:03 ovpn-server2[20785]: /usr/sbin/ip addr add dev tun22 10.16.0.1/24 broadcast 10.16.0.255
May 14 09:52:03 ovpn-server2[20785]: updown.sh tun22 1500 1621 10.16.0.1 255.255.255.0 init
May 14 09:52:03 ovpn-server2[20785]: WARNING: Failed running command (--up/--down): external program exited with error status: 2
May 14 09:52:03 ovpn-server2[20785]: Exiting due to fatal error

Now I get

May 14 20:39:28 kernel: ADDRCONF(NETDEV_CHANGE): tun22: link becomes ready
May 14 20:39:28 ovpn-server2[9031]: /usr/sbin/ip addr add dev tun22 10.16.0.1/24 broadcast 10.16.0.255
May 14 20:39:28 ovpn-server2[9031]: updown.sh tun22 1500 1621 10.16.0.1 255.255.255.0 init
May 14 20:39:28 (updown.sh): Executed

IM very happy - so obviously this openvpn-event file is causing issues - Can I leave that 1 liner in the openvpn-event file going forward?

Glad we got to the root cause.

Unless you have a need to execute a custom script based on any of the possible OpenVPN event triggers, then you should be able to delete '/jffs/scripts/openvpn-event' although you may get the following message

NOTE: starting with OpenVPN 2.1, '--script-security 2' or higher is required to call user-defined scripts or executables

in Syslog during the OpenVPN servers initialisation.

So the $64,000 question is.... who created the faulty '/jffs/scripts/openvpn-event' script, or more importantly, what are its dodgy contents?

Another hypothesis, assuming you retained the faulty script under a different name, the worst-case scenario would be that you have a bad /jffs and the area occupied by the original script can't be read correctly. In which case it may be necessary in the near future to backup the contents of /jffs and explicitly reformat /jffs etc.

P.S. Some clearly have no understanding of root cause / effect analysis i.e. the HND only BCM SDK issue occurs after the OpenVPN process has successfully initialised, yet silently fails to execute a custom script.

 

[joe scian]

Thank you for all your help Martineau - so very much appreciated -

 

[ZoloN]

hi all,

I use site2site OpenVPN connection between two RT-AC66U B1 and after short internet outage on server side I lost the VPN connection. in the log I could find following messages:

Mar 13 10:46:03 ovpn-server1[1690]: Multiple --up scripts defined. The previously configured script is overridden.
Mar 13 10:46:03 ovpn-server1[1691]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Mar 13 10:46:03 ovpn-server1[1691]: WARNING: Failed running command (--up/--down): external program exited with error status: 2

after investigation I found out, that something has overwritten my Custom Configuration - there were following commands:

up "/bin/sh /jffs/etc/profile"
script-security 3

instead of the right command entered by me:

reneg-sec 432000
username-as-common-name
push "route 192.168.xx.0 255.255.255.0"
client-config-dir /jffs/configs/openvpn/ccd1/
route 192.168.yy.0 255.255.255.0

has anyone idea, what could overwritten the config?

 

[ColinTaylor]

after investigation I found out, that something has overwritten my Custom Configuration - there were following commands:

up "/bin/sh /jffs/etc/profile"
script-security 3

We've seen this reported on more than one occasion. It's believed to be malware.

Factory reset your router, update it to the latest firmware and manually configure it. Do not enable Web Access from WAN (Administration - System).