Need a script that auto reboot if internet is down

[semsem]

Hi,

I have merlin 380.64 on asus ac68u and sometimes internet goes down while I am away from home and need to reboot the router to be up again. Can someone please help with a simple script that checks for internet connectivity every few minutes and if it’s down, perform a reboot? Thanks in advance.

 

[Martineau]

Can someone please help with a simple script that checks for internet connectivity every few minutes and if it’s down, perform a reboot?

There are many such scripts on the forum/Google search etc., and most use PING to a target host.

Mine is no exception but optionally additionally initiates a 12MB file download (or a 15 Byte IP retrieval!), just in case a false positive PING is encountered.
(Also using curl/wget allows an explicit WAN interface to be used - such as eth0/ppp0 or even a VPN tun1x interface - to verify the physical data transfer request)
EDIT: 29/11/2018 v1.10 Fix 'Monitoring WAN connection using PING method to xxx.xxx.xxx.xxx check FAILED' message
EDIT: 14/10/2018 v1.09 Add 'curlrate=' option
EDIT: 25/09/2018 v1.08 Add 'i=','ping=', 'tries=' ,'fails=','forcebig|forcesmall' and 'noaction' options
EDIT: 11/08/2018 v1.07 Ignore local IP PING targets, and add 'googleonly' option
EDIT: ??/??/???? v1.06 Add display of actual transfer time for curl data transmission
EDIT: 11/05/2018 v1.05 Renamed as ChkWAN (wan-check was too similar to wan-start), and also added 'quiet' and 'nowait' options
EDIT: 12/01/2018 v1.04 now will honour a cron/cru schedule (if found) rather than running continuously in the background and checking every 30secs!
WAN Check script

Spoiler: Script help

#============================================================================================ © 2016-2018 Martineau v1.09
#
# Monitor WAN connection state using PINGs to multiple hosts, or a single cURL 15 Byte data request and optionally a 12MB/500B WGET/CURL data transfer.
#         NOTE: The cURL data transfer rate/perfomance threshold may also be checked e.g. to switch from a 'slow' (Dual) WAN interface.
#         Usually the Recovery action (REBOOT or restart the WAN) occurs in about 90 secs (PING ONLY) or in about 03:30 mins for 'force' data download
#
# Usage:    ChkWAN  [help|-h]
#                   [reboot | wan | noaction] [force[big | small]] [nowait] [quiet] [once] [i={[wan0|wan1]}] [googleonly] [curl] [ping='ping_target[,..]']
#                   [tries=number] [fails=number] [curlrate=number]
#
#           ChkWAN
#                   Will REBOOT router if the PINGs to ALL of the hosts FAILS
#           ChkWAN  force
#                   Will REBOOT router if the PINGs to ALL of the hosts FAIL, but after each group PING attempt, a physical 12MByte data download is attempted.
#           ChkWAN  forcesmall
#                   Will REBOOT router if the PINGs to ALL of the hosts FAIL, but after each group PING attempt, a physical 500Byte data download is attempted.
#                   (For users on a metered connection assuming that the 15Byte cURL is deemed unreliable?)
#           ChkWAN  wan
#                   Will restart the WAN interface (instead of a FULL REBOOT) if the PINGs to ALL of the hosts FAIL
#           ChkWAN  curl
#                   Will REBOOT router if cURL (i.e. NO PINGs attempted) fails to retrieve the remote end-point IP address of the WAN (Max 15bytes)
#           ChkWAN  cron
#                   Will REBOOT router if the PINGs to ALL of the hosts FAILS, cron entry is added: Runs every 5mins.
#                   'cru a ChkWAN "*/5 * * * * /jffs/scripts/ChkWAN.sh"'
#           ChkWAN  nowait
#                   By default the script will wait 10 secs before actually starting the check; 'nowait' (when used by cron) skips this delay.
#           ChkWAN  googleonly
#                   Only the two Google DNS severs will be PING'd - WAN Gateway/local DNS config will be ignored
#           ChkWAN  i=wan1 noaction
#                   In a Dual-WAN environment check WAN1 interface, but if it's DOWN simply return RC=99
#           ChkWAN  ping=1.2.3.4,1.1.1.1
#                   PING the two hosts 1.2.3.4 and 1.1.1.1, rather than the defaults.
#           ChkWAN  tries=1 fails=1
#                   Reduce the number of retry attempts to 1 instead of the default 3 and maximum number of fails is 1 rather than 3
#           ChkWAN  force curlrate=1000000
#                   If the 12MB average curl transfer rate is <1000000 Bytes per second (1MB), then treat this as a FAIL

The script should be called asynchronously at the end of wan-start (or v1.04 allows scheduling via cron/cru)
e.g. PING only

sh /jffs/scripts/wan-check.sh &

or force the data download to verify PING result.

sh /jffs/scripts/wan-check.sh  "force" &

Once invoked, the script will run continuously in the background (default using sleep commands is check WAN connectivity every 30 secs) however, if you have scheduled the script via cron/cru then the executing script instance will terminate once it has found the WAN connection to be UP.

The default script variables

TRIES=3
INTERVAL_ALL_FAILED_SECS=10

and the 5 target PING hosts usually initiates the recovery ACTION within 90 secs or within <4 mins if the data download is requested as part of the sanity check.

 

[Goobi]

@Martineau I installed this script but don’t want to reboot after placing “sh /jffs/scripts/wan-check.sh &” in wan-start. If I run that command from the shell, it won’t stay running if I log off. It also prints to the terminal every 30 seconds. Anyway to invoke the script and make sure it stays running in the background even when I log off?

Also, if I want to restart the wan as opposed to doing a full reboot, would that be accomplished by setting the action to “wan” in the script?

Thanks in advance.

 

[Martineau]

@Martineau I installed this script but don’t want to reboot after placing “sh /jffs/scripts/wan-check.sh &” in wan-start. If I run that command from the shell, it won’t stay running if I log off.

The script was written to be called as a background task from wan-start but you could try the 'nohup' command etc. or simply restart the wan?

service restart_wan

It also prints to the terminal every 30 seconds.

I've uploaded v1.05 to allow you to specify the 'quiet' option, although when running the script manually via the command line (as a foreground task) then the echo statements are usually deemed useful.

Also, if I want to restart the wan as opposed to doing a full reboot, would that be accomplished by setting the action to “wan” in the script?

As per the script help: Use the 'wan' option when starting the script:

sh /jffs/scripts/wan-check.sh wan &

 

[Goobi]

Simply restarting the wan allowed the script to start in the background as you suggested. Thanks!

 

[Goobi]

@Martineau I have a issue that I am wondering if this script slightly modified could help me with.

I run dnscrypt and occasionally it crashes causing my internet to go down. If I restart it I am back in business.

I checked in the dnscrypt thread and was told that it does a self check every 12 hours and restarts itself if it is down. This is too wide of a gap to have it restart itself in my opinion.

I was wondering if I could leverage this script to perform it’s checking for a know good domain address like www.google.com and if it fails, run this command which terminates and restarts dnscrypt:

"/jffs/dnscrypt/manager dnscrypt-start"

I figured that would be less aggressive then restarting the router or restarting the wan.

Problem is I have no idea how to go about coding that and wanted to check to see if you got any pointers on how to go about doing that if not too complex?

Thanks in advance.

 

[HuskyHerder]

@Martineau

I am able to run the script manually and it runs every 30 seconds as desired for a manual run. However I have a few questions etc.

I was able to schedule the script by manually entering the scheduling parameters from my ssh client. I am not sure if that was proper or should have been accomplished via the script ? If it was to be accomplished via the script thats where I failed.

One question is in the posts above (v1.04) it displays the script as "wan-check" when using "help".

When I use the help command (v1.05) all references are to "ChkWAN" I am assuming there were naming changes that took place ? I named mine as "ChkWAN", maybe its a non issue but I was trying to maintain uniformity on my end.

Thanks,

HH

 

[Martineau]

@Martineau I was able to schedule the script by manually entering the scheduling parameters from my ssh client. I am not sure if that was proper or should have been accomplished via the script ? If it was to be accomplished via the script thats where I failed.

Using cron could cause (in some edge cases) unexpected invocation of the ChkWAN.sh script, so whilst I tried to make the script user-friendly by removing the need for novice-users to learn about cron/cru, I did attempt to include self scheduling, which may have been broken in the recent updates).

Clearly, for users that are comfortable with cron/cru, they can indeed customise the scheduling of the script to suit their requirements.

However, it probably makes more sense for the ChkWAN.sh script to be invoked by wan-start when a known successful WAN connection should be monitored.

One question is in the posts above (v1.04) it displays the script as "wan-check" when using "help".

When I use the help command (v1.05) all references are to "ChkWAN" I am assuming there were naming changes that took place ? I named mine as "ChkWAN", maybe its a non issue but I was trying to maintain uniformity on my end.

Apologies, I decided 'wan-check' was too similar to the system script name 'wan-start' and as I also have scripts 'ChkVPN.sh' etc. I did indeed rename the script.

 

[Martineau]

@Martineau I run dnscrypt and occasionally it crashes causing my internet to go down. If I restart it I am back in business.

I checked in the dnscrypt thread and was told that it does a self check every 12 hours and restarts itself if it is down. This is too wide of a gap to have it restart itself in my opinion.

I was wondering if I could leverage this script to perform it’s checking for a know good domain address like www.google.com and if it fails, run this command which terminates and restarts dnscrypt.

I don't run dnscrypt, but in answer to your query, then yes the script could be modified - although it's off-topic.

Personally, I would suggest that as dnscrypt is obviously critical, rather than poll its state on a regular basis, it would be better to have an event driven script to restart dnscrypt in real-time when it 'crashes' (assuming of course that the root cause of the crash cannot be identified and fixed by the author).

i.e. You could use my crude SyslogEventMonitor.sh script assuming there is some dnscrypt fail message available in Syslog that could be used to trigger an immediate restart of dnscrypt?

 

[Goobi]

Thanks for your help with this and apologies for originally posting here as I don’t want to take this thread off topic.

 

[xtropodx]

There are many such scripts on the forum/Google search etc., and most use PING to a target host.

Mine is no exception but optionally additionally initiates a 12MB file download (or a 15 Byte IP retrieval!), just in case a false positive PING is encountered.
(Also using curl/wget allows an explicit WAN interface to be used - such as eth0/ppp0 or even a VPN tun1x interface - to verify the physical data transfer request)
EDIT: 11/05/2018 v1.05 Renamed as ChkWAN (wan-check was too similar to wan-start), and also added 'quiet' and 'nowait' options
EDIT: 12/01/2018 v1.04 now will honour a cron/cru schedule (if found) rather than running continuously in the background and checking every 30secs!
WAN Check and reboot script

Spoiler: Script help

#============================================================================================ © 2016-2018 Martineau v1.05
#
# Monitor WAN connection state using PINGs to multiple hosts, or a single curl data request and optionally a 12MB WGET/CURL data transfer.
#         Usually the Recovery action (REBOOT or restart the WAN) occurs in about 90 secs (PING ONLY) or in about 03:30 mins for 'force' data download
#
# Usage:    ChkWAN  [help|-h] |  [reboot | wan] [force | curl] [nowait] [quiet] [once]
#
#           ChkWAN
#                      Will REBOOT router if the PINGs to ALL of the hosts FAILS
#           ChkWAN  force
#                      Will REBOOT router if the PINGs to ALL of the hosts FAIL, but after each group PING attempt, a physical 12MB data download is attempted.
#           ChkWAN  wan
#                      Will restart the WAN interface (instead of a FULL REBOOT) if the PINGs to ALL of the hosts FAIL
#           ChkWAN  curl
#                      Will REBOOT router if curl (i.e. NO PINGs attempted) fails to retrieve the remote end-point IP address of the WAN
#           ChkWAN  cron
#                      Will REBOOT router if the PINGs to ALL of the hosts FAILS, cron entry is added: Runs every 5mins.
#                      'cru a ChkWAN "*/5 0 * * * /jffs/scripts/ChkWAN.sh"'
#           ChkWAN  nowait
#                      By default the script will wait 10 secs before actually starting the check; 'nowait' (when used by cron) skips this delay.

The script should be called asynchronously at the end of wan-start (or v1.04 allows scheduling via cron/cru)
e.g. PING only

sh /jffs/scripts/wan-check.sh &

or force the data download to verify PING result.

sh /jffs/scripts/wan-check.sh  "force" &

Once invoked, the script will run continuously in the background (default using sleep commands is check WAN connectivity every 30 secs) however, if you have scheduled the script via cron/cru then the executing script instance will terminate once it has found the WAN connection to be UP.

The default script variables

TRIES=3
INTERVAL_ALL_FAILED_SECS=10

and the 5 target PING hosts usually initiates the recovery ACTION within 90 secs or within <4 mins if the data download is requested as part of the sanity check.

I'm trying to get this to run, but having an issue as it's not working.

I have set up;
/jffs/scripts/ChkWAN.sh

And inside following;
/jffs/scripts/wan-start

#!/bin/sh
sh /jffs/scripts/ChkWAN.sh wan &
TRIES=3
INTERVAL_ALL_FAILED_SECS=10

I've restarted wan but how can I check that this script is running? I don't see anything in the logs. I've also tested it by removing connection from router & replugging it back in. And still nothing. I'm still having to manually turn off/on WAN.

What am I missing?
I'm using RT-AC3200 merlin bridged behind modem.

Thanks!

 

[Martineau]

I'm trying to get this to run, but having an issue as it's not working.

I have set up;
/jffs/scripts/ChkWAN.sh

And inside following;
/jffs/scripts/wan-start

#!/bin/sh
sh /jffs/scripts/ChkWAN.sh wan &
TRIES=3
INTERVAL_ALL_FAILED_SECS=10

I've restarted wan but how can I check that this script is running? I don't see anything in the logs. I've also tested it by removing connection from router & replugging it back in. And still nothing. I'm still having to manually turn off/on WAN.

What am I missing?
I'm using RT-AC3200 merlin bridged behind modem.

Thanks!

There should be messages in Syslog...

e.g. I have a cron job

0 * * * * /jffs/scripts/ChkWAN.sh force wan once nowait #WAN_Check_1hr#

that runs every hour and produces

grep ChkWAN /tmp/syslog.log

12:00:01 RT-AC68U user.warn (ChkWAN.sh): 30913 Monitoring WAN connection using 5 target PING hosts (92.2.208.1 9.9.9.9 208.67.222.222 8.8.8.8 8.8.4.4) (Tries=3)
12:00:01 RT-AC68U user.warn (ChkWAN.sh): 30913 Monitoring pass 1 out of 3
12:00:02 RT-AC68U user.warn (ChkWAN.sh): 30913 Starting curl data transfer.....(Expect 12MB approx @3.1MB/sec on 20Mbps download = 00:04 secs)
12:00:07 RT-AC68U user.warn (ChkWAN.sh): 30913 Curl 12MB transfer took: 4.804527 secs
12:00:08 RT-AC68U user.warn (ChkWAN.sh): 30913 Monitoring WAN connection OK.....(Successful ping to '92.2.208.1'); Terminating due to ACTIVE cron schedule

What happens if you run the script manually from the command prompt?

 

[xtropodx]

No, no message. And when I try to run it from the WinSCP command prompt, assuming I'm running/starting it correctly, I get following & then nothing.

 

[octopus]

Seems you have missed shebang at top of script and have you set write premission on file. Make sure you have copy and paste script right.

 

[xtropodx]

Seems you have missed shebang at top of script and have you set write premission on file. Make sure you have copy and paste script rihght.

Thanks. Seems this may have been the issue as copied/pasted again got it working.

@Martineau Is there a way to code in following?
-check WAN every hour;
-if above fails, stop/start WAN first - repeat x times/hours.
-Only if previous fails, after x tries/hours, reboot.

So, basically restart WAN first & then reboot as a fail safe?

Also, just confirming pinging my modem comes back as "Successful ping" despite no internet connection? Is it possible to change script so I could ping www.google.com instead & if so how/where would I amend this?

Thanks!

 

[xtropodx]

So last night this was put to the test & it failed.
Logged off my PC last night & this morning for whatever reason internet wouldn't work (looks like it stopped working middle of the night).

Get the following repeatedly in the logs (over several hours);

Aug  9 03:30:31 (ChkWAN.sh): 513 Monitoring WAN connection using PING method to xxx.xxx.x.x check FAILED
Aug  9 03:30:31 (ChkWAN.sh): 513 Monitoring WAN connection OK.....(Successful ping to '192.168.20.1'). Will check WAN again in 30 secs

But I can't tell/see anywhere where the WAN has automatically been turned off/on & I can't see it pinging or attempting to download any file for false positives. How can I check that the WAN is auto-restarting, as I've had to do it manually?

Thanks.

 

[CaptainSTX]

So last night this was put to the test & it failed.
Logged off my PC last night & this morning for whatever reason internet wouldn't work (looks like it stopped working middle of the night).

Get the following repeatedly in the logs (over several hours);

Aug  9 03:30:31 (ChkWAN.sh): 513 Monitoring WAN connection using PING method to xxx.xxx.x.x check FAILED
Aug  9 03:30:31 (ChkWAN.sh): 513 Monitoring WAN connection OK.....(Successful ping to '192.168.20.1'). Will check WAN again in 30 secs

But I can't tell/see anywhere where the WAN has automatically been turned off/on & I can't see it pinging or attempting to download any file for false positives. How can I check that the WAN is auto-restarting, as I've had to do it manually?

Thanks.

If you can't get the script working to your satisfaction you might look at the 3G Remote Power Switch

https://www.amazon.com/gp/product/B00LLVQ9OI/?tag=snbforums-20

It pings up to seven IPs either on the WAN or the LAN. If it times out based on a time delay you set and the number of failures it then shuts down your modem and your router. It then restarts my modem and after a delay I set restarts the router. The switch will if necessary repeat the process until www connectivity is restored.

My power switch is setup to connect using a VPN client on the router so even if the www connection is good but the VPN has crashed the 3G power switch will reboot both the modem and the router and hopefully reestablish the VPN client when the router starts.

It has worked well for me. While I was out of town I was able to force a reboot cycle when I loss contact with one of my IoT devices. I did this by logging into my Comcast account and then having Comcast send a modem reboot signal to my modem. There also are options on the switch for remote access but if the modem has dropped off line that won't work.

My wife loves this device because if there is a glitch with our Internet connection and I'm not around then this device will automatically function to try and restablish a connection without her have to remember what plugs to pull and which lights to watch on the modem before plugging the router back in. Having a happy non technical wife has made this device worth every penny of the $99 I paid for the switch.

 

[xtropodx]

That's an interesting option/device, but don't think I'm there just yet. Not really keen on adding yet another device, especially once it converts to AUD cost .

 

[Martineau]

So last night this was put to the test & it failed.
Logged off my PC last night & this morning for whatever reason internet wouldn't work (looks like it stopped working middle of the night).

Get the following repeatedly in the logs (over several hours);

Aug  9 03:30:31 (ChkWAN.sh): 513 Monitoring WAN connection using PING method to xxx.xxx.x.x check FAILED
Aug  9 03:30:31 (ChkWAN.sh): 513 Monitoring WAN connection OK.....(Successful ping to '192.168.20.1'). Will check WAN again in 30 secs

But I can't tell/see anywhere where the WAN has automatically been turned off/on & I can't see it pinging or attempting to download any file for false positives. How can I check that the WAN is auto-restarting, as I've had to do it manually?

The script appears to have determined it is unable to PING the obfuscated xxx.xxx.x.x address, but in order to prevent false positives, when the script attempted to PING the next target (192.168.20.1) it was successful so it assumed that the first PING failure was a false positive and thus assumed the WAN is UP.

Clearly 192.168.20.1 is not a WAN address, so either you are using a double NAT / local DNS server? or somehow the script is seemingly erroneously picking a LAN device I/P to PING??

NOTE: If I find time I will create v1.07 of the script to ensure that LAN I/Ps cannot be used as a PING target.

 

[xtropodx]

The script appears to have determined it is unable to PING the obfuscated xxx.xxx.x.x address, but in order to prevent false positives, when the script attempted to PING the next target (192.168.20.1) it was successful so it assumed that the first PING failure was a false positive and thus assumed the WAN is UP.

Clearly 192.168.20.1 is not a WAN address, so either you are using a double NAT / local DNS server? or somehow the script is seemingly erroneously picking a LAN device I/P to PING??

NOTE: If I find time I will create v1.07 of the script to ensure that LAN I/Ps cannot be used as a PING target.

Pretty sure 192.168.20.1 IP is the modem that the router is bridged behind, so the same target/device as xxx.xxx.x.x , as I use completely different IP range for LAN & any network devices. Guess this is why I asked about having script ping google.com, or where I'd change the script to do this.

NAT: It is enabled, not sure about double NAT? How/where would I double check that?
Local DNS Server: Pretty sure no. LAN -> DNS Server 1 & 2 is blank. I do have VPN Server on the router but that wouldn't be related?

Thanks.