What's new

Solved Push syslog msg from nat-start

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

ZebMcKayhan

Very Senior Member
Hello,

I'm trying to create a script that restores ipsets after reboot. As this have been troublesome before with consecutive runs on boot, I've used a technique with lock file creation from @eibgrad so consecutive executions are run sequentially. This works great as far as I have tested. I've also added a retry every second to wait maximum 30 sec for the usb drive to mount.
Everything is working and my ipsets gets restored every boot.
For reference, here is the script:
Code:
#!/bin/sh 
############################################################ 
# required for serialization when reentry is possible 
LOCK="/tmp/$(basename $0).lock" 
# acquire_lock # one instance at a time 
while ! mkdir $LOCK &>/dev/null; do sleep 2; done;
logger -t $(basename $0) "Started"
############################################################ 

## 
## Put existing nat-start directives here 
## 

IPSET_LIST="NETFLIX-DNS NETFLIX-DNS6 wg11-mac" #List of ipsets to restore
DIR="/opt/tmp" #directory for store ipset
MAX_TRIES=30 #Retries every second [MAX_TRIES] amount of times. 


## Normally nothing need to be changed below ##
TRIES="0" 
while [ "$TRIES" -lt "$MAX_TRIES" ]; do # Wait for target (usb drive) ready
    if [ -d "$DIR" ]; then # target ready?
        for IPSET_NAME in $IPSET_LIST; do # Each ipset in list
            if [ "$(ipset list -n "$IPSET_NAME" 2>/dev/null)" != "$IPSET_NAME" ]; then #if ipset does not already exist 
                if [ -s "$DIR/$IPSET_NAME" ]; then #if a backup file exists 
                    ipset restore -! <"$DIR/$IPSET_NAME" #restore ipset 
                    cru a "$IPSET_NAME" "0 2 * * * ipset save $IPSET_NAME > $DIR/$IPSET_NAME" >/dev/null 2>&1 # create cron job for autosave 
                    logger -t $(basename $0) "IPSET restored: $IPSET_NAME" 
                else
                    logger -t $(basename $0) "Warning: Failed to find IPSET restore file: $IPSET_NAME"
                fi 
            fi 
        done 
        break
    else 
        sleep 1 
        TRIES=$((TRIES + 1)) 
        if [ "$TRIES" -eq "$MAX_TRIES" ]; then 
            logger -t $(basename $0) "Warning: Failed to detect mounted USB-Drive within $MAX_TRIES seconds! IPSET not restored!" 
        fi 
    fi 
done 
############################################################ 
# exit (any concurrent instance(s) may now run) 
rmdir $LOCK &>/dev/null;
logger -t $(basename $0) "Completed [$@]"

but in my syslog, I dont get all messages. I can see nat-start executed before ntp is synced but then klogd is exited and after that there are no syslog messages from the initial run that restored my ipsets.
If I destroy my ipsets and run nat-start manually all syslog messages shows as they should.

This is my syslog relevant parts from boot:
Code:
May  5 07:05:20 custom_script: Running /jffs/scripts/nat-start
May  5 07:05:20 nat-start: Started 

<snip> 

May  5 07:05:21 ntpd: Started ntpd
Oct 15 00:43:39 ntpd: Initial clock set

<snip>

Oct 15 00:43:43 custom_script: Running /jffs/scripts/post-mount (args: /tmp/mnt/UsbDrv)
Oct 15 00:43:43 Diversion: Starting Entware and Diversion services on /tmp/mnt/UsbDrv
Oct 15 00:43:44 kernel: klogd: exiting
Oct 15 00:43:44 RT-AC86U-D7D8 kernel: ip_set: protocol 6
Oct 15 00:43:44 RT-AC86U-D7D8 rc_service: service 3421:notify_rc restart_dnsmasq
Oct 15 00:43:44 RT-AC86U-D7D8 rc_service: waitting "restart_dnsmasq" via  ...
Oct 15 00:43:45 RT-AC86U-D7D8 rc_service: udhcpc_wan 2445:notify_rc stop_samba
Oct 15 00:43:45 RT-AC86U-D7D8 rc_service: waitting "restart_dnsmasq" via  ... 

<snip>

Even tough there are no syslog messages that nat-start completed all ipsets are restored, cron job created and lock file removed so it must have completed.

Is this caused by klogd exited? Or with time sync? Are there any way around it?

I'm running scribe if that could affect this.

Regards
Zeb
 
Last edited:
It's quite possible that your messages are getting lost when Scribe restarts syslogd. To confirm, try disabling Scribe and rebooting. Another possibility is that you've configure Scribe in such a way that it's filtering out those messages from the standard log.
 
Hello,

I'm trying to create a script that restores ipsets after reboot. As this have been troublesome before with consecutive runs on boot, I've used a technique with lock file creation from @eibgrad so consecutive executions are run sequentially. This works great as far as I have tested. I've also added a retry every second to wait maximum 30 sec for the usb drive to mount.
Everything is working and my ipsets gets restored every boot.
For reference, here is the script:
Code:
#!/bin/sh
############################################################
# required for serialization when reentry is possible
LOCK="/tmp/$(basename $0).lock"
# acquire_lock # one instance at a time
while ! mkdir $LOCK &>/dev/null; do sleep 2; done;
logger -t $(basename $0) "Started"
############################################################

##
## Put existing nat-start directives here
##

IPSET_LIST="NETFLIX-DNS NETFLIX-DNS6 wg11-mac" #List of ipsets to restore
DIR="/opt/tmp" #directory for store ipset
MAX_TRIES=30 #Retries every second [MAX_TRIES] amount of times.


## Normally nothing need to be changed below ##
TRIES="0"
while [ "$TRIES" -lt "$MAX_TRIES" ]; do # Wait for target (usb drive) ready
    if [ -d "$DIR" ]; then # target ready?
        for IPSET_NAME in $IPSET_LIST; do # Each ipset in list
            if [ "$(ipset list -n "$IPSET_NAME" 2>/dev/null)" != "$IPSET_NAME" ]; then #if ipset does not already exist
                if [ -s "$DIR/$IPSET_NAME" ]; then #if a backup file exists
                    ipset restore -! <"$DIR/$IPSET_NAME" #restore ipset
                    cru a "$IPSET_NAME" "0 2 * * * ipset save $IPSET_NAME > $DIR/$IPSET_NAME" >/dev/null 2>&1 # create cron job for autosave
                    logger -t $(basename $0) "IPSET restored: $IPSET_NAME"
                else
                    logger -t $(basename $0) "Warning: Failed to find IPSET restore file: $IPSET_NAME"
                fi
            fi
        done
        break
    else
        sleep 1
        TRIES=$((TRIES + 1))
        if [ "$TRIES" -eq "$MAX_TRIES" ]; then
            logger -t $(basename $0) "Warning: Failed to detect mounted USB-Drive within $MAX_TRIES seconds! IPSET not restored!"
        fi
    fi
done
############################################################
# exit (any concurrent instance(s) may now run)
rmdir $LOCK &>/dev/null;
logger -t $(basename $0) "Completed [$@]"

but in my syslog, I dont get all messages. I can see nat-start executed before ntp is synced but then klogd is exited and after that there are no syslog messages from the initial run that restored my ipsets.
If I destroy my ipsets and run nat-start manually all syslog messages shows as they should.

This is my syslog relevant parts from boot:
Code:
May  5 07:05:20 custom_script: Running /jffs/scripts/nat-start
May  5 07:05:20 nat-start: Started

<snip>

May  5 07:05:21 ntpd: Started ntpd
Oct 15 00:43:39 ntpd: Initial clock set

<snip>

Oct 15 00:43:43 custom_script: Running /jffs/scripts/post-mount (args: /tmp/mnt/UsbDrv)
Oct 15 00:43:43 Diversion: Starting Entware and Diversion services on /tmp/mnt/UsbDrv
Oct 15 00:43:44 kernel: klogd: exiting
Oct 15 00:43:44 RT-AC86U-D7D8 kernel: ip_set: protocol 6
Oct 15 00:43:44 RT-AC86U-D7D8 rc_service: service 3421:notify_rc restart_dnsmasq
Oct 15 00:43:44 RT-AC86U-D7D8 rc_service: waitting "restart_dnsmasq" via  ...
Oct 15 00:43:45 RT-AC86U-D7D8 rc_service: udhcpc_wan 2445:notify_rc stop_samba
Oct 15 00:43:45 RT-AC86U-D7D8 rc_service: waitting "restart_dnsmasq" via  ...

<snip>

Even tough there are no syslog messages that nat-start completed all ipsets are restored, cron job created and lock file removed so it must have completed.

Is this caused by klogd exited? Or with time sync? Are there any way around it?

I'm running scribe if that could affect this.

Regards
Zeb

I had similar issue, but with init-start logs not recorded. I also run scribe. I suspect this happen around the time klogd/syslogd exited during bootup.
For nat-start, I add a loop to wait for NTP sync. Not a good practice as recently there is another thread discuss about too many scripts waiting for NTP. However, with this I can see nat-start logs in syslogs.
I suppose can ignore the lack of such logs during bootup, since you have tested ipset are properly restored.

Code:
logger -t $(basename $0) "Started [$@]"

# check NTP readiness
logger -t $(basename $0) "Check NTP readiness..."
ntptimer=0
ntptimeout=60
while [ "$(nvram get ntp_ready)" -eq "0" ] && [ "$ntptimer" -le "$ntptimeout" ]; do
    ntptimer=$((ntptimer+1))
    sleep 1
done

# optional to log NTP status
if [ "$ntptimer" -eq "0" ]; then
   logger -t $(basename $0) "System time already synced"
elif [ "$ntptimer" -le "$ntptimeout" ]; then
   logger -t $(basename $0) "Waited ${ntptimer}s for NTP to sync date"
else
   logger -t $(basename $0) "[$@] NTP sync failed after one minute"
fi
 
It's quite possible that your messages are getting lost when Scribe restarts syslogd. To confirm, try disabling Scribe and rebooting.
Good point Ill try that tonight, when rebooting is allowed by family members.

Another possibility is that you've configure Scribe in such a way that it's filtering out those messages from the standard log.
Nope, since I could destroy my ipsets and run nat-start manually they all show up:
Code:
Oct 15 00:01:46 RT-AC86U-D7D8 nat-start: Started
Oct 15 00:01:46 RT-AC86U-D7D8 nat-start: IPSET restored: NETFLIX-DNS6
Oct 15 00:01:46 RT-AC86U-D7D8 nat-start: IPSET restored: MYIP
Oct 15 00:01:46 RT-AC86U-D7D8 nat-start: IPSET restored: MYIP6
Oct 15 00:01:46 RT-AC86U-D7D8 nat-start: Completed

For nat-start, I add a loop to wait for NTP sync. Not a good practice as recently there is another thread discuss about too many scripts waiting for NTP. However, with this I can see nat-start logs in syslogs.
Yea, Im not sure that would help... Ive tried adding a sleep 10 at the start of nat-start, then no logs shows from nat-start at all. It is like if nat-start was executed in a different enviroment (or sticking to the old enviroment after klogd/syslogd is killed off).

Maybee not a biggie as you say but its bugging me not to be able to use the logger function in nat-start. The confirmation messages may not be important but if it fails I would like to see the messages. And right now I dont trust that I will.
 
Last edited:
Yea, Im not sure that would help... Ive tried adding a sleep 10 at the start of nat-start, then no logs shows from nat-start at all. It is like if nat-start was executed in a different enviroment (or sticking to the old enviroment after klogd/syslogd is killed off).

Maybee not a biggie as you say but its bugging me not to be able to use the logger function in nat-start. The confirmation messages may not be important but if it fails I would like to see the messages. And right now I dont trust that I will.
You are right. Probably it won't help. Now that I recall why I wait for NTP sync in nat-start.

I have this in nat-start
Code:
# create RPDB rules
if [ -z "$(ip rule | grep "9880:[[:space:]]from all fwmark 0x8000/0x8000 lookup main")" ]; then
  ##ip rule del prio 9880 2>/dev/null
   ip rule add from all fwmark 0x8000/0x8000 table main prio 9880        # WAN fwmark
   logger -t $(basename $0) "Add ip rule 9880 for main routing table"
fi

Without waiting for NTP sync, it gets run again in the second sequential run.
Code:
May  5 13:05:25 custom_script: Running /jffs/scripts/nat-start
May  5 13:05:25 nat-start: Started []
May  5 13:05:25 nat-start: Add ip rule 9880 for main routing table
Sep  9 09:58:36 custom_script: Running /jffs/scripts/nat-start
Sep  9 09:58:49 RT-AC86U-DBA8 nat-start: Completed []
Sep  9 09:58:49 RT-AC86U-DBA8 nat-start: Started []
Sep  9 09:58:49 RT-AC86U-DBA8 nat-start: Add ip rule 9880 for main routing table
Sep  9 09:58:55 RT-AC86U-DBA8 nat-start: Completed []

After add the wait for NTP sync, it only run once.
Code:
May  5 13:05:25 custom_script: Running /jffs/scripts/nat-start
May  5 13:05:25 nat-start: Started []
May  5 13:05:25 nat-start: Check NTP readiness...
Sep  9 17:01:20 nat-start: Waited 4s for NTP to sync date
Sep  9 17:01:20 nat-start: Add ip rule 9880 for main routing table
Sep  9 17:01:24 custom_script: Running /jffs/scripts/nat-start
Sep  9 17:01:32 RT-AC86U-DBA8 nat-start: Completed []
Sep  9 17:01:32 RT-AC86U-DBA8 nat-start: Started []
Sep  9 17:01:32 RT-AC86U-DBA8 nat-start: Check NTP readiness...
Sep  9 17:01:32 RT-AC86U-DBA8 nat-start: System time already synced
Sep  9 17:01:39 RT-AC86U-DBA8 nat-start: Completed []
 
Last edited:
Before scribe starts, entware starts from post-mount. All the logged messages up to that point are in /tmp/syslog.log. When scribe starts, it appends everything in syslog.log to messages.
 
Last edited:
It's quite possible that your messages are getting lost when Scribe restarts syslogd. To confirm, try disabling Scribe and rebooting.
I disabled scribe by chmod -x S01syslog-ng and commented the uiScribe entries in post-mount and can confirm that I now get all syslog messages:
Code:
May  5 07:05:20 nat-start: Started

<snip>

May  5 07:05:22 ntpd: Started ntpd
Oct 15 18:14:31 ntpd: Initial clock set

<snip>

Oct 15 18:14:36 usb: USB ext4 fs at /dev/sda1 mounted on /tmp/mnt/UsbDrv.
Oct 15 18:14:36 custom_script: Running /jffs/scripts/post-mount (args: /tmp/mnt/UsbDrv)

<snip>

Oct 15 18:14:37 nat-start: IPSET restored: NETFLIX-DNS
Oct 15 18:14:37 S61unbound: start Unbound DNS server  /opt/etc/init.d/S61unbound
Oct 15 18:14:37 nat-start: IPSET restored: NETFLIX-DNS6
Oct 15 18:14:37 rc_service: service 3181:notify_rc restart_dnsmasq
Oct 15 18:14:37 custom_script: Running /jffs/scripts/service-event (args: restart dnsmasq)
Oct 15 18:14:37 admin: Started unbound from /jffs/scripts/post-mount.
Oct 15 18:14:37 nat-start: IPSET restored: MYIP
Oct 15 18:14:37 Diversion: created br0:pixelserv-tls for 192.168.1.2
Oct 15 18:14:37 custom_config: Appending content of /jffs/configs/hosts.add.
Oct 15 18:14:37 nat-start: IPSET restored: MYIP6
Oct 15 18:14:37 custom_config: Appending content of /jffs/configs/dnsmasq.conf.add.
Oct 15 18:14:37 custom_script: Running /jffs/scripts/dnsmasq.postconf (args: /etc/dnsmasq.conf)
Oct 15 18:14:37 nat-start: Completed

Wonder if it would be better to restore ipset in post-mount script instead of nat-start?? It would make more sense and I dont have to wait for usb-drive. The firmware wouldnt flush ipsets at a nat-start event anyway right?
 
Before scribe starts, entware starts from post-mount. All the logged messages up to that point are in /tmp/syslog.log. When scribe starts, it appends everything in syslog.log
So Im guessing my nat-start are still outputting to /tmp/syslog.log since it was initiated before scribe started (or how does this work)? Which is why the later syslog entries are not included in scribe syslog?
 
May 5 13:05:25 nat-start: Started [] May 5 13:05:25 nat-start: Check NTP readiness... Sep 9 17:01:20 nat-start: Waited 4s for NTP to sync date Sep 9 17:01:20 nat-start: Add ip rule 9880 for main routing table Sep 9 17:01:24 custom_script: Running /jffs/scripts/nat-start Sep 9 17:01:32 RT-AC86U-DBA8 nat-start: Completed []
Apparently your nat-start was initiated before klogd/syslogd are killed, yet you got the final "completed" msg after scribe took over... Im started to think that I get problems because both my script and scribe starts at the same time. Both when usbdrive are mounted.... so maybee this is not a general problem but a much more specific one.

I'll try to change the ipset restoration to post-mount script instead, after everything else.
 
I moved the ipset restoration to last in post-mount instead. This feels like a much better solution as I only need to verify correct target, so I ended up with:
/jffs/scripts/post-mount
Code:
#!/bin/sh 
swapon /tmp/mnt/UsbDrv/myswap.swp # Added by amtm
. /jffs/addons/diversion/mount-entware.div # Added by amtm
/jffs/scripts/uiScribe startup "$@" & # uiScribe
/bin/sleep 2s
/bin/sleep 2s
/jffs/addons/unbound/unbound_stats.sh startup "$@" & # Unbound_Stats.sh
/jffs/addons/wireguard/wg_manager.sh init "" & # WireGuard Manager
/jffs/scripts/uiDivStats startup "$@" & # uiDivStats
cru a logrotate "5 0 * * * /opt/sbin/logrotate /opt/etc/logrotate.conf >> /opt/tmp/logrotate.daily 2>&1" # added by scribe

IPSET_LIST="NETFLIX-DNS NETFLIX-DNS6 MYIP MYIP6" #List of ipsets to restore
DIR="/opt/tmp" #directory for store ipset

## Normally nothing need to be changed below ##
if [ -d "$DIR" ]; then # our target ready?
    for IPSET_NAME in $IPSET_LIST; do # Each ipset in list
        if [ "$(ipset list -n "$IPSET_NAME" 2>/dev/null)" != "$IPSET_NAME" ]; then #if ipset does not already exist 
            if [ -s "$DIR/$IPSET_NAME" ]; then #if a backup file exists 
                ipset restore -! <"$DIR/$IPSET_NAME" #restore ipset 
                cru a "$IPSET_NAME" "0 2 * * * ipset save $IPSET_NAME > $DIR/$IPSET_NAME" >/dev/null 2>&1 # create cron job for autosave 
                logger -t $(basename $0) "IPSET restored: $IPSET_NAME" 
            else
                logger -t $(basename $0) "Warning: Failed to find IPSET restore file: $DIR/$IPSET_NAME"
            fi 
        fi 
    done 
else 
    logger -t $(basename $0) "Warning: Failed to detect $DIR in mounted USB-Drive! IPSET not restored!" 
fi

Syslog shows at bootup:
Code:
Oct 15 21:15:08 RT-AC86U-D7D8 post-mount: IPSET restored: NETFLIX-DNS
Oct 15 21:15:08 RT-AC86U-D7D8 post-mount: IPSET restored: NETFLIX-DNS6
Oct 15 21:15:08 RT-AC86U-D7D8 post-mount: IPSET restored: MYIP
Oct 15 21:15:08 RT-AC86U-D7D8 post-mount: IPSET restored: MYIP6

My interpretation of the original problem is that I attemp to add syslog messages virtually the same seconds that scribe shifts syslog manager, which seems to make these messages dissappear.

I could have digged deeper into this but I believe this to be a better solution for my problem.
 
So, scribe is the control & management side, but syslog-ng is doing the logging. Scribe sets it up with a source that reads unix dgrams and a fake file; klogd and syslogd never figure in syslog-ng. From boot, syslogd and klogd are handling the logs and writing to /tmp/syslog.log. Post mount starts entware, and S01syslog-ng is the first thing that starts. It starts by killing syslogd and klogd, and from that point on no messages are going to syslog.log. Then, it appends syslog.log to messages (note these don't go through syslog-ng), then deletes syslog.log and converts it into a directory. Then syslog-ng starts.

Syslog-ng starts by reading in the configuration files and ordering and parsing the sources, destinations, filters and templates (etc.) That may or may not include the scl files. Then it starts polling the sources and running them through the logging statements alphabetically. At the very end, any messages that haven't been processed by some other statement goes to messages.

In my case, from the time S01 kills the internal logging, and syslog-ng starts polling sources, including the dgrams and /proc, takes about 40 seconds. That isn't representative for most folks, because I've done a lot of tomfoolery with my scribe setup and include scl. But in that window a lot of other things have kicked off.

During that window, too, messages are piling up in the sources. I do not know what the internal buffer is, but it is entirely possible that logging statements are being lost. I think Zeb is right about this, but I'm not sure moving later in post-mount necessarily fixes things, because it is a race between how fast post-mount gets to S01 and whether later post-mount calls are generating logging statements in that window.

This is all because syslog-ng is the first entware script called, and scribe preprocesses things. But there isn't any particular reason why syslog-ng has to be S01 rather than S99, because the internal log is running. It would end up in the same place if it started well down the road when the 900 initial boot logging statements have all been collected by the internal log.

In fact, I wonder if it makes sense not to start syslog-ng in the rc process at all, but later in post mount starting with a timer to let the dust settle. Syslog-ng is an entware package; is it possible to run it outside the rc process?
 
Last edited:
Just as an aside, @ZebMcKayhan, you are going to get grief from the script gurus about putting all your stuff in post-mount, instead of putting a call in post-mount to another file that does all your stuff. One line is your allotment. :)
 
Just as an aside, @ZebMcKayhan, you are going to get grief from the script gurus about putting all your stuff in post-mount, instead of putting a call in post-mount to another file that does all your stuff. One line is your allotment. :)
Thanks for the heads-up. I can understand that with all scripts adding/removing strings in these files. I'll put it in a separate file instead.

And thanks for sharing your insight on logging, I really appreciate it!
 
...
This is all because syslog-ng is the first entware script called, and scribe preprocesses things. But there isn't any particular reason why syslog-ng has to be S01 rather than S99, because the internal log is running. It would end up in the same place if it started well down the road when the 900 initial boot logging statements have all been collected by the internal log.

In fact, I wonder if it makes sense not to start syslog-ng in the rc process at all, but later in post mount starting with a timer to let the dust settle. Syslog-ng is an entware package; is it possible to run it outside the rc process?

It's certainly possible to start an Entware service outside of the standard "rc.unslung" startup script. I've done that before when checking out a couple of services that I wanted to test/verify at my leisure without them being started up automatically when the USB disk was mounted or during bootup.

The method I've used is to put the following code at the top of the service script ("S01syslog-ng" in your particular case), so it looks like this:
Bash:
#!/bin/sh

#----------------------------------------------------#
# Delay the start of the service for a later time.
#----------------------------------------------------#
DelayStart=1
if [ "$DelayStart" -eq 1 ] && [ $# -gt 1 ] && [ "$1" = "start" ]
then
    if [ -n "$2" ]
    then
        logger -st "INFO_$0_$$" "DELAYING START."
        shift ; set check "$@"
    fi
fi

...[REST OF THE ORIGINAL SCRIPT]...
With the above addition to the original script, the service is no longer started automatically as usual; but you can start it up at some later point from another script, whenever it suits your needs or objectives.

For example:
Bash:
#!/bin/sh
sleep 5m ; /opt/etc/init.d/S01syslog-ng start

My 2 cents.

EDIT:
Note that I don't have Scribe installed on my router so while the above code has worked well on other Entware services that run *standalone*, it might not yield the results that you want (i.e. not lose some of the system log messages generated during reboot) with "syslog-ng" given that there may be interdependencies on changes made to the system when integrated with Scribe (e.g. making a symbolic link "/tmp/syslog.log" to an Entware location, creating "/jffs/syslog.log" & "/jffs/syslog.log-1" as directories) which syslog-ng may or may not be aware of, and I don't know whether those system changes are removed when the USB drive is unmounted and then recreated upon remounting. So keep that in mind when testing/trying the method described above when rebooting.
 
Last edited:
It's certainly possible to start an Entware service outside of the standard "rc.unslung" startup script. I've done that before when checking out a couple of services that I wanted to test/verify at my leisure without them being started up automatically when the USB disk was mounted or during bootup.

The method I've used is to put the following code at the top of the service script ("S01syslog-ng" in your particular case), so it looks like this:
Bash:
#!/bin/sh

#----------------------------------------------------#
# Delay the start of the service for a later time.
#----------------------------------------------------#
DelayStart=1
if [ "$DelayStart" -eq 1 ] && [ $# -gt 1 ] && [ "$1" = "start" ]
then
    if [ -n "$2" ]
    then
        logger -st "INFO_$0_$$" "DELAYING START."
        shift ; set check "$@"
    fi
fi

...[REST OF THE ORIGINAL SCRIPT]...
With the above addition to the original script, the service is no longer started automatically as usual; but you can start it up at some later point from another script, whenever it suits your needs or objectives.

For example:
Bash:
#!/bin/sh
sleep 5m ; /opt/etc/init.d/S01syslog-ng start

My 2 cents.

EDIT:
Note that I don't have Scribe installed on my router so while the above code has worked well on other Entware services that run *standalone*, it might not yield the results that you want (i.e. not lose some of the system log messages generated during reboot) with "syslog-ng" given that there may be interdependencies on changes made to the system when integrated with Scribe (e.g. making a symbolic link "/tmp/syslog.log" to an Entware location, creating "/jffs/syslog.log" & "/jffs/syslog.log-1" as directories) which syslog-ng may or may not be aware of, and I don't know whether those system changes are removed when the USB drive is unmounted and then recreated upon remounting. So keep that in mind when testing/trying the method described above when rebooting.
Thanks for the suggestion. But scribe itself ties into more scripts, as logrotate. Preventing start of syslog-ng will likely mess up something. Besides, this would have to be re-applied whenever Entware updates syslog-ng. Im not too happy about this way.

However... since my script does not have any Entware dependencies I should be able to start it before Entware. So basically at the top of post-mount and use "$@" and relative paths.

Somehing like this:
Code:
#!/bin/sh
/jffs/scripts/restore_ipsets.sh "$@"
swapon /tmp/mnt/UsbDrv/myswap.swp # Added by amtm 
. /jffs/addons/diversion/mount-entware.div # Added by amtm 
/jffs/scripts/uiScribe startup "$@" & # uiScribe 
/bin/sleep 2s 
/bin/sleep 2s
...

/jffs/scripts/restore_ipsets.sh:
Code:
#!/bin/sh 

IPSET_LIST="NETFLIX-DNS NETFLIX-DNS6 MYIP MYIP6" #List of ipsets to restore
DIRECTORY="tmp" #directory for store ipset (omit /opt so tmp=/opt/tmp)

## Normally nothing need to be changed below ##
for TARGET in "$@"; do #more targets?
    DIR="$TARGET/entware/$DIRECTORY" #Path to directory on mounted usb drive.
    if [ -d "$DIR" ]; then # our target ready?
        for IPSET_NAME in $IPSET_LIST; do # Each ipset in list
            if [ "$(ipset list -n "$IPSET_NAME" 2>/dev/null)" != "$IPSET_NAME" ]; then #if ipset does not already exist 
                if [ -s "$DIR/$IPSET_NAME" ]; then #if a backup file exists 
                    sleep 1
                    ipset restore -! <"$DIR/$IPSET_NAME" #restore ipset 
                    cru a "$IPSET_NAME" "0 3 * * * ipset save $IPSET_NAME > $DIR/$IPSET_NAME" >/dev/null 2>&1 # create cron job for autosave 
                    logger -t $(basename $0) "IPSET restored: $IPSET_NAME" 
                else
                    logger -t $(basename $0) "Warning: Failed to find IPSET restore file: $DIR/$IPSET_NAME"
                fi 
            fi 
        done 
    else 
        logger -t $(basename $0) "Warning: Failed to detect $DIR in mounted USB-Drive! IPSET not restored!" 
    fi 
done

Or will I mess something else up with that?
 
Last edited:
However... since my script does not have any Entware dependencies I should be able to start it before Entware. So basically at the top of post-mount and use "$@" and relative paths.
...
...
Or will I mess something else up with that?
Apologies for the belated response. I was on an 18-day family vacation (finally, after 2 years without one due to COVID-19); and once I got home, I was very busy catching up with work (a major s/w release is scheduled for early next month).

Anyway, FWIW, here is my take. If your script doesn't have dependencies on any Entware utilities or services, and the only dependency is on the disk partition (where Entware is installed) being properly mounted (which it should be when the "/jffs/scripts/post-mount" script is called with the correct Entware mount point), I don't see any issues at all with your script being run before starting the Entware services. You also get the benefit of getting the log messages from the script into the built-in system logs (and not losing them) before syslog-ng gets started.

Thanks for the suggestion. But scribe itself ties into more scripts, as logrotate. Preventing start of syslog-ng will likely mess up something. Besides, this would have to be re-applied whenever Entware updates syslog-ng. Im not too happy about this way
Yeah, I get it - it does seem like an inconvenience to keep track of that. OTOH, I see it as an opportunity to write a script (or two, actually) to make it an automatic task. The old saying: "If all you have is a hammer, everything looks like a nail." also tends to apply to many s/w developers like myself. ;)
 
Apologies for the belated response. I was on an 18-day family vacation (finally, after 2 years without one due to COVID-19); and once I got home, I was very busy catching up with work (a major s/w release is scheduled for early next month).

Anyway, FWIW, here is my take. If your script doesn't have dependencies on any Entware utilities or services, and the only dependency is on the disk partition (where Entware is installed) being properly mounted (which it should be when the "/jffs/scripts/post-mount" script is called with the correct Entware mount point), I don't see any issues at all with your script being run before starting the Entware services. You also get the benefit of getting the log messages from the script into the built-in system logs (and not losing them) before syslog-ng gets started.


Yeah, I get it - it does seem like an inconvenience to keep track of that. OTOH, I see it as an opportunity to write a script (or two, actually) to make it an automatic task. The old saying: "If all you have is a hammer, everything looks like a nail." also tends to apply to many s/w developers like myself. ;)
Thanks! Ive been running the script now for a couple of weeks and it seems to do the job. My only fear I guess is that if my script contains a typo or something crashes or enters some infinate loop then Entware may not start.
 
My only fear I guess is that if my script contains a typo or something crashes or enters some infinate loop then Entware may not start.
Ah OK, I understand. You could gain more confidence in the robustness of your script by running some validation testing to exercise all the linearly independent logic paths that the code can take via control flow conditionals (in s/w development we call this cyclomatic complexity). The more of these "paths" the code has, the more complex the code is, and the more chances of failure exist.

Your script is fairly straightforward so a validation test script can be something like this (just as an example to try, it's *not* an exhaustive test so modify as you see fit for your own environment):
Bash:
#!/bin/sh
##################################################################
# TEST_restore_ipsets.sh
##################################################################

# The expected/valid mount point #
MountPoint="/tmp/mnt/UsbDrv"
ScriptFile="/jffs/scripts/restore_ipsets.sh"

IPSETpath="${MountPoint}/entware/tmp"
IPSETname="NETFLIX-DNS"
IPSETfile="${IPSETpath}/$IPSETname"

# IMPORTANT: FIRST Argument MUST BE "-testmode" #
TestNum=1

#-------------------------------------------------------#
echo "Testing with NO Mount Points [$((TestNum++))]"
$ScriptFile -testmode
echo

#-------------------------------------------------------#
echo "Testing with an EMPTY Mount Point [$((TestNum++))]"
$ScriptFile -testmode ""
echo

#-------------------------------------------------------#
echo "Testing with an INVALID Mount Point [$((TestNum++))]"
$ScriptFile -testmode "/tmp/mnt/DummyUSB1"
echo

#-------------------------------------------------------#
echo "Testing with INVALID Mount Points [$((TestNum++))]"
$ScriptFile -testmode "/tmp/mnt/DummyUSB1" "/tmp/mnt/DummyUSB2"
echo

#-------------------------------------------------------#
echo "Testing with EXPECTED Mount Point [$((TestNum++))]"
$ScriptFile -testmode "$MountPoint"
echo

#-------------------------------------------------------#
# RENAME THE IPSET BACKUP FILE #
if [ -s "$IPSETfile" ] ; then
   mv -f "$IPSETfile" "${IPSETfile}.TEMP_OFF"
fi

echo "Testing with ONE IPSET Backup File RENAMED [$((TestNum++))]"
$ScriptFile -testmode "$MountPoint"
echo

# RESTORE THE IPSET BACKUP FILE #
if [ -f "${IPSETfile}.TEMP_OFF" ] ; then
   mv -f "${IPSETfile}.TEMP_OFF" "$IPSETfile"
fi

#EOF#

Then you edit your own script to add the following code snippet after the 1st line (e.g. #!/bin/sh):
Bash:
LogFlags="-t"
if [ $# -gt 0 ] && [ -n "$1" ] && [ "$1" = "-testmode" ]
then LogFlags="-s -t" ; shift ; fi

Now, replace all the existing logger calls in your script.
FROM:
logger -t

TO:
logger $LogFlags

Your newly-modified script would look like this:
Bash:
#!/bin/sh

LogFlags="-t"
if [ $# -gt 0 ] && [ -n "$1" ] && [ "$1" = "-testmode" ]
then LogFlags="-s -t" ; shift ; fi

IPSET_LIST="NETFLIX-DNS NETFLIX-DNS6 MYIP MYIP6" #List of ipsets to restore
DIRECTORY="tmp" #directory for store ipset (omit /opt so tmp=/opt/tmp)

## Normally nothing need to be changed below ##
for TARGET in "$@"; do #more targets?
    DIR="$TARGET/entware/$DIRECTORY" #Path to directory on mounted usb drive.
    if [ -d "$DIR" ]; then # our target ready?
        for IPSET_NAME in $IPSET_LIST; do # Each ipset in list
            if [ "$(ipset list -n "$IPSET_NAME" 2>/dev/null)" != "$IPSET_NAME" ]; then #if ipset does not already exist
                if [ -s "$DIR/$IPSET_NAME" ]; then #if a backup file exists
                    sleep 1
                    ipset restore -! <"$DIR/$IPSET_NAME" #restore ipset
                    cru a "$IPSET_NAME" "0 3 * * * ipset save $IPSET_NAME > $DIR/$IPSET_NAME" >/dev/null 2>&1 # create cron job for autosave
                    logger $LogFlags $(basename $0) "IPSET restored: $IPSET_NAME"
                else
                    logger $LogFlags $(basename $0) "Warning: Failed to find IPSET restore file: $DIR/$IPSET_NAME"
                fi
            fi
        done
    else
        logger $LogFlags $(basename $0) "Warning: Failed to detect $DIR in mounted USB-Drive! IPSET not restored!"
    fi
done

Now, you can run the test script "TEST_restore_ipsets.sh" to verify that you get the results that you expect/want. If later on, you add more conditional logic or feature to the script, you can add the corresponding test call to validate the new code.

Just my 2 cents.
 
Ah OK, I understand. You could gain more confidence in the robustness of your script by running some validation testing to exercise all the linearly independent logic paths that the code can take via control flow conditionals (in s/w development we call this cyclomatic complexity). The more of these "paths" the code has, the more complex the code is, and the more chances of failure exist.

Your script is fairly straightforward so a validation test script can be something like this (just as an example to try, it's *not* an exhaustive test so modify as you see fit for your own environment):
Bash:
#!/bin/sh
##################################################################
# TEST_restore_ipsets.sh
##################################################################

# The expected/valid mount point #
MountPoint="/tmp/mnt/UsbDrv"
ScriptFile="/jffs/scripts/restore_ipsets.sh"

IPSETpath="${MountPoint}/entware/tmp"
IPSETname="NETFLIX-DNS"
IPSETfile="${IPSETpath}/$IPSETname"

# IMPORTANT: FIRST Argument MUST BE "-testmode" #
TestNum=1

#-------------------------------------------------------#
echo "Testing with NO Mount Points [$((TestNum++))]"
$ScriptFile -testmode
echo

#-------------------------------------------------------#
echo "Testing with an EMPTY Mount Point [$((TestNum++))]"
$ScriptFile -testmode ""
echo

#-------------------------------------------------------#
echo "Testing with an INVALID Mount Point [$((TestNum++))]"
$ScriptFile -testmode "/tmp/mnt/DummyUSB1"
echo

#-------------------------------------------------------#
echo "Testing with INVALID Mount Points [$((TestNum++))]"
$ScriptFile -testmode "/tmp/mnt/DummyUSB1" "/tmp/mnt/DummyUSB2"
echo

#-------------------------------------------------------#
echo "Testing with EXPECTED Mount Point [$((TestNum++))]"
$ScriptFile -testmode "$MountPoint"
echo

#-------------------------------------------------------#
# RENAME THE IPSET BACKUP FILE #
if [ -s "$IPSETfile" ] ; then
   mv -f "$IPSETfile" "${IPSETfile}.TEMP_OFF"
fi

echo "Testing with ONE IPSET Backup File RENAMED [$((TestNum++))]"
$ScriptFile -testmode "$MountPoint"
echo

# RESTORE THE IPSET BACKUP FILE #
if [ -f "${IPSETfile}.TEMP_OFF" ] ; then
   mv -f "${IPSETfile}.TEMP_OFF" "$IPSETfile"
fi

#EOF#

Then you edit your own script to add the following code snippet after the 1st line (e.g. #!/bin/sh):
Bash:
LogFlags="-t"
if [ $# -gt 0 ] && [ -n "$1" ] && [ "$1" = "-testmode" ]
then LogFlags="-s -t" ; shift ; fi

Now, replace all the existing logger calls in your script.
FROM:
logger -t

TO:
logger $LogFlags

Your newly-modified script would look like this:
Bash:
#!/bin/sh

LogFlags="-t"
if [ $# -gt 0 ] && [ -n "$1" ] && [ "$1" = "-testmode" ]
then LogFlags="-s -t" ; shift ; fi

IPSET_LIST="NETFLIX-DNS NETFLIX-DNS6 MYIP MYIP6" #List of ipsets to restore
DIRECTORY="tmp" #directory for store ipset (omit /opt so tmp=/opt/tmp)

## Normally nothing need to be changed below ##
for TARGET in "$@"; do #more targets?
    DIR="$TARGET/entware/$DIRECTORY" #Path to directory on mounted usb drive.
    if [ -d "$DIR" ]; then # our target ready?
        for IPSET_NAME in $IPSET_LIST; do # Each ipset in list
            if [ "$(ipset list -n "$IPSET_NAME" 2>/dev/null)" != "$IPSET_NAME" ]; then #if ipset does not already exist
                if [ -s "$DIR/$IPSET_NAME" ]; then #if a backup file exists
                    sleep 1
                    ipset restore -! <"$DIR/$IPSET_NAME" #restore ipset
                    cru a "$IPSET_NAME" "0 3 * * * ipset save $IPSET_NAME > $DIR/$IPSET_NAME" >/dev/null 2>&1 # create cron job for autosave
                    logger $LogFlags $(basename $0) "IPSET restored: $IPSET_NAME"
                else
                    logger $LogFlags $(basename $0) "Warning: Failed to find IPSET restore file: $DIR/$IPSET_NAME"
                fi
            fi
        done
    else
        logger $LogFlags $(basename $0) "Warning: Failed to detect $DIR in mounted USB-Drive! IPSET not restored!"
    fi
done

Now, you can run the test script "TEST_restore_ipsets.sh" to verify that you get the results that you expect/want. If later on, you add more conditional logic or feature to the script, you can add the corresponding test call to validate the new code.

Just my 2 cents.
Thanks for the test bench. I will take it out for a test spin!
 

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top