Scribe scribe - syslog-ng and logrotate installer

[elorimer]

@Butterfly Bones: you still have a problem. In your log excerpt, there should be no "BLOCKED" messages above the hourly roll-up. Go into skynet and point its log setting to the skynet-0 log. You'll have to go back and redo the symlink too.

 

[Butterfly Bones]

@Butterfly Bones: you still have a problem. In your log excerpt, there should be no "BLOCKED" messages above the hourly roll-up. Go into skynet and point its log setting to the skynet-0 log. You'll have to go back and redo the symlink too.

You were right, Skynet log was as "Default", and I changed it to /opt/var/log/skynet-0.log. I forget what files need to be linked with the symlink. Is it

ln -s /opt/var/log/skynet-0.log /tmp/mnt/USB_name/skynet/skynet.log

EDIT - I think it is ok now. I ran "(s)how scribe status" and got "checking directory links - updated", so looks like cmkelley has some code magic in there too. If I run "(s)how scribe status" now I get "checking directory links - present"

 

[cmkelley]

You were right, Skynet log was as "Default", and I changed it to /opt/var/log/skynet-0.log. I forget what files need to be linked with the symlink. Is it

ln -s /opt/var/log/skynet-0.log /tmp/mnt/USB_name/skynet/skynet.log

EDIT - I think it is ok now. I ran "(s)how scribe status" and got "checking directory links - updated", so looks like cmkelley has some code magic in there too. If I run "(s)how scribe status" now I get "checking directory links - present"

No, directory links are for setting ups /tmp/syslog.log and /tmp/syslog.log-1 and making their /jffs counterparts directories.

Oops, apparently I only tell skynet to change it's logging location when initially setting up syslog-ng, and when uninstalling syslog-ng. So if it gets reset or Skynet is installed after scribe, the directories never get fixed. Gonna have to fix that logic!!!

 

[elorimer]

The GUI displays /tmp/syslog.log. So scribe creates a symlink for that that points to /opt/var/log/messages. You can do that yourself with that command after removing the /tmp/syslog.log file. You don't really need to do anything with log-1 or the /jffs files. That is just to stop writing the log to /jffs, and you already have a permanent copy of the log on the USB drive.

EDIT: to indicate you have to delete the log file before the link.

 

[Butterfly Bones]

The GUI displays /tmp/syslog.log. So scribe creates a symlink for that that points to /opt/var/log/messages. You can do that yourself with that command. You don't really need to do anything with log-1 or the /jffs files. That is just to stop writing the log to /jffs, and you already have a permanent copy of the log on the USB drive.

Just to confirm, it was fixed without me issuing that direct command, maybe in manually changing the Skynet log output(?) dunno, but all good now.

user@RT-AC86U-4608:/tmp/home/root# ln -s /opt/var/log/skynet-0.log /tmp/syslog.log
ln: /tmp/syslog.log: File exists

Also the /opt/var/log/skynet-0.log is correct now with no logrotate file needed. Maybe the fact that the router sat in the box in storage for five weeks while I was gone for the apartment rehab. When I returned, I just plugged it back in and turned it on, with no real attention to details. I had boxes to unpack and reconstruct my living space.

-rw-------    1 user_name root         30287 Oct 20 06:29 skynet-0.log

 

[elorimer]

I'm not sure the output of ln shows that the link is made. I think that is an error message that there already is a file.

If the -f option is not specified, ln shall write a diagnostic message to standard error, do nothing more with the current source_file

rm /tmp/syslog.log and then run the ln command, or run it with -fs instead of -s.

I edited my post above to fix that.

 

[Butterfly Bones]

The symlink is there and the ln -s command is correct stating the file exists. I did not have to remove it first.

user@RT-AC86U-4608:/tmp/home/root# ls -al /tmp/
 %< (snipped)
lrwxrwxrwx    1 user_name root            21 Oct 19 16:47 syslog.log -> /opt/var/log/messages

 

[Goobi]

Any pointers on creating a filter for my dnsmasq.log?
They are pretty big

25.1M Oct 22 18:05 /opt/var/log/dnsmasq.log
-rw-r----- 1 nobody root 23.3M Oct 21 05:20 /opt/var/log/dnsmasq.log1
-rw-r----- 1 nobody root 54.1M Oct 21 05:20 /opt/var/log/dnsmasq.log2

Thanks!

 

[elorimer]

Any pointers on creating a filter for my dnsmasq.log?

If you have Diversion installed, do not touch the dnsmasq.log files. Diversion covers rolling those.

 

[Goobi]

If you have Diversion installed, do not touch the dnsmasq.log files. Diversion covers rolling those.

Thanks much, yes using Diversion so these will stay put.

 

[shark]

Did a clean installation of everything yesterday and my skynet logs dont work properly anymore:

 

[Butterfly Bones]

Did a clean installation of everything yesterday and my skynet logs dont work properly anymore:

What does your filter file have in it, like this?

# this MUST BE the file configured in Skynet as the syslog.log location
# DO NOT use /tmp/syslog.log or /opt/var/log/messages here!
destination d_skynet { 
    file("/opt/var/log/skynet-0.log");
};
# logs everything from Skynet to /opt/var/log/skynet-0.log
filter f_skynet {
    program("Skynet") or
    message("BLOCKED -") or
    message("DROP IN=");
};
# final flag stops processing of messages matching the f_skynet filter
log {
    source(src);
    filter(f_skynet);
    destination(d_skynet);
    flags(final);
};
#eof

 

[Ayitaka]

tl;dr: At the hour mark, skynet doesnt leave a stat line after purging skynet-0.log and then skynet logging stops working entirely.

Different problem from shark I think... I recently reset to factory defaults, did an M&M configuration, and reinstalled all the scripts via amtm. Everything works fine except that every hour when skynet does its hourly save, it purges /opt/var/log/skynet-0.log leaving an empty file that no longer gets logged to (via syslog-ng).

Before I reset, when skynet saved, it would leave 1+ lines in skynet-0.log similar to:

Oct 26 14:34:44 Skynet: [#] 130264 IPs (+0) -- 40232 Ranges Banned (+0) || 49 Inbound -- 0 Outbound Connections Blocked! [save] [8s]

But that doesn't happen now. I have tried:

• Restarting skynet which doesnt resume logging.
• Disable/Enable logging via firewall->settings doesnt resume logging.
• Restarting syslog-ng does restart normal logging until the hour-mark again.
• Reinstalling skynet + scribe (in that order), and also just scribe, and there is no difference.

All my other logs work fine. The /jffs/syslog.log/ dir and /tmp/syslog.log link are both fine, before and after the hour.

I have unsuccessfully googled and searched the forums for any similar issues. I am not sure what to do to fix this and any help would be greatly appreciated.

If I can provide anything that might help, lemme know. In case these are helpful...

Spoiler: skynet.cfg

################################################
## Generated By Skynet - Do Not Manually Edit ##
## Oct 26 21:00:14                            ##

## Installer ##
model="RT-AC5300"
localver="v6.9.1"
autoupdate="enabled"
banmalwareupdate="daily"
forcebanmalwareupdate=""
logmode="enabled"
filtertraffic="all"
swaplocation="/tmp/mnt/sda1/myswap.swp"
swappartition=""

## Counters / Lists ##
blacklist1count="130264"
blacklist2count="40232"
customlisturl=""
customlist2url=""
countrylist="cn tw ve br vn id in ru ua ar bd bg hk lv es tr dz uz kp"
excludelists=""

## Settings ##
unbanprivateip="enabled"
loginvalid="disabled"
banaiprotect="enabled"
securemode="enabled"
extendedstats="enabled"
fastswitch="disabled"
syslogloc="/opt/var/log/skynet-0.log"
syslog1loc="/tmp/syslog.log-1"
iotblocked="disabled"
iotports=""
iotproto="udp"
lookupcountry="enabled"
cdnwhitelist="enabled"

################################################

Spoiler: Default /opt/etc/syslog-ng.d/skynet

# this MUST BE the file configured in Skynet as the syslog.log location
# DO NOT use /tmp/syslog.log or /opt/var/log/messages here!
destination d_skynet {
        file("/opt/var/log/skynet-0.log");
};

# logs everything from Skynet to /opt/var/log/skynet-0.log
filter f_skynet {
    program("Skynet") or
    message("BLOCKED -") or
    message("DROP IN=");
};

# final flag stops processing of messages matching the f_skynet filter
log {
    source(src);
    filter(f_skynet);
    destination(d_skynet);
    flags(final);
};

#eof

Spoiler: Default /opt/etc/syslog-ng.conf

#############################################################################
# OpenWrt syslog-ng.conf specific file
# which collects all local logs into a single file called /var/log/messages.
# More details about these settings can be found here:
# https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/release-notes/global-options
#
# non-scl version with "@include "scl.conf"" and call to default-network-drivers function (part of scl) removed
#   moved sourcing /proc/kmsg from "source kernel" to "source src" and added directives to /proc/kmsg and /dev/log sourcing
#   log_msg_size increased to 16K to ensure even large messages are completely captured
#   stats_freq changed to every 6 hours
#   source net function to get udp log messages from local network commented out
#   destination log_server function added (commented out) to illustrate sending udp log messages to server on local network

@version: 3.23
@include "/opt/etc/syslog-ng.d/" # Put any customization files in this directory

options {
    chain_hostnames(no); # Enable or disable the chained hostname format.
    create_dirs(yes);
    keep_hostname(yes); # Enable or disable hostname rewriting.
    log_fifo_size(256); # The number of messages that the output queue can store.
    log_msg_size(16384); # Maximum length of a message in bytes.
    stats_freq(21600); # The period between two STATS messages (sent by syslog-ng, containing statistics about dropped logs) in seconds. (21,600 seconds = 6 hours)
    flush_lines(0); # How many lines are flushed to a destination at a time.
    use_fqdn(no); # Add Fully Qualified Domain Name instead of short hostname.
};

# syslog-ng gets messages from the system, kernel, and syslog-ng (internal)
# DO NOT use system() source; causes issues on HND routers
# so_rcvbuf = maximum number of messages per second * 1024
source src {
    unix-dgram("/dev/log" so_rcvbuf(65536) flags(syslog-protocol));
    file("/proc/kmsg" program_override("kernel") flags(kernel));
    internal();
};

# uncomment these three lines to get udp log messages from local network
# must also uncomment "source(net);" below
#source net {
#    udp(ip(192.168.x.y) port(514));
#};

destination messages {
    file("/opt/var/log/messages");
};

# uncomment these three lines to send udp log messages to local network
# must also uncomment "destination(log_server);" below
#destination log_server {
#    udp("192.168.x.y" port(514));
#};

log {
    source(src);
#    source(net); # uncomment this and "source net" function above to get udp log messages from local network
    destination(messages);
#    destination(log_server); # uncomment this and "destination log_server" function above to send udp log messages to local network
};

Edit: Added syslog-ng.conf

 

[elorimer]

@Ayitaka : I'm not having the same problem, and I don't see anything obvious in your files. It sounds like skynet's hourly scrub is holding the file open and preventing syslog-ng from writing to it, but I can't think why. I assume the skynet messages aren't showing up in messages either.

I notice in your syslog-ng.conf file that this is version 3.23. I'm on 3.20. I don't see anything obvious in the release notes from 3.20 onward that would affect this, but perhaps there is something. Also, skynet seems to have been updated--auto update for me this morning-- so you might try updating.

You might also try uninstalling scribe and leaving skynet, and then seeing if its behavior is as expected on the syslog.log file. I suspect that would rule out something about skynet's sed logic.

 

[shark]

What does your filter file have in it, like this?

# this MUST BE the file configured in Skynet as the syslog.log location
# DO NOT use /tmp/syslog.log or /opt/var/log/messages here!
destination d_skynet {
    file("/opt/var/log/skynet-0.log");
};
# logs everything from Skynet to /opt/var/log/skynet-0.log
filter f_skynet {
    program("Skynet") or
    message("BLOCKED -") or
    message("DROP IN=");
};
# final flag stops processing of messages matching the f_skynet filter
log {
    source(src);
    filter(f_skynet);
    destination(d_skynet);
    flags(final);
};
#eof

# this MUST BE the file configured in Skynet as the syslog.log location
# DO NOT use /tmp/syslog.log or /opt/var/log/messages here!
destination d_skynet {
    file("/opt/var/log/skynet-0.log");
};

# logs everything from Skynet to /opt/var/log/skynet-0.log
filter f_skynet {
    program("Skynet") or
    message("BLOCKED -") or
    message("DROP IN=");
};

# final flag stops processing of messages matching the f_skynet filter
log {
    source(src);
    filter(f_skynet);
    destination(d_skynet);
    flags(final);
};

#eof

It stoped working after last update from skynet and scribe... thats why i did a clean install since it was also due time, so i though why not but then it did not fix the issue!

 

[Butterfly Bones]

# this MUST BE the file configured in Skynet as the syslog.log location
# DO NOT use /tmp/syslog.log or /opt/var/log/messages here!
destination d_skynet {
    file("/opt/var/log/skynet-0.log");
};

# logs everything from Skynet to /opt/var/log/skynet-0.log
filter f_skynet {
    program("Skynet") or
    message("BLOCKED -") or
    message("DROP IN=");
};

# final flag stops processing of messages matching the f_skynet filter
log {
    source(src);
    filter(f_skynet);
    destination(d_skynet);
    flags(final);
};

#eof

It stoped working after last update from skynet and scribe... thats why i did a clean install since it was also due time, so i though why not but then it did not fix the issue!

I vaguely remember having the same or similar issue way back in the early days of testing and configuring Scribe. I just spent considerable time searching this thread and find nothing about that, darn.

I think @cmkelley pointed me in the correct direction back then. Did you double check the settings in Skynet for log location? Maybe not the issue, but it cannot hurt to check, that tripped me up just a few days ago.

 

[cmkelley]

So, I apologize for not having an answer, life has gotten in the way of fun lately.

I don't generally look much at the logs, but after reading here, I noticed a separate problem, EVERYTHING stopped logging for me late the night of the 19th. Not sure what caused that. I updated syslog-ng to 3.23 on the 17th, so it doesn't seem like that is the issue. Maybe it's file permissions thing? I need to investigate what changed between 3.20 and 3.23.

Sadly, life is still "interesting" (nobody is sick or hurt or anything, just everything happening at once it seems), and I don't know when it will let up. If I get a chance I'll see what I can dig up.

 

[Butterfly Bones]

So, I apologize for not having an answer, life has gotten in the way of fun lately.

I don't generally look much at the logs, but after reading here, I noticed a separate problem, EVERYTHING stopped logging for me late the night of the 19th. Not sure what caused that. I updated syslog-ng to 3.23 on the 17th, so it doesn't seem like that is the issue. Maybe it's file permissions thing? I need to investigate what changed between 3.20 and 3.23.

Sadly, life is still "interesting" (nobody is sick or hurt or anything, just everything happening at once it seems), and I don't know when it will let up. If I get a chance I'll see what I can dig up.

I think that version change might have something to do with this issue. I just checked and am on syslog-ng - 3.20.1-1. Sorry @cmkelley that this is an inconvenient time.

I see others on 3.23 like @shark and @Ayitaka that might be it. Can you two run this in a shell and report the result?

opkg list-installed | grep syslog

I'll do some research on downgrading to 3.20 if that resolves this until scribe can be updated.

 

[cmkelley]

I think that version change might have something to do with this issue. I just checked and am on syslog-ng - 3.20.1-1. Sorry @cmkelley that this is an inconvenient time.

I see others on 3.23 like @shark and @Ayitaka that might be it. Can you two run this in a shell and report the result?

opkg list-installed | grep syslog

I'll do some research on downgrading to 3.20 if that resolves this until scribe can be updated.

There's some weird stuff going on here ... I tried re-starting Skynet but now I get the lock file detected message and that won't go away even after several minutes. I see a couple people have tried M&M to no avail, so it would seem not to be one of those that goes away with that. Past my bedtime already ...

 

[Ayitaka]

@elorimer : That is correct, with scribe installed nothing is going to messages even when logging for skynet stops. When I get a little time I will retry uninstalling scribe and letting skynet run normally logging to messages and report back with the results.

@cmkelley:

user@router:/# opkg list-installed | grep syslog
syslog-ng - 3.23.1-1

For what its worth, I then downloaded the ipk's and manually --force-downgrad'ed to syslog-ng 3.20.1-1 and libjson-c - 0.12.1-1 (got an error running scribe status with current version of libjson) and skynet logging almost works correctly again, with the exception that the stats line from skynet still isnt left in the log file like it use to be and it takes a good 30+ seconds to resume logging and misses any messages during that time.

user@router:/tmp/mnt/sda1/entware/var/log# opkg list-installed | grep syslog
syslog-ng - 3.20.1-1