Scribe scribe - syslog-ng and logrotate installer

[Butterfly Bones]

So just to beat on this a bit, my 86U and 56U are both on kernel 2.6.36, and @cmkelley has a 86U which I guess is on 4.1. So that might be why his system() expands differently than mine does.

Just to confirm.

username@RT-AC86U-4608:/tmp/home/root# uname -r
4.1.27

 

[Goobi]

Is there anyway to remove these entries from the syslog and have them dumped off like skynet-0.log/wlceventd.log?:

kernel: protocol 0800 is buggy, dev eth0

Thanks!

 

[Butterfly Bones]

Is there anyway to remove these entries from the syslog and have them dumped off like skynet-0.log/wlceventd.log?:

kernel: protocol 0800 is buggy, dev eth0

Thanks!

Try this, name it protocol and place it in the syslog-ng.d directory. I *think* it will work, if not let us know and we can modify it.

EDIT - made changes per elorimer post just below (#384)

# put protocol "buggy" messages into /opt/var/log/protocol.log
destination d_protocol {
    file("/opt/var/log/protocol.log");
};
filter f_protocol {
    program("kernel") and
    message("buggy");
};
log {
    source(src);
    filter(f_protocol);
    destination(d_protocol);
    flags(final);
};
#eof

 

[elorimer]

I think you need to delete this line in the filter: "program("protocol") and", since the program is kernel, not protocol. Or change the program reference to "kernel". Not sure whether any non-kernel log messages would include "buggy".

You could also delete the destination to just drop the messages to begin with.

 

[joe scian]

That looks like the one from the Loggly site. I looked at it, but then used the one @elorimer posted. Here it is named 0loggly - place it in syslog.d directory:

### Syslog-ng Logging Directives for Loggly.com ###
template LogglyFormat { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [PLACE_YOUR_TOKEN_HERE@41058 tag=\"YOUR_TAG\" ] $MSG\n");
    template_escape(no);
};
destination d_loggly {
    tcp("logs-01.loggly.com" port(514) template(LogglyFormat));
};
log {
    source(src);
    destination(d_loggly);
};
### END Syslog-ng Logging Directives for Loggly.com ###

When I run scribe config or the syslog-ng command "syslog-ng -Fevd" I see it, yes.

Have you succeeded with Loggly? I was out early this morning, and just now reading messages.

No I have not - I tried your filter before I attempt elorimers filter mods and i get this when running diags.

18.267382] Finishing include; filename='/opt/etc/syslog-ng.d/expandlog', depth='1'
[2019-04-19T00:50:18.267479] Starting to read include file; filename='/opt/etc/syslog-ng.d/logrotate', depth='1'
[2019-04-19T00:50:18.269364] Finishing include; filename='/opt/etc/syslog-ng.d/logrotate', depth='1'
[2019-04-19T00:50:18.269455] Starting to read include file; filename='/opt/etc/syslog-ng.d/openvpn', depth='1'
[2019-04-19T00:50:18.271515] Finishing include; filename='/opt/etc/syslog-ng.d/openvpn', depth='1'
[2019-04-19T00:50:18.271626] Starting to read include file; filename='/opt/etc/syslog-ng.d/pixelserv', depth='1'
[2019-04-19T00:50:18.273088] Finishing include; filename='/opt/etc/syslog-ng.d/pixelserv', depth='1'
[2019-04-19T00:50:18.273194] Starting to read include file; filename='/opt/etc/syslog-ng.d/skynet', depth='1'
[2019-04-19T00:50:18.275242] Finishing include; filename='/opt/etc/syslog-ng.d/skynet', depth='1'
[2019-04-19T00:50:18.275362] Starting to read include file; filename='/opt/etc/syslog-ng.d/syslogng', depth='1'
[2019-04-19T00:50:18.276679] Finishing include; filename='/opt/etc/syslog-ng.d/syslogng', depth='1'
[2019-04-19T00:50:18.276776] Starting to read include file; filename='/opt/etc/syslog-ng.d/wlceventd', depth='1'
[2019-04-19T00:50:18.278208] Finishing include; filename='/opt/etc/syslog-ng.d/wlceventd', depth='1'
[2019-04-19T00:50:18.279320] Module loaded and initialized successfully; module='system-source'
[2019-04-19T00:50:18.279658] system(): Enabling Linux kernel log device; device='/proc/kmsg', format='(null)'
[2019-04-19T00:50:18.281650] Module loaded and initialized successfully; module='appmodel'
[2019-04-19T00:50:18.282032] Finishing include; content='parser generator app-parser', depth='2'
[2019-04-19T00:50:18.282246] Finishing include; content='parser generator app-parser', depth='2'
[2019-04-19T00:50:18.282501] Finishing include; content='source generator system', depth='1'
[2019-04-19T00:50:18.285774] Module loaded and initialized successfully; module='syslogformat'
[2019-04-19T00:50:18.331625] Running application hooks; hook='1'
[2019-04-19T00:50:18.331726] Running application hooks; hook='3'
[2019-04-19T00:50:18.331874] syslog-ng starting up; version='3.19.1'
[2019-04-19T00:50:18.333948] Syslog connection failed; fd='15', server='AF_INET(192.168.2.3:514)', error='Connection refused (111)', time_reopen='60'
[2019-04-19T00:51:18.342541] Syslog connection failed; fd='20', server='AF_INET(192.168.2.3:514)', error='Connection refused (111)', time_reopen='60'
^C[2019-04-19T00:51:59.781034] Running application hooks; hook='4'
[2019-04-19T00:51:59.781090] syslog-ng shutting down; version='3.19.1'
[2019-04-19T00:51:59.908904] Running application hooks; hook='5'

192.168.2.3 is my pixelserv ip address

 

[joe scian]

syslog-ng.log file

Apr 19 11:18:25 RT-AC5300 syslog-ng[14734]: Syslog connection failed; fd='17', server='AF_INET(192.168.2.3:514)', error='Connection refused (111)', time_reopen='60'
Apr 19 11:19:25 RT-AC5300 syslog-ng[14734]: Syslog connection failed; fd='5', server='AF_INET(192.168.2.3:514)', error='Connection refused (111)', time_reopen='60'
Apr 19 11:20:25 RT-AC5300 syslog-ng[14734]: Syslog connection failed; fd='21', server='AF_INET(192.168.2.3:514)', error='Connection refused (111)', time_reopen='60'
Apr 19 11:21:25 RT-AC5300 syslog-ng[14734]: Syslog connection failed; fd='5', server='AF_INET(192.168.2.3:514)', error='Connection refused (111)', time_reopen='60'
Apr 19 11:22:25 RT-AC5300 syslog-ng[14734]: Syslog connection failed; fd='5', server='AF_INET(192.168.2.3:514)', error='Connection refused (111)', time_reopen='60'
Apr 19 11:23:25 RT-AC5300 syslog-ng[14734]: Syslog connection failed; fd='21', server='AF_INET(192.168.2.3:514)', error='Connection refused (111)', time_reopen='60'
Apr 19 11:24:25 RT-AC5300 syslog-ng[14734]: Syslog connection failed; fd='5', server='AF_INET(192.168.2.3:514)', error='Connection refused (111)', time_reopen='60'

If I remove 0loggly no more error messages - strange

 

[Butterfly Bones]

No I have not - I tried your filter before I attempt elorimers filter mods and i get this when running diags.

18.267382] Finishing include; filename='/opt/etc/syslog-ng.d/expandlog', depth='1'
[2019-04-19T00:50:18.267479] Starting to read include file; filename='/opt/etc/syslog-ng.d/logrotate', depth='1'
[2019-04-19T00:50:18.269364] Finishing include; filename='/opt/etc/syslog-ng.d/logrotate', depth='1'
[2019-04-19T00:50:18.269455] Starting to read include file; filename='/opt/etc/syslog-ng.d/openvpn', depth='1'
[2019-04-19T00:50:18.271515] Finishing include; filename='/opt/etc/syslog-ng.d/openvpn', depth='1'
[2019-04-19T00:50:18.271626] Starting to read include file; filename='/opt/etc/syslog-ng.d/pixelserv', depth='1'
[2019-04-19T00:50:18.273088] Finishing include; filename='/opt/etc/syslog-ng.d/pixelserv', depth='1'
[2019-04-19T00:50:18.273194] Starting to read include file; filename='/opt/etc/syslog-ng.d/skynet', depth='1'
[2019-04-19T00:50:18.275242] Finishing include; filename='/opt/etc/syslog-ng.d/skynet', depth='1'
[2019-04-19T00:50:18.275362] Starting to read include file; filename='/opt/etc/syslog-ng.d/syslogng', depth='1'
[2019-04-19T00:50:18.276679] Finishing include; filename='/opt/etc/syslog-ng.d/syslogng', depth='1'
[2019-04-19T00:50:18.276776] Starting to read include file; filename='/opt/etc/syslog-ng.d/wlceventd', depth='1'
[2019-04-19T00:50:18.278208] Finishing include; filename='/opt/etc/syslog-ng.d/wlceventd', depth='1'
[2019-04-19T00:50:18.279320] Module loaded and initialized successfully; module='system-source'
[2019-04-19T00:50:18.279658] system(): Enabling Linux kernel log device; device='/proc/kmsg', format='(null)'
[2019-04-19T00:50:18.281650] Module loaded and initialized successfully; module='appmodel'
[2019-04-19T00:50:18.282032] Finishing include; content='parser generator app-parser', depth='2'
[2019-04-19T00:50:18.282246] Finishing include; content='parser generator app-parser', depth='2'
[2019-04-19T00:50:18.282501] Finishing include; content='source generator system', depth='1'
[2019-04-19T00:50:18.285774] Module loaded and initialized successfully; module='syslogformat'
[2019-04-19T00:50:18.331625] Running application hooks; hook='1'
[2019-04-19T00:50:18.331726] Running application hooks; hook='3'
[2019-04-19T00:50:18.331874] syslog-ng starting up; version='3.19.1'
[2019-04-19T00:50:18.333948] Syslog connection failed; fd='15', server='AF_INET(192.168.2.3:514)', error='Connection refused (111)', time_reopen='60'
[2019-04-19T00:51:18.342541] Syslog connection failed; fd='20', server='AF_INET(192.168.2.3:514)', error='Connection refused (111)', time_reopen='60'
^C[2019-04-19T00:51:59.781034] Running application hooks; hook='4'
[2019-04-19T00:51:59.781090] syslog-ng shutting down; version='3.19.1'
[2019-04-19T00:51:59.908904] Running application hooks; hook='5'

192.168.2.3 is my pixelserv ip address

Yes, there is a line seems incorrect in the destination part of your 0loggly file.
Note <<----- (here) below.

destination d_loggly {
   tcp("logs-01.loggly.com" port(514) template(LogglyFormat));    <<--- (here)
};

Make sure it is - tcp("logs-01.loggly.com" port(514) template(LogglyFormat));

This is part is the destination URL - logs-01.loggly.com

This is what I see in the syslog-ng.log or running diagnostics.

Apr 17 11:40:47 RT-AC86U-4608 syslog-ng[30254]: Syslog connection established; fd='17', server='AF_INET(52.73.201.133:514)', local='AF_INET(0.0.0.0:0)'

 

[cmkelley]

So just to beat on this a bit, my 86U and 56U are both on kernel 2.6.36, and @cmkelley has a 86U which I guess is on 4.1. So that might be why his system() expands differently than mine does.

Yep. 4.1.27

 

[joe scian]

Yes, there is a line seems incorrect in the destination part of your 0loggly file.
Note <<----- (here) below.

destination d_loggly {
   tcp("logs-01.loggly.com" port(514) template(LogglyFormat));    <<--- (here)
};

Make sure it is - tcp("logs-01.loggly.com" port(514) template(LogglyFormat));

This is part is the destination URL - logs-01.loggly.com

This is what I see in the syslog-ng.log or running diagnostics.

Apr 17 11:40:47 RT-AC86U-4608 syslog-ng[30254]: Syslog connection established; fd='17', server='AF_INET(52.73.201.133:514)', local='AF_INET(0.0.0.0:0)'

Its exactly as per your example above copied and pasted with my token. I still get the problem

 

[Butterfly Bones]

Its exactly as per your example above copied and pasted with my token. I still get the problem

Go into Diversion and whitelist that domain of "logs-01.loggly.com" (no quote marks)

I remember reading somewhere that is in one of Diversion blocking list. It think elorimer posted that early in this thread about Loggly, or I saw it in the old syslog-ng thread of kvic.

 

[joe scian]

I think i know whats the cause - its diversion blocking logs-01.loggly.com - im whitelisting it now.
seems we think alike !!
working now thanks

 

[elorimer]

It was a while ago: scribe - syslog-ng and logrotate installer [v0.9_1 (Beta)]

 

[elorimer]

I redid my syslog-ng.conf to delete system() as a source, and added back a source following the /dev/log stream, and added back reading the file "dev/kmsg" instead of "/proc/kmsg". That errored out with "no such file". So my kernel 2.6 can't read /dev/kmsg the way 4.1 can.

Then, I redid it to read /dev/kmsg as a stream, like /dev/log, and that errored out as unable to bind the unix socket. So /dev/kmsg is not workable for routers on 2.6.

It looks to me like using system() as a source is smart enough to decide whether the router is capable of following /dev/kmsg, and if not, following /proc/kmsg. It may be that the HND routers can follow /dev/kmsg without killing klogd, but there doesn't seem to be any advantage to doing so. An alternative would be to have a more complicated syslog-ng.conf that read /dev/kmsg or /proc/kmsg, depending on the router. But that is what system() seems to be doing, so no point.

Conclusion: using system() as a source makes the script compatible across routers. I think the entware version of syslog-ng.conf might do well to follow suit.

 

[Butterfly Bones]

Here is an update on getting Skynet logging working on my RT-AC86u. After four uninstall scribe, let things run, reinstall scribe, I have almost all logging that I want to see working the way it does without scribe.

My final step that worked is to disable Skynet before scribe install. After adding my own filter files to logrotate.d and syslog-ng.d, verifying all permissions chmod 600, then I restarted Skynet and went into setting there and changed logging to custom /opt/var/log/skynet-0.log (without Skynet running scribe did not change log location, as designed I assume). It seems that there is an issue installing scribe on the AC86u with Skynet running, at least for me.

Here is Skynet info from today and yesterday from various sources.

cat /mnt/SNB/skynet/skynet.log
Apr 19 00:00:03 RT-AC86U-4608 Skynet[1880]: [#] 145203 IPs (+0) -- 1737 Ranges Banned (+0) || 487 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 19 01:00:03 RT-AC86U-4608 Skynet[15681]: [#] 145203 IPs (+0) -- 1737 Ranges Banned (+0) || 595 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 19 03:00:03 RT-AC86U-4608 Skynet[10666]: [#] 145203 IPs (+0) -- 1737 Ranges Banned (+0) || 801 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 19 04:00:03 RT-AC86U-4608 Skynet[24062]: [#] 145203 IPs (+0) -- 1737 Ranges Banned (+0) || 944 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 19 07:00:04 RT-AC86U-4608 Skynet[32336]: [#] 145203 IPs (+0) -- 1737 Ranges Banned (+0) || 1307 Inbound -- 0 Outbound Connections Blocked! [save] [3s]

cat /mnt/SNB/skynet/events.log
Apr 18 14:00:03 Skynet: [#] 145203 IPs (+0) -- 1737 Ranges Banned (+0) || 116 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 18 15:00:03 Skynet: [#] 145203 IPs (+0) -- 1737 Ranges Banned (+0) || 228 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 18 16:00:03 Skynet: [#] 145203 IPs (+0) -- 1737 Ranges Banned (+0) || 340 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 18 17:00:03 Skynet: [#] 145203 IPs (+0) -- 1737 Ranges Banned (+0) || 461 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 18 18:00:03 Skynet: [#] 145203 IPs (+0) -- 1737 Ranges Banned (+0) || 605 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 18 19:00:03 Skynet: [#] 145203 IPs (+0) -- 1737 Ranges Banned (+0) || 735 Inbound -- 0 Outbound Connections Blocked! [save] [3s]

I see the ongoing debug events from Skynet using tail on /opt/var/log/skynet-0.log, just no SAVE or BANMALWARE events. Since I know something strange occurs with the two native Skynet logs, i have not tried to search those in Skynet menu.

and from Loggly which verifies [save] works! I can export from Loggly, but I've not learned how to preserve timestamps, so a screen cap will have to do.

Also now I see some events that I have no desire to filter, Diversion actions, service events show in the webGUI syslog, along with a plethora of empty kernel: events. I know this is being looked at, so this is not a complaint.

Apr 18 22:00:00 RT-AC86U-4608 rc_service[6671]: service 6671:notify_rc restart_leds
Apr 18 22:00:00 RT-AC86U-4608 custom_script[1]: Running /jffs/scripts/service-event (args: restart leds) - max timeout = 120s
Apr 18 22:07:51 RT-AC86U-4608 kernel: x 14
Apr 19 00:00:00 RT-AC86U-4608 rc_service[1315]: service 1315:notify_rc restart_httpd
Apr 19 00:00:00 RT-AC86U-4608 custom_script[1]: Running /jffs/scripts/service-event (args: restart httpd) - max timeout = 120s
Apr 19 00:00:00 RT-AC86U-4608 RT-AC86U[1]: start https:8443
Apr 19 00:00:00 RT-AC86U-4608 RT-AC86U[1]: start httpd:80
Apr 19 00:07:58 RT-AC86U-4608 kernel: x 45

Apr 19 05:20:03 RT-AC86U-4608 Diversion[9873]: rotated dnsmasq log files, from /opt/share/diversion/file/rotate-logs.div
Apr 19 05:38:31 RT-AC86U-4608 kernel: x 4

Apr 19 06:00:00 RT-AC86U-4608 rc_service[18303]: service 18303:notify_rc restart_httpd
Apr 19 06:00:00 RT-AC86U-4608 custom_script[1]: Running /jffs/scripts/service-event (args: restart httpd) - max timeout = 120s
Apr 19 06:00:00 RT-AC86U-4608 RT-AC86U[1]: start https:8443
Apr 19 06:00:00 RT-AC86U-4608 RT-AC86U[1]: start httpd:80
Apr 19 06:27:41 RT-AC86U-4608 kernel: x 4

Apr 19 07:00:00 RT-AC86U-4608 rc_service[31800]: service 31800:notify_rc restart_leds
Apr 19 07:00:01 RT-AC86U-4608 custom_script[1]: Running /jffs/scripts/service-event (args: restart leds) - max timeout = 120s
Apr 19 07:06:16 RT-AC86U-4608 kernel: x 20

The only thing I want to resolve is the Banmalware cron job from Skynet. I can see using "cru l" that it is currently set to run at 01:25 but I can find anything in any log nor in Loggly. Is there a way to see if a cron ran other than the logs? (I know, really dumb question.)

I know others have set up their own cron jobs to run Skynet banmalware more often, maybe I'll try that.

So, thanks to @cmkelley, @elorimer, @tomsk and others, I am *finally* content to just sit on my hands and let this thing run. I'm also more than willing to play guinea pig to test and see if I can break it again!

 

[cmkelley]

Here is an update on getting Skynet logging working on my RT-AC86u. After four uninstall scribe, let things run, reinstall scribe, I have almost all logging that I want to see working the way it does without scribe.

My final step that worked is to disable Skynet before scribe install. After adding my own filter files to logrotate.d and syslog-ng.d, verifying all permissions chmod 600, then I restarted Skynet and went into setting there and changed logging to custom /opt/var/log/skynet-0.log (without Skynet running scribe did not change log location, as designed I assume). It seems that there is an issue installing scribe on the AC86u with Skynet running, at least for me.

If Skynet isn't present when scribe is installed, the syslog-ng helper file to sort the iptables output to /opt/var/log/skynet-0.log isn't installed and those should be going to messages (/opt/var/log/skynet-0.log would remain empty). Are you copying over the skynet helper file after setting up Skynet?

This is very curious. I'm struggling with how it's possible that the order of install makes a difference. Does it keep getting every hourly entry after a reboot?

 

[Butterfly Bones]

If Skynet isn't present when scribe is installed, the syslog-ng helper file to sort the iptables output to /opt/var/log/skynet-0.log isn't installed and those should be going to messages (/opt/var/log/skynet-0.log would remain empty). Are you copying over the skynet helper file after setting up Skynet?

This is very curious. I'm struggling with how it's possible that the order of install makes a difference. Does it keep getting every hourly entry after a reboot?

I don't understand the "skynet helper file". Do you mean what I call the "filter file" in syslog-ng.d? I copied it over from the one saved on my computer that matched the one from the github in /syslog-ng.share/ named "skynet", along with the other "filter" files that I use, not on your github, then ran "scribe restart".

Then I opened Skynet via AMTM, entered Settings (11) then Syslog location (10) then syslog.log (1) and chose Custom (2), and typed in "/opt/var/log/skynet-0.log" pressed Enter key and backed out of Skynet to AMTM.

I know on previous scribe installs with skynet running, I remember seeing the scribe script output about changing the skynet logging (no captures, not sure of exact wording). I watched for it this time as well, and of course did not see it, which I understand.

I just had a reboot from my ChkWAN script and yes, I am still seeing the [SAVE] messages from Skynet in Loggly, but nowhere in logs from my AC86u. I see all of them from today from 0000 to 1100 hours. The ChkWAN reboot was at 10:52.

 

[cmkelley]

Pushed v0.9_2

I think this is feature complete for 1.0; I don't intend to add more features, just squash bugs.

updates from 9_1

• logrotate is now run once after install, primarily to create its log files that would not be created until it ran overnight.
• provide a check that logrotate ran correctly in the above
• add checks that the hooks in the various scripts haven't been written over
• added logrotate file for wlceventd
• updated logrotate file for firewall per suggestion from B00ze64 on GitHub
• greatly increased output verbosity

Should be able to update with a simple 'scribe update' ... except the logrotate files.

 

[cmkelley]

I don't understand the "skynet helper file". Do you mean what I call the "filter file" in syslog-ng.d? I copied it over from the one saved on my computer that matched the one from the github in /syslog-ng.share/ named "skynet", along with the other "filter" files that I use, not on your github, then ran "scribe restart".

Yes, that's what I meant.

 

[elorimer]

Upgraded to 0.9.2 fine.

 

[Butterfly Bones]

Upgraded to 0.9.2 fine.

Me too, and still getting Skynet hourly [save] lines in Loggy, just nowhere on my router that I can find.

I know they have to be *somewhere* or they would never get sent to Loggly. I just have no idea where.