Scribe scribe - syslog-ng and logrotate installer

[elorimer]

So without loggly [or equivalent] is the script "as is" useful for us noobs?

Totally worth it.

I set it up to work with a free loggly account mostly out of curiosity and proof of concept. One day I will get around to making it useful or trying one of the alternatives (like, um, the other "scribe"), perhaps in a jail on my NAS getting fed by other devices.

I have pixelserv-tls set at log level 2 because I can collect statistics from it. But it generates thousands of messages as a byproduct, and once tabulated, are useless. So using syslog-ng to pull those to one file means the collecting process only operates on that file, and then logrotate deletes the older messages automatically. Presto.

I have syslog-ng pulling out the skynet messages for essentially the same process for its statistics, including the tons of messages from the last hour until they get destroyed. Presto.

I have syslog-ng sending openvpn messages to its own file, because sometimes I want to see those messages, and generally don't want to see all the refused connections as someone trolls the ports.

What I have left is a webgui log that has mostly routine messages but is much easier to scan. Scribe leaves you just with handling configurations you might want to set up, or try, in syslog-ng.d and a companion logrotate in logrotate.d . But the @cmkelley has done all the heavy lifting (and the light lifting too) evidenced by over 500 messages in this thread.

 

[kernol]

Totally worth it.

I set it up to work with a free loggly account mostly out of curiosity and proof of concept. One day I will get around to making it useful or trying one of the alternatives (like, um, the other "scribe"), perhaps in a jail on my NAS getting fed by other devices.

I have pixelserv-tls set at log level 2 because I can collect statistics from it. But it generates thousands of messages as a byproduct, and once tabulated, are useless. So using syslog-ng to pull those to one file means the collecting process only operates on that file, and then logrotate deletes the older messages automatically. Presto.

I have syslog-ng pulling out the skynet messages for essentially the same process for its statistics, including the tons of messages from the last hour until they get destroyed. Presto.

I have syslog-ng sending openvpn messages to its own file, because sometimes I want to see those messages, and generally don't want to see all the refused connections as someone trolls the ports.

What I have left is a webgui log that has mostly routine messages but is much easier to scan. Scribe leaves you just with handling configurations you might want to set up, or try, in syslog-ng.d and a companion logrotate in logrotate.d . But the @cmkelley has done all the heavy lifting (and the light lifting too) evidenced by over 500 messages in this thread.

Many thanks for that input.
I had installed and applied precisely what you have suggested above before seeing your message .
What I may have missed though - is including them in logrotate.d ... so have just done so {thumbs-Up}.

Just an observation for @cmkelley ... following fresh install I was alerted to the fact that syslog-ng was running in "compatibility mode" and was advised to update and change a line in the syslog-ng config file. For the sake of "noobs" not familiar with such edits - may be good to automate/fix that? The current version of syslog-ng is 3.20 not 3.19 as set in the syslog-ng config file provided in the scribe install.

Now to find a simple web interface to pull the log files for easy access. Tried the Synology NAS option - but not overly impressed.
I'm spoilt by the web based log reporting of our office Sophos XG Firewall Appliance!

 

[cmkelley]

Many thanks for that input.
I had installed and applied precisely what you have suggested above before seeing your message .
What I may have missed though - is including them in logrotate.d ... so have just done so {thumbs-Up}.

Just an observation for @cmkelley ... following fresh install I was alerted to the fact that syslog-ng was running in "compatibility mode" and was advised to update and change a line in the syslog-ng config file. For the sake of "noobs" not familiar with such edits - may be good to automate/fix that? The current version of syslog-ng is 3.20 not 3.19 as set in the syslog-ng config file provided in the scribe install.

Now to find a simple web interface to pull the log files for easy access. Tried the Synology NAS option - but not overly impressed.
I'm spoilt by the web based log reporting of our office Sophos XG Firewall Appliance!

They must have just pushed that out. I completely wiped and re-installed on my testing router a couple times Sunday and it was still 3.19 then. Reading the changelog it doesn't seem like it should be a problem to roll the version in the syslog-ng.conf file, but I'll have to play with it a bit on my test router before pushing a change to fix that.

Entware's current version may be 3.20, but the actual current version is 3.21. Syslog-ng appears to push out releases every ~2 months, I'd guess Entware will always be a release behind the current.

 

[cmkelley]

I set it up to work with a free loggly account mostly out of curiosity and proof of concept. One day I will get around to making it useful or trying one of the alternatives (like, um, the other "scribe"), perhaps in a jail on my NAS getting fed by other devices.

Now to find a simple web interface to pull the log files for easy access. Tried the Synology NAS option - but not overly impressed.
I'm spoilt by the web based log reporting of our office Sophos XG Firewall Appliance!

I don't know anything about it, but syslog-ng 3.20 added support for the collectd daemon (https://collectd.org/), which is available in Entware, that appears at first blush to do some log massaging to get useful data. I'm not interested enough to delve into it, but it might be interesting for someone trying to pull more data from their logs. It seems to have active development, although the current release is about 18 months old.

 

[cmkelley]

Many thanks for that input.
I had installed and applied precisely what you have suggested above before seeing your message .
What I may have missed though - is including them in logrotate.d ... so have just done so {thumbs-Up}.

Just an observation for @cmkelley ... following fresh install I was alerted to the fact that syslog-ng was running in "compatibility mode" and was advised to update and change a line in the syslog-ng config file. For the sake of "noobs" not familiar with such edits - may be good to automate/fix that? The current version of syslog-ng is 3.20 not 3.19 as set in the syslog-ng config file provided in the scribe install.

Now to find a simple web interface to pull the log files for easy access. Tried the Synology NAS option - but not overly impressed.
I'm spoilt by the web based log reporting of our office Sophos XG Firewall Appliance!

Oh, joy. The syslog-ng.conf file that Entware is providing still says 3.19 as well.

EDIT: Ahh, I see (and remember now) ... Entware gets their syslog-ng.conf from OpenWrt, which is still on 3.19 ... not sure why Entware bothered to update the executable though.

Looking at the 3.20 archive from balabit on github, they've a mish-mash of different config files, many of which try to duplicate the traditional mapping into various log files (e.g. mail, cron, debug, etc.). I can't say I blame Entware OpenWrt for just sticking with what they've got.

 

[cmkelley]

Just an observation for @cmkelley ... following fresh install I was alerted to the fact that syslog-ng was running in "compatibility mode" and was advised to update and change a line in the syslog-ng config file. For the sake of "noobs" not familiar with such edits - may be good to automate/fix that? The current version of syslog-ng is 3.20 not 3.19 as set in the syslog-ng config file provided in the scribe install.

To solve your immediate issue, ssh into your router and type:

sed -i "s/3.19/3.20/" /opt/etc/syslog-ng.conf

That will at least shut up the error message until I have time to add a fix to scribe.

 

[kernol]

To solve your immediate issue, ssh into your router and type:

sed -i "s/3.19/3.20/" /opt/etc/syslog-ng.conf

That will at least shut up the error message until I have time to add a fix to scribe.

No need ... I simply edited line 14 of your syslog-ng.conf file to "@version: 3.20" from "@version: 3.19" and all runs well without error messages !

 

[kernol]

Oh, joy. The syslog-ng.conf file that Entware is providing still says 3.19 as well.

EDIT: Ahh, I see (and remember now) ... Entware gets their syslog-ng.conf from OpenWrt, which is still on 3.19 ... not sure why Entware bothered to update the executable though.

Looking at the 3.20 archive from balabit on github, they've a mish-mash of different config files, many of which try to duplicate the traditional mapping into various log files (e.g. mail, cron, debug, etc.). I can't say I blame Entware OpenWrt for just sticking with what they've got.

I have no idea how version 3.20 was installed on my router.
I did a full factory reset and USB format with recent install of 384.11-2 - installed amtm plus add-ons seen in my signature for the AC5300 and a week or so later upgraded firmware to 384.12-Alpha [2nd edition] after removing Xentrk Stubby and Jack Yaz ntpMerlin [because I am now using firmware built in versions].
No custom upgrades or updates done - so must have come down with Diversion's install of Entware?

 

[elorimer]

I think we've seen sneaky upgrades of syslog-ng before. Some scripts are very careful--I think the stubby script was-- to only upgrade their entware packages. Others did a simple opkg update;opkg upgrade and bob's your uncle. Scribe now has less to do with the syslog-ng.conf file, and after getting the @include statement moved in the entware version there is much less trobule in this area.

I did the same edit of the version line, and that is the only thing I see that changed with 3.20. Fooling around with collectd now though. I note that this is going to involve the scl directory so it may be some distance down the road. The combination of that and sqlite being enabled could bring about some really interesting things.

 

[cmkelley]

I have no idea how version 3.20 was installed on my router.
I did a full factory reset and USB format with recent install of 384.11-2 - installed amtm plus add-ons seen in my signature for the AC5300 and a week or so later upgraded firmware to 384.12-Alpha [2nd edition] after removing Xentrk Stubby and Jack Yaz ntpMerlin [because I am now using firmware built in versions].
No custom upgrades or updates done - so must have come down with Diversion's install of Entware?

When you did the fresh install of scribe it pulled down the latest version from Entware. Generally it's better to have the latest version of things, often security vulnerabilities are addressed.

I'll probably add an option to the menu version to adjust the syslog-ng.conf @version line to match the installed syslog-ng, hopefully syslog-ng won't break forward compatibility with a point version change.

 

[cmkelley]

V1.3_1 is up

Automagically changes the '@version' line in syslog-ng.conf to match the installed syslog-ng version without so much as a "by your leave".

"scribe update" to update.

 

[L&LD]

V1.3_1 is up

Automagically changes the '@version' line in syslog-ng.conf to match the installed syslog-ng version without so much as a "by your leave".

"scribe update" to update.

I installed this update on my main RT-AC86U w/RMerlin 384.12 Alpha 1 (latest today) installed.

Showed 3.19 installed, no issues.

Then I started amtm, Diversion, 'ep', '6', '6' and updated all Entware packages installed. No issues.

I then went back to scribe and ran a 'scribe status' command and it showed 3.20 installed, no issues.

Thank you again for these boringly working as intended updates!

 

[kernol]

... Fooling around with collectd now though. I note that this is going to involve the scl directory so it may be some distance down the road. The combination of that and sqlite being enabled could bring about some really interesting things.

Good to hear ... I also had a look at collectd ... can see interesting possibilities ... for those with coding skills [- which I seriously lack ].
Keep us posted if you find it works for you.

 

[cmkelley]

I installed this update on my main RT-AC86U w/RMerlin 384.12 Alpha 1 (latest today) installed.

Showed 3.19 installed, no issues.

Then I started amtm, Diversion, 'ep', '6', '6' and updated all Entware packages installed. No issues.

I then went back to scribe and ran a 'scribe status' command and it showed 3.20 installed, no issues.

Thank you again for these boringly working as intended updates!

As opposed to the pre-v1.0_0 updates which were as likely to break new things as they were to fix problems.

 

[L&LD]

As opposed to the pre-v1.0_0 updates which were as likely to break new things as they were to fix problems.

I certainly didn't find that to be the case for me.

I think I've had only one issue, but it was quickly resolved and didn't really affect me except that I knew my install was 'broken' for a few minutes.

 

[cmkelley]

scribe v2.0_0 is available. This version includes a menu system.

First and most importantly: DO NOT BUG thelonelycoder about adding scribe to amtm! He is very well aware of it, however, he has a much more pressing real life (i.e. work) issue he has to complete first. He will get to it when he can and feels inclined, and not before. Thank you.

To update from v1.1_0 or later, simply issue "scribe update" at the command line. Note that it may ask you twice about updating filters. This will only happen this time.

To update from v1.0_0 or before, you must use:

/usr/sbin/curl --retry 3 "https://raw.githubusercontent.com/cynicastic/scribe/master/scribe" -o "/jffs/scripts/scribe" && chmod 0755 /jffs/scripts/scribe && /jffs/scripts/scribe install

After updating, just starting scribe will bring up the menu. The command line options will still work.

Other than the menu, the only real change from previous versions is automatically matching the syslog-ng.conf "@version=" line to the currently installed version of syslog-ng. This is necessitated by Entware choosing not to roll the version # of their distributed syslog-ng.conf, resulting in an error message when checking the syntax of syslog-ng.conf.

The change to the menu structure required a couple logic changes, I think I've got it all sorted, let me know if you see anything strange.

N.B. it is late here on the west coast of USizicstania, I've got work tomorrow, and coding is not my day job. So response time and fixing issues may be delayed.

 

[Butterfly Bones]

scribe v2.0_0 is available. This version includes a menu system.

First and most importantly: DO NOT BUG thelonelycoder about adding scribe to amtm! He is very well aware of it, however, he has a much more pressing real life (i.e. work) issue he has to complete first. He will get to it when he can and feels inclined, and not before. Thank you.

To update from v1.1_0 or later, simply issue "scribe update" at the command line. Note that it may ask you twice about updating filters. This will only happen this time.

To update from v1.0_0 or before, you must use:

/usr/sbin/curl --retry 3 "https://raw.githubusercontent.com/cynicastic/scribe/master/scribe" -o "/jffs/scripts/scribe" && chmod 0755 /jffs/scripts/scribe && /jffs/scripts/scribe install

After updating, just starting scribe will bring up the menu. The command line options will still work.

Other than the menu, the only real change from previous versions is automatically matching the syslog-ng.conf "@version=" line to the currently installed version of syslog-ng. This is necessitated by Entware choosing not to roll the version # of their distributed syslog-ng.conf, resulting in an error message when checking the syntax of syslog-ng.conf.

The change to the menu structure required a couple logic changes, I think I've got it all sorted, let me know if you see anything strange.

N.B. it is late here on the west coast of USizicstania, I've got work tomorrow, and coding is not my day job. So response time and fixing issues may be delayed.

That was painless, thank goodness. I was without scribe for almost a week, troubleshooting why it was not grabbing the syslog. Most events but not all were not being pulled out. I kept removing and removing scripts to get as bare bones as was comfortable. Never did figure out what the heck was going on. Thinking it was the 384.12 alpha 1, I decided to reflash it and low and behold all worked. Must have been something weird in the flash that affected so little it was like a needle in a haystack.

Of course, I needed to jump into the deep end with both feet immediately, so here I am on 384.12 alpha 2 with the update to scribe v.2

All is running and scribe / syslog-ng is cleaning the syslog twinkly clean! Both VPN client and server up and running. Staying with amtm/Diversion, Skynet, scribe only for a few days. Since I run Linux, the x-term has tabs, so I just added one for "scribe" to the two I already had setup for "amtm" and "terminal". Roll the mouse wheel over the window to change tabs.

 

[L&LD]

Updated to v2.0 no issues (also on Alpha 2, like Butterfly Bones above).

 

[thelonelycoder]

First and most importantly: DO NOT BUG thelonelycoder about adding scribe to amtm! He is very well aware of it, however, he has a much more pressing real life (i.e. work) issue he has to complete first. He will get to it when he can and feels inclined, and not before. Thank you

I thank you! Problem identified, solution found, implemented and tested. Happy customer, happy employer, satisfied me. Flying home from Toronto this evening.

 

[akb]

I am sorry but how do i use this script. The menu system does not display any logs, my web guis does not show any logs. What is the point of this if I can not as a stupid user even find any ways to look at logs?