Scribe scribe - syslog-ng and logrotate installer

[cmkelley]

Feel free to clone the code in https://github.com/Xentrk/Asuswrt-Merlin-Linux-Shell-Scripts/blob/master/dhcpstaticlist.sh

Look at the functions Save_DHCP_Staticlist and Make_Backukp

I'm lazy. After a bit of a think I decided to wipe it on install & uninstall. Install or re-install is the only time it gets checked anyways, and anything that gets written to it should be immediately picked up by syslog-ng.

 

[cmkelley]

v1.2_1 is up

• delete logrotate.daily just before running logrotate for the first time when (re-)installing to get rid of stale messages (h/t Xentrk)
  • also deletes it when uninstalling scribe
• when checking the version of scribe on github, it had been retrieving the same file 3 times; now only gets it once
  • which exposed a bug where stopping grep at the first instance (-m1) was needed but missing

 

[Xentrk]

I'm lazy. After a bit of a think I decided to wipe it on install & uninstall. Install or re-install is the only time it gets checked anyways, and anything that gets written to it should be immediately picked up by syslog-ng.

The reason I suggested a backup rather than a replace or removal is for those who may already have syslog-ng and logratate installed. Perhaps they want to test with Scribe before deciding to cut over. They need the ability to restore their prior config. I am in this category. I may be a fringe case though. Most people may be coming from ground zero. I did copy over the config files after the first install attempt failed. I also have jffs backups to fall back on.

Thanks again for your work and effort on scribe. syslog-ng is not the easiest to understand. I applaud your work and am grateful for the project.

 

[Xentrk]

v1.2_1 is up

• delete logrotate.daily just before running logrotate for the first time when (re-)installing to get rid of stale messages (h/t Xentrk)
  • also deletes it when uninstalling scribe
• when checking the version of scribe on github, it had been retrieving the same file 3 times; now only gets it once
  • which exposed a bug where stopping grep at the first instance (-m1) was needed but missing

Updated went well. No error messages in /opt/tmp/logrotate.daily.

 

[Xentrk]

Looks like my old syslog-ng config file got wiped out. The one feature it had was logging pixelserv-tls to a separate log file. I looked at my history and there is not that much data. But for some people testing a new version of pixelserv-tls, it may be useful. I do have the config posted here. Please consider it for a future item. Thanks.

 

[Butterfly Bones]

Looks like my old syslog-ng config file got wiped out. The one feature it had was logging pixelserv-tls to a separate log file. I looked at my history and there is not that much data. But for some people testing a new version of pixelserv-tls, it may be useful. I do have the config posted here. Please consider it for a future item. Thanks.

The pixelserv-tls filter file ("helper file" per cmkelley terminology) is included with the install, at least it is there in syslog-ng/share.
https://github.com/cynicastic/scribe/blob/master/syslog-ng.share/pixelserv

 

[cmkelley]

The reason I suggested a backup rather than a replace or removal is for those who may already have syslog-ng and logratate installed. Perhaps they want to test with Scribe before deciding to cut over. They need the ability to restore their prior config. I am in this category. I may be a fringe case though. Most people may be coming from ground zero. I did copy over the config files after the first install attempt failed. I also have jffs backups to fall back on.

Thanks again for your work and effort on scribe. syslog-ng is not the easiest to understand. I applaud your work and am grateful for the project.

So, the "logrotate.daily" file isn't a standard logrotate file. I just made that name up as a place to redirect any console output from the nightly logrotate run. "logrotate.status" is the file that logrotate creates when it runs. It shouldn't exist before scribe is installed, except for the very edge case where someone created a cron job for logrotate, decided to redirect the console output, and choose the same filename I did.

 

[cmkelley]

The pixelserv-tls filter file ("helper file" per cmkelley terminology) is included with the install, at least it is there in syslog-ng/share.
https://github.com/cynicastic/scribe/blob/master/syslog-ng.share/pixelserv

Yep, it's just not automatically copied to /opt/etc/syslog-ng.d ... only Skynet gets that special treatment since it produces so many logs.

And I can't say I'm particularly consistent about my terminology. "filter" is probably better wording.

 

[elorimer]

I don't know that filter is better wording. In most cases this is a custom configuration defining filter, destination and log{}. I like that you don't automatically copy the examples to the .d directories, since my configuration differs. But I now routinely create a backup of the .d directories before updating scribe, just in case.

 

[cmkelley]

I don't know that filter is better wording. In most cases this is a custom configuration defining filter, destination and log{}. I like that you don't automatically copy the examples to the .d directories, since my configuration differs. But I now routinely create a backup of the .d directories before updating scribe, just in case.

That, or something like that, is already on my todo list. scribe ToDo list

 

[cmkelley]

Looks like my old syslog-ng config file got wiped out. The one feature it had was logging pixelserv-tls to a separate log file. I looked at my history and there is not that much data. But for some people testing a new version of pixelserv-tls, it may be useful. I do have the config posted here. Please consider it for a future item. Thanks.

A quick note, as Butterfly Bones said, I do have a filter for pixelserv in /opt/share/syslog-ng/examples, however that only filters pixelsrv logs, where your example filtered pixelsrv and "DROP IN=" messages. I filter the "DROP IN=" messages through either the "Skynet" filter, which is automatically installed if Skynet is detected on installation, or the "firewall" filter, also found in /opt/share/syslog-ng/examples, which would need to be copied to /opt/etc/syslong-ng.d _IF_ Skynet is not installed. Having both firewall and Skynet filters at the same time will result in messages with "DROP IN=" not going to Skynet (and thereby making that statistics useless), since they are processed alphabetically as near as I can determine.

 

[cmkelley]

Looks like my old syslog-ng config file got wiped out. The one feature it had was logging pixelserv-tls to a separate log file. I looked at my history and there is not that much data. But for some people testing a new version of pixelserv-tls, it may be useful. I do have the config posted here. Please consider it for a future item. Thanks.

Just updating shouldn't have touched your syslog-ng.conf file. If you did an install without uninstalling it should have renamed your syslog-ng.conf to syslog-ng.conf-prev.

Uninstalling however would have removed syslog-ng.conf.

 

[elorimer]

Having both firewall and Skynet filters at the same time will result in messages with "DROP IN=" not going to Skynet (and thereby making that statistics useless), since they are processed alphabetically as near as I can determine.

Just to clarify this, in my understanding it works like this:

Syslog-ng processes log statements in the order in which they appear in the configuration file. Where the configuration file includes a file or directory by an "include" statement, those files are included at that place in the configuration file, and are processed in alphabetic order. In the scribe universe, the include statement is in the beginning of syslog-ng.conf, and the log statement for messages is at the end, so all the log statements throughout /syslog-ng.d will be processed first, and everything not terminated earlier will fall to the messages file. The log statements throughout /syslog-ng.d are not processed themselves alphabetically, but in order of the alphabetic files in which they appear.

In my setup, the log statement in /syslog-ng.d/0loggly sends everything to loggly (no final flag) and then onward to the reset of the /syslog-ng.d configuration files before everything not flagged final goes to messages.

As I think you found early in this process, it doesn't matter where the source, filter, destination, template, etc, statements are, as long as they are defined throughout the included configuration files once and only once. If they are not defined or defined twice, I think you pass the syntax check and fail in -Fevd.

 

[cmkelley]

scribe v1.3_0 is up.

If you have not upgraded from v1.0_0 (hopefully that's nobody at this point) please do so very soon. At this point I'm ready to start working on a menu interface (i.e. version 2.0_0) and the internal versioning system has to change for compatibility with amtm. All versions since v1.0_0 are able to detect either internal version numbering scheme.

Notable changes (at least to me)

• Determine syslog-ng version with 'syslog-ng --version' rather than the syslog-ng.conf file
• If Skynet isn't installed, copies the syslog-ng "firewall" filter to /opt/etc/syslog-ng.d
• Improve updating .d and examples directories
  • for .d files, only add new files, don't change existing ones
  • only way to update existing .d file is to force re-installation
  • for examples files, add new and update any existing
  • creates backups with date and time of existing files in examples
• create backup of -opkg file with time and date on re-install in case it changes
• corrects an error where original -opkg file would get overwritten with copy of modified file
• some UI tweaks
  • that's probably going to continue forever

Barring any "oopsies" I'm going to work on the menu interface.

 

[L&LD]

scribe v1.3_0 is up.

If you have not upgraded from v1.0_0 (hopefully that's nobody at this point) please do so very soon. At this point I'm ready to start working on a menu interface (i.e. version 2.0_0) and the internal versioning system has to change for compatibility with amtm. All versions since v1.0_0 are able to detect either internal version numbering scheme.

Notable changes (at least to me)

• Determine syslog-ng version with 'syslog-ng --version' rather than the syslog-ng.conf file
• If Skynet isn't installed, copies the syslog-ng "firewall" filter to /opt/etc/syslog-ng.d
• Improve updating .d and examples directories
  • for .d files, only add new files, don't change existing ones
  • only way to update existing .d file is to force re-installation
  • for examples files, add new and update any existing
  • creates backups with date and time of existing files in examples
• create backup of -opkg file with time and date on re-install in case it changes
• corrects an error where original -opkg file would get overwritten with copy of modified file
• some UI tweaks
  • that's probably going to continue forever

Barring any "oopsies" I'm going to work on the menu interface.

Thank you!

Just updated to v1.3_0 from v1.2_1. No issues so far.

 

[elorimer]

When you started I thought this would be a simple thing, and this is so much more comprehensive than I could have imagined.

My only divergence at this point is that on start up I don't cat the syslog to messages, I cat it to a new file and flush it through syslog-ng.

 

[cmkelley]

When you started I thought this would be a simple thing, and this is so much more comprehensive than I could have imagined.

You and me both. Not sure I would have started down this road if I had realized how complex it would become. I honestly thought "install, uninstall, update, a couple checks, how hard could this be?" Yeah ....

My only divergence at this point is that on start up I don't cat the syslog to messages, I cat it to a new file and flush it through syslog-ng.

Did you post your approach to that somewhere? Might be something to think about post-2.0.

 

[elorimer]

Did you post your approach to that somewhere?

If I did it was only a fragment.

1. In rc.func.syslog-ng, I use this line, which gets messed up on scribe update:

       cat /tmp/syslog.log > /opt/var/log/startupmessages

That overwrites the file when syslog.log is getting deleted.
2. Then I have a configuration file /opt/etc/syslog-ng.d/0startup:

source s_startup {
    file("/opt/var/log/startupmessages");
};
log {
   source(s_startup);
    filter(f_pixelserv-tls);
    destination(d_pixelserv-tls);
    flags(final);
};
log {
    source(s_startup);
    filter(f_openvpn);
    destination(d_openvpn);
    flags(final);
};
log {
    source(s_startup);
    filter(f_skynet);
    destination(d_skynet);
    flags(final);
};
log {
    source(s_startup);
    destination(messages);
    flags(final);
};

I also include that source in my 0loggly configuration file. So, since they get processed by syslog-ng first, all of syslog.log gets sent to 0loggly, then it gets processed by syslog-ng once to messages. Syslog.log only ever has these three nonkernel messages, which I strip out, but basically my loggly and messages file contains the original syslog.log statements but now with the router name attached. This helps with loggly, because I have two routers sending logs. Cat directly to messages means the starting sequence doesn't get sent to loggly.

0startup has to be processed before any of the other configuration files or else the nonkernel messages will creep in during the non-trivial time while the startupmessages file is being processed, but after it has processed it isn't written to again so it is out of the picture, and syslog-ng goes on to process everything else with the src source. I use the the final flag sending stuff to messages, but probably don't need to since that source doesn't appear in another place.

 

[kernol]

@cmkelley - a quote from the last lines of your Github page ...

Seriously?
Are you really still reading this? I'd have probably lost interest about halfway through!

Yep ... read ALL of it ... PLUS ... all 29 pages of this thread [most of the posts flying way over my head ] and I'm STILL interested .
Clearly this whole syslog-ng thingy would be well beyond my skillset.

So here's the thing?
Your script offering does a wonderful job of splitting up the traditional syslog into separate logs in appropriate functional folders - cleaning up nicely what remains in syslog itself - but is the real advantage only to be found in running something like loggly to scoop up the logs and present them in easy to access reports??? So without loggly [or equivalent] is the script "as is" useful for us noobs?

 

[cmkelley]

I've never used loggy so I can't comment on it. I just look at the logs I'm interested in on the router, I haven't seen the need to use loggy.

The original motivation for me for using syslog-ng was to easily filter out the dcd crash log entries from the system log so I could find other events I was interested in more easily, and then it grew to filtering the iptables entries from Skynet, and then it kinda went from there.

Is it useful without loggy? I guess it depends on what you want to use it for. If you want to go around looking for things "wrong" with your router, please don't. There are tons of messages that look "bad" but are really just informational. I use it to separate stuff out that I know I want to look at separately (openvpn logs) or stuff I don't want to see at all (dcd crashes, wlceventd logs, etc.), leaving the default log (messages) easier to search if something weird does happen, or I want to see the result of something without wading through unrelated stuff.

If you want to learn about the system and part of that is examining the logs, I'd actually recommend starting off with the firmware syslog output for a while until (or maybe syslog-ng with just the dcd crash filter) so you see stuff somewhat in context. Then as you get more knowledgeable about what you're seeing, you can decide on your own what other filters you want to add.