Scribe scribe - syslog-ng and logrotate installer

[elorimer]

Or am I better of deleting them and change the config to avoid these large sizes (don't know yet how) ?

You can exclude that log from the webui display for starters. If you don't need those log messages, you can just discard them by deleting (or commenting out) the destination in the log statement.

 

[cmkelley]

Y'all - what do we think is a good maxsize? I'm going to add a maxsize argument to the A00global file, this will cause logs over maxsize to be rotated regardless of if they have hit their rotation time or not. I've got an 86U, so mine probably isn't a good benchmark, I've got one currently that is 4 megs without a problem, but I don't know what would happen if they all grew that large. I would hope that only a couple at most would get anywhere close to that before the rotation schedule kicked in.

2 megs?
4 megs?
8 megs?

I'm thinking 4, personally.

 

[L&LD]

@cmkelley what is the impact of larger files? Could this be customized for the router platform and/or the user (preference)?

 

[cmkelley]

@cmkelley what is the impact of larger files? Could this be customized for the router platform and/or the user (preference)?

If you're using uiScribe, beyond some size, the webUI becomes unresponsive when you try to view the log files.

_COULD_ it be customized? Yeah, I mean, I know how I could do it, and give some options, but to be honest, I think that's a lot of work for very little gain.

 

[L&LD]

@cmkelley thanks for the information. Just curious and wanted to know if it was a hard programming, storage or other issue.

 

[joe scian]

Y'all - what do we think is a good maxsize? I'm going to add a maxsize argument to the A00global file, this will cause logs over maxsize to be rotated regardless of if they have hit their rotation time or not. I've got an 86U, so mine probably isn't a good benchmark, I've got one currently that is 4 megs without a problem, but I don't know what would happen if they all grew that large. I would hope that only a couple at most would get anywhere close to that before the rotation schedule kicked in.

2 megs?
4 megs?
8 megs?

I'm thinking 4, personally.

4 is good - 8 is too big and performance suffers

 

[cmkelley]

@cmkelley thanks for the information. Just curious and wanted to know if it was a hard programming, storage or other issue.

I'm a little less tired this morning ... the hard part is really figuring out (a) does the size really matter that much from model to model, and (b) if it does, what is the right size for each model. Simply implementing a choice wouldn't be super difficult, but that begs the question that the user already knows they want to change it - look at the questions and differing opinions on swap file size as an example. At that point it's not too big a stretch to assume they would be sophisticated enough to change it on their own with little to no instruction. Particularly so if the answer to (a) is "no".

 

[elorimer]

I guess we need data, which is kind of hard to generate with logrotate.

But, we could all put maxsize 8192k in A00, and comment out all the weekly, monthly, daily lines we might have in logrotate.conf and the logrotate.d files. The log files will all generate at different rates, but maybe after a week we could see at what point our uiScribe display gets sluggish and report back with our model.

 

[Butterfly Bones]

I'm a little less tired this morning ... the hard part is really figuring out (a) does the size really matter that much from model to model, and (b) if it does, what is the right size for each model. Simply implementing a choice wouldn't be super difficult, but that begs the question that the user already knows they want to change it - look at the questions and differing opinions on swap file size as an example. At that point it's not too big a stretch to assume they would be sophisticated enough to change it on their own with little to no instruction. Particularly so if the answer to (a) is "no".

I agree. I tend to read and learn about the scripts / apps I have installed and then if I have an issue (I did with log sizes some time back). So I customized all my logrotate files based on log levels producing more or less data, and what and how much history I wanted.

Then scribe began defining more parameters, I found it tedious to keep running diffs on all syslog-ng filters and logrotate files, so I gave up and just installed defaults. That was giving me different results than I wanted, so I went back and edited all again to produce what I wanted.

I'm not a fan of global setting for the most part, but know it is useful very often. For the logrotate files, I would answer no, do not set a global maxsize. Mine, where set are only 1 or 2, for me 4 is way too big;

 

[cmkelley]

I agree. I tend to read and learn about the scripts / apps I have installed and then if I have an issue (I did with log sizes some time back). So I customized all my logrotate files based on log levels producing more or less data, and what and how much history I wanted.

Then scribe began defining more parameters, I found it tedious to keep running diffs on all syslog-ng filters and logrotate files, so I gave up and just installed defaults. That was giving me different results than I wanted, so I went back and edited all again to produce what I wanted.

I'm not a fan of global setting for the most part, but know it is useful very often. For the logrotate files, I would answer no, do not set a global maxsize. Mine, where set are only 1 or 2, for me 4 is way too big;

So, the problem with not setting any global maxsize, is that there are cases where less technically knowledgeable users end up with an unresponsive webUI when something unexpectedly starts spamming one of the logs and it gets huge really quick. I think with the ability to view the diff before merging changes, people like yourself who intentionally change things can set them the way you like, while still protecting people who want a "set it and forget it" approach. If logs don't reach the maxsize, they'll still rotate at the time period (daily, weekly, monthly), unless they have a minsize set, and then they won't rotate until both the time period and minsize checks are met.

Is 4 way too big for you because it slows your webUI or because it's just too much information? I haven't had issues with logs that big on my 86U.

 

[Butterfly Bones]

So, the problem with not setting any global maxsize, is that there are cases where less technically knowledgeable users end up with an unresponsive webUI when something unexpectedly starts spamming one of the logs and it gets huge really quick. I think with the ability to view the diff before merging changes, people like yourself who intentionally change things can set them the way you like, while still protecting people who want a "set it and forget it" approach. If logs don't reach the maxsize, they'll still rotate at the time period (daily, weekly, monthly), unless they have a minsize set, and then they won't rotate until both the time period and minsize checks are met.

Is 4 way too big for you because it slows your webUI or because it's just too much information? I haven't had issues with logs that big on my 86U.

I agree, best to set if for the less technically knowledgeable. One plus on the global maxsize is I only need to edit one file to get what works best for me, so that is a plus plus. I have found having generally 1-2 MB file size works best if I show all nine eleven syslog segments in uiScribe (I do occasionally, not always). Keeps that entire page lively, more than that gets sluggish, but I am just being picky.

What I found works best is to set minsize 1024k in all files, than customize rotate as daily/weekly/monthly to keep the sizes manageable. I monitored them daily to determine how much data I wanted to store, based on the log importance to me.

I do have to agree with your logic here. My maxsize is personal preference only.

 

[L&LD]

I was having issues with unbound.log being handled by Scribe.

When I enabled logging in unbound_manager by using 3, scribe, the router GUI would freeze when accessing System Log.

I got around that by using uiScribe to exclude the unbound.log file, but of course, the GUI wouldn't show the log.

I'm not sure what sequence of events I did that showed the unbound.log 'box' in the GUI, but with no logging, but I thought I had it working until yesterday when I was shown in another thread that I should be seeing something there!

I tried many combinations of st and rs in Scribe but nothing worked.

I finally had a chance to do some more detective work and cleanup on my own router today.

The unbound.log file was 130MB!

Using WinSCP, I located it in /opt/var/log/ and deleted it.

In unbound_manager, I once again ran 3, scribe. I also used uiScribe to include the excluded unbound.log file and now the GUI and the unbound log are showing life. Success!

My only question now is that in a very short time (less than 10 minutes), the unbound.log file is at 1MB already.

My question is should I be concerned about how fast the log file is growing? Or will Scribe take care of this now for me?

 

[cmkelley]

I was having issues with unbound.log being handled by Scribe.

When I enabled logging in unbound_manager by using 3, scribe, the router GUI would freeze when accessing System Log.

I got around that by using uiScribe to exclude the unbound.log file, but of course, the GUI wouldn't show the log.

I'm not sure what sequence of events I did that showed the unbound.log 'box' in the GUI, but with no logging, but I thought I had it working until yesterday when I was shown in another thread that I should be seeing something there!

I tried many combinations of st and rs in Scribe but nothing worked.

I finally had a chance to do some more detective work and cleanup on my own router today.

The unbound.log file was 130MB!

Using WinSCP, I located it in /opt/var/log/ and deleted it.

In unbound_manager, I once again ran 3, scribe. I also used uiScribe to include the excluded unbound.log file and now the GUI and the unbound log are showing life. Success!

My only question now is that in a very short time (less than 10 minutes), the unbound.log file is at 1MB already.

My question is should I be concerned about how fast the log file is growing? Or will Scribe take care of this now for me?

Yes, you should be concerned, because logrotate still only runs once a day, at 00:05, so the log will become way too huge before rotation. Even hourly isn't going to help if it's running 6 to 10 MB/hour. That seems like an insane amount of logging.

I don't use unbound, so I don't know what unbound_manger is doing.

 

[joe scian]

I was having issues with unbound.log being handled by Scribe.

When I enabled logging in unbound_manager by using 3, scribe, the router GUI would freeze when accessing System Log.

I got around that by using uiScribe to exclude the unbound.log file, but of course, the GUI wouldn't show the log.

I'm not sure what sequence of events I did that showed the unbound.log 'box' in the GUI, but with no logging, but I thought I had it working until yesterday when I was shown in another thread that I should be seeing something there!

I tried many combinations of st and rs in Scribe but nothing worked.

I finally had a chance to do some more detective work and cleanup on my own router today.

The unbound.log file was 130MB!

Using WinSCP, I located it in /opt/var/log/ and deleted it.

In unbound_manager, I once again ran 3, scribe. I also used uiScribe to include the excluded unbound.log file and now the GUI and the unbound log are showing life. Success!

My only question now is that in a very short time (less than 10 minutes), the unbound.log file is at 1MB already.

My question is should I be concerned about how fast the log file is growing? Or will Scribe take care of this now for me?

Crazy that yours is 1 MByte in less than 10mins. Mine is only 3K been running for 2 days. Did you modify the logging or verbosity in unbound.conf or just using default unbound.conf from Martineaus install file? Im only getting "starting" messages in log. What are you getting - dont tell me you are getting all the dns queries as well?

 

[L&LD]

@joe scian I don't think I changed anything specifically.

My unbound.conf file is here.

https://www.snbforums.com/threads/u...-caching-dns-server.58967/page-53#post-547423

 

[kernol]

@joe scian I don't think I changed anything specifically.

My unbound.conf file is here.

https://www.snbforums.com/threads/u...-caching-dns-server.58967/page-53#post-547423

My unbound.conf file identical to yours except for IP6 settings. I don't have an issue with unbound log in Scribe and my unbound.log file in /opt/var/log is less than 3k after several hours.

That is not where the unbound LIVE log file lives - its default directory is /opt/var/lib/unbound/unbound.log.
If the LIVE log file has found its way into Scribe - it would certainly explain why you will have webUI crashes.

This is what your unbound script should be in /opt/etc/syslog-ng.d

# log all unbound logs to /opt/var/log/unbound.log and stop processing unbound logs

destination d_unbound {
    file("/opt/var/log/unbound.log");
};

filter f_unbound {
    program("unbound");
};

log {
    source(src);
    filter(f_unbound);
    destination(d_unbound);
    flags(final);
};
#eof

 

[SomeWhereOverTheRainBow]

got this when opening uiScribe today don't know if it relates to scribe or UiScribe

[2020-02-05T05:37:32.353122] WARNING: Configuration file format is too old, syslog-ng is running in compatibility mode. Please update it to use the syslog-ng 3.25 format at your time of convenience. To upgrade the configuration, please review the warnings about incompatible changes printed by syslog-ng, and once completed change the @version header at the top of the configuration file; config-version='3.23'

 

[SomeWhereOverTheRainBow]

updating scribe fixed the issue..

 

[dave14305]

@joe scian I don't think I changed anything specifically.

My unbound.conf file is here.

https://www.snbforums.com/threads/u...-caching-dns-server.58967/page-53#post-547423

You probably should not leave query logging enabled all the time. It can affect performance per the official documentation.

And if you have both Diversion and Unbound logging DNS queries simultaneously its more activity on your USB.

I enable logging ad-how when needed to confirm things are working. I setup profile commands here:
Unbound - Authoritative Recursive Caching DNS Server

 

[L&LD]

@dave14305 thank you. I thought this was like Skynet where logging should be on by default.

I had already turned it off yesterday when I found it growing so fast.

I will be trying what @kernol suggested to me in his post directly above with scribe.