What's new

Scribe scribe - syslog-ng and logrotate installer

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

uiScribe still overrides the default System Log page. We could mount to a separate page, but you'd then have 2 logging tabs - one for default syslog (i.e. messages) and another tab for everything else.

That wouldn't be great UX.
 
uiScribe still overrides the default System Log page. We could mount to a separate page, but you'd then have 2 logging tabs - one for default syslog (i.e. messages) and another tab for everything else.

That wouldn't be great UX.
Does it do that through the new API (like I thought) or is it still the same method as before?

I have a hard enough time remembering how my script works, I need to learn to keep my mouth shut about other people's scripts ...
 
Does it do that through the new API (like I thought) or is it still the same method as before?

I have a hard enough time remembering how my script works, I need to learn to keep my mouth shut about other people's scripts ...
no problem. We're all here to learn! Since uiscribe is just overriding a page and doesn't need a new tab it's dead simple. The complex stuff is when overriding unused pages and needing to insert a tab. the new api makes adding a tab much simpler, and the user mount points removes the need for me to override obscure unused pages
 
In response to the recent issues uncovered with upgrading syslog-ng, and a couple things I've picked at, scribe 2.4.2 is up.
This is dumb silly really stupid question I need to clarify, sorry I'm on some serious pain killers and my mind is fuzzy, and will be for some time. :oops:

The new afpd filter, the - are removed and the + are added?
Code:
 Update available for /opt/etc/logrotate.d/afpd.
    (a)ccept, (r)eject, or (v)iew diff for this file? v

--- /opt/etc/logrotate.d/afpd
+++ /opt/share/logrotate/examples/afpd
@@ -1,6 +1,4 @@
 /opt/var/log/afpd.log {
-minsize 1024k
-   daily
    rotate 2
    postrotate
        /usr/bin/killall -HUP syslog-ng
I loose minsize 1024k for the default 4MB max? and no daily rotate, only after the maxsize of 4MB?

This line totally looses me. :confused:
Code:
@@ -1,6 +1,4 @@
Thanks
 
This is dumb silly really stupid question I need to clarify, sorry I'm on some serious pain killers and my mind is fuzzy, and will be for some time. :oops:
Until you're used to them diffs can be a bit difficult to understand, even without pain meds.
The new afpd filter, the - are removed and the + are added?
Code:
 Update available for /opt/etc/logrotate.d/afpd.
    (a)ccept, (r)eject, or (v)iew diff for this file? v
[/quote]
Yes, more accurately, the lines marked - are only in the first file (/opt/etc/logrotate.d/afpd), the lines marked + are only in the second file (/opt/share/logrotate/examples/afpd), and unmarked lines are in both files.
[quote]
--- /opt/etc/logrotate.d/afpd
+++ /opt/share/logrotate/examples/afpd
@@ -1,6 +1,4 @@
 /opt/var/log/afpd.log {
-minsize 1024k
-   daily
    rotate 2
    postrotate
        /usr/bin/killall -HUP syslog-ng
I loose minsize 1024k for the default 4MB max? and no daily rotate, only after the maxsize of 4MB?
Not quite. Minsize means it has to be at least that big to rotate, regardless of the period specified. So as it was before, the file had to be at least 1M, AND it had to have been at least a day since the log was last rotated. In other words, both conditions had to be satisfied to rotate the log. Yes, on our routers, we only run logrotate once a day at 00:05, but on other systems, logrotate might run every hour, or even more often.

In /opt/etc/logrotate.conf, there is a global default of weekly log rotation, so removing daily just changes it to weekly. With maxsize set to 4M, and the period to weekly, the log will be rotated either when it reaches 4M, OR when it's been a week since the last log was rotated, whichever comes first.

The original file could run for months without rotating the log if the log file filled very slowly. The revised file will rotate every week, but can rotate sooner if the log file fills rapidly.

So:
  • Time period (daily, weekly, monthly, etc) sets the normal log rotation time.
  • Minsize requires that the log be at least some size before being rotated, even if it is past the time period. But if it passes minsize before the time period is up, the log won't be rotated until the time period passes.
  • Maxsize causes the log to be rotated when it reaches that size, even if the time period is not up yet. In this case if the time period is up before maxisize is reached, the log file will still be rotated.
All three of these can be used in a single logrotate directive. Between the default logrotate.conf and A00global, we now have a default time period of weekly, and a default maxsize of 4M. There is no default minsize, although the 'notifempty' directive in A00global prevents rotating empty logs, so you could say that minsize is "1 byte".

Rotating logs based only on size is possible, but I don't make use of it in the files I supply with scribe.
This line totally looses me. :confused:
Code:
@@ -1,6 +1,4 @@
Thanks
The first number pair (1,6) is the starting line number(1) and number of lines shown that are part of the first file(6). The second pair is the starting line number(1) and the number of lines shown that are part of the second file(4). The pairs are preceded by - and + as a reminder of which file the pair refers to. Obviously, this isn't important in a files as short as these, but if I'm looking a diff between, say, one version of scribe and another, and I see a difference I want to investigate, the first number of each pair tells me where to find that section in each file.
 
Back making trouble.:D

I have in the past in this thread described my method (here: scribe - syslog-ng and logrotate installer) of pouring the startup log messages back into syslog-ng to be processed and put into messages. That adds the host tag. I still have the May 5 time stamp for those messages that go to tmp/syslog.log before the time sync takes place.

I realized I could change the messages destination in syslog-ng.conf like so:
Code:
destination messages {
    file("/opt/var/log/messages" template("${R_DATE} ${HOST} ${MSGHDR}${MESSAGE}\n") );
};
This changes the timestamp of the log statements going to the messages file from the system date to the date that syslog-ng received the message. As a result, all the log statements in messages are in order starting with the time I rebooted the router, starting with all the startup messages with a time stamp of when syslog-ng started.

I also added
Code:
rm -f /tmp/mnt/USBNAME/entware/var/log/messages
to my post-mount script, with the result that my messages file now starts in order from the last time I rebooted the router. This means, though, that I can't recover error messages if something went wrong and I had to reboot.
 
Last edited:
Did somebody already write a filter for NextDNS?
This filter should work, right?
Code:
# log all NextDNS logs to nextdns.log

destination d_nextdns {
    file("/opt/var/log/nextdns.log");
};

filter f_nextdns {
    program("nextdns");
};

log {
    source(src);
    filter(f_nextdns);
    destination(d_nextdns);
    flags(final);
};

#eof
 
I'm still getting heavy lag in the browser when viewing scribe logs. I saw comments saying reduce the log sizes. But I am not finding it clear how to go about that. Can someone explain what I need to do to set a default setting that won't lag the browser?
 
I'm still getting heavy lag in the browser when viewing scribe logs. I saw comments saying reduce the log sizes. But I am not finding it clear how to go about that. Can someone explain what I need to do to set a default setting that won't lag the browser?
From the scribe main menu, use "uf" to update the filters. One of the filters it should ask you about is /opt/etc/logrotate.d/A00global. Updating A00global will set the maximum log size to 4M. That should trim them down at the next logrotate run. If you've done this and are STILL having problems, post or PM me the results of
Code:
ls -lart /opt/var/log
Thanks.
 
If you've done this and are STILL having problems, post or PM me the results...

Yeah I think I have been running with the default 4MB. I was trying to lower it to 1MB to see if that helped but wasn't sure how to do it properly or if it would resolve the issue. Here is the ls output:

Code:
username@DSL-AC68U-0628:/tmp/home/root# ls -lart /opt/var/log
-rw-r-----    1 nobody   root        640839 Feb 12 05:18 dnsmasq.log1
-rw-r-----    1 nobody   root       3120859 Feb 12 05:20 dnsmasq.log2
drwxr-xr-x    2 username root          4096 Feb 12 06:43 .
drwxr-xr-x    7 username root          4096 Feb 12 17:44 ..
-rw-------    1 username root        382640 Feb 12 20:32 messages
-rw-r-----    1 nobody   root        473747 Feb 12 20:34 dnsmasq.log
-rw-------    1 username root          3722 Feb 12 20:35 logrotate.log
-rw-------    1 username root         48219 Feb 12 20:35 syslog-ng.log
-rw-------    1 username root      10353438 Feb 12 20:35 skynet-0.log
 
Yeah I think I have been running with the default 4MB. I was trying to lower it to 1MB to see if that helped but wasn't sure how to do it properly or if it would resolve the issue. Here is the ls output:

Code:
username@DSL-AC68U-0628:/tmp/home/root# ls -lart /opt/var/log
-rw-r-----    1 nobody   root        640839 Feb 12 05:18 dnsmasq.log1
-rw-r-----    1 nobody   root       3120859 Feb 12 05:20 dnsmasq.log2
drwxr-xr-x    2 username root          4096 Feb 12 06:43 .
drwxr-xr-x    7 username root          4096 Feb 12 17:44 ..
-rw-------    1 username root        382640 Feb 12 20:32 messages
-rw-r-----    1 nobody   root        473747 Feb 12 20:34 dnsmasq.log
-rw-------    1 username root          3722 Feb 12 20:35 logrotate.log
-rw-------    1 username root         48219 Feb 12 20:35 syslog-ng.log
-rw-------    1 username root      10353438 Feb 12 20:35 skynet-0.log
The skynet log is too big and probably isn't being collapsed hourly through skynet. I think therein lies your problem.
 
Yeah I think I have been running with the default 4MB. I was trying to lower it to 1MB to see if that helped but wasn't sure how to do it properly or if it would resolve the issue. Here is the ls output:

Code:
username@DSL-AC68U-0628:/tmp/home/root# ls -lart /opt/var/log
-rw-r-----    1 nobody   root        640839 Feb 12 05:18 dnsmasq.log1
-rw-r-----    1 nobody   root       3120859 Feb 12 05:20 dnsmasq.log2
drwxr-xr-x    2 username root          4096 Feb 12 06:43 .
drwxr-xr-x    7 username root          4096 Feb 12 17:44 ..
-rw-------    1 username root        382640 Feb 12 20:32 messages
-rw-r-----    1 nobody   root        473747 Feb 12 20:34 dnsmasq.log
-rw-------    1 username root          3722 Feb 12 20:35 logrotate.log
-rw-------    1 username root         48219 Feb 12 20:35 syslog-ng.log
-rw-------    1 username root      10353438 Feb 12 20:35 skynet-0.log
The only large one is skynet-0.log, and it isn't touched by logrotate anyways. Skynet should clear it hourly. If you're getting over 10 gigs in 35 minutes though, that's HUGE, so I suspect for some reason the Skynet scrub isn't happening. It's just past the hour now, how big is skynet-0.log now?
 
I just reset the logs on Skynet, and set the log file location using this command:
Code:
sh /jffs/scripts/firewall settings syslog /opt/var/log/skynet-0.log
Not sure if that will fix it, I don't see a command to set the log to collapse hourly in the Skynet command.
 
I just reset the logs on Skynet, and set the log file location using this command:
Code:
sh /jffs/scripts/firewall settings syslog /opt/var/log/skynet-0.log
Not sure if that will fix it, I don't see a command to set the log to collapse hourly in the Skynet command.
What's the output of
Code:
cru -l
(that's an 'el' not a 'one')
 
What's the output of
Code:
cru -l
(that's an 'el' not a 'one')

Code:
username@DSL-AC68U-0628:/tmp/home/root# cru l
00 2 * * Thu sh /opt/share/diversion/file/update-bl.div reset #Diversion_UpdateBL#
20 5 * * * sh /opt/share/diversion/file/rotate-logs.div #Diversion_RotateLogs#
20 17 * * * diversion count_ads count #Diversion_CountAds#
5 0 * * * /opt/sbin/logrotate /opt/etc/logrotate.conf >> /opt/tmp/logrotate.daily 2>&1 #logrotate#
25 5 * * * sh /jffs/scripts/firewall banmalware #Skynet_banmalware#
25 1 * * Mon sh /jffs/scripts/firewall update #Skynet_autoupdate#
0 * * * * sh /jffs/scripts/firewall save #Skynet_save#
40 */12 * * * sh /jffs/scripts/firewall debug genstats #Skynet_genstats#
*/5 * * * * /jffs/scripts/connmon generate daily #connmon_daily#
2 * * * * /jffs/scripts/connmon generate weekly #connmon_weekly#
3 */3 * * * /jffs/scripts/connmon generate monthly #connmon_monthly#
12,42 * * * * /jffs/scripts/spdmerlin generate #spdMerlin#
 
Code:
username@DSL-AC68U-0628:/tmp/home/root# cru l
00 2 * * Thu sh /opt/share/diversion/file/update-bl.div reset #Diversion_UpdateBL#
20 5 * * * sh /opt/share/diversion/file/rotate-logs.div #Diversion_RotateLogs#
20 17 * * * diversion count_ads count #Diversion_CountAds#
5 0 * * * /opt/sbin/logrotate /opt/etc/logrotate.conf >> /opt/tmp/logrotate.daily 2>&1 #logrotate#
25 5 * * * sh /jffs/scripts/firewall banmalware #Skynet_banmalware#
25 1 * * Mon sh /jffs/scripts/firewall update #Skynet_autoupdate#
0 * * * * sh /jffs/scripts/firewall save #Skynet_save#
40 */12 * * * sh /jffs/scripts/firewall debug genstats #Skynet_genstats#
*/5 * * * * /jffs/scripts/connmon generate daily #connmon_daily#
2 * * * * /jffs/scripts/connmon generate weekly #connmon_weekly#
3 */3 * * * /jffs/scripts/connmon generate monthly #connmon_monthly#
12,42 * * * * /jffs/scripts/spdmerlin generate #spdMerlin#
The firewall save job should scrub the log, so I wonder why it isn't running. On the other hand I don't have that genstats entry.
 
The firewall save job should scrub the log, so I wonder why it isn't running. On the other hand I don't have that genstats entry.
That is the new webgui stats generation to get a nice graphs page under the Firewall tab.
https://github.com/Adamm00/IPSet_ASUS
Code:
( sh /jffs/scripts/firewall settings webui enable|disable ) Enable/Disable WebUI
 
Code:
username@DSL-AC68U-0628:/tmp/home/root# cru l
00 2 * * Thu sh /opt/share/diversion/file/update-bl.div reset #Diversion_UpdateBL#
20 5 * * * sh /opt/share/diversion/file/rotate-logs.div #Diversion_RotateLogs#
20 17 * * * diversion count_ads count #Diversion_CountAds#
5 0 * * * /opt/sbin/logrotate /opt/etc/logrotate.conf >> /opt/tmp/logrotate.daily 2>&1 #logrotate#
25 5 * * * sh /jffs/scripts/firewall banmalware #Skynet_banmalware#
25 1 * * Mon sh /jffs/scripts/firewall update #Skynet_autoupdate#
0 * * * * sh /jffs/scripts/firewall save #Skynet_save#
40 */12 * * * sh /jffs/scripts/firewall debug genstats #Skynet_genstats#
*/5 * * * * /jffs/scripts/connmon generate daily #connmon_daily#
2 * * * * /jffs/scripts/connmon generate weekly #connmon_weekly#
3 */3 * * * /jffs/scripts/connmon generate monthly #connmon_monthly#
12,42 * * * * /jffs/scripts/spdmerlin generate #spdMerlin#
The firewall save job should scrub the log, so I wonder why it isn't running. On the other hand I don't have that genstats entry.
Yeah, at this point I think it's either (a) you have an issue with your Skynet installation, or (b) you're really getting ~20 gigs per hour of Skynet logs which is massive, and I'd think your internet would barely be functional at that rate. Judging from what I have now, I get about 20 megs/hour of Skynet logs - 3 orders of magnitude less!
 
I keep getting this error in syslog.ng-log:

Code:
Feb 13 08:47:01 AC3100 syslog-ng[10549]: The current log file has a mismatching size/inode information, restarting from the beginning; state='affile_sd_curpos(/var/lib/logrotate.status)', stored_inode='7925593', cur_file_inode='7937689', stored_size='207', cur_file_size='207', raw_stream_pos='0'

Not sure how to fix it, or if its even a scribe issue, but I see it in uiScribe in the WebUI which lately has been locking up and becoming non-responsive until I restart scribe.

I've tried two complete uninstalls, reboots, etc.

RT-AC3100 (armv7l) FW-384.15
 

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top