What's new

scribe - syslog-ng and logrotate installer

elorimer

Very Senior Member
On the sda filter, I'm guessing you have another file that also defines a kernel filter. Syslog-ng won't start if a definition occurs twice, unless you specifically allow duplicates. So it wasn't scribe or uiScribe that was choking on the sda file, it was syslog-ng itself. But that is why your logs weren't updating; syslog-ng didn't start. I don't think you need that filter in your sda file in the first place, since the messages seem so specific. So try it with just the message filter. If you need it, delete it from all of your log files and put it in its own file once. Don't put it in syslog-ng.conf, cuz that will get overwritten.

A general debugging tool is to run "syslog-ng -Fevd" from a terminal prompt. If syslog-ng isn't starting, it will show you exactly why.

A couple of minor things. You don't need to create the log files themselves; syslog-ng will create them if they don't exist. You don't need to reboot to start syslog-ng--you can do it directly from scribe or a command line (but scribe is much easier).

The five hour thing looks like a time zone thing. Not ringing a bell immediately.

For a log to be listed in uiScribe, you have to run uiScribe again to reset the logs.

Last, if you just want to delete a log message, you don't need to send it to its own file. Just leave out the destination specification. If no destination is specified, syslog-ng deletes it.
 
Last edited:

Megahurtz

New Around Here
Thanks for the tip about restarting syslog from scribe. I'm still having trouble getting the message section correct. I'm following the example from the crash filter that has multiple messages using "or" to add to the list, but syslog keeps throwing a syntax error at the closing curly bracket for that section. I tried changing the text string for the messages thinking that may have been part of the problem. Didn't make any difference.

sda:
destination d_sda {
file("/opt/var/log/sda.log");
};

filter f_mymsg {
message("Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE") or
message("Sense Key : Illegal Request [current]") or
message("Add. Sense: Invalid command operation code") or
message("CDB: Write same(16)") or
message("end_request: I/O error, dev sda") or
message("xhci_hcd 0000:00:0c.0: WARN: Stalled endpoint");
};

log {
source(src);
filter(f_mymsg);
destination(d_sda);
flags(final);
};
Here is what I see when trying to restart syslog:
Starting syslog-ng... done.

Restarting uiScribe ...[2020-09-14T03:27:49.690485] WARNING: Duplicate configuration objects (sources, destinations, ...) are not allowed by default starting with syslog-ng 3.3, add "@define allow-config-dups 1" to your configuration to re-enable;

Error parsing config, duplicate filter definition in /opt/etc/syslog-ng.d/sda:12:1-12:2:
7 message("Sense Key : Illegal Request [current]") or
8 message("Add. Sense: Invalid command operation code") or
9 message("CDB: Write same(16)") or
10 message("end_request: I/O error, dev sda") or
11 message("xhci_hcd 0000:00:0c.0: WARN: Stalled endpoint");
12----> };
12----> ^
13
14 log {
15 source(src);
16 filter(f_mymsg);
17 destination(d_sda);

Included from /opt/etc/syslog-ng.conf:12:1-12:1:
7 # Release notes: https://github.com/syslog-ng/syslog-ng/releases
8
9 @version: 3.27
10 #@include "scl.conf" # uncomment this line to for additional functionality, see syslog-ng documentation
11 @include "/opt/etc/syslog-ng.d/" # Put any customization files in this directory
12---->
12----> ^
13 options {
14 chain_hostnames(no); # Enable or disable the chained hostname format.
15 create_dirs(yes);
16 keep_hostname(yes); # Enable or disable hostname rewriting.
17 log_fifo_size(256); # The number of messages that the output queue can store.

syslog-ng documentation: https://www.syslog-ng.com/technical-documents/list/syslog-ng-open-source-edition
contact: https://lists.balabit.hu/mailman/listinfo/syslog-ng
done.
checking syslog-ng daemon ... dead.
the system logger (syslogd) ... is not running!
 
Last edited:

elorimer

Very Senior Member
In one of your other configuration files you have already defined a filter called f_mymsg. It will be one alphabetically earlier than sda. You have it defined in the afpd file you posted earlier.

As a quick fix, change the name of the filter to f_sda in the definition and the log statement.
 
Last edited:

Megahurtz

New Around Here
You were right! I had another message filter with the same name in a different file. I've renamed it and now all files process correctly. I think whatever was going on with my kernel time being off sorted itself out too. However, I'm still having a couple of those lines from the logs slipping through my filter.

These two entries:
kernel: sd 0:0:0:0: [sda] Sense Key : Illegal Request [current]
kernel: sd 0:0:0:0: [sda] CDB: Write same(16): 93 08 00 00 00 00 95 ae 9c d8 00 00 00 78 0
When I tried just putting "sd 0:0:0:0: [sda]" into a message filter, it didn't catch any of those messages. I broke them down into their individual messages, which captured most of them, but the two above still aren't being filtered to sda.log. For the second entry, the hex value is changing all the time. Here is my message section:

filter f_sdaspam{
message("Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE") or
message("Sense Key : Illegal Request [current]") or
message("Add. Sense: Invalid command operation code") or
message("CDB: Write same(16)") or
message("end_request: I/O error") or
message("xhci_hcd 0000:00:0c.0");
};
How can I add those two to my sda log?
 

elorimer

Very Senior Member
Hmm. The messages filter in syslog-ng is pretty powerful but slow, and the more specific you make it, the slower I suspect it is. These routers don't generate enough messages to be a problem, I suspect. But you might try to strip out some of the specifics to skinny down the filter to the minimum. Like, does anything else in the messages.log file use the phrase "CDB"? or "Sense"? Maybe that's all you need.

But I'm wondering if you should explore why these messages are being generated. I say this because I don't recall any other posts in this forum like this. I'm not sure they are "spam"; This drive is generating so many error messages that you want to make them stop. But maybe you have a fire alarm going on that you should investigate first.
 

Megahurtz

New Around Here
I haven't researched it extensively, but I recall reading in a thread somewhere that other people said it was a common occurrence with the 4TB WD portable hard drives.

Regarding slimming down my message filter, I initially tried with just "sd 0:0:0:0: [sda]", but that didn't catch any of the four messages I repeatedly have that start with it. Is there a reason all four of those messages would slip past that message filter?
 

elorimer

Very Senior Member
I'll think on the filtering some. There must be something, maybe the colons, maybe the brackets, that is throwing off the match.

On the idea of a 4tb portable hard drive, this is pushing a router a bit, and can I say, good on you for scoring such a thing. I had for some time a 4tb seagate desktop drive hooked up without issue, and I've had some 1 and 2tb portables, and I have a 4tb 2.5" inch drive in my tivo, with some shoehorning. But a 4tb portable external is a rare goose to start with, more so if driven off the router. Why I'm thinking these error messages are something to attend to. We might start with asking what router you are using, off which port.

Not being critical here. My first thought was that the router is on 24/7, so there would be a power savings if I hung a big disk off the router. When I got to 4Tb, I started to do some thinking. So now I am what "datahoarders" call a "babyhoarder", and I will not rely on a biggly amount of data hung off the router, ever again.
 

cmkelley

Very Senior Member
I haven't researched it extensively, but I recall reading in a thread somewhere that other people said it was a common occurrence with the 4TB WD portable hard drives.

Regarding slimming down my message filter, I initially tried with just "sd 0:0:0:0: [sda]", but that didn't catch any of the four messages I repeatedly have that start with it. Is there a reason all four of those messages would slip past that message filter?
So, the way I would approach it is to copy expandlog from /opt/share/syslog-ng/examples to /opt/etc/syslog-ng.d and restart syslog-ng from scribe. This will create a very large log pretty quickly, but with 4 TB I'm guessing you can stand to create a big file. You can then search the resulting expanded-syslog.log file for, say "0:0:0:0:" and see if that gives you hint why it's not working. Perhaps the message isn't quite what it seems? Don't forget to delete the filter file and restart syslog-ng again after you find the culprit. Also, don't go into the UIScribe GUI page with the expandlog filter running. The UI can't handle such large files (this is not UIScribe's fault, it's probably to do with the webserver used in the firmware).

Also RMerlin had this to say previously about running 4TB drives off the router (https://www.snbforums.com/threads/w...d-drive-that-ac68u-can-run.35763/#post-291173):
This is a bad idea. First, the router doesn't even have enough RAM to properly cache the metadata from such a large volume, or run any filesystem check in case of a problem.
True, that message is from 4 years ago, and I don't know what router you're using, but unless you're running a really recent router, I'll bet the same applies today.

My (unasked for) $.02 - Your messages may be indicative that the router or hard drive are not happy with the set up. A "common occurrence" doesn't mean it's harmless. I'd think a 4GB Raspberry Pi 4 would make a MUCH better NAS than the router, and the RPi will have nice GUI to make it easier. There are plenty of articles on how to do this on the web.
 

cmkelley

Very Senior Member
Im stuck in a loop. Getting the following

Code:
Updated list of available packages in /opt/var/opkg-lists/entware

Installing syslog-ng (3.27.1-1) to root...
Downloading http://bin.entware.net/armv7sf-k2.6/syslog-ng_3.27.1-1_armv7-2.6.ipk
Configuring syslog-ng.
syslog-ng: error while loading shared libraries: /opt/lib/librt.so.1: invalid ELF header

syslog-ng version 3.19 or higher required!
Please update your Entware packages and run scribe install again.

Removing package syslog-ng from root...
Any suggestions what to do? Updating Entware says all upto date.
syslog-ng is now at 3.27.1-2 in entware, maybe try installing syslog-ng again?
 

Megahurtz

New Around Here
So, the way I would approach it is to copy expandlog from /opt/share/syslog-ng/examples to /opt/etc/syslog-ng.d and restart syslog-ng from scribe. This will create a very large log pretty quickly, but with 4 TB I'm guessing you can stand to create a big file. You can then search the resulting expanded-syslog.log file for, say "0:0:0:0:" and see if that gives you hint why it's not working. Perhaps the message isn't quite what it seems? Don't forget to delete the filter file and restart syslog-ng again after you find the culprit. Also, don't go into the UIScribe GUI page with the expandlog filter running. The UI can't handle such large files (this is not UIScribe's fault, it's probably to do with the webserver used in the firmware).

Also RMerlin had this to say previously about running 4TB drives off the router (https://www.snbforums.com/threads/w...d-drive-that-ac68u-can-run.35763/#post-291173):


True, that message is from 4 years ago, and I don't know what router you're using, but unless you're running a really recent router, I'll bet the same applies today.

My (unasked for) $.02 - Your messages may be indicative that the router or hard drive are not happy with the set up. A "common occurrence" doesn't mean it's harmless. I'd think a 4GB Raspberry Pi 4 would make a MUCH better NAS than the router, and the RPi will have nice GUI to make it easier. There are plenty of articles on how to do this on the web.
I had an extra 4TB WD My Passport drive laying around and decided to throw it on the router when I installed Merlin. My primary purpose was to set up a 1TB partition to use as a secondary Time Machine backup location for my MacBook, but figured while I had it, I may as well use it to store some video files. If the whole thing craps itself, I'd be disappointed but not terribly upset. It was more just something to tinker with given hardware I already had at hand.

I did figure out how to get all of those sda messages filtered out. This is finally what worked for me:
filter f_sdaspam{
message("Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE") or
message("Sense Key : Illegal Request") or
message("Add. Sense: Invalid command operation code") or
message("CDB: Write same") or
message("end_request: I/O error") or
message("xhci_hcd 0000:00:0c.0");
};

It isn't quite as specific as I wanted, but it doesn't appear to be inadvertently trapping anything else. All 6 messages are now going to their own log file, time machine messages are sent to their own log, and transmission is separated as well. I'm very happy with it now! The main sys log is very readable. Thanks for your help and suggestions, everybody :D
 

elorimer

Very Senior Member
syslog-ng is now at 3.27.1-2 in entware, maybe try installing syslog-ng again?
We were at 3.27-1-1. weren't we? I wonder what is new. I see github is up to 3.29-1
 

Butterfly Bones

Very Senior Member

giant46man46

New Around Here
Skynet isn't necessary.

Scribe is a script that handles configuring syslog-ng on the router to replace the native syslogd. From there syslog-ng can take you in a lot of different directions, including parsing out the single system log into separate logs. But it doesn't have anything to do with anything else other than logging.

UiScribe is a script that builds on scribe by displaying in the GUI the different logs that scribe creates.

Skynet is something entirely different. That is a firewall program that denies inbound and outbound traffic to addresses it thinks are nasty. It generates log messages each time it does that. If scribe/syslog-ng isn't active, those messages go to the system log. If scribe/syslog-ng is active, it will send those messages to its own log. Hourly it purges those messages from the log destination (whichever one) and logs a summary. There was a bit of interaction there, because the purge process had the unintended effect of stopping syslog-ng and restarting syslogd. That has long since been fixed, and that is the only interaction between scribe and skynet.

As to this:
you're going to have to finish that sentence.
I do apologize for not finishing my thought. The only log that works is the system messages. The rest of the logs do not work. They do not display anything on the screen. Can you please help me?
 

giant46man46

New Around Here
I'm not quite sure I understand your problem, but neither scribe nor Skynet rely on each other work. If you have, or are going to use Skynet, you should install Skynet first to ensure scribe properly handles Skynet's logs, but that's it.

I'm sorry, I can't understand the second part of your question. Is there a problem with the other logs?

Forget the part about Skynet. I realize they are two different things. The only reason I even mentioned the two things in the same sentence is because during the install of Scribe, I was asked if I wanted to install Skynet with scribe. Please forget the Skynet. My problem is the only log that works in Scribe is the System Messages log. The other logs are blank. I have attached a screenshot to this post just in case my problem is still unclear.
 

Attachments

elorimer

Very Senior Member
From the screenshot, it looks like nothing is working; the last system message is the built-in log daemons exiting, and there should be stuff after that.

Login from a terminal and run "syslog-ng -Fevd" to see why syslog-ng isn't starting.
 

giant46man46

New Around Here
From the screenshot, it looks like nothing is working; the last system message is the built-in log daemons exiting, and there should be stuff after that.

Login from a terminal and run "syslog-ng -Fevd" to see why syslog-ng isn't starting.
Thanks elorimer. I have attached a screenshot of the result of the syslog-ng -Fevd command. Please let me know what my next steps are. Thank you so much.
 

Attachments

elorimer

Very Senior Member
Thanks elorimer. I have attached a screenshot of the result of the syslog-ng -Fevd command. Please let me know what my next steps are. Thank you so much.
Can't read it unfortunately. But there should be a place where it indicates what character of what line of what file it is choking on.
 

giant46man46

New Around Here
Can't read it unfortunately. But there should be a place where it indicates what character of what line of what file it is choking on.
I just attached a text file containing the log. The name of the file is Log.txt. I didn't see any line item that it is choking on, but mine are the eyes of inexperience. Are you able to read my attachment?

Ken
 

Attachments

elorimer

Very Senior Member
Yes, that worked to read, but it doesn't seem to be loading.

This is new to me. It seems to be loading fine, but it ends wrong, because it should end with a message that says syslog-ng is starting.

So, what model router do you have, what version of Merlin do you have?

Also, what files, if any, do you see in /opt/var/log?

EDIT: So could others check their log files? The recent Entware update borked my unbound, but looking at my files I can see some odd things. (So [email protected] if you've picked up on something.)
 
Last edited:

giant46man46

New Around Here
Yes, that worked to read, but it doesn't seem to be loading.

This is new to me. It seems to be loading fine, but it ends wrong, because it should end with a message that says syslog-ng is starting.

So, what model router do you have, what version of Merlin do you have?

Also, what files, if any, do you see in /opt/var/log?
I only see one file and the name of that file is "messages". I have an ASUS AC1900, RT-AC68U, Merlin version 384.19
 

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top