Scribe scribe - syslog-ng and logrotate installer

[cmkelley]

I tried all ways (updated example files, updated script, and the script and example files), but no change in /opt/var/log.

I'm sorry, I don't understand what you're saying here. Is nothing logging?

 

[L&LD]

I'm sorry, I don't understand what you're saying here. Is nothing logging?

Sorry, yes, there is logging happening, but it is also repeated in the router's GUI too (like, messages, for example).

What I'm trying to get is the logging Butterfly Bones indicates, like Diversion, etc.

There are 9 files in /opt/var/log/, dnsmasq.log also has .log1 and .log2. Hope this is clear now.

 

[Butterfly Bones]

@L&LD here are the filter files that I have that are not included in scribe. Let me know it there is something else. (ignore the ---------- dividing lines only) Cut and paste each one into its own file, use the default ones from scribe as an example if you need. I think some of these are already in a new install, if so, no need to replace what is already there.

# put ChkWAN messages into /opt/var/log/chkwan.log
destination d_chkwan {
    file("/opt/var/log/chkwan.log");
};
filter f_chkwan {
    program("ChkWAN.sh");
};
log {
    source(src);
    filter(f_chkwan);
    destination(d_chkwan);
    flags(final);
};
#eof
--------------------------------------------------------
# log ethernet change to /opt/var/log/ethernet.log only
destination d_ethernet {
    file("/opt/var/log/ethernet.log");
};
filter f_ethernet {
    message("eth1") or
    message("eth2") or
    message("eth3") or
    message("eth4") or
    message("eth5") or
    message("eth6") or
    message("eth7") or
    message("eth8") or
    message("br0:");
};
filter f_kernel  { program("kernel") ; };
log {
    source(src);
    filter(f_ethernet);
    filter(f_kernel);
    destination(d_ethernet);
    flags(final);
};
#eof
----------------------------------------------------
# log all divstats logs to /opt/var/log/divstats.log and stop processing divstats logs
destination d_divstats {
   file("/opt/var/log/divstats.log");
};
filter f_divstats {
   program("uiDivStats");
};
log {
   source(src);
   filter(f_divstats);
   destination(d_divstats);
   flags(final);
};
#eof
------------------------------------------------------
# log all rc_service server and client logs into one file - /opt/var/log/rc_service.log and stop processing rc_service logs
destination d_rc_service {
    file("/opt/var/log/rc_service.log");
};
filter f_rc_service {
    program("rc_service") or
    program("custom_script") or
    program("stubby") or
    program("wan") or
    program("nat") or
    program("ntpd") or
    program("inadyn") or
    program("Diversion") or
    program("custom_config") or
    program("start_ddns") or
    program("dhcp_client") or
    program("BWDPI");
};
log {
    source(src);
    filter(f_rc_service);
    destination(d_rc_service);
    flags(final);
};
#eof
----------------------------------------------------
# put VPN_Failover VPN Client Monitor: messages into /opt/var/log/vpnfailover.log
destination d_vpnfailover {
    file("/opt/var/log/vpnfailover.log");
};
filter f_vpn_failover {
    program("VPN_Failover.sh");
};
log {
    source(src);
    filter(f_vpn_failover);
    destination(d_vpnfailover);
    flags(final);
};
#eof

 

[cmkelley]

Sorry, yes, there is logging happening, but it is also repeated in the router's GUI too (like, messages, for example).

What I'm trying to get is the logging Butterfly Bones indicates, like Diversion, etc.

There are 9 files in /opt/var/log/, dnsmasq.log also has .log1 and .log2. Hope this is clear now.

Right, messages is symlinked to /tmp/syslog.log explicitly so that it does show up in the router GUI. But that should be the only one repeated in the GUI. I'm not sure Diversion creates any logs. Skynet does, and those should be routed to /opt/var/log/skynet-0.log and skynet pointed to that file.

 

[Butterfly Bones]

Right, messages is symlinked to /tmp/syslog.log explicitly so that it does show up in the router GUI. But that should be the only one repeated in the GUI. I'm not sure Diversion creates any logs. Skynet does, and those should be routed to /opt/var/log/skynet-0.log and skynet pointed to that file.

It does, but not many, I clean all of those up with my rc_service filter (see above).

May  5 05:20:03 RT-AC86U-4608 Diversion[2480]: rotated dnsmasq log files, from /opt/share/diversion/file/rotate-logs.div
May  5 07:21:01 RT-AC86U-4608 Diversion[18573]: restarted Dnsmasq to apply settings, from /jffs/scripts/dnsmasq.postconf
May  5 07:21:02 RT-AC86U-4608 Diversion[18695]: created br0:pixelserv-tls 192.168.1.2, from /opt/etc/init.d/S80pixelserv-tls
May  5 09:34:28 RT-AC86U-4608 Diversion[6852]: added 'codeload.github.com' to whitelist (user action), from /opt/bin/diversion
May  5 10:33:43 RT-AC86U-4608 Diversion[32128]: restarted Dnsmasq to apply settings, from /jffs/scripts/dnsmasq.postconf
May  5 10:33:58 RT-AC86U-4608 Diversion[32448]: restarted Dnsmasq to apply settings, from /jffs/scripts/dnsmasq.postconf
May  5 10:35:06 RT-AC86U-4608 Diversion[2156]: updated Standard+ blocking list from 6 hosts files, 508211 domains are now blocked, from /opt/share/diversion/file/update-bf.div
May  5 12:56:29 RT-AC86U-4608 Diversion[22888]: added 'raw.githubusercontent.com' to whitelist (user action), from /opt/bin/diversion

 

[Jack Yaz]

Right, messages is symlinked to /tmp/syslog.log explicitly so that it does show up in the router GUI. But that should be the only one repeated in the GUI. I'm not sure Diversion creates any logs. Skynet does, and those should be routed to /opt/var/log/skynet-0.log and skynet pointed to that file.

It logs all dnsmasq queries, if you configure it to do so

 

[cmkelley]

@L&LD here are the filter files that I have that are not included in scribe. Let me know it there is something else. (ignore the ---------- dividing lines only) Cut and paste each one into its own file, use the default ones from scribe as an example if you need. I think some of these are already in a new install, if so, no need to replace what is already there.

# put ChkWAN messages into /opt/var/log/chkwan.log
destination d_chkwan {
    file("/opt/var/log/chkwan.log");
};
filter f_chkwan {
    program("ChkWAN.sh");
};
log {
    source(src);
    filter(f_chkwan);
    destination(d_chkwan);
    flags(final);
};
#eof
--------------------------------------------------------
# log ethernet change to /opt/var/log/ethernet.log only
destination d_ethernet {
    file("/opt/var/log/ethernet.log");
};
filter f_ethernet {
    message("eth1") or
    message("eth2") or
    message("eth3") or
    message("eth4") or
    message("eth5") or
    message("eth6") or
    message("eth7") or
    message("eth8") or
    message("br0:");
};
filter f_kernel  { program("kernel") ; };
log {
    source(src);
    filter(f_ethernet);
    filter(f_kernel);
    destination(d_ethernet);
    flags(final);
};
#eof
----------------------------------------------------
# log all divstats logs to /opt/var/log/divstats.log and stop processing divstats logs
destination d_divstats {
   file("/opt/var/log/divstats.log");
};
filter f_divstats {
   program("uiDivStats");
};
log {
   source(src);
   filter(f_divstats);
   destination(d_divstats);
   flags(final);
};
#eof
------------------------------------------------------
# log all openvpn server and client logs into one file - /opt/var/log/openvpn.log and stop processing openvpn logs
destination d_openvpn {
    file("/opt/var/log/openvpn.log");
};
filter f_openvpn {
    program("ovpn-server1") or
    program("ovpn-server2") or
    program("ovpn-client1") or
    program("ovpn-client2") or
    program("ovpn-client3") or
    program("ovpn-client4") or
    program("ovpn-client5") or
    program("openvpn-routing");
};
log {
    source(src);
    filter(f_openvpn);
    destination(d_openvpn);
    flags(final);
};
#eof
----------------------------------------------------
# log all pixelserv-tls logs to /opt/var/log/pixelserv.log and stop processing pixelserv-tls logs
destination d_pixelserv {
    file("/opt/var/log/pixelserv.log");
};
filter f_pixelserv {
    program("pixelserv-tls");
};
log {
    source(src);
    filter(f_pixelserv);
    destination(d_pixelserv);
    flags(final);
};
#eof
-----------------------------------------------
# log all rc_service server and client logs into one file - /opt/var/log/rc_service.log and stop processing rc_service logs
destination d_rc_service {
    file("/opt/var/log/rc_service.log");
};
filter f_rc_service {
    program("rc_service") or
    program("custom_script") or
    program("stubby") or
    program("wan") or
    program("nat") or
    program("ntpd") or
    program("inadyn") or
    program("Diversion") or
    program("custom_config") or
    program("start_ddns") or
    program("dhcp_client") or
    program("BWDPI");
};
log {
    source(src);
    filter(f_rc_service);
    destination(d_rc_service);
    flags(final);
};
#eof
----------------------------------------------------
# put VPN_Failover VPN Client Monitor: messages into /opt/var/log/vpnfailover.log
destination d_vpnfailover {
    file("/opt/var/log/vpnfailover.log");
};
filter f_vpn_failover {
    program("VPN_Failover.sh");
};
log {
    source(src);
    filter(f_vpn_failover);
    destination(d_vpnfailover);
    flags(final);
};
#eof

openvpn and pixelserv filters are included with 0.10_2, and I just now added your ethernet to the repository.

 

[cmkelley]

It logs all dnsmasq queries, if you configure it to do so

Oh. Missed that. Thanks.

 

[L&LD]

I am really messing up this script, sorry.

These are the steps I do:

1. Uninstall fully using the scribe uninstall YES menu sequence. Router reboots and I wait 10 minutes for it to become idle.
2. Install using the curl command from post 1.
3. At this point, everything is working. If I use scribe status, all present and alive v0.10_2 (master).
4. Now, I copy the example files from the /opt/share/syslog-ng/examples/ directory to the /opt/etc/syslog-ng.d/ directory.
5. I do the same for the files located in the /opt/share/logrotate/ directory to the /opt/etc/logrotate.d/ directory.

At this point, in the /opt/var/log/ directory, there are no new log files created (ever, even after a reboot and 24 hours pass).

When I check 'scribe status' it says that the daemon is dead.

If I skip steps 4 and 5 and try adding the Butterfly Bones scripts for sc_service and uidivStats, same failure. I make sure to have then be only 'rw----------' files (sorry, didn't count the '-' dashes).

At this point, nothing I can do will bring scribe back to life. I have to go to step 1. and fully uninstall it and reinstall it.

What am I missing?

How can I get all the log files I want to show up in the /opt/var/log/ directory?

 

[Butterfly Bones]

I am really messing up this script, sorry.

These are the steps I do:

1. Uninstall fully using the scribe uninstall YES menu sequence. Router reboots and I wait 10 minutes for it to become idle.
2. Install using the curl command from post 1.
3. At this point, everything is working. If I use scribe status, all present and alive v0.10_2 (master).
4. Now, I copy the example files from the /opt/share/syslog-ng/examples/ directory to the /opt/etc/syslog-ng.d/ directory.
5. I do the same for the files located in the /opt/share/logrotate/ directory to the /opt/etc/logrotate.d/ directory.

At this point, in the /opt/var/log/ directory, there are no new log files created (ever, even after a reboot and 24 hours pass).

When I check 'scribe status' it says that the daemon is dead.

If I skip steps 4 and 5 and try adding the Butterfly Bones scripts for sc_service and uidivStats, same failure. I make sure to have then be only 'rw----------' files (sorry, didn't count the '-' dashes).

At this point, nothing I can do will bring scribe back to life. I have to go to step 1. and fully uninstall it and reinstall it.

What am I missing?

How can I get all the log files I want to show up in the /opt/var/log/ directory?

Everything sounds correct. The files not being correct chmod (600) tripped me up, but you have that covered. Run the command that @Martineau posted, it will show you where you have an issue.

syslog-ng -Fevd

 

[L&LD]

Everything sounds correct. The files not being correct chmod (600) tripped me up, but you have that covered. Run the command that @Martineau posted, it will show you where you have an issue.

syslog-ng -Fevd

Does that command never end?

 

[Butterfly Bones]

Does that command never end?

If it does not show something with a caret, yes it can run for a long time. Then end of mine loads the connection to Loggly and then shows everything being sent there, I have to do a Ctrl-C to stop it. If you don't send to Loggly or another server it should end, but it is very long, it checks everything about syslog-ng and logrotate.

 

[L&LD]

If it does not show something with a caret, yes it can run for a long time. Then end of mine loads the connection to Loggly and then shows everything being sent there, I have to do a Ctrl-C to stop it. If you don't send to Loggly or another server it should end, but it is very long, it checks everything about syslog-ng and logrotate.

Thank you. Okay, it is still going, but right now I had forgotten I had stopped at my step 3.

I will let it finish and then do the remaining steps and see where I kill it.

 

[cmkelley]

I am really messing up this script, sorry.

These are the steps I do:

1. Uninstall fully using the scribe uninstall YES menu sequence. Router reboots and I wait 10 minutes for it to become idle.
2. Install using the curl command from post 1.
3. At this point, everything is working. If I use scribe status, all present and alive v0.10_2 (master).
4. Now, I copy the example files from the /opt/share/syslog-ng/examples/ directory to the /opt/etc/syslog-ng.d/ directory.
5. I do the same for the files located in the /opt/share/logrotate/ directory to the /opt/etc/logrotate.d/ directory.

At this point, in the /opt/var/log/ directory, there are no new log files created (ever, even after a reboot and 24 hours pass).

When I check 'scribe status' it says that the daemon is dead.

If I skip steps 4 and 5 and try adding the Butterfly Bones scripts for sc_service and uidivStats, same failure. I make sure to have then be only 'rw----------' files (sorry, didn't count the '-' dashes).

At this point, nothing I can do will bring scribe back to life. I have to go to step 1. and fully uninstall it and reinstall it.

What am I missing?

How can I get all the log files I want to show up in the /opt/var/log/ directory?

Couple things ...

Please do not blindly copy everything from the examples directories to their respective .d directories. If you're just copying everything from /opt/share/syslog-ng/examples/ to /opt/etc/syslog-ng.d/, you've also copied syslog-ng.conf-opkg and syslog-ng.conf-scribe, which should NOT be copied into that directory. Having those in there would definitely prevent syslog-ng from starting. Maybe I should put READMEs in those directories explaining what each file is. Also, the expandlog filter is not designed to be used continuously, it's used for troubleshooting and will create very large log files. I don't have a logrotate file for expandlog because it is not designed to be used continuously.

I assume that after step 5 you're executing "scribe restart"? Files in the /opt/etc/syslog-ng.d/ folder are not added to configuration until syslog-ng is restarted. Rebooting of course has the same effect.

There's now files in the github repository for uiDivStats (included in diversion) as well as ethernet.

 

[L&LD]

Couple things ...

Please do not blindly copy everything from the examples directories to their respective .d directories. If you're just copying everything from /opt/share/syslog-ng/examples/ to /opt/etc/syslog-ng.d/, you've also copied syslog-ng.conf-opkg and syslog-ng.conf-scribe, which should NOT be copied into that directory. Having those in there would definitely prevent syslog-ng from starting. Maybe I should put READMEs in those directories explaining what each file is. Also, the expandlog filter is not designed to be used continuously, it's used for troubleshooting and will create very large log files. I don't have a logrotate file for expandlog because it is not designed to be used continuously.

I assume that after step 5 you're executing "scribe restart"? Files in the /opt/etc/syslog-ng.d/ folder are not added to configuration until syslog-ng is restarted. Rebooting of course has the same effect.

There's now files in the github repository for uiDivStats (included in diversion) as well as ethernet.

Yes, I was using 'scribe restart' afterward and I had tried rebooting multiple times.

But I did just blindly copy everything from the example directories!

The 'syslog-ng -Fevd' command is still going (about an hour and 15 minutes so far), I think I'll kill it now.

Thank you! You may have saved me from running -Fevd needlessly.

 

[L&LD]

Okay, as a test I only copied the new 'diversion' file to the /opt/etc/syslog-ng.d/ directory.

No log file created in the /opt/var/log/ directory. I force updated Diversion, but still, nothing shows up.

'scribe status' at least shows everything still working.

 

[Butterfly Bones]

Okay, as a test I only copied the new 'diversion' file to the /opt/etc/syslog-ng.d/ directory.

No log file created in the /opt/var/log/ directory. I force updated Diversion, but still, nothing shows up.

'scribe status' at least shows everything still working.

Try refreshing uiDivStats, it gets scraped by the same diversion filter file.

Oh, yeah, and welcome to the deep end of the pool!

 

[L&LD]

I too got impatient. I copied a few more over to /opt/etc/syslog-ng.d/.

the above is what should be installed (as I understand it).

The above is what is actually available.

I think I'll give up on this for tonight.

@cmkelley, sorry for mangling your script so badly.

It does seem to work for me when I just leave the install defaults though.


Edit: seems like Butterfly Bones has me on the right path. Please see my post below.

 

[L&LD]

Try refreshing uiDivStats, it gets scraped by the same diversion filter file.

Oh, yeah, and welcome to the deep end of the pool!

Thank you, the waters are warm!

I'll have to go and edit my post above! Diversion log now shows.

I also noticed that it takes 30 seconds to generate those statistics now.

How do I get the rest to show?

Like Ethernet, firewall, OpenVPN, and pixelserv?

 

[Butterfly Bones]

Thank you, the waters are warm!

I'll have to go and edit my post above! Diversion log now shows.

I also noticed that it takes 30 seconds to generate those statistics now.

How do I get the rest to show?

Like Ethernet, firewall, OpenVPN, and pixelserv?

Depends on what goes on in your router. My ethernet does not collect much, mostly I have a smart TV with smartcast and a chromecast, which like to restart overnight, who knows why. Since I don't think it important enough to watch, I scrape them. Here is an example, only see 6-8 per day.

May  5 02:54:16 RT-AC86U-4608 kernel: eth2 (Ext switch port: 1) (Logical Port: 9) Link DOWN.
May  5 02:54:16 RT-AC86U-4608 kernel: br0: port 2(eth2) entered disabled state
May  5 02:54:31 RT-AC86U-4608 kernel: eth2 (Ext switch port: 1) (Logical Port: 9) Link UP 100 mbps full duplex
May  5 02:54:31 RT-AC86U-4608 kernel: br0: port 2(eth2) entered listening state
May  5 02:54:31 RT-AC86U-4608 kernel: br0: port 2(eth2) entered listening state

I do not use firewall since it is meant for those who do NOT use skynet, so it is unneeded if you have Skynet installed and the skynet filter file.

The openvpn log is minimal, and only shows when I reboot, or if my connection fails and then the VPNFailover.sh script from Martineau restarts it. I go days with no entries there, and mostly at reboot. You can increase the openvpn log verbosity in the VPN client settings, also did you use the trick that cmkelley posted earlier in this thread to add this line to the custom config portion of the VPN?

log-append /opt/var/log/openvpn.log

Pixelserv does not log unless you have added addition switches for more verbose output. I use Diversion to add the -l4 switch and then pixelserv logs every single connection from every single device, wired or wireless. It gets a bazillion messages a day with the log level 4 switch, very chatty, oftimes with a bunch of binary blabber.

May  2 11:01:32 RT-AC86U-4608 pixelserv-tls[14965]: handshake failed: shutdown after ServerHello. client 192.168.1.14:64150 server e.crashlytics.com
May  2 11:01:32 RT-AC86U-4608 pixelserv-tls[14965]: handshake failed: shutdown after ServerHello. client 192.168.1.14:64151 server e.crashlytics.com
May  2 11:02:45 RT-AC86U-4608 pixelserv-tls[14965]: 192.168.1.1 rgom10-en.url.trendmicro.com:80 GET /T/108/QyTZh5Yt5wz_Y633P0ZK7150QjriATlOBZxijFne_VB48m1Zx90MEWhHL375PP41YEJdJCmdmsTzJtD05bTzw6yDVqthChXQlYro-Gs6JBI= HTTP/1.1 tls_none