Scribe scribe - syslog-ng and logrotate installer

[cmkelley]

unable to leave well enough alone, v0.10_1 is up now.

• while syslog-ng.conf may need customizing for your local setup, /opt/etc/init.d/rc.func.syslog-ng shouldn't, so now/opt/etc/init.d/rc.func.syslog-ng is now updated whenever scribe is updated; only /opt/etc/syslog-ng.conf remains to be copied if a reinstall is not forced

 

[cmkelley]

Just thought I'd drop in again to mention my setup with klogd still running (I also wrote my own similar service-start script loop to kill syslogd - actually it calls scribe's own rc.func-syslog-ng, albeit an older versoin without the killall klogd) and it hasn't missed a beat. Was thinking that if you were inclined to do so with scribe, it should allow both 2.6 kernal and the HND kernel to read the kernel (klogd) messages without issue as they are then just collecting from the same place.
Plus, at least for me, I feel this has the added advantage of not messing too much with an embedded system where unlike a typical linux distro, klogd is linked into many other parts of the firmware (eg the time service).

Either way, I appreciate all the work you've put in @cmkelley, thanks mate.

So, I made a conscious decision to only run syslog-ng, and kill both klogd and syslogd in an effort to minimize the number of services running. Given the increasing number of additional services people are running with the scripts, I think it's the right choice for the majority of users, particularly since using syslog-ng really only makes sense if you're running the additional services.

I can't recall if klogd had the problem with occasional garbled log entries from IPTables, but the latest changes I've made to syslog-ng.conf appear to eliminate those when using just syslog-ng.

 

[Goobi]

Trying to update to latest from 0.9.2 and get the following:

curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

scribe GitHub repository is unavailable! -- Aborting.

 

[cmkelley]

Trying to update to latest from 0.9.2 and get the following:

curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

scribe GitHub repository is unavailable! -- Aborting.

That error is not a scribe error, it's an error between your computer and GitHub. I double checked it worked on my test router so I think it's going to take someone with a lot more knowledge about curl and certificates to answer your issue.

 

[elorimer]

Looks like huge progress. I'm thousands of miles away from my router and too dependent on it to try this out yet.

A few comments: if I understand it correctly, kmsg is a read-once read-only socket that exists as a way to get kernel messages out. klogd scarfs them up and delivers them to the system log source. The skynet garbling I think was because syslog-ng was reading kmsg as a file source at the same time klogd was running, and they couldn't both be reading it at the same time. Your path to a solution is to kill klogd; @Cam's path is to leave it running but stop syslog-ng from reading that source and instead get kernel messages from the regular system source. I think we've shown adequately that either works.

The system() source seems to pick up the difference between the 2.6 kernel and the 4. kernel kernel handling just by seeing if /dev/kmsg exists, but I guess you've found a different problem with it with the HND implementation. In any case, using system() as a source means you have to take the path to kill klogd. Replicating it the way you do means a user can easily comment out a line or two and take @Cam's path.

 

[xilitol]

I had the same issue as Goobi.

Trying different DNS configurations (with or without DOT - the router is AC86U with 384.11_beta2, no stubby installed like for previous version), the error changed like this:

syslog-ng and logrotate installation
v0.9_3 (master) Coded by cynicastic


scribe installed version: v0.9_3
scribe GitHub version: v0.10_1


New version available!
Do you wish to upgrade [y|n] y

Do you want to update syslog-ng and logrotate example files? [y|n] y

fetching scribe from GitHub master branch ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 122 0 122 0 0 263 0 --:--:-- --:--:-- --:--:-- 293
0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0
curl: (7) Failed to connect to codeload.github.com port 443: No route to host

scribe GitHub repository is unavailable! -- Aborting.

Finally, trying the install code in the first post, scribe updated but at the end was the same error.

m@RT-AC86U-4988:/tmp/home/root# /usr/sbin/curl --retry 3 "https: //raw.githubusercontent.com/cynicastic/scribe/master/scribe" -o "/jffs/scripts/scribe" && chmod 0755 /jffs/scripts/scribe && /jffs/scripts/scribe install
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 22177 100 22177 0 0 46202 0 --:--:-- --:--:-- --:--:-- 52802
_
_ ( )
___ ___ _ __ (_)| |_ __
/',__) /'___)( '__)| || '_`\ /'__`\
\__, \( (___ | | | || |_) )( ___/
(____/`\____)(_) (_)(_,__/'`\____)
syslog-ng and logrotate installation
v0.10_1 (master) Coded by cynicastic


fetching scribe from GitHub master branch ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 122 0 122 0 0 162 0 --:--:-- --:--:-- --:--:-- 176
0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0
curl: (7) Failed to connect to codeload.github.com port 443: No route to host

scribe GitHub repository is unavailable! -- Aborting.

When I check with scribe status, it shows me that is installed version v0.10_1 - I don't know how right this is.

 

[WNorkle]

I'm also running into the same issue as Goobi. Tried uninstalling and reinstalling using "--insecure" in the curl command string bug (as per the link) - still getting

curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

 scribe GitHub repository is unavailable!  -- Aborting.

I am able to download the files to my PC as a ZIP file. So, I don't think that the issue is related to the certs on my computer.

 

[Butterfly Bones]

I installed v0.10_1 this morning. I did not re-install v.0.9_3 a few days back, after my own bumbling with some other router settings causing need for full router and USB reset / format/ install.

No issues with cURL download. Somewhere I remember someone reporting an issue with certs and RMerlin stated that the router time was off, I *think* because of new ntp. Darned if I can find the post now. I think it was with the 384.11_x beta with the new ntp implementation. Those with cURL issues, check that your router clock is correct. Then again, I may be completely wrong

Just to be certain, I did a force install immediately to be sure all files updated to the most recent. I have uncommented the line in 'blankmsg' (and commented out the dev/null one). Now time to let and run and see. Quick check with tail -F on skynet-0.log shows the [save] lines from the last few hours, also showing as expected in Loggly.

Thank you, @cmkelley. I see no hurry to get our of beta to version 1.0. Those of us with HND routers have had struggles getting syslog-ng to work well, in my case for a year. This is light years ahead of what I accomplished previously.

 

[cmkelley]

I installed v0.10_1 this morning. I did not re-install v.0.9_3 a few days back, after my own bumbling with some other router settings causing need for full router and USB reset / format/ install.

No issues with cURL download. Somewhere I remember someone reporting an issue with certs and RMerlin stated that the router time was off, I *think* because of new ntp. Darned if I can find the post now. I think it was with the 384.11_x beta with the new ntp implementation. Those with cURL issues, check that your router clock is correct. Then again, I may be completely wrong

Just to be certain, I did a force install immediately to be sure all files updated to the most recent. I have uncommented the line in 'blankmsg' (and commented out the dev/null one). Now time to let and run and see. Quick check with tail -F on skynet-0.log shows the [save] lines from the last few hours, also showing as expected in Loggly.

Thank you, @cmkelley. I see no hurry to get our of beta to version 1.0. Those of us with HND routers have had struggles getting syslog-ng to work well, in my case for a year. This is light years ahead of what I accomplished previously.

Not a big deal, but you don't have to comment out the /dev/null line ... it just sends it to /dev/null as well as to the file. That's actually what I'm doing now, I'm going to keep tabs on the blank file for a couple weeks and make sure it really is only blank messsages.

The issue with the router time was someone reporting an error where the certificate wasn't yet valid. It doesn't seem likely (but I'm no expert) that a mis-set clock could cause a problem with a self-signed certificate. I'm still googling, but I'm at a loss as to why people are having issue with my script and not say @Jack Yaz's stuff (and yes Jack, that is a passive-aggressive plea for your help since you host a lot of scripts on GitHub)

 

[Butterfly Bones]

Not a big deal, but you don't have to comment out the /dev/null line ... it just sends it to /dev/null as well as to the file. That's actually what I'm doing now, I'm going to keep tabs on the blank file for a couple weeks and make sure it really is only blank messsages.

I did exactly that, but no blankmsg.log file was generated after about an hour, so I went back and commented the line out. It is there now, timestamp May 4 8:55 but I did the v.10_1 install just before 7am my time. (I've learned to get at least one cuppa in me, before I post in the morning.)

I'll leave it for now, and just watch it as you are. The only thing I don't scrape our of syslog / messages now is dropbear and other unusual errors. I created a filter /helper file to clean out the rc_service & custom_script messages that have arrived with 384.11_ betas, I'll share if anyone wants it, in fact here is a listing of my filter / helper files, most of them are provided by scribe, of course:

 0loggly
 blankmsg
 chkwan
 crash
 divstats
 ethernet
 expandlog
 logrotate
 openvpn
 pixelserv
 rc_service
 skynet
 syslogng
 vpnfailover
 wlceventd

 

[cmkelley]

Looks like huge progress. I'm thousands of miles away from my router and too dependent on it to try this out yet.

A few comments: if I understand it correctly, kmsg is a read-once read-only socket that exists as a way to get kernel messages out. klogd scarfs them up and delivers them to the system log source. The skynet garbling I think was because syslog-ng was reading kmsg as a file source at the same time klogd was running, and they couldn't both be reading it at the same time. Your path to a solution is to kill klogd; @Cam's path is to leave it running but stop syslog-ng from reading that source and instead get kernel messages from the regular system source. I think we've shown adequately that either works.

The system() source seems to pick up the difference between the 2.6 kernel and the 4. kernel kernel handling just by seeing if /dev/kmsg exists, but I guess you've found a different problem with it with the HND implementation. In any case, using system() as a source means you have to take the path to kill klogd. Replicating it the way you do means a user can easily comment out a line or two and take @Cam's path.

Heh, yeah I wouldn't mess with anything like this remotely.

Technically, it's using /proc/kmsg or /dev/kmsg that necessitates killing klogd. system() just seems to be a macro of some sort that expands to one of those depending on architecture. Also, /proc/kmsg is read-once read-only and /dev/kmsg is read-multiple read/write. I think either would be usable by syslog-ng IF restarting klogd wasn't hard-coded into the firmware. But since it is, either /proc/kmsg has to be used, or else neither used and let klogd run.

I don't mean to say @Cam's solution isn't valid, just that it isn't the one I chose, and I believe mine is the correct choice for the general use case. I think, but I can't be arsed to check, that using klogd instead of reading from /proc/kmsg means you can't filter the Skynet / IPTables messages to their own file.

 

[cmkelley]

I did exactly that, but no blankmsg.log file was generated after about an hour, so I went back and commented the line out. It is there now, timestamp May 4 8:55 but I did the v.10_1 install just before 7am my time. (I've learned to get at least one cuppa in me, before I post in the morning.)

I'll leave it for now, and just watch it as you are. The only thing I don't scrape our of syslog / messages now is dropbear and other unusual errors. I created a filter /helper file to clean out the rc_service & custom_script messages that have arrived with 384.11_ betas, I'll share if anyone wants it, in fact here is a listing of my filter / helper files, most of them are provided by scribe, of course:

Werid, I definitely have both the blank.log file and /dev/null lines uncommeted and my blank.log fills up nicely.

I think the rc_services & custom_script messages have always been there. I see them in my logs on my router running 384.10_2.

 

[cmkelley]

I'm also running into the same issue as Goobi. Tried uninstalling and reinstalling using "--insecure" in the curl command string bug (as per the link) - still getting

curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

 scribe GitHub repository is unavailable!  -- Aborting.

I am able to download the files to my PC as a ZIP file. So, I don't think that the issue is related to the certs on my computer.

Well, not quite; curl is complaining about self-signed certs in the chain. Whatever you used for downloading the zip file onto your PC may not care about self-signed certs.

Using the "--insecure" flag in the install command lets you get the one file, but it still uses curl internally to get the version number on GitHub and to get the zip file.

 

[cmkelley]

I had the same issue as Goobi.

Trying different DNS configurations (with or without DOT - the router is AC86U with 384.11_beta2, no stubby installed like for previous version), the error changed like this:

syslog-ng and logrotate installation
v0.9_3 (master) Coded by cynicastic


scribe installed version: v0.9_3
scribe GitHub version: v0.10_1


New version available!
Do you wish to upgrade [y|n] y

Do you want to update syslog-ng and logrotate example files? [y|n] y

fetching scribe from GitHub master branch ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 122 0 122 0 0 263 0 --:--:-- --:--:-- --:--:-- 293
0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0
curl: (7) Failed to connect to codeload.github.com port 443: No route to host

scribe GitHub repository is unavailable! -- Aborting.

Finally, trying the install code in the first post, scribe updated but at the end was the same error.

m@RT-AC86U-4988:/tmp/home/root# /usr/sbin/curl --retry 3 "https: //raw.githubusercontent.com/cynicastic/scribe/master/scribe" -o "/jffs/scripts/scribe" && chmod 0755 /jffs/scripts/scribe && /jffs/scripts/scribe install
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 22177 100 22177 0 0 46202 0 --:--:-- --:--:-- --:--:-- 52802
_
_ ( )
___ ___ _ __ (_)| |_ __
/',__) /'___)( '__)| || '_`\ /'__`\
\__, \( (___ | | | || |_) )( ___/
(____/`\____)(_) (_)(_,__/'`\____)
syslog-ng and logrotate installation
v0.10_1 (master) Coded by cynicastic


fetching scribe from GitHub master branch ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 122 0 122 0 0 162 0 --:--:-- --:--:-- --:--:-- 176
0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0
curl: (7) Failed to connect to codeload.github.com port 443: No route to host

scribe GitHub repository is unavailable! -- Aborting.

When I check with scribe status, it shows me that is installed version v0.10_1 - I don't know how right this is.

The script was updated to 0.10_1 from the install code (getting from raw.githubusercontent.com) but none of the other files were updated.

 

[Jack Yaz]

Not a big deal, but you don't have to comment out the /dev/null line ... it just sends it to /dev/null as well as to the file. That's actually what I'm doing now, I'm going to keep tabs on the blank file for a couple weeks and make sure it really is only blank messsages.

The issue with the router time was someone reporting an error where the certificate wasn't yet valid. It doesn't seem likely (but I'm no expert) that a mis-set clock could cause a problem with a self-signed certificate. I'm still googling, but I'm at a loss as to why people are having issue with my script and not say @Jack Yaz's stuff (and yes Jack, that is a passive-aggressive plea for your help since you host a lot of scripts on GitHub)

Your curl commands look identical to others, so I'm not sure why it's having errors - I wonder if the affected users have the ca-certificates package from Entware and it may need updating?

 

[RMerlin]

Make sure your clock is accurate - this is essentials when dealing with SSL/TLS.

 

[cmkelley]

@xilitol, @WNorkle, and others having issues with "self signed certificate in certificate chain";

What are the first few lines of:

openssl s_client  -connect www.github.com:443

It should look something like:

CONNECTED(00000003)
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 Extended Validation Server CA
verify error:num=20:unable to get local issuer certificate
---
Certificate chain
 0 s:/businessCategory=Private Organization/jurisdictionC=US/jurisdictionST=Delaware/serialNumber=5157550/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 Extended Validation Server CA
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 Extended Validation Server CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
---

There is some information at https://github.com/Microsoft/Git-Credential-Manager-for-Windows/issues/646 where firewall settings (on a Windows computer) were intercepting the ssl certificates and causing issues.

 

[faria]

Make sure your clock is accurate - this is essentials when dealing with SSL/TLS.

My clock is correct and im also having the same issue.
Weird thing , if i run the reinstall command it gives the same error but it updates anyway.
Rt-ac87u 384.11_beta2

 

[faria]

@xilitol, @WNorkle, and others having issues with "self signed certificate in certificate chain";

What are the first few lines of:

openssl s_client  -connect www.github.com:443

It should look something like:

CONNECTED(00000003)
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 Extended Validation Server CA
verify error:num=20:unable to get local issuer certificate
---
Certificate chain
 0 s:/businessCategory=Private Organization/jurisdictionC=US/jurisdictionST=Delaware/serialNumber=5157550/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 Extended Validation Server CA
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 Extended Validation Server CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
---

There is some information at https://github.com/Microsoft/Git-Credential-Manager-for-Windows/issues/646 where firewall settings (on a Windows computer) were intercepting the ssl certificates and causing issues.

jose@RT-AC86U:/tmp/home/root# openssl s_client  -connect www.github.com:443
CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assu                                                                             rance EV Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 Exte                                                                             nded Validation Server CA
verify return:1
depth=0 businessCategory = Private Organization, jurisdictionC = US, jurisdictio                                                                             nST = Delaware, serialNumber = 5157550, C = US, ST = California, L = San Francis                                                                             co, O = "GitHub, Inc.", CN = github.com
verify return:1
---
Certificate chain
 0 s:/businessCategory=Private Organization/jurisdictionC=US/jurisdictionST=Dela                                                                             ware/serialNumber=5157550/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=g                                                                             ithub.com
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 Extended Validati                                                                             on Server CA
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 Extended Validati                                                                             on Server CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root                                                                              CA
---

 

[RMerlin]

What is the location of your curl command?

which curl