Scribe scribe - syslog-ng and logrotate installer

[Jack Yaz]

Yes, that is an interesting question. For me it seems that there is some conflict. The log below shows the time when I installed scribe and uiscribe at 12:18 klogd exiting...

Mar 09 12:18:28 RT-AC68U_WHITE kernel klogd: exiting
Mar 09 15:21:38 RT-AC68U_WHITE kernel klogd started: BusyBox v1.25.1 (2020-02-08 13:38:20 EST)
Mar 09 15:21:39 RT-AC68U_WHITE kernel klogd: exiting
Mar 09 15:21:39 RT-AC68U_WHITE rc_service httpds 255:notify_rc restart_logger
Mar 09 15:21:39 RT-AC68U_WHITE custom_script Running /jffs/scripts/service-event (args: restart logger)
Mar 09 15:21:40 RT-AC68U_WHITE kernel klogd started: BusyBox v1.25.1 (2020-02-08 13:38:20 EST)
Mar 09 15:21:43 RT-AC68U_WHITE kernel klogd: exiting
Mar 09 15:33:38 RT-AC68U_WHITE kernel klogd started: BusyBox v1.25.1 (2020-02-08 13:38:20 EST)
Mar 09 15:33:39 RT-AC68U_WHITE kernel klogd: exiting

Then I uninstalled uiscribe, and I got back to the built-in UI with syslog server address. I clicked on APPLY at 15:21 klogd started...
As you can see, it stops after a few seconds. The link to the syslog server (papertrail) works, these logs are copied from its UI.
Each time I APPLY, it starts and exits.

Scribe installed, uiscribe uninstalled.

Scribe kills the built in loggger daemon. You would need to configure syslog-ng to forward logs to an external log server as the destination in a syslog-ng config file (see /opt/etc/syslog-ng.d/ for examples)

 

[elorimer]

You might start here: scribe - syslog-ng and logrotate installer

 

[JohnDeere]

Scribe kills the built in loggger daemon. You would need to configure syslog-ng to forward logs to an external log server as the destination in a syslog-ng config file (see /opt/etc/syslog-ng.d/ for examples)

Thank you, I will dig into this and learn how to set it up correctly. As far as I understand, it is possible to send logs to external log server while running scribe and uiscribe in the same time.
By the way, I like this package and I find it very useful!

 

[JohnDeere]

You might start here: scribe - syslog-ng and logrotate installer

Thanks a lot for pointing me right to where to start I will take my time and learn

 

[cmkelley]

Hey y'all ... I'm getting reports of extremely high memory usage (~300 meg) for HND routers (AC86U, AC88U) ... I'm currently down to just my AC3200 which isn't seeing this, I get ~15K usage! Likely not unrelated, these reports are also seeing extremely high memory usage for pixelserv (~200 meg) while I'm only seeing 6K on my AC3200.

I'm not near smart enough to figure this out, is anyone seeing anything OTHER than the above? I.e. high syslog-ng and pixelserve memory usage on non-HND routers or low memory usage on HND routers?

 

[joe scian]

Hey y'all ... I'm getting reports of extremely high memory usage (~300 meg) for HND routers (AC86U, AC88U) ... I'm currently down to just my AC3200 which isn't seeing this, I get ~15K usage! Likely not unrelated, these reports are also seeing extremely high memory usage for pixelserv (~200 meg) while I'm only seeing 6K on my AC3200.

I'm not near smart enough to figure this out, is anyone seeing anything OTHER than the above? I.e. high syslog-ng and pixelserve memory usage on non-HND routers or low memory usage on HND routers?

My AC5300 on Beta 1 is just fine - no adverse memory consumption - works fine - like it always has.

 

[XIII]

How do you want us to check these numbers?

(Which tool/command should we use?)

 

[cmkelley]

How do you want us to check these numbers?

(Which tool/command should we use?)

Just top, then 'm' sorts the list by memory usage.

 

[XIII]

Just top, then 'm' sorts the list by memory usage.

RT-AC86u, 384.16 Beta 1 (since yesterday, so recent reboot):

* syslog-ng: 156m
* pixelserv-tls: 15120

Biggest offender is NextDNS on my router:

* nextdns: 782m

 

[HardCat]

RT-AX88U

  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 3800  3799 hardcat  S     299m 33.9   0  0.0 syslog-ng
18793     1 nobody   S    66324  7.3   3  0.0 unbound -c /opt/var/lib/unbound/unbound.conf
28568     1 nobody   S    63340  7.0   0  0.0 pixelserv-tls 192.168.xxx.xxx -l 2

 

[Smokey613]

RT-AC86U and using 782m

 

[XIII]

I might need a Linux memory 101...

For example for NextDNS: VIRT is 728M, but RES is only 9020 in htop.

(for syslog-ng: VIRT is 156m, RES is 5384)

 

[Smokey613]

 

[Smokey613]

https://www.tldp.org/LDP/sag/html/memory-management.html

 

[elorimer]

86U: syslog-ng 155M, Pixelserv 51K
87U: syslog-ng 18K, pixelserv 9K

 

[Butterfly Bones]

Here is my AC86U only one user online right now.

 

[juched]

On my RT-AX88U running alpha2.

PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
27805 27804 juched S 276m 31.2 0 0.0 syslog-ng

I have been doing some reading on this. The config is 16K of memory per line, and 256 lines, and if I multiply that by my 10 destinations (I think!) I end up with ~40MB of memory to be needed.
https://support.oneidentity.com/kb/311729/memory-utilization-is-high-on-syslog-ng-servers

How are destinations defined?

 

[elorimer]

The config is 16K of memory per line, and 256 lines, and if I multiply that by my 10 destinations (I think!) I end up with ~40MB of memory to be needed.

I went down this road too. I think most of us have around 10 or 12 destinations. That calculation gives you the total size of the output buffers the config sets up. In my case my 87U has exactly the same config as my 86U, but the latter uses 10,000 times more memory, more or less stable. So the cache the config sets up doesn't seem to be the issue.

There is also the receive buffer for the dgram portion of the src source, which is 64 messages per second or 64K in size. That doesn't seem to be it either.

Poking at this a little more, I added stats_level(2) to the config file of both routers, and then ran the stats. The output was roughly similar, except the 86U had these lines:

global;msg_allocated_bytes;;a;value;18446744073709550560
global;scratch_buffers_count;;a;queued;51539607552

The 87U didn't have those. I think the first is a running total of the size of all messages but that remains an unimaginable number. Not sure what the second is, but also unimaginably large.

 

[juched]

I went down this road too. I think most of us have around 10 or 12 destinations. That calculation gives you the total size of the output buffers the config sets up. In my case my 87U has exactly the same config as my 86U, but the latter uses 10,000 times more memory, more or less stable. So the cache the config sets up doesn't seem to be the issue.

There is also the receive buffer for the dgram portion of the src source, which is 64 messages per second or 64K in size. That doesn't seem to be it either.

Poking at this a little more, I added stats_level(2) to the config file of both routers, and then ran the stats. The output was roughly similar, except the 86U had these lines:

global;msg_allocated_bytes;;a;value;18446744073709550560
global;scratch_buffers_count;;a;queued;51539607552

The 87U didn't have those. I think the first is a running total of the size of all messages but that remains an unimaginable number. Not sure what the second is, but also unimaginably large.

Are you able to get syslog-ng-ctl to work? It has tools for seeing memory usage, etc.

I get this error and am trying to figure out how to configure:

Error connecting control socket, socket='/opt/var/syslog-ng.ctl', error='Connection refused'

That file doesn't exist, or when it does it is empty.

--- edit ---

ok, if I kill and relaunch

syslog-ng --control /var/run/syslog-ng.ctl

now I can see stats like you

SourceName;SourceId;SourceInstance;State;Type;Number                                                                                                                           
destination;messages;;a;processed;0                                                                                                                                           
global;payload_reallocs;;a;processed;0                                                                                                                                         
source;s_lr_status;;a;processed;0                                                                                                                                             
global;scratch_buffers_bytes;;a;queued;0                                                                                                                                       
destination;d_skynet;;a;processed;44                                                                                                                                           
global;msg_clones;;a;processed;0                                                                                                                                               
source;s_lr_daily;;a;processed;0                                                                                                                                               
global;internal_queue_length;;a;processed;0                                                                                                                                   
global;sdata_updates;;a;processed;0                                                                                                                                           
destination;d_logrotate;;a;processed;0                                                                                                                                         
destination;d_spdmerlin;;a;processed;2                                                                                                                                         
destination;d_ntpd;;a;processed;0                                                                                                                                             
center;;received;a;processed;47                                                                                                                                               
destination;d_diversion;;a;processed;0                                                                                                                                         
source;s_ntpd;;a;processed;0                                                                                                                                                   
destination;d_syslogng;;a;processed;1                                                                                                                                         
destination;d_openvpn;;a;processed;0                                                                                                                                           
destination;dt_ntpd;;a;processed;0                                                                                                                                             
global;scratch_buffers_count;;a;queued;8589934592                                                                                                                             
center;;queued;a;processed;47                                                                                                                                                 
source;src;;a;processed;47                                                                                                                                                     
destination;d_wlceventd;;a;processed;0                                                                                                                                         
src.internal;src#2;;a;processed;1                                                                                                                                             
src.internal;src#2;;a;stamp;1584293686                                                                                                                                         
destination;d_unbound;;a;processed;0

 

[elorimer]

Are you able to get syslog-ng-ctl to work?

Yes, no issue. Just ran in a terminal

syslog-ng.ctl stats