1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

Configuring syslog-ng with merlin firmware

Discussion in 'Asuswrt-Merlin' started by tomsk, Oct 7, 2016.

  1. tomsk

    tomsk Very Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    570
    I fancy having a go at using syslog-ng after reading a post in the NTP daemon thread
    is there anyone who can demonstrate or have a quick and dirty configuration to set it up to play nicely with asus-merlin setups? i found the config file in /opt/etc/syslog-ng.conf

    i found what i guess is the right admin guide for the entware version https://my.balabit.com/downloads/archived_documents/syslog-ng-2.0-guides
     
    joegreat likes this.
  2. kvic

    kvic Part of the Furniture

    Joined:
    Aug 11, 2014
    Messages:
    2,438
    Location:
    22.4399N 114.2222E
    I take it as you summoning me..lol. I'm very thankful to Entware-ng. Without it my Asus is pretty much half dead. So let me do a favour to Entware.

    To get syslog-ng up and running in AsusWRT or merlin derivative, the easy part is simply "kill -9" syslog. Then start syslog-ng through Entware's init.d script. So you see you could embed "kill-9" in the script to make it in one go..

    syslog-ng likes to dance along with logrotate (also available from Entware-ng). I would install it.

    From there on, everything will be syslog-ng. Standard Entware or Linux stuff.

    The nasty bit. You have to make a symbolic link in /tmp/syslog.log to the syslog-ng log file, /opt/var/log/messages - the defacto log file if not filtered into other files. With the symbolic link, usual log messages will continue to display on WebUI.

    A second nasty bit. Stop AsusWRT writing a copy of /tmp/syslog.log to /jffs. Search the forum. A couple of threads discussed that. IMO, simply disable this feature regardless you run syslog or syslog-ng.

    The optimisation bit. When messages file gets big, Asus' stupid WebUI chokes when loading it. Hence, you want to limit messages through logrotate to below 1MB.

    The challenging bit. When Asus boots up, logging starts immediately and to its usual place, /tmp/syslog.log. When logging transferred to syslog-ng upon Entware loading, you want to include the portion of log from /tmp/syslog.log in /opt/var/log/messages before transforming /tmp/syslog.log into a symbolic link..

    Without the last bit, syslog-ng still runs very well on AsusWRT or merlin. I got the last bit solved but really not have the appetite to write about the nitty gritty nor a step-by-step guide to set up syslog-ng.

    If you work it through, you might document the steps and share with the folks here. Assistance will be available..maybe me if I see fit. Good luck!
     
  3. tomsk

    tomsk Very Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    570
    Hi Kvic ..... sorry if my post appeared to be directed specifically at you. It was simply that your post had peaked my interest. I was throwing the question out to the forum in general because I'm sure there are others who have been down this path too.
    I realised i would need logrotate to stop the logs bloating yesterday, and already downloaded that through entware. That looks pretty straight forward so the man pages should be enough info to configure it.
    Thanks for all the pointers. I will have a fish around in the forums and try to piece it all together.
    I will certainly document the steps for everyones benefit if i can work it out.
    I've learnt so much from others on this forum already and it would be great if if i could give something back.

    Thanks :)
     
    rotorbudd and kvic like this.
  4. tomsk

    tomsk Very Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    570
    I have killed syslog and started syslog-ng
    Code:
    kill -9 $(pidof syslogd)
    /opt/etc/init.d/S01syslog-ng start
    I can see /opt/var/log/messages file being populated... so it looks like the default source is ok in the syslog-ng.conf
    Code:
    Oct  8 10:21:08 RT-AC68U-4690 syslog-ng[17663]: syslog-ng starting up; version='2.1.4'
    Oct  8 14:30:01 RT-AC68U-4690 crond[461]: crond: USER tOmsK pid 17712 cmd /jffs/bin/ntpstats.sh
    Oct  8 14:35:01 RT-AC68U-4690 crond[461]: crond: USER tOmsK pid 17754 cmd /jffs/bin/ntpstats.sh
     
  5. tomsk

    tomsk Very Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    570
    The /tmp/syslog.log file is recreated every boot , so i can simply transfer the entire content to the /opt/var/log/messages file before creating the symbolic link. Whats the fastest way to avoid loosing any potential logs between stopping syslog and starting syslog-ng? something like sed?
     
  6. kvic

    kvic Part of the Furniture

    Joined:
    Aug 11, 2014
    Messages:
    2,438
    Location:
    22.4399N 114.2222E
    cat /tmp/syslog.log >> /opt/var/log/messages

    This shall work okay. Lost a few lines not a big deal on rare occasions during the process..
     
  7. tomsk

    tomsk Very Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    570
    Thanks Kvic ... yup i was playing around with that yesterday and it works ok...just seemed to be a little slow.
    Entware will start syslog-ng up first anyway because of the 01 in the script name when rc.unslung sorts them right?
    Heres what i got so far.....thought of dumping it in the syslog-ng launch script.. or linking it to that somehow ( not sure how to pass command arguments from one script to another)
    Code:
    # check syslog-ng init.d argument for start
    # if S1 = start ... do the following
    # stop the syslog daemon ('syslogd -m 0" a better option??)
    kill -9 $(pidof syslogd)
    # copy syslog to syslog-ng
    cat /tmp/syslog.log >> /opt/var/log/messages
    # delete the syslog logfile to allow symlink creation
    rm /tmp/syslog.log
    # create symbolic link
    ln -s /opt/var/log/messages /tmp/syslog.log
    # possibly have to restart syslog-ng?
    /opt/etc/init.d/S01syslog-ng restart
     
    Last edited: Oct 10, 2016
  8. kvic

    kvic Part of the Furniture

    Joined:
    Aug 11, 2014
    Messages:
    2,438
    Location:
    22.4399N 114.2222E
    That's right. syslog-ng is the first thing gets started in Entware-ng space..for the obvious reason. Getting everything into syslog-ng init.d script is not a bad option.

    You can create a function, say kill_syslog. Put all the steps you've figured out so far in there. Call the function from the right place automatically by rc.func by adding the line in the same script:

    PRECMD="kill_syslog"
     
  9. tomsk

    tomsk Very Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    570
    You mean something like this? ..... or have i got hold of the wrong end of the stick and i have to put the function in rc.func? The only place i see PRECMD in rc.func is in the start function where any value is sent down the black hole.... $PRECMD > /dev/null 2>&1

    Code:
    #!/bin/sh
    
    ENABLED=yes
    PROCS=syslog-ng
    ARGS=""
    PREARGS=""
    DESC=$PROCS
    PRECMD="kill_syslog"
    
    PATH=/opt/sbin:/opt/bin:/opt/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    
    function kill_syslog() {
        blah blah some code here
    }
    
    . /opt/etc/init.d/rc.func
     
    Last edited: Oct 10, 2016
  10. kvic

    kvic Part of the Furniture

    Joined:
    Aug 11, 2014
    Messages:
    2,438
    Location:
    22.4399N 114.2222E
    @tomsk I think you're right on track. You can try something like this (I dug out from my backup storage):

    Code:
    #!/bin/sh
    kill_syslogd () {
        if [[ ! -z `pidof syslogd` ]]; then
            killall syslogd
            cat /tmp/syslog.log >> /opt/var/log/messages
            rm  /tmp/syslog.log /tmp/syslog.log-1
            ln -s /opt/var/log/messages /tmp/syslog.log
        fi
    }
    
    ENABLED=yes
    PROCS=syslog-ng
    ARGS=""
    PREARGS=""
    PRECMD="kill_syslogd"
    DESC=$PROCS
    PATH=/opt/sbin:/opt/bin:/opt/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    
    . /opt/etc/init.d/rc.func
    
     
    MarCoMLXXV likes this.
  11. kvic

    kvic Part of the Furniture

    Joined:
    Aug 11, 2014
    Messages:
    2,438
    Location:
    22.4399N 114.2222E
    The above shall work well enough.

    The other option if you dont want to compile your own firmware, disassemble the FW image which is in squashfs. Replace the necessary files with symbolic links. One of the links will be a redirection to the syslog-ng binary in Entware. So that when AsusWRT calls syslogd, it actually executes syslog-ng. Reassemble everything back to squashfs image. Flash it.
     
  12. tomsk

    tomsk Very Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    570
    Im happy that i seem to be fumbling my way in the right direction with some gentle nudging :)
    A question about doing it this way..... won't this precmd be applied irrespective of starting or stopping syslog-ng? Im thinking about the shutdown process now and if there is a need to re-invoke syslog to capture those logs?
     
  13. kvic

    kvic Part of the Furniture

    Joined:
    Aug 11, 2014
    Messages:
    2,438
    Location:
    22.4399N 114.2222E
    Function only runs something when syslogd is alive. Once gone, the world is a better place and we don't have to worry about it..including shutdown. Everything handled by syslog-ng.


    Very nice indeed. I was hoping someone caught that post..didn't realise it'll happen a year later.
     
    tomsk likes this.
  14. tomsk

    tomsk Very Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    570
    Thats waaaaaay beyond anything I've done before. my previous experience of compiling anything from source was trying to build binaries for X windows for redhat 5 and mandrake maybe 20 years ago.... all i can remember is getting frustrated from endless library dependancy problems.
     
    kvic likes this.
  15. tomsk

    tomsk Very Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    570
    Looking at this post to set up log rotate
    To add the cron job, would it be easier to use the "cru" command instead?

    Code:
    cru a LogRotate "0 0 * * * /opt/sbin/logrotate /opt/etc/logrotate.conf"
     
  16. tomsk

    tomsk Very Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    570
    I have a syslog-ng.conf file set up per below, which is nicely filtering out cron and pixelserv logs into their own files and putting unfiltered messages into the messages log
    Code:
    options {
            chain_hostnames(off);
            sync(0);
    
            # The default action of syslog-ng 1.6.0 is to log a STATS line
            # to the file every 10 minutes.  That's pretty ugly after a while.
            # Change it to every 12 hours so you get a nice daily update of
            # how many messages syslog-ng missed (0).
            stats(43200);
    };
    
    source src { unix-stream("/dev/log"); internal(); };
    
    filter f_cron { facility(cron); };
    filter f_pixelserv { match("pixelserv"); };
    
    destination messages { file("/opt/var/log/messages"); };
    destination cron { file("/opt/var/log/cron"); };
    destination pixelserv-tls { file("/opt/var/log/pixelserv-tls"); };
    
    log { source(src); filter(f_cron); destination(cron); };
    log { source(src); filter(f_pixelserv); destination(pixelserv-tls); };
    log { source(src); destination(messages); flags(fallback); };
    When logs rotate syslog-ng has to be restarted, hence the "killall -HUP syslog-ng" postrotate command in the log rotate config file

    Code:
    # system-specific logs may be also be configured here.
    
    /opt/var/log/messages {
    su tOmsK root
    maxsize 1024k
    weekly
    rotate 9
    postrotate
    killall -HUP syslog-ng
    endscript
    }
    If i have different logs set for rotate at other maxfile size or intervals, will i have to kill and restart syslog-ng for every instance?
     
  17. rotorbudd

    rotorbudd Occasional Visitor

    Joined:
    Sep 22, 2016
    Messages:
    29
    @tomsk Thanks for these posts. They allowed me to understand how to get syslog-ng and logrotate working together.
     
  18. tomsk

    tomsk Very Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    570
    Hi Rotorbudd.. I'm happy someone made some use of my bumbling discoveries :) .. i didn't like the idea of restarting syslog-ng overtime i rotated a file, and so did some poking around in the internet looking for some ideas and came across this interesting article
    http://sflanders.net/2012/05/12/logrotate-limitations-revisited/#more-189
    The idea here is to use the macros built into syslog-ng to dynamically change the location that logging is done to as the date changes. The advantage is that there will be no need to restart syslog-ng after rotation because logrotate will not be rotating a live file, however the disadvantage is that a new folder is going to be created daily which i dislike the idea of managing equally as much.
    If i come up with a better way of doing this i will post here, but for now i am continuing to signal syslog-ng to restart postrotate.
     
    rotorbudd likes this.
  19. kvic

    kvic Part of the Furniture

    Joined:
    Aug 11, 2014
    Messages:
    2,438
    Location:
    22.4399N 114.2222E
    Entware-ng updated syslog-ng to v3.8. A huge jump from the previous v2.1. Config file requires migration obviously.

    Here is what I need to migrate: http://kazoo.ga/migrate-to-syslog-ng-3-8/

    For people want to start your adventure on syslog-ng, it serves as a skeleton config to start your own.
     
    tomsk likes this.
  20. tomsk

    tomsk Very Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    570
    the filter example for pixelserv doesn't work for me
    Code:
    filter f_pixelserv { facility(daemon) and program("pixelserv"); };
    i tried program('pixelserv-tls") as well.... no good.
    do i have to point it at the entware directory?
    The global options were already configured in the entware install :)