What's new

Syslogd, logstash or other?

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

KenZ71

Senior Member
So I am looking to extend my geek level and use my NAS4Free to store logs for multiple systems - EdgeRouter, UAP Access points and NAS of course.

Has anyone done similar?

I currently have NAS4Free setup as a syslog server but that seems to put all logs in one. So maybe I'll give logstash a try, unless others have feedback.
 
rsyslog, syslog-ng...

The 'consensus' is that rsyslog is more elegant. Easier to organise filters (?). Unfortunately not available on Entware-ng.

Edgerouter X is running so well and smooth. I don't bother to look at the log :) Not sent to remote logger either. Its default log level is minimal anyway. Hey..no news is good news! So you might have to increase verbosity.

I do have syslog-ng running locally on my AP. Primary purpose is to catch who poke at my openvpn port and collect the IP addresses for ban. Also my pixelserv-tls has logging enabled all the time. I pull ad url's from there, sent it over to another process which pull the actual ad for analysis e.g. accounting how much data I saved by blocking that adserver. Yes, this way I'm not actually saving bandwidth but it's home broadband... :)
 
Well, the reason I am looking to log all this is mostly to find out why VPN to office drops on occasion.

Sometimes it is solid for 2 or 3 days. Others it drops several times a day. I have a shell script to ping my router, google dns and office vpn, none show anything odd.
 

Or you can download the free version of Nagios Log Server, which is based on the ELK stack. The free version will only log 500 Mb per day, but I doubt most of our home networks generate that much data.

We switched from Splunk to this at work (Splunk was horribly expensive for the volume we needed), so I am trying out the free version on my home ESXi box to get more familiar with it.
 
ELK and NLS are similar on the front end to Splunk - we were a Splunk shop to monitor applications - it's pretty efficient once one gets the indexes sorted - and along with Splunk, we also used Teoco's SONAR platform for certain applications as well...

And yes, both platforms can be terribly costly - so one has to look at the cost vs. benefit on them - for us, having the Analytics was were the value was - in terms of platform sizing, licensing, and demand/capacity forecasting...
 
ELK and NLS are similar on the front end to Splunk - we were a Splunk shop to monitor applications - it's pretty efficient once one gets the indexes sorted - and along with Splunk, we also used Teoco's SONAR platform for certain applications as well...

And yes, both platforms can be terribly costly - so one has to look at the cost vs. benefit on them - for us, having the Analytics was were the value was - in terms of platform sizing, licensing, and demand/capacity forecasting...

The problem with Splunk pricing is it is based on volume of log data you collect - 1GB/day costs $5K (+ annual maintenance) and up from there. NLS is priced per instance ($2K for single instance, unlimited log volume). I agree Splunk can do a lot IF you have someone that knows how to set up the reports and/or program custom dashboards. We never had someone that could do that for us.

NLS does take a little tweeking out-of-the-box) to collect on the traditional syslog port (514/udp), as it has logstash running as a regular user. You have to go and change it to run as root, or do a Java hack, to be able to listen on privileged ports.
 
No doubt - but the magic of Splunk is on the back end - it's money well spent once one gets beyond a certain point. I agree, it's not cheap... My Ops guys were the big fans of Splunk - I had an appreciation of that instance..

I was more about SONAR and index/structured data and cross correlation between OSS/BSS/NSS, in that application SONAR was hella faster - and we front-ended that with SAP Business Objects (which Splunk, back in the day, couldn't support) - BO lets one slice/dice things pretty much on the fly...

Nothings perfect however... Analytics need to scale as much as actual traffic - our instances for Splunk and SONAR were scaled out to around 20M subs...
 
Well, the reason I am looking to log all this is mostly to find out why VPN to office drops on occasion.

Sometimes it is solid for 2 or 3 days. Others it drops several times a day. I have a shell script to ping my router, google dns and office vpn, none show anything odd.

Ah. Even the reason appears in the log and you manage to find it, the cause of drop would be boring I'm sure...I would script the openvpn client side to redial to simply workaround it.

If ports aren't blocked, consider an ipsec tunnel, I think this technology has better built-in resilience. Kids use openvpn. Dad's use ipsec. :)
 
I am locked in to Cisco Anyconnect. It is the employer's network that I am connecting to and therefore their choice of VPN clients.

I use Splunk at work quite a bit, a fantastic tool. However it is beyond out of my budget for home use.

Logstash looks interesting. Could be useful at work as well.
 
I use Splunk at work quite a bit, a fantastic tool. However it is beyond out of my budget for home use.

Logstash looks interesting. Could be useful at work as well.

One can mimic a lot of what Splunk does with the ELK stack...
 
I am locked in to Cisco Anyconnect. It is the employer's network that I am connecting to and tan dherefore their choice of VPN clients.

I use Splunk at work quite a bit, a fantastic tool. However it is beyond out of my budget for home use.

Logstash looks interesting. Could be useful at work as well.

If all you need is to see why the VPN is disconnecting, rsyslog with filters to separate by device (Ubuntu does this by default) is just fine. You can easily extract what you need with basic Unix tools along with some standard log parsing tools like like logwatch/fwlogwatch.

Logstash, NLS and Splunk are for when you want to look at long term trends and dig deeper in to the data, or have a LOT of devices to look at. However with that comes complexity - with Logstash, you need to write input filters in their scripting language to be able to parse logs into usable database fields. Then you have to write the queries on the front end to be able to generate the reports you want.
 
Logstash, NLS and Splunk are for when you want to look at long term trends and dig deeper in to the data, or have a LOT of devices to look at. However with that comes complexity - with Logstash, you need to write input filters in their scripting language to be able to parse logs into usable database fields. Then you have to write the queries on the front end to be able to generate the reports you want.

True, true...

There are some folks that do have a bit more resources off-board - and collecting data there, for post processing...

grep is always a friend when looking at local system logs - most likely in linux platforms that'll be over in /var/log/syslog - and there one can do a "less" and chase thru it, or a tail -f and watch realtime..
 

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top