Scribe Scribe no longer starting - symbol error

CaptnDanLKW

Senior Member
So my wife said she thought the internet was down last night, so I went to check my logs and i found that scribe was not running and did not start after a reboot 4 1/2 days ago.

No changes to firmware or any addons, the reboot was to re-do the thermal pads inside the router.

Scribe wont start, checking it throws a symbol error... odd

syslog-ng.conf version check ...syslog-ng: symbol lookup error: /opt/lib/libsyslog-ng-3.36.so.0: undefined symbol: _xmd_scanner_end_element
in sync. ()
syslog-ng.conf syntax check ... FAILED!

I was going to force a reinstall and restore the config but I wanted to see if anyone else knows whats up?
 

elorimer

Very Senior Member
So my wife said she thought the internet was down last night, so I went to check my logs and i found that scribe was not running and did not start after a reboot 4 1/2 days ago.

No changes to firmware or any addons, the reboot was to re-do the thermal pads inside the router.

Scribe wont start, checking it throws a symbol error... odd

syslog-ng.conf version check ...syslog-ng: symbol lookup error: /opt/lib/libsyslog-ng-3.36.so.0: undefined symbol: _xmd_scanner_end_element
in sync. ()
syslog-ng.conf syntax check ... FAILED!

I was going to force a reinstall and restore the config but I wanted to see if anyone else knows whats up?
I'm guessing here. When scribe starts, I think it does a check of the version number in syslog-ng.conf, and if it doesn't match the version of syslog-ng, it updates the version number. That does nothing other than supress an error message, and I think maybe alert you to the idea that syslog-ng has been updated and maybe the .conf needs to change accordingly. We haven't had that in a while.

I think the current version in entware is 3.36. You might look at your syslog-ng.conf and see if that is what listed, and if not just change it and restart scribe. If it doesn't, start it in the debug mode and see where it chokes.

Scribe not starting shouldn't affect internet though.
 

kernol

Very Senior Member
I [and probably many others] have encountered this version mismatch issue which causes Scribe to falter
So - whenever Entware is updated - I have a habit of immediately opening Scribe menu from amtm and taking option "s" to Show scribe status".
That portion of the script finds the version mismatch and auto-corrects it ... job done. :).
 

elorimer

Very Senior Member
It looks to me like this error is triggered in line 363 of the script by the command "syslog-ng --version". So this isn't really about the script, but some deeper problem with syslog-ng, perhaps some mismatch between the version you have installed and the library referenced in the error message.

I think I would uninstall syslog-ng and then reinstall it using opkg.
 

CaptnDanLKW

Senior Member
I'm guessing here. When scribe starts, I think it does a check of the version number in syslog-ng.conf, and if it doesn't match the version of syslog-ng, it updates the version number. That does nothing other than supress an error message, and I think maybe alert you to the idea that syslog-ng has been updated and maybe the .conf needs to change accordingly. We haven't had that in a while.

I think the current version in entware is 3.36. You might look at your syslog-ng.conf and see if that is what listed, and if not just change it and restart scribe. If it doesn't, start it in the debug mode and see where it chokes.
I forgot about this and came back again to troubleshoot. my conf file has been around a while, so I just backed it up and copied the -opkg to a new conf file.

Same Symbol error issue.
Scribe not starting shouldn't affect internet though.
Yea, never suspect it - just when I hear from the wife 'the internet had an issue', I go and look at logs. its only then did I realize syslog-ng wasnt working.

I guess I'll just remove and reinstall.


____________________________________________

Please select an option: rs

syslog-ng NOT running! Starting ...
Starting syslog-ng... failed.
Restarting uiScribe ...syslog-ng: symbol lookup error: /opt/lib/libsyslog-ng-3.36.so.0: undefined symbol: _xmd_scanner_end_element
Mounting WebUI page for uiScribe

Mounted uiScribe WebUI page as Main_LogStatus_Content.asp

done.

checking syslog-ng daemon ... dead.
the system logger (syslogd) ... is not running!

Type syslog-ng -Fevd at shell prompt or select sd
from scribe utilities menu (su) to view syslog-ng
debugging data.

checking system for necessary scribe hooks ...

checking S01syslog-ng ... present.
checking post-mount ... present.
checking unmount ... present.

checking syslog-ng configuration ...

syslog-ng.conf version check ...syslog-ng: symbol lookup error: /opt/lib/libsyslog-ng-3.36.so.0: undefined symbol: _xmd_scanner_end_element
in sync. ()
syslog-ng.conf syntax check ... FAILED!


scribe installed version: v2.4_3 (master)
scribe GitHub version: v2.4_3 (master)
scribe is up to date!
 

CaptnDanLKW

Senior Member
OK.

I did the full remove, reboot, reinstall, reboot, and when I restored the config, the same problem came back, so I went with the vanilla conf file and re-edited to enable remote logging and my custom /opt/etc/syslog.d and logrotate.d files remained.

The structure of syslog.conf has changed a few times over the years and while this one seems simpler, yet

Problem #1

Remote logging is working but is going to the default syslog log (and uiScribe's config window for my remote nodes only contains old entries since before the issue - as expected)

Maybe the syntax of the other files needs changing? I'll reexample the samples and see if i can figure it out, if not I'll be back here.

Ugh, I hate re-learning things that only occasionally are needed.

Problem? #2

I see this error (bold, below) when starting... I don't pretend to code but I think I can interpret what it's trying (listen on IPv6 loopback if present, else listen on IPv4) to do and it shouldn't matter to my config.

The problem is sysctl is not a valid binary. I thought this was a standard busybox function, but I can't remember the nuances between the older 32bit models and the newer 64bit HND models. I don't know if the entware version of syslog-ng incorrectly makes assumptions about the underlying platform...

As I said, I'm not a coder but every once in a while I stare at the Matix and can momentarily 'see' it.

if it is a entware/syslog-ng bug, I wouldn't even know how to report it.

Thanks for reading and the insight. An old mantra of mine goes something like this.

'It's when it doesn't work we learn the most and I've learned a lot over the years'

_____________________________________________

Please select an option: rs

syslog-ng NOT running! Starting ...
Starting syslog-ng... done.
Restarting uiScribe .../opt/share/syslog-ng/include/scl/network_localhost/detect.sh: line 6: sysctl: not found
Mounting WebUI page for uiScribe
...
...

scribe is up to date!
 

elorimer

Very Senior Member
Youza. First, mine is working fine, so there must be a path back. I collect logs from my TrueNAS servers, and a node and another Asus router connected via vpn, and it's been a while since I've used the scribe configuration.

The syslog-ng.conf that entware tries to install includes the scl directory and all that good stuff, which includes the default network drivers, and uses them to create a network source. That source collects the messages other devices send to the router.

Scribe does it a different way, and uses the deprecated udp driver to define a source to collect network messages, and doesn't include the scl directory. But it also comments out that source, so by default scribe does not collect network messages.

I've modified my .conf file to include the default network drivers, so I do it the -ng way, not the Scribe way. It sounds to me like your config is doing the same, or at least trying to configure the default network drivers by including scl, and failing. Because it fails, it isn't starting the network drivers and that's why you aren't hearing the remote logging. But maybe it is also seeing the scribe parts that comment out those logging statements and source definitions.
 

CaptnDanLKW

Senior Member
Youza. First, mine is working fine, so there must be a path back. I collect logs from my TrueNAS servers, and a node and another Asus router connected via vpn, and it's been a while since I've used the scribe configuration.

The syslog-ng.conf that entware tries to install includes the scl directory and all that good stuff, which includes the default network drivers, and uses them to create a network source. That source collects the messages other devices send to the router.
interesting...

Scribe does it a different way, and uses the deprecated udp driver to define a source to collect network messages, and doesn't include the scl directory. But it also comments out that source, so by default scribe does not collect network messages.
Couple of Interesting differences between my original (used to be) working -conf file (start I see the newer syslog-ng.conf file from 3.36

1) new has @include "scl.conf" code line enabled and in my old version its commented out.
2) new -conf is lacking most of the helpful #comments and the only comment is
log {
source(src);
source(net);
source(kernel);
destination(messages);

# uncomment this line to open port 514 to receive messages
source(s_network);

So I did...

I've modified my .conf file to include the default network drivers, so I do it the -ng way, not the Scribe way. It sounds to me like your config is doing the same, or at least trying to configure the default network drivers by including scl, and failing. Because it fails, it isn't starting the network drivers and that's why you aren't hearing the remote logging. But maybe it is also seeing the scribe parts that comment out those logging statements and source definitions.

Sorry, I wasn't clear... Remote logging IS working, but they are mixed in with the regular syslog.log (/opt/var/log/messages)
1654516759430.png


What isn't working are any of the /opt/etc/syslog.d/ filters. I checked the 3 I use with built-in examples - ioctl, ethernet and wlceventd, so their syntax requirements didn't change.

I feel the difference is how the new syslog-ng.conf structure and how its passed though the internal plumbing.

In the original syslog-ng.conf, these comments were very helpful, and showed two distinctly different ways to use the remote listener.

Snippet from old -conf

# syslog-ng gets messages from the system, kernel, and syslog-ng (internal)
# DO NOT use system() source; causes issues on HND routers
# so_rcvbuf = maximum number of messages per second * 1024
source src {
unix-dgram("/dev/log" so_rcvbuf(65536) flags(syslog-protocol));
file("/proc/kmsg" program_override("kernel") flags(kernel));
internal();
# udp(ip(192.168.x.y) port(514)); # uncomment this line to pass all network messages through syslog-ng filters
};

# if you only want to pass network messages through some syslog-ng filters, uncomment the source line below
# then add "soource(net);" to the log statement in any filter you want to pass network messages through
#source net { udp(ip(192.168.x.y) port(514)); };
source net { network( transport(udp) ); };

I chose the 2nd option because I created a new filter to capture everything from my mesh nodes in to one file.

I'm fairly down in the weeds now, not sure how much help there is, I may just nuke and start all over, re-learning and trying each step on by one. And for the benefit of other who may have some anomaly like this.


Thanks
 

CaptnDanLKW

Senior Member
So, for those who read and follow.

The TL;DR version is that at some point running the 'status' command in scribe must have downloaded an incompatible version of syslog-ng.conf, because after further investigation of the structural differences between them, I notice the header on the non-functioning config said:
"# OpenWrt syslog-ng.conf specific file"

When I nuked and cleaned all scribe components and installed fresh, the structure of the -conf file once again looked familiar and the header specifically said "# syslog-ng.conf customized for scribe on Asuswrt-Merlin firmware"

So, that ends this saga. Wiser now.
 

Similar threads

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top