Cross compile static version of conntrack-tools

[mad_ady]

Hello all,

I'm trying to cross-compile my own version of conntrack-tools (in order to invalidate established sessions based on custom needs - e.g. when removing time-based firewall rules). I got my dev environment up by following this guide: https://github.com/RMerl/asuswrt-merlin/wiki/Compile-Firmware-from-source-using-Ubuntu, except I didn't use VirtualBox, but set up a lxc container instead.

I got everything set up and working although I had to do an extra step:

# cat /etc/ld.so.conf.d/arm.conf
/root/asuswrt-merlin/release/src-rt-6.x.4708/toolchains/hndtools-arm-linux-2.6.36-uclibc-4.5.3/lib

Next I got contrack-tools and all its dependencies (about 5 related libraries) and cross compiled all of them in the order required with something like this:

root@ubuntu12:~/asuswrt-merlin/release/src-rt/conntrack/conntrack-tools-1.4.3# export PKG_CONFIG_PATH=/root/asuswrt-merlin/release/src-rt/conntrack/libnfnetlink-1.0.1:/root/asuswrt-merlin/release/src-rt/conntrack/libmnl-1.0.3:/root/asuswrt-merlin/release/src-rt/conntrack/libnetfilter_conntrack-1.0.5:/root/asuswrt-merlin/release/src-rt/conntrack/libnetfilter_cttimeout-1.0.0:/root/asuswrt-merlin/release/src-rt/conntrack/libnetfilter_queue-1.0.2:/root/asuswrt-merlin/release/src-rt/conntrack/libnetfilter_cthelper-1.0.0; ./configure --host=arm-brcm-linux-uclibcgnueabi --enable-static; make;

It's messy, I agree, but in the end it compiles the conntrack binary that I need.

Now comes the problem part. I wanted to build a static binary, but it isn't static. I have two options - try to make it static, or move the dependent libraries on the router. It seems I failed both tasks and I require RMerlin's expertise...

root@ubuntu12:~/asuswrt-merlin/release/src-rt/conntrack/conntrack-tools-1.4.3# file src/conntrack
src/conntrack: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), stripped

I tried moving the libs + symlinks to the router inside /opt/lib, and the binary to /opt/bin

admin@LHS 288:/tmp/home/root# ldd /opt/bin/conntrack
   libnetfilter_conntrack.so.3 => not found
   libmnl.so.0 => not found
   libnfnetlink.so.0 => not found
   libc.so.0 => /lib/libc.so.0 (0x401ad000)
   ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x400ec000)
admin@LHS 288:/tmp/home/root#
admin@LHS 288:/tmp/home/root# ls -l /opt/lib/libnetfilter_conntrack.so.3* /opt/lib/libmnl.so.0* /opt/lib/libnfnetlink.so.0*
lrwxrwxrwx  1 admin  root  15 Dec 16 15:27 /opt/lib/libmnl.so.0 -> libmnl.so.0.1.0
-rwxr-xr-x  1 admin  root  47678 Dec 16 15:27 /opt/lib/libmnl.so.0.1.0
lrwxrwxrwx  1 admin  root  31 Dec 16 15:17 /opt/lib/libnetfilter_conntrack.so.3 -> libnetfilter_conntrack.so.3.5.0
-rwxr-xr-x  1 admin  root  480741 Dec 16 15:17 /opt/lib/libnetfilter_conntrack.so.3.5.0
lrwxrwxrwx  1 admin  root  21 Dec 16 15:29 /opt/lib/libnfnetlink.so.0 -> libnfnetlink.so.0.2.0
-rwxr-xr-x  1 admin  root  69362 Dec 16 15:28 /opt/lib/libnfnetlink.so.0.2.0

... but even after a reboot, ldd still doesn't see the libs.

So, question:
1. How can I convince it to link statically?
2. If I can't convince it to do it statically, why can't ldd see the libs I've put in /opt/lib? I can't run ldconfig on the router...

Thanks!

 

[mad_ady]

I've tried to trick it with LD_LIBRARY_PATH and LD_PRELOAD, but none work:

admin@LHS 288:/tmp/home/root# LD_LIBRARY_PATH=/opt/lib conntrack
conntrack: '/opt/lib/libc.so.6' library contains unsupported TLS
conntrack: can't load library 'libc.so.6'
admin@LHS 288:/tmp/home/root# LD_LIBRARY_PATH=/opt/lib ldd conntrack
   libnetfilter_conntrack.so.3 => not found
   libmnl.so.0 => not found
   libnfnetlink.so.0 => not found
   libc.so.0 => /lib/libc.so.0 (0x401b5000)
   ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x400f8000)
admin@LHS 288:/tmp/home/root# LD_PRELOAD=/opt/lib/libnetfilter_conntrack.so.3:/opt/lib/libmnl.so.0:/opt/lib/libnfnetlink.so.0 conntrack
conntrack: can't load library 'libnetfilter_conntrack.so.3'
admin@LHS 288:/tmp/home/root# file /opt/lib/libnetfilter_conntrack.so.3
/opt/lib/libnetfilter_conntrack.so.3: symbolic link to `libnetfilter_conntrack.so.3.5.0'
admin@LHS 288:/tmp/home/root# file /opt/lib/libnetfilter_conntrack.so.3.5.0
/opt/lib/libnetfilter_conntrack.so.3.5.0: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, not stripped

 

[lancethepants]

Do you also have optware/entware installed in /opt? It looks as though from the error you posted, that when using LD_LIBRARY_PATH that it's trying to link to a different c library than the one it was built with.

What I would try first is recompiling from scratch, but when compiling the dependency libraries, configure them with the additional arguments "--enable-static --disable-shared". That way way it comes time to link them, those libraries will be statically linked, leaving only the libraries already included in asuswrt.

 

[mad_ady]

I have entware already working in /opt and I'm running it on a RT-AC56 which is arm (and I trust I used the correct arm compiler). I have compiled all the libraries with --enable-static, but I haven't tried --disable-shared. I'll do that as well. Thanks!

Update: I recompiled everything from scratch with:

# export PKG_CONFIG_PATH=/root/asuswrt-merlin/release/src-rt/conntrack/libnfnetlink-1.0.1:/root/asuswrt-merlin/release/src-rt/conntrack/libmnl-1.0.3:/root/asuswrt-merlin/release/src-rt/conntrack/libnetfilter_conntrack-1.0.5:/root/asuswrt-merlin/release/src-rt/conntrack/libnetfilter_cttimeout-1.0.0:/root/asuswrt-merlin/release/src-rt/conntrack/libnetfilter_queue-1.0.2:/root/asuswrt-merlin/release/src-rt/conntrack/libnetfilter_cthelper-1.0.0; ./configure --host=arm-brcm-linux-uclibcgnueabi --enable-static --disable-shared

... and it still produces a binary with shared libraries:

root@ubuntu12:~/asuswrt-merlin/release/src-rt/conntrack/conntrack-tools-1.4.3/src# ls -l conntrack
-rwxr-xr-x 1 root root 161355 Dec 18 10:41 conntrack
root@ubuntu12:~/asuswrt-merlin/release/src-rt/conntrack/conntrack-tools-1.4.3/src# file conntrack
conntrack: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), not stripped

 

[lancethepants]

The question is, do you want a completely static binary? or just to statically link just those libraries that are not already included in asuswrt? If you compiled all those libraries with --enable-static --disable-shared, then hopefully all those libraries should now be included statically in the binary. If that's the case, then the binary should run fine in the asuswrt environment, and without using LD_LIBRARY_PATH or anything like that. The binary may not be completely static, but asuswrt supplies all the other libararies that are needed, ie libc, libm, libpthread....
What does 'readelf -d conntrack' return?

Now if you want a completely static binary, sometimes that can be a little tricky, and you might have to get creative.
Usually when configuring, I might try something like this.

LDFLAGS="-Wl,-static -static -static-libgcc -static-libstdc++" ./configure ...

Some binaries are built with libtool which can make things tricky. In those cases I might this after configuring

make LDFLAGS="-all-static"

There's plenty of exceptions to these two methods, and sometime that I've had to edit Makefiles, or just manually run the linking command throwing '-static' into the mix.

 

[szpunk]

I compiled on tomatoware success. The script 'copy' from @lancethepants 's old unbound static compile script. You can get from: http://pastebin.com/4bLDaPdx

 

[mstombs]

Why would you want a completely static compile? Surely this userspace app just communicates with the kernel to manipulate kernel filtering tables? An extended version of iptables, needs good co-ordination between kernel modules and userspace configuration tools?

 

[mad_ady]

Thank you all for your suggestions. I'll give it another try next week. The idea for the static binary was to have a self-contained binary (I was really interested in adding just the custom libs, not libc), but I could have lived with a dynamic binary as long as the dependencies would load... I'm still not sure why they don't load..

 

[mad_ady]

Sorry I haven't replied in a while - been busy with life. Conntrack finally compiled with that script (thanks!), but doesn't seem to work:

admin@arcturus:/tmp/home/root# conntrack -L
conntrack v1.4.3 (conntrack-tools): Operation failed: invalid parameters
admin@arcturus:/tmp/home/root# strace conntrack -L
execve("/opt/sbin/conntrack", ["conntrack", "-L"], [/* 16 vars */]) = 0
open("/dev/urandom", O_RDONLY)  = 3
read(3, "L\205}~", 4)  = 4
close(3)  = 0
ioctl(0, TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
brk(0)  = 0x5c000
brk(0x5d000)  = 0x5d000
socket(PF_NETLINK, SOCK_RAW, NETLINK_NETFILTER) = 3
getsockname(3, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 0
gettimeofday({1453213545, 927867}, NULL) = 0
bind(3, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
getsockname(3, {sa_family=AF_NETLINK, pid=31232, groups=00000000}, [12]) = 0
bind(3, {sa_family=AF_NETLINK, pid=31232, groups=00000000}, 12) = 0
sendto(3, "$\0\0\0\1\1\1\3jG\236V\0\0\0\0\2\0\0\0\10\0\10\0\0\0\0\0\10\0\25\0"..., 36, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 36
recvfrom(3, "8\0\0\0\2\0\0\0jG\236V\0z\0\0\352\377\377\377$\0\0\0\1\1\1\3jG\236V"..., 8192, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 56
close(3)  = 0
write(2, "conntrack", 9conntrack)  = 9
write(2, " v", 2 v)  = 2
write(2, "1.4.3", 51.4.3)  = 5
write(2, " (conntrack-tools): ", 20 (conntrack-tools): )  = 20
write(2, "Operation failed: ", 18Operation failed: )  = 18
write(2, "invalid parameters", 18invalid parameters)  = 18
write(2, "\n", 1
)  = 1
exit(1)  = ?
+++ exited with 1 +++

I'll have to check prerequisites - maybe there's some kernel module I have to load to allow AF_NETLINK.

 

[mad_ady]

Update. The strace output from a working conntrack installation looks like this:

...
socket(PF_NETLINK, SOCK_RAW, 12)  = 3
getsockname(3, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 0
bind(3, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
getsockname(3, {sa_family=AF_NETLINK, pid=4601, groups=00000000}, [12]) = 0
bind(3, {sa_family=AF_NETLINK, pid=4601, groups=00000000}, 12) = 0
sendto(3, "$\0\0\0\1\1\1\3RS\237V\0\0\0\0\2\0\0\0\10\0\10\0\0\0\0\0\10\0\25\0"..., 36, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 36
recvfrom(3, "\244\0\0\0\0\1\2\0RS\237V\371\21\0\0\2\0\0\0004\0\1\200\24\0\1\200\10\0\1\0"..., 8192, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 3684
fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(136, 15), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3740b23000
write(1, "udp  17 14 src=192.168.100.31 "..., 150udp  17 14 src=192.168.100.31 dst=192.168.100.255 sport=137 dport=137 [UNREPLIED] src=192.168.100.255 dst=192.168.100.31 sport=137 dport=137 mark=0 use=1
) = 150

Unfortunately I can't check the kernel config, but on the system where it works I have these enabled:

root@frost:~# cat /boot/config-4.2.0-23-generic | grep NETLINK
CONFIG_COMPAT_NETLINK_MESSAGES=y
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_ACCT=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NF_CT_NETLINK_TIMEOUT=m
CONFIG_NF_CT_NETLINK_HELPER=m
CONFIG_NETFILTER_NETLINK_QUEUE_CT=y
CONFIG_NETLINK_MMAP=y
CONFIG_NETLINK_DIAG=m
CONFIG_SCSI_NETLINK=y
CONFIG_QUOTA_NETLINK_INTERFACE=y

On the system where it works, I have the nf_conntrack_netlink module loaded:

root@frost:~# lsmod | grep nf_conntrack_netlink
nf_conntrack_netlink  36864  0
nfnetlink  16384  1 nf_conntrack_netlink
nf_conntrack  106496  6 nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4

On the ASUS I don't have it:

admin@arcturus:/tmp/home/root# lsmod | grep nf_conntrack_netlink
admin@arcturus:/tmp/home/root# find / -name "nf_conntrack_netlink*"
admin@arcturus:/tmp/home/root#

Question - Do we have the kernel sources to compile this as a module? Can anyone point me in the right direction?

Thanks

 

[RMerlin]

Kernel sources are part of the Git repo.

 

[mad_ady]

Thanks. Is this procedure correct for my RT-56 running RT-AC56U_380.57_0?

cd ~/asuswrt-merlin/release/src-rt-6.x.4708
make rt-ac56u

Or should I be looking in some other directory?

I've started the make process - I expect I can change .config (do a menuconfig?) and recompile later, right? I'm hoping I can get away with just a few modules, but if I need to recompile the kernel can I/should I flash it, or are we missing binary blobs?

 

[RMerlin]

Thanks. Is this procedure correct for my RT-56 running RT-AC56U_380.57_0?

cd ~/asuswrt-merlin/release/src-rt-6.x.4708
make rt-ac56u

Or should I be looking in some other directory?

I've started the make process - I expect I can change .config (do a menuconfig?) and recompile later, right? I'm hoping I can get away with just a few modules, but if I need to recompile the kernel can I/should I flash it, or are we missing binary blobs?

That's the correct SDK directory, yes.

All kernel setting changes must be made to linux/linux2.6/config_base .

Be warned that any kernel change that causes a kernel symbol change risks breaking compatibility with the closed-source kernel modules (wifi driver, NTFS driver, Trend Micro DPI engine, etc...).

Kernel modules will also need to be copied manually in src/router/Makefile, they aren't automatically included in the generated image if you add new modules.

The kernel is part of the firmware, it's not compiled separately.

 

[mad_ady]

Thanks! If I'm only missing modules I won't build a new firmware, I'll just load them with insmod.

Update: The kernel is missing CONFIG_NF_CT_NETLINK. I will try to compile it as a module and use it, but if it's not taking too much space could you add it in your new builds?

 

[mad_ady]

Update: Hmm, strange, I've added CONFIG_NF_CT_NETLINK=m in both linux/linux-2.6/config_base (which didn't exist) and linux/linux-2.6/config_rt-ac56u and ran the make rt-ac56u command, but no module is built and the resulting linux/linux-2.6/config_rt-ac56u has CONFIG_NF_CT_NETLINK commented out. It looks like I'm doing it wrong. I tried the same edit in linux/linux-2.6/.config, but it was overwritten by the build process. Where exactly can I change kernel config?

 

[RMerlin]

Sorry - complete filename is config_base.6a.

https://github.com/RMerl/asuswrt-me...rt-6.x.4708/linux/linux-2.6.36/config_base.6a

 

[mad_ady]

Ok, module compiled fine, but it won't load into the kernel

admin@arcturus:/jffs# file nf_conntrack_netlink.ko
nf_conntrack_netlink.ko: ELF 32-bit LSB relocatable, ARM, EABI5 version 1 (SYSV), BuildID[sha1]=52f7284545ddc0d276f0610d3cf6d7d238d9d2e6, not stripped
admin@arcturus:/jffs# ls -l nf_conntrack_netlink.ko
-rw-------  1 admin  root  340240 Jan 22 14:44 nf_conntrack_netlink.ko
admin@arcturus:/jffs# insmod nf_conntrack_netlink.ko
insmod: can't insert 'nf_conntrack_netlink.ko': Device or resource busy
admin@arcturus:/jffs# dmesg | tail -3
ctnetlink_init: cannot register notifier.
ctnetlink v0.93: registering with nfnetlink.
ctnetlink_init: cannot register notifier.

Back to the drawing board...