firmware mod kit does not do the job

[slobodan]

Here is what I want to do: unpack John's fork 25E1, replace dnsmasq with dnsmasq from John fork 24E3. I have RT-AC56U.

What I did till now:

Used firmware mod kit to extract both builds to different folders. Worked ok. I have replaced dnsmasq. I have rebuilt the firmware. It worked. I have hex edited the resulting firmware in order to have the proper footer. I have applied the proper checksum to this firmware. I have flashed it to the router. It has flashed ok but the router gets bricked.

Hint: I know that the older dnsmasq works ok, since I am using it from USB stick.

 

[Fitz Mutch]

I think the correct way to pack a firmware is:
make image

You must also have the correct .config file in place (i.e. rt-ac68u, etc) and have your firmware unpacked at the expected location.

How did you unpack the .trx file in the first place?

 

[slobodan]

I was not compiling it, I was seeking to replace an already compiled binary with an already compiled binary, then repack the firmware.

I have unpacked and repacked it with https://github.com/openwrt-stuff/firmware-mod-kit

In case someone wonder why I want to do it, see:

/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 57 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 57 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 63 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 63 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 50 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 50 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 52 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 52 bytes

 

[ColinTaylor]

In case someone wonder why I want to do it, see:

/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes
/tmp/syslog.log:May 19 13:59:44 dnsmasq[4034]: failed to allocate 65 bytes

That message indicates that you have run out of RAM. So I'd suggest that your problem might not be with dnsmasq but all the other customisations you are running.

 

[slobodan]

That message indicates that you have run out of RAM. So I'd suggest that your problem might not be with dnsmasq but all the other customisations you are running.

Right, out of RAM happened because I had two different dnsmasq processes, each with large hosts files.

But anyway, dnsmasq from 25E1 gave me problems even if nothing were reported about it in the log, it just died unexpectedly (especially while I was using Zemana AntiMalware Portable).

 

[Fitz Mutch]

... out of RAM happened ...

I think the firmware repackaging tool is only supposed to change the filesystem. What version of the Linux kernel do you see?

What is the free RAM normally?

 

[slobodan]

I think the firmware repackaging tool is only supposed to change the filesystem. What version of the Linux kernel do you see?

What is the free RAM normally?

I have now 96628 free out of 255632 (163980 free without buffers/cache).

I have double-checked with the two dnsmasq processes, and, no, it did not run out of RAM, since there always remained more than 50000 free out of 255632.

 

[kvic]

I think the correct way to pack a firmware is:
make image

"firmware mod kit" is supposed to cover multiple platforms but it seems not widely tested. For ASUSWRT, the easier is as you suggested but that's too geek speak. Even for some geeks, they might not want to bother with a build system + the whole firmware repository.

An interesting project would be to extract the bit of build scripts "make image" and package as a standalone tool like "firmware mod kit" but solely supports ASUSWRT and its derivatives.

I expect it should be only a few shell utilities + a couple of command lines. The usage shall be tremendous IMO. It gives users freedom to re-package their firmware by customising, removing junks, replacing with better parts (e.g. utilities and newer drivers from ASUS or 3rd party) and adding new features from the community.

Do you have interest to pick up this project?

 

[Fitz Mutch]

An interesting project would be to extract the bit of build scripts "make image"...

I have tested this for RT-AC68U firmware and it seems to work just fine. It's exactly how you say it, however it's new.
https://github.com/blackfuel/asuswrt-mod-kit

Looking for some brave souls to test the other Asus ARM routers?

 

[kvic]

I have tested this for RT-AC68U firmware and it seems to work just fine. It's exactly how you say it, however it's new.
https://github.com/blackfuel/asuswrt-mod-kit

Looking for some brave souls to test the other Asus ARM routers?

Very nice! Great minds think ahead.

My one-year old FW build:
$ ./asuswrt-version.sh  RT-AC56U_60A1_Kp6.trx

FIRMWARE=RT-AC56U_60A1_Kp6.trx
BUILD_NAME=RT-AC56U
KERNEL_VER=3.0
FS_VER=0.4
SERIALNO=60
EXTENDNO=60

Latest stock build:
$ ./asuswrt-version.sh RT-AC56U_3.0.0.4_380_7378-g7a25649.trx

FIRMWARE=RT-AC56U_3.0.0.4_380_7378-g7a25649.trx
BUILD_NAME=RT-AC56U
KERNEL_VER=3.0
FS_VER=0.4
SERIALNO=380
EXTENDNO=7378

Seems none standard SERIALNO and EXTENDNO can't be reported correctly. My build was 60A1 and Kp6 respectively.

I also looked at asuswrt-extract.sh. I'm little worried about dependency on ".config". I believe it's no use other than making 'make' happy. Do you see a real need? Custom builds are surely to have different .config than the stock one..

Future suggestions: it's a little heavy lifting to download GPl tarballs. For RT-AC56U/RT-AC68U, it's possible to prepare a tiny subset of necessary binaries for the image itself and tools for reassembling the image. RT-AC87U will be another set. RT-AC88U/RT-AC3100/RT-AC5300(?) perhaps a third set. RT-AC3200 is an orphan..(not sure you want to put in more effort than currently done). You can make these much smaller sets of download from your GitHub repository.

 

[slobodan]

It works on my AC56U! I have changed however:

 diff asuswrt-extract.sh ../firmware-mod-kit/asuswrt-extract.sh
134,136c134,136
<   mv -f _*/squashfs-root "$TARGETDIR"
<   mv -f _*/1C "$IMAGEDIR/piggy"
<   rm -rf _*
---
>   mv -f /root/firmware-mod-kit/squashfs-root "$TARGETDIR"
>   mv -f /root/firmware-mod-kit/1C "$IMAGEDIR/piggy"
> #  rm -rf _*

Before running it I have manually entered:

mkdir /opt/firmware-mod-kit
ln -s /root/firmware-mod-kit /opt/firmware-mod-kit/trunk

I have done this on a fresh install of Debian 7 32-bit.

 

[slobodan]

Resulting firmware works ok and it is available at http://members.ziggo.nl/slobodan/

 

[john9527]

I still think you should spend the time tracking down the real conflict....no one but you is reporting a problem with the later dnsmasq.

 

[slobodan]

I still think you should spend the time tracking down the real conflict....no one but you is reporting a problem with the later dnsmasq.

Well, this is what I know:

It did not happen in builds before 25B7.

It does not happen in the latest build if I use Sixxs instead of Hurricane Electric for 6in4 tunneling. (Sixxs will shut down soon, so using it will be no longer an option.)

It mostly happens when I use Zemana AntiMalware Portable and press the scan button very shortly after starting the program. But it happens spontaneously in other scenarios as well, though less frequently. Sometimes the newer dnsmasq works for several hours, then it dies and does not come back.

I have ab-solution installed with a large hosts package (20.1 M).

I also have the user root, but I don't know why this would have anything to do with dnsmasq (which runs as nobody).

I have configured the router recently after resetting its settings to factory defaults.

I didn't mess with the bootloader.

I have checked 98M RAM and it is ok, so I assume there are no memory problems.

The router isn't overclocked.

It has nothing to do with CTF/FA settings.

It has nothing to do with dnsmasq.conf.add.

I have these lines in the firewall:

ACCEPT     icmp --  arc.he.net           anywhere
ACCEPT     icmp --  arc.he.net           anywhere
ACCEPT     icmp --  tserv1.ams1.he.net   anywhere

 

[slobodan]

Speaking of messing with the CFE... I had 1.0.27. I have upgraded to 1.0.29, then did a last try with the 25E1 build: still the same problem. I have reverted to the firmware having an older dnsmasq.

 

[slobodan]

Well, I had such problems once before, but it was due to enabling DNSSEC for dnscrypt for Cisco (which does not support DNSSEC). But this time DNSSEC isn't enabled.

 

[kvic]

I have ab-solution installed with a large hosts package (20.1 M).

More than 90% of the hosts you won't hit during your lifetime. It's insane for a home. Is your dnsmasq serving a large local community?

 

[slobodan]

More than 90% of the hosts you won't hit during your lifetime. It's insane for a home. Is your dnsmasq serving a large local community?

ab-solution is an ad blocking solution. Also blocks known bad websites (adware, spyware, viruses, phishing). Since I cannot know in advance which of the websites I will hit, I choose to have them all, since if I were to remove hosts, I would not know which hosts to remove.

And no, it is just for my own flat (with occasional guests having mobile phones, tablets or laptops), not for a local community.

If you ask me what's the benefit: I use this solution in my home and in my parents home, and they had some virus infections, I also had a virus infection and the viruses did not do much harm, since the viruses found themselves in a jail (hosts blocking known bad websites at router level as well as within Windows, malware-block script blocking known bad IPs, PeerBlock for blocking ads and spyware IPs, OpenDNS filtering mandatory enforced, CryptoPrevent applied, antivirus and MBAM enabled -- even if they did not detect the virus, they blocked bad IPs). E.g. my PayPal username and password are stored on my PC and my PayPal money did not get stolen.

This makes sense since nowadays viruses are mostly bots (from botnets) or trojan downloaders, since writing viruses has become business.

 

[kvic]

Since I cannot know in advance which of the websites I will hit, I choose to have them all, since if I were to remove hosts, I would not know which hosts to remove.

I recall seeing someone posted on this forum a "mother of all adblock lists" that cover 100M (?) hosts. You might be interested in giving that a try. A quick search didn't find me the thread..so I can't link to it for you...lol

Each online host list describes its purpose (adserver or malicious hosts). So you can control precisely what goes into Dnsmasq.

I run a no-name adblock script written by myself. It has 143 lines in total. 11 lines are the description of the script in the header; another 11 lines are codes to email myself the summary of a run; 22 lines are the 22 host lists that I source.

The rest (~100 lines) is the meat (including blank lines, comments, profiling codes and an 'advanced' algorithm to trim junks). If I cheat a bit by sacrificing readability, the number could reduce down to one third.

This no-name script takes 10 seconds to run (+3 more if email summary is desirable). That includes downloading online lists, processing and preparing host files. Currently it blocks >70k domains. Dnsmasq returns local query in less than 0ms (i.e. not measurable). A local query is answered from Dnsmasq's cache. So at the moment I've no plan to switch to something better.

 

[slobodan]

My block list is now blocking 373175 hosts (each listed twice: once for IPv4 and once for IPv6).