Help with unbricking RT-AC66u from serial

[donbowman]

I'm not sure what happened, but as part of 380.57 upgrade my RT-AC66u is semi-bricked.

I have serial access, but it will not go to recovery.
The reason is the CFE won't bring up the interface:

CFE> ifconfig
Network interface has not been configured
*** command status = 0
CFE> ifconfig -addr=192.168.1.1 -mask=255.255.255.0 -hwaddr=00:00:00:01:02:03
Network interface has not been configured
CFE> ifconfig -addr=192.168.1.1 -mask=255.255.255.0 -hwaddr=00:00:00:01:02:03 eth0
Could not activate network interface 'eth0': Error
*** command status = -1
CFE> ifconfig -addr=192.168.1.1 -mask=255.255.255.0 -hwaddr=00:00:00:01:02:03 eth1
Could not activate network interface 'eth1': Error
*** command status = -1

Same affect if I hold the reset button while powering on.
its hw rev 1.30, and on 1.0.1.4 CFE.

I'm thinking (?) that the nvram is corrupt, i wonder if some kind soul would post or could point me @ a known good AC66U nvram show output?

from this environment, it would appear I can 'boot' or 'load' from serial (via s-records I guess?) but that would presumably be a binary image. Has anyone done this and has some pointers? E.g. I would not load a trx I guess, but what would I load?

I don't have a jtag programmer, but I guess I could get one, does anyone have a pointer @ instructions on programming the flash that way?

or, is the best approach to somehow 'load' the trx file into memory and then do a flash from memory? the cfe seems to suggest some support there on both load and flash.

Any other suggestions as to why the CFE won't bring up the ethernet interface for recovery? Is my nvram corrupt assumption a good one?

Decompressing...done


CFE version 6.30.39.29 (r338244) based on BBP 1.0.37 for BCM947XX (32bit,SP,LE)
Build Date: ¸ 10 15 10:41:41 CST 2012 (yau@wireless-pub2)
Copyright (C) 2000-2008 Broadcom Corporation.

Init Arena
Init Devs.
Boot partition size = 262144(0x40000)
Found an ST compatible serial flash with 32 64KB blocks; total size 2MB
Found a Samsung NAND flash with 2048B pages or 128KB blocks; total size 128MB
CPU type 0x19749: 600MHz
Tot mem: 131072 KBytes

CFE mem:    0x80700000 - 0x8094EDE0 (2420192)
Data:       0x80738B70 - 0x8073BE10 (12960)
BSS:        0x8073BE10 - 0x8074CDE0 (69584)
Heap:       0x8074CDE0 - 0x8094CDE0 (2097152)
Stack:      0x8094CDE0 - 0x8094EDE0 (8192)
Text:       0x80700000 - 0x80738B70 (232304)

Null Rescue Flag.
boot the image...
Invalid boot block on disk
Invalid boot block on disk
Check 2 trx result: -31, -31
Hello!! Enter Rescue Mode: (Check error)

Reading :: TFTP Server.
Failed.: Error
Loader:raw Filesys:raw Dev:nflash0.os File: Options:(null)
Loading: raw_fileop_uninit: warning: refcnt not zero
Entry at 0x00000000
CFE>

 

[RMerlin]

If there's some corruption at the nvram level then it can fail to initialize the Ethernet interface. Try wiping out nvram, then reboot and try again.

nvram erase

 

[donbowman]

So i got mostly back.
i manually erased the nvram, then did a 'strings' on the cfe binary, took all the fields, changed the mac address approp, and did an nvram set on each.
So this got my cfe back and willing to talk tftp.
i reflashed the flash1.trx.
i boot, and it comes up, but i'm missing eth1 and has reset the pci/2/1/macaddr to XX:XX:XX:XX:XX:XX. Its set correctly in the cfe env, but gets reset on bootup.

This is a 1.30 router running 1.0.1.4 cfe.

I'm sure how to correct this. There were 147 lines in the cfe binary for the 'fallback' nvram.

I'm still doing some experiments, but it seems that the 380_58 behaves different than 378_56_2, the latter seems to have come up once correctly.

Early in its life this went through the 32k/64k change, i wonder if that's hurting its recovery.

if i do a mtd-erase -d nvram and then reset, the kernel panics.
if i do a nvram erase; powercycle the kernel panics.

if i then set my nvram in line by line through cfe, we're good again, except that eth1 is MIA when the kernel boots.
Attached is what i am putting in for my nvram (these are the compiled in defaults of my cfe, w/ the MAC set in each spot).

anyone else have a 1.30 version, 1.0.1.4 cfe ac66 who could share their 'nvram show' so i can check what i'm missing? I assumed it was the pci/2/1 lines.

In the cfe, if i do nvram get pci/2/1/macaddr, I get AC:22:0B:CE:69:24.
If i then let the kernel boot, it is now reset to XX:XX:XX:XX:XX:XX. And now going back to cfe and its XX:XX again.

so booting the firmware is eating that line.

e.g. in cfe:

nvram get pci/2/1/macaddr ==> good
go
<wait for firmware boot>
nvram get pci/2/1/macaddr ==> bad

in cfe:
nvram set pci/2/1/macaddr
nvram commit
reboot to cfe
nvram get ==> good

so it is definitely the linux boot that is eating the field.

 

[bbsc]

My 5 cents:
Since some FW version some variables are taken from CFE, not nvram. So you can try to flash an older FW, say 374.43. You'll get access to the router and will be able to reflash CFE and upgrade FW later.

Looks like your CFE is modified but not successfully.
The most radical solution is to unsolder the SPI chip with nvram ans CFE, to reflash it with a programmer (only CFE is needed) and to solder it back.

I have 1.30 router with 1.0.1.4 CFE as you asked.

 

[donbowman]

i have access to the router, i can flash it ok
[i also have spi programmer for jtag port but not needed here].

the cfe has the default vars in it ok, but something is happening as it books that it undoes those.

i don't support u could post or PM or email me the output of 'nvram show'? Then I can hunt for what i'm missing from my flash partition of 'nvram'

 

[bbsc]

Ok, I'll try to explain once more.

The practical side of the problem.

1. Q. Why does my router not boot and kernel panics?
A. Because kernel gets a wrong variable value from CFE.

2. Q. What should I do to get it working?
A. You have access to the router an it makes things more easy.
Get current CFE with cat /dev/mtd0 > cfe.old.
Edit cfe.old to correct the wrong values.
Save as cfe.new.
Flash with mtd-write -i cfe.new -d pmon.
Clear nvram with mtd-erase -d nvram.
Switch the router off and on. It should boot normally.
Editing can be done with any hex editor or a tool taken from this topic.

The theoretical side of the problem (optional).

3. Q. The CFE fallback defaults seem to be Ok in my current CFE.
A. No, pci/2/1/macaddr=00:00:00:01:02:02 is not the right value. At least.

4. Q. Where does the kernel gets variable values from?
A. From nvram and CFE. Beginning from some FW version several values are taken form CFE omitting nvram.

5. Q. Is nvram necessary for kernel to boot?
A. No. All values will be taken form CFE. Nvram space may be blank at all. Nothing. Just blank space.

6. Q. Why does the kernel take pci/2/1/macaddr form the CFE if I've set it in nvram?
A. I don't know. Either it considers nvram as corrupt or see p.4.

7. Q. Why did it work before?
A. I don't know. Maybe you did not restore factory defaults for a long time. Or maybe the newer FW kernels check more values now.

8. Q. How have the wrong values get there into my current CFE?
A. It happened when updating CFE. You've wrote in this topic that you was using CFE v. 1.0.1.1.
Now you're using 1.0.1.4.
Asus routers do not update their CFE's by their selves, somebody had to do it manually.
Everything was Ok, but he forgot to correct one value at least.
Nothing special, to err is human.