As long the Broadcom wireless drivers will remain closed-source, you will always have very limited control over whatever happens in that black box I'm afraid.I think that the wireless drivers in the router can only accept certain values, and has a number of preset values set. So if you type in 5mW, the driver will set the lowest value it can, which I think is 50mW.
It's one of the reasons it has been requested to have a real GUI for this type of thing, instead of the magical mystery tour numeric input we have now.
One thing I will say, is that it may not be a good idea to run the router without antennas attached. I don't think it would be good for the power amps, but this needs answering by an electrical engineer, or real RF expert.