Unfortunately, the only correct answer to this is: it depends.
My general advice:
- Factory default reset when going from Merlin to stock Asus
- Factory default when going to a much older build (2-3 revisions older in general)
- Factory default when going from Asus to Merlin AND there is a large version jump (for instance, going from a one year old Asus to the latest version of my firmware)
Otherwise, try it, and be prepared to do a factory default reset IF you notice anything unusual. Personally, I try to implement nvram upgrade code (Asus does too) when an nvram format has changed in a new version, but that will obviously not work when downgrading. And there are cases where there's some low-level wireless change that, while things might still work fine with your older settings, they won't run as optimally as they would after you do a factory default reset to fully apply those new values.
I do factory default resets maybe once a year on average on my own routers, and that's usually if I do encounter something unusual.
The only sure rule there is: try it, and if something is odd, do a factory default reset.