Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

Asuswrt-Merlin - Project update

Discussion in 'Asuswrt-Merlin' started by RMerlin, Sep 4, 2017.

  1. RMerlin

    RMerlin Part of the Furniture

    Joined:
    Apr 14, 2012
    Messages:
    24,635
    Location:
    Canada
    The short version:
    Asuswrt-merlin is getting re-implemented on top of the GPL 382 codebase. Only the RT-AC86U will be supported initially, followed by other models as Asus migrates them to 382. Ultimately, all development will be done on the 382 codebase, and the 380 codebase will be abandoned.

    Some features will change or need to be abandoned. Some models might be dropped, unsure which yet.

    There is no timetable/ETA for any of these stages, as they depend on many factors, some of which outside of my control (Asus' own timetable in migrating to 382).


    And the details:

    Introduction
    As many of you know by now, Asus has spent the past 18+ months working on a major update to Asuswrt, usually referred to as "382" (the current version used by most routers is 3.0.0.4.380_xxxx). The GT-AC5300 was the first router released based on this new code base, followed by the RT-AC86U.

    Asus intends to gradually upgrade existing models to this new 382 codebase. The first Broadcom models to get the upgrade so far are the RT-AC88U and RT-AC3100. Others will progressively follow over the coming months, as this new code base gains in maturity (so far, these first AC88/AC3100 builds seem to have a number of issues).


    Challenges introduced with it

    GPL 382 contains about 18 months of parallel development from Asus. There's a lot of changes in it, making it more difficult to merge than past GPL releases.

    I started studying the 382 codebase these past few weeks, migrating a few webui-related features to Asuswrt-Merlin 380.68. After receiving a "native" 382 device to work with (an RT-AC86U provided by Asus), I started experimenting with the code itself, and decided to try something different: rather than attempt to merge the 382 GPL on top of Asuswrt-Merlin (like I used to do in the past), I decided to try the other way around: starting with a clean GPL 382 code base, and merging Asuswrt-Merlin's changes on top of it. So far, this method has shown good promises, as the bulk of Asuswrt-Merlin's changes to the core components (httpd, shared and rc) have now been ported on top of 382. This is still a long and difficult process, and much still remains to be done. Some features will have to be abandoned, others will need to be reworked. But so far things are progressing well.


    The plan: project AM382
    AM382 is the "codename" by which I refer to this new Asuswrt-Merlin 382.xx codebase. (the official name will remain Asuswrt-Merlin, this is just to more easily distinguish it from the existing codebase, which I call AM380).

    I have begun the work of porting AM380's code on top of a fresh new 382 code repository. During this early development phase the new repo will remain private, as I occasionally have to rewrite previous code commits to ensure that once the code goes public, we have a clean foundation to build upon. The repo will be made public once the code reaches a more advanced stage.

    During this initial work, I started discovering which parts of Asuswrt-Merlin would have to be abandoned. For instance, some of the enhancements made to WAN routing will have to be abandoned (that code is now closed source). IPTraffic will probably not be possible for the RT-AC86U (due to its newer kernel, IPTraffic relies on an old kernel module). If you're a kernel hacker and feel interested in porting ipt_account to kernel 4.1, be my guests.

    Also, OpenVPN development will now fork away from Asus's own OpenVPN support, as with 382 they moved their code into a closed-source library. This means that future developments made by Asus for OpenVPN won't be available in Asuswrt-Merlin, however since our current OpenVPN implementation is already more advanced (offering OpenVPN 2.4 support for instance), I believe this is the best course of action.

    While the initial focus will be on getting a first usable build of AM382 (which will become Asuswrt-Merlin 382.1), very little development will be done on the AM380 codebase (this being a one-man project.) Development will pick up on AM380 after I have a first usable beta build of AM382 for the RT-AC86U.

    The next step will be to merge the first models upgraded by Asus (RT-AC88U and RT-AC3100).

    At that point, development time will be split between AM382 (AC88, AC3100 and AC86) and AM380 (all existing models). Once the majority of models have been migrated to AM382, development on AM380 will gradually slow down, until all development is made only on AM382.


    Model support
    AM382 will initially support only the RT-AC86U. (for now I have no plan to support the GT-AC5300 - too many unique features to deal with). The RT-AC88U and RT-AC3100 will be the first two models added to AM382, followed by other models as Asus releases 382 firmwares for them.

    I haven't made any final decision yet as to whether I will drop any model, and if I do, which ones will be dropped. It will depend in part on how frequently Asus makes new releases for them on the AM382 codebase (as each new GPL release will require new binary blobs for every single model I wish to compile the code for, this means I need frequent new GPL for every model I want to actively support). Right now, what I am considering is to abandon the MIPS platform (that means the RT-N66U and RT-AC66U). If I go ahead with that, it will be for the following reasons:

    1) Dropping the MIPS architecture will make a lot of things easier for me, those older models with their older kernel are already starting to fall behind ARM models in terms of features (no ipset v6, no fq_codel, etc...).

    2) Those models wouldn't be entirely abandoned, as @john9527 is very actively developing for these two models. While his firmware does not have exactly the same feature set as mine, it's generally rather close for these two specific models. It also already offers better wifi performance for these models (especially the RT-N66U).

    3) I suspect that Asus might not release new 382 GPLs as often for these two older models as they do for newer models, which means that quite often, I would be unable to make new firmware releases for these models as their GPL would be out-of-sync with newer models.

    4) Having these models on a separate code base actually opens the door to occasional bugfix releases, or having someone fork the project and taking over maintenance of these older models.

    In any case, I won't be making any final decision however until I get a clearer picture as to what Asus's own support will be like concerning 382 for all the models I currently support.


    Conclusion
    Having two separate codebases will allow the transition to 382 to be done in stages, and the current AM382 development method is providing good results in terms of merging things together. This plan will allow the project to continue, with a few changes to its statement (feature parity with stock firmware will no longer be ensured, namely for OpenVPN support at this time), and some of the older models being possibly dropped. Doing it this way will also ensure the project can remain manageable for one single developer.
     
    pepemosca, il2, TurboFreak68 and 50 others like this.
  2. jrmwvu04

    jrmwvu04 Senior Member

    Joined:
    Mar 29, 2016
    Messages:
    252
    Not that you were asking, but it seems like a set of reasonable decisions to me. You can't really expect to move mountains if the Asus plan isnt going to be conducive to keeping things as they were. Your hobby has a pretty huge userbase at this point, but you don't owe anything to any of us. I think you're quite aware of that but sometimes it helps to hear someone else say it, so I did.
     
  3. Vexira

    Vexira Very Senior Member

    Joined:
    Jan 20, 2017
    Messages:
    860
    Location:
    Australia
    Does that mean the WAN rules you added in 380.66 that allow for multiple console support will be gone completely and impossible to re implement?
     
  4. RMerlin

    RMerlin Part of the Furniture

    Joined:
    Apr 14, 2012
    Messages:
    24,635
    Location:
    Canada
    Rooter and Ryansr like this.
  5. Vexira

    Vexira Very Senior Member

    Joined:
    Jan 20, 2017
    Messages:
    860
    Location:
    Australia
    Oh awesome it still works, but it's saddens me that the fix will be missing for those who need it, I hope asus opens up that part of the code again, but if they don't I hope they apply the fix.
    Also may I ask what rule you implemented in ip tables I'm considering testing to see if I can use one to fix gta v and battle born.
     
  6. fearz

    fearz Regular Contributor

    Joined:
    Jun 20, 2012
    Messages:
    66
    Thanks a lot for your hard work, can you please shed some light on the 382 update? What’s so major about it?


    Sent from my iPhone using Tapatalk
     
  7. Vexira

    Vexira Very Senior Member

    Joined:
    Jan 20, 2017
    Messages:
    860
    Location:
    Australia
    new major set of code, like a major update tot he firmware it self.
     
  8. RMerlin

    RMerlin Part of the Furniture

    Joined:
    Apr 14, 2012
    Messages:
    24,635
    Location:
    Canada
    Asus started working on that code roughly 18 months ago (from what I can deduce, might have been even longer than that). Following the FTC settlement, they spent some time reviewing various parts of the code to better secure it.

    382 also adds support for a new SDK, which is used by the GT-AC5300 and RT-AC86U - this new SDK is radically different from past SDKs. That's in addition to the various Qualcomm devices that were also added to the core code, and the Lantiq/Quantenna devices that are also still under development.

    382 adds a lot of new features (not all are available for every models): IPSEC, Notification Center,, a new Trend Micro engine (with more advanced IPS), VLAN support (for the BRT-AC828 only at this time), VPNFusion, Protection Service (which was partly backported to 380). Probably a few others I'm forgetting right now.

    The code that generates the webui menu was redesigned, using a more modular approach over the messy ball of hacks that they were using before.

    Both the networkmap and OpenVPN code saw a lot of changes, and were made closed-source. Various other portions were also moved to closed-source binaries throughout the firmware.

    And there's all the code related to the Lyra that was added throughout the firmware, to handle config sync between devices among other things.

    And there are a number of new features that seem to still be under development: Let's Encrypt support, SSL certificate management, AMAS (see the public beta in the Official Asuswrt forum here). I also see some traces of nvram encryption that might be another work-in-progress.

    Even the long list of default nvram settings were rewritten, using a new data format that seem to indicate Broadcom might eventually implement lowlevel validation of value length and encryption. Just updating the structure of this list with my own additions took me nearly two hours.

    So it's a lot of changes throughout the whole firmware.
     
    Rooter, MacG32, ChriscomIT and 2 others like this.
  9. RMerlin

    RMerlin Part of the Furniture

    Joined:
    Apr 14, 2012
    Messages:
    24,635
    Location:
    Canada
    To compare, a normal GPL merge contains roughly 1-2 months of work, and can take me roughly two nights to merge in (provided there's no usual major change in it).
     
    Rooter likes this.
  10. thelonelycoder

    thelonelycoder Part of the Furniture

    Joined:
    Jan 23, 2014
    Messages:
    3,349
    Location:
    Switzerland
    Let's Encrypt will be very helpful and welcome but I fear it will be bound to the routers IP and domain.
    Let's hope the code is reusable for another IP/Domain e.g. pixelserv-tls.
    What I'm slightly worried about is Nvram encryption. Here I hope only encryption-worthy entries fall in this category.
    Or the decryption code is accessible to scripts.

    Overall I am very happy you continue your project despite the hurdles and challenges.
    Will the initial AM382 support the same models as 380? My RT-AC1900P would jump out of happiness.
     
    Rooter likes this.
  11. martinr

    martinr Very Senior Member

    Joined:
    Nov 27, 2014
    Messages:
    1,020
    Location:
    United Kingdom
    @RMerlin But this is a massive undertaking: getting AM382 up and running and then dealing with all the posts it will generate will be huge, and won't be accomplished overnight. You surely you won't try to maintain your present level of attention to the existing project and all the posts it generates. That would be impossible, surely, and your good health takes precedence over everything. We all had a taster earlier this year of what might happen should your health suffer. Nobody wants that on their conscience, and it's clear where the future lies.
     
    Last edited: Sep 5, 2017
  12. JDB

    JDB Senior Member

    Joined:
    Aug 28, 2016
    Messages:
    224
    Thanks for the detailed update Merlin. Amazing commitment as always.
    I'm assuming option 60/61 on the WAN won't be lost as a result of the closed source routing?

    Either way, I'm going to go make another donation to you as it's the only real way I have to help and sounds like you'll need a couple of beers during this process!


    Sent from my iPhone using Tapatalk
     
  13. ColinTaylor

    ColinTaylor Part of the Furniture

    Joined:
    Mar 31, 2014
    Messages:
    3,888
    Location:
    UK
    Already answered in post #1 under Model support.
     
  14. thelonelycoder

    thelonelycoder Part of the Furniture

    Joined:
    Jan 23, 2014
    Messages:
    3,349
    Location:
    Switzerland
    Argh, my brain read 68U, whitch is also the firmware for the 1900P.
    Too bad.
     
  15. David B

    David B Regular Contributor

    Joined:
    May 2, 2017
    Messages:
    56
    I have a question about this:
    What exactly has Asus made closed-source about their implementation of OpenVPN?

    By the way, thanks for the great work @RMerlin .
     
  16. RMerlin

    RMerlin Part of the Furniture

    Joined:
    Apr 14, 2012
    Messages:
    24,635
    Location:
    Canada
    Everything that manages nvram settings related to OpenVPN, generate the client and server config files, start and stop OpenVPN. Basically all the functionality previously found in rc/openvpn.c . The code was moved to a closed-source library.

    What worries me more is the move of networkmap to a closed source component. Over the years, networkmap is probably the piece of code where I've fixed the largest amount of bugs and security issues over the years. Now, we'll have to rely 100% on Asus to maintain that piece of code.
     
    Last edited: Sep 5, 2017
    Rooter and Andrewski like this.
  17. RMerlin

    RMerlin Part of the Furniture

    Joined:
    Apr 14, 2012
    Messages:
    24,635
    Location:
    Canada
    It's not affected, these are parameters passed to udhcpc, that code remains intact. It's the code that creates routes for the WAN interface that was closed, probably because they wanted to close parts of their Dual WAN code to prevent people from running that code on non-Asus products...
     
    JDB likes this.
  18. RMerlin

    RMerlin Part of the Furniture

    Joined:
    Apr 14, 2012
    Messages:
    24,635
    Location:
    Canada
    The webui code for it was added to the DDNS page, which sound like an odd location to me (I added my own SSL management code to the Administration page instead, next to the other httpd settings).
     
    Rooter likes this.
  19. David B

    David B Regular Contributor

    Joined:
    May 2, 2017
    Messages:
    56
    I guess the folks at Asus didn't appreciate that you maintained it better than they did. Either way, is it possible for you to maintain networkmap separately in the same way that you plan to maintain OpenVPN?
     
  20. mstombs

    mstombs Very Senior Member

    Joined:
    Jul 18, 2012
    Messages:
    534
    All very clear and sensible, I find it hard to believe Asus also haven't moved the MIPS based routers into a security only 'care and maintenance' role, they have kept updating for a very long time compared to other manufacturers, I hope they appreciate the number of customer recommendations that this buys! I am not a big fan of their move to more closed source or the use of 3rd party to do the web protection - wouldn't be surprised that is at no cost to Asus, Trend-Micro get the big data! Suspect a lot of pressure to make Asus proprietary firmware unusable on other manufacturers hardware, I hope they are being careful with the strict interpretations of the various GPL licences, but I'm not sure it is still in the spirit...
     

Share This Page