The RT-N66U from the box has to be connected by a network cable to your computer. When you power up the router, the computer usually gets automatically an IP address from the router.
Open your browser and at the address bar type: 192.168.1.1 <hit ENTER>
Logon information to the router is Username: admin, Password: admin.
The router setup wizzard will guide you through the most important parts.
The router tries to automatically connect to the internet (it tries different "standards" ways to access your modem, most times it will succeed, depending of the connection type it may ask for the credentials for your ISP).
It will further ask for a new Admin password, create one at your own wishes (use 8 alpha-numeric characters, Aa-Zz, 0-9, no spaces and odd characters).
It will ask for your wireless ID's and authentication method. Set a different SSID for 2.4 GHz and 5 GHz, so you know to what frequency you connect a client.
Set wireless authentication method to WPA2 Personal and create a Pre Shared Key code (again, use 8 alpha-numeric characters, Aa-Zz, 0-9, no spaces and odd characters), the code for 2.4 GHz and 5 GHz can be kept the same for your convenience.
Anything else is basically optional, totally depending of your own unique situation.
I suggest to start like this and see what it brings.
For the wireless connection you need to connect to either your 2.4 GHz or 5 GHz channel, by the SSID as you have created and your authentication code.
For wired connections, no further setup is usually required.
After the setup wizzard, the standard configuration pages of the router appear, which I would not touch at this point.
According different sources you may need additional settings for the XBox (I have no experience with that box), I know the PS3 here at home works well without any special settings in the router.
For Merlin firmware the above applies, with more special features.