What's new

Wireguard Session Manager - Discussion thread (CLOSED/EXPIRED Oct 2021 use http://www.snbforums.com/threads/session-manager-discussion-2nd-thread.75129/)

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

Hello,

Wondering how to enterpret the stats (which by the way have not produced any negative numbers!)
For example 2 consecutive statistics:
Code:
Jun 21 18:59:00 RT-AC86U-D7D8 (wg_manager.sh): 676 Clients [97m2[95m, Servers [97m0
Jun 21 18:59:01 RT-AC86U-D7D8 (wg_manager.sh): 676 wg11:[97m transfer: 13.41 GiB received, 688.76 MiB sent        [97m2 days 08:54:10 from 2021-06-19 10:04:51 >>>>>>[0m
Jun 21 18:59:01 RT-AC86U-D7D8 (wg_manager.sh): 676 wg11: period : 6.41 GiB received, 311.53 MiB sent (Rx=6885967133;Tx=326662880)
Jun 21 18:59:02 RT-AC86U-D7D8 (wg_manager.sh): 676 wg12:[97m transfer: 373.07 MiB received, 14.10 MiB sent        [97m2 days 08:54:09 from 2021-06-19 10:04:53 >>>>>>[0m
Jun 21 18:59:02 RT-AC86U-D7D8 (wg_manager.sh): 676 wg12: period : 107.78 MiB received, 7.46 MiB sent (Rx=113020599;Tx=7825491)
Jun 21 19:59:00 RT-AC86U-D7D8 (wg_manager.sh): 3564 Clients [97m2[95m, Servers [97m0
Jun 21 19:59:01 RT-AC86U-D7D8 (wg_manager.sh): 3564 wg11:[97m transfer: 13.44 GiB received, 691.04 MiB sent        [97m2 days 09:54:10 from 2021-06-19 10:04:51 >>>>>>[0m
Jun 21 19:59:01 RT-AC86U-D7D8 (wg_manager.sh): 3564 wg11: period : 7.03 GiB received, 379.51 MiB sent (Rx=7545122982;Tx=397945079)
Jun 21 19:59:02 RT-AC86U-D7D8 (wg_manager.sh): 3564 wg12:[97m transfer: 373.08 MiB received, 14.10 MiB sent        [97m2 days 09:54:09 from 2021-06-19 10:04:53 >>>>>>[0m
Jun 21 19:59:02 RT-AC86U-D7D8 (wg_manager.sh): 3564 wg12: period : 265.30 MiB received, 6.64 MiB sent (Rx=278182135;Tx=6959431)

What is the relation between transfer and period, I thoughts period was since last stats but wg11 above is
1: 13.41 GiB received - Period: 6.41 GiB
2: 13.44 GiB received - Period: 7.03 GiB

How come the Period numbers increase faster than transfer... or could someone explain the mechanism. Or are there some arithmetic/logic error?

//Zeb

Edit: aah, get it... 13.44-6.41=7.03 only delta values stored in database so period = transfer - prevDelta.

@ZebMcKayhan

I've uploaded wg_manager v4.11b7 with a patch to correct the traffic metrics.

You need to manually issue the following commands:
Code:
sqlite3 /opt/etc/wireguard.d/WireGuard.db "ALTER TABLE traffic ADD COLUMN rxtotal UNSIGNED BIG INT;" 
sqlite3 /opt/etc/wireguard.d/WireGuard.db "ALTER TABLE traffic ADD COLUMN txtotal UNSIGNED BIG INT;"
then apply the patch
Code:
e  = Exit Script [?]

E:Option ==> uf dev
then restart ALL active 'client' Peers

You may check the corrected metrics by manually requesting the stats (rather than wait until the scheduled cron job executes 'xx:59')
Code:
e  = Exit Script [?]

E:Option ==> generatestats
and
Code:
e  = Exit Script [?]

E:Option ==> diag sql traffic

    DEBUG: SQL '/opt/etc/wireguard.d/WireGuard.db'

    Table:traffic
Peer    Timestamp            RX           TX          RX Total     TX Total
<snip>
wg11    2021-06-22 19:15:57  *            *           *            *
wg11    2021-06-22 19:16:14  0            0           0            0
wg11    2021-06-22 19:16:33  52132        18319       52132        18319
wg11    2021-06-22 19:17:22  25610        61604       77742        79923
wg11    2021-06-22 19:29:36  1841152      645049      1918894      724972
wg11    2021-06-22 19:31:11  115343       41809       2034237      766781
wg11    2021-06-22 19:38:14  2348811      1686887     4383048      2453668
where the first '* * * *' record denotes the 'client' Peer was terminated and the second '0 0 0 0' record denotes the 'client' Peer was initialised.

So the 3rd record shows the metrics produced by the first manually executed generatestats command, and the 4th record shows the results of the second generatestats command

Hopefully it can be seen that the RX Total (77742) matches the sum of the two previous 'period/interval' RX values (52132 + 25610)
 
@ZebMcKayhan

I've uploaded wg_manager v4.11b7 with a patch to correct the traffic metrics.

You need to manually issue the following commands:
Code:
sqlite3 /opt/etc/wireguard.d/WireGuard.db "ALTER TABLE traffic ADD COLUMN rxtotal UNSIGNED BIG INT;"
sqlite3 /opt/etc/wireguard.d/WireGuard.db "ALTER TABLE traffic ADD COLUMN txtotal UNSIGNED BIG INT;"
then apply the patch
Code:
e  = Exit Script [?]

E:Option ==> uf dev
then restart ALL active 'client' Peers

You may check the corrected metrics by manually requesting the stats (rather than wait until the scheduled cron job executes 'xx:59')
Code:
e  = Exit Script [?]

E:Option ==> generatestats
and
Code:
e  = Exit Script [?]

E:Option ==> diag sql traffic

    DEBUG: SQL '/opt/etc/wireguard.d/WireGuard.db'

    Table:traffic
Peer    Timestamp            RX           TX          RX Total     TX Total
<snip>
wg11    2021-06-22 19:15:57  *            *           *            *
wg11    2021-06-22 19:16:14  0            0           0            0
wg11    2021-06-22 19:16:33  52132        18319       52132        18319
wg11    2021-06-22 19:17:22  25610        61604       77742        79923
wg11    2021-06-22 19:29:36  1841152      645049      1918894      724972
wg11    2021-06-22 19:31:11  115343       41809       2034237      766781
wg11    2021-06-22 19:38:14  2348811      1686887     4383048      2453668
where the first '* * * *' record denotes the 'client' Peer was terminated and the second '0 0 0 0' record denotes the 'client' Peer was initialised.

So the 3rd record shows the metrics produced by the first manually executed generatestats command, and the 4th record shows the results of the second generatestats command

Hopefully it can be seen that the RX Total (77742) matches the sum of the two previous 'period/interval' RX values (52132 + 25610)
Wow! Thanks for the update.
I got mostly errors when issuing your commands so I did:
Code:
diag sqlX
ALTER TABLE traffic ADD COLUMN rxtotal UNSIGNED BIG INT;
ALTER TABLE traffic ADD COLUMN txtotal UNSIGNED BIG INT;
.quit
I read the database to make sure the 2 columns have been created, which they had, and they were of type BIG int.

So I restarted the peers using "6" but again, got error messages.

Tried to generate stats, got error messages.

At this time syslog looked like:
Code:
Jun 22 21:19:20 RT-AC86U-D7D8 (wg_manager.sh): 27478 wg11:[97m transfer: 14.03 KiB received, 9.13 KiB sent        [97m0 Days, 00:00:26 from 2021-06-22 21:18:54 >>>>>>[0m
Jun 22 21:19:20 RT-AC86U-D7D8 (wg_manager.sh): 27478 wg11: period : N/A received, N/A sent (Rx=;Tx=)
Jun 22 21:19:20 RT-AC86U-D7D8 (wg_manager.sh): 27478 wg12:[97m transfer: 92 B received, 180 B sent            [97m0 Days, 00:00:25 from 2021-06-22 21:18:55 >>>>>>[0m
Jun 22 21:19:20 RT-AC86U-D7D8 (wg_manager.sh): 27478 wg12: period : N/A received, N/A sent (Rx=;Tx=)
Jun 22 21:22:04 RT-AC86U-D7D8 (wg_manager.sh): 2409 wg11:[97m transfer: 542.29 KiB received, 230.10 KiB sent        [97m0 Days, 00:03:10 from 2021-06-22 21:18:54 >>>>>>[0m
Jun 22 21:22:04 RT-AC86U-D7D8 (wg_manager.sh): 2409 wg11: period : N/A received, N/A sent (Rx=;Tx=)
Jun 22 21:22:04 RT-AC86U-D7D8 (wg_manager.sh): 2409 wg12:[97m transfer: 184 B received, 584 B sent            [97m0 Days, 00:03:09 from 2021-06-22 21:18:55 >>>>>>[0m
Jun 22 21:22:04 RT-AC86U-D7D8 (wg_manager.sh): 2409 wg12: period : N/A received, N/A sent (Rx=;Tx=)

Stopped the peers using "5"
Started the peers using "4"

Some error messages during these commands (same as before).

"generatestats" and no error messages! After this all data seemed to be fine!

Sorry, didn't snip the errors and now they don't appear anymore.

Syslog:
Code:
Jun 22 21:25:15 RT-AC86U-D7D8 (wg_manager.sh): 5159 wg11:[97m transfer: 26.54 MiB received, 733.25 KiB sent        [97m0 Days, 00:01:08 from 2021-06-22 21:24:07 >>>>>>[0m
Jun 22 21:25:15 RT-AC86U-D7D8 (wg_manager.sh): 5159 wg11: period : 26.54 MiB received, 733.25 KiB sent (Rx=27829207;Tx=750848)
Jun 22 21:25:15 RT-AC86U-D7D8 (wg_manager.sh): 5159 wg12:[97m transfer: 92 B received, 244 B sent            [97m0 Days, 00:01:08 from 2021-06-22 21:24:07 >>>>>>[0m
Jun 22 21:25:15 RT-AC86U-D7D8 (wg_manager.sh): 5159 wg12: period : 92 Bytes received, 244 Bytes sent (Rx=92;Tx=244)
Jun 22 21:47:09 RT-AC86U-D7D8 (wg_manager.sh): 26257 wg11:[97m transfer: 181.95 MiB received, 6.10 MiB sent        [97m0 Days, 00:23:02 from 2021-06-22 21:24:07 >>>>>>[0m
Jun 22 21:47:10 RT-AC86U-D7D8 (wg_manager.sh): 26257 wg11: period : 155.41 MiB received, 5.38 MiB sent (Rx=162959196;Tx=5645466)
Jun 22 21:47:10 RT-AC86U-D7D8 (wg_manager.sh): 26257 wg12:[97m transfer: 1.08 KiB received, 3.83 KiB sent        [97m0 Days, 00:23:03 from 2021-06-22 21:24:07 >>>>>>[0m
Jun 22 21:47:10 RT-AC86U-D7D8 (wg_manager.sh): 26257 wg12: period : 1014 Bytes received, 3.59 KiB sent (Rx=1014;Tx=3678)
Jun 22 21:59:01 RT-AC86U-D7D8 (wg_manager.sh): 6993 Clients [97m2[95m, Servers [97m0
Jun 22 21:59:01 RT-AC86U-D7D8 (wg_manager.sh): 6993 wg11:[97m transfer: 243.53 MiB received, 9.57 MiB sent        [97m0 Days, 00:34:54 from 2021-06-22 21:24:07 >>>>>>[0m
Jun 22 21:59:01 RT-AC86U-D7D8 (wg_manager.sh): 6993 wg11: period : 61.58 MiB received, 3.47 MiB sent (Rx=64571310;Tx=3638558)
Jun 22 21:59:02 RT-AC86U-D7D8 (wg_manager.sh): 6993 wg12:[97m transfer: 1.53 KiB received, 5.58 KiB sent        [97m0 Days, 00:34:55 from 2021-06-22 21:24:07 >>>>>>[0m
Jun 22 21:59:02 RT-AC86U-D7D8 (wg_manager.sh): 6993 wg12: period : 461 Bytes received, 1.75 KiB sent (Rx=461;Tx=1792)

And:
wg11 2021-06-22 20:59:01 8694026734 468168215
wg12 2021-06-22 20:59:02 278192621 7085258
wg11 2021-06-22 21:18:54 * * * *
wg11 2021-06-22 21:18:54 0 0 0 0
wg12 2021-06-22 21:18:55 * * * *
wg12 2021-06-22 21:18:55 0 0 0 0
wg11 2021-06-22 21:24:00 * * * *
wg12 2021-06-22 21:24:01 * * * *
wg11 2021-06-22 21:24:07 0 0 0 0
wg12 2021-06-22 21:24:07 0 0 0 0
wg11 2021-06-22 21:25:14 27829207 750848 27829207 750848
wg12 2021-06-22 21:25:15 92 244 92 244
wg11 2021-06-22 21:47:09 162959196 5645466 190788403 6396314
wg12 2021-06-22 21:47:10 1014 3678 1106 3922
wg11 2021-06-22 21:59:01 64571310 3638558 255359713 10034872
wg12 2021-06-22 21:59:02 461 1792 1567 5714

Wonder if it would make sense to issue
Code:
drop table traffic;
drop table session;
.quit
Initdb keep

To start over and maybee not get as many errors.

Anyway, altough the migration may not be perfect the changes appears to be working and that's what is important!

//Zeb

Edit: just realized you probably meant issuing:
Code:
sqlite3 /opt/etc/wireguard.d/WireGuard.db "ALTER TABLE traffic ADD COLUMN rxtotal UNSIGNED BIG INT;" 
sqlite3 /opt/etc/wireguard.d/WireGuard.db "ALTER TABLE traffic ADD COLUMN txtotal UNSIGNED BIG INT;"
At the prompt. I run this in wgm. Probably why these were not working. Should have spotted that.
 
Last edited:
just added another vpn client with another country output, wg12. I figure I'll try without doing anything to the config file from my supplier. I couldnt believe it when it just connected.
further, I could'nt really wrap my head around how to make the DNS work with 3 different outbound connections (with unbound stuck in the middle, bound to one vpn client)... well, @Martineau was way ahead of me.

Using Yazfi to make one of the guest ssid: "SurfinUSA" using DNS to router. obviously custom config to allow wl1.1 to wg12 (guessing it is only a matter of time until Yazfi supports wireguard). the only custom config needed at wireguard_manager side was actually to masquarade this subnet as it was leaving wg12.

Thank you @Martineau for making this possible!

//Zeb

(ps. now I only need to figure out why I would ever need to surf-the-usa)
Thanks to your post, just realized I can create another client wg12 to peer to a different NordVPN server. Didn’t know this is possible as both clients have the same private ip peers.
I have setup both clients in policy mode. Are you able to divert unbound to any of the wg clients in policy mode?
 
Are you able to divert unbound to any of the wg clients in policy mode?
Yes, my "ordinary" LAN 192.168.1.x is routed out wg11 in policy mode.

if I remember correctly I changed this line in unbound.conf:
Code:
#outgoing-interface: xxx.xxx.xxx.xxx        # v1.08 Martineau Use VPN tunnel to hide Root server queries from ISP (or force WAN ONLY)
to
Code:
outgoing-interface: 192.168.1.1        # v1.08 Martineau Use VPN tunnel to hide Root server queries from ISP (or force WAN ONLY)

without really knowing the mechanisms behind this I figured that if the source address of the packages sent will be this, then "ip route" will make sure they end up at wg11. wheither or not it is right it is working. according to dnsleaks.com my resolving adress is in fact wg11 public adress.

using multiple VPN's for different country to get the full experience we should resolve roughly from the same site (or atleast same continent). there is an interesting thread here: Unbound DNS VPN Client w/policy rules

//Zeb
 
Last edited:
I've uploaded wg_manager v4.11b7 with a patch to correct the traffic metrics.

You need to manually issue the following commands:
Code:
sqlite3 /opt/etc/wireguard.d/WireGuard.db "ALTER TABLE traffic ADD COLUMN rxtotal UNSIGNED BIG INT;"
sqlite3 /opt/etc/wireguard.d/WireGuard.db "ALTER TABLE traffic ADD COLUMN txtotal UNSIGNED BIG INT;"
then apply the patch
Code:
e  = Exit Script [?]

E:Option ==> uf dev
then restart ALL active 'client' Peers

You may check the corrected metrics by manually requesting the stats (rather than wait until the scheduled cron job executes 'xx:59')
Code:
e  = Exit Script [?]

E:Option ==> generatestats
and
Code:
e  = Exit Script [?]

E:Option ==> diag sql traffic

    DEBUG: SQL '/opt/etc/wireguard.d/WireGuard.db'

    Table:traffic
Peer    Timestamp            RX           TX          RX Total     TX Total
<snip>
wg11    2021-06-22 19:15:57  *            *           *            *
wg11    2021-06-22 19:16:14  0            0           0            0
wg11    2021-06-22 19:16:33  52132        18319       52132        18319
wg11    2021-06-22 19:17:22  25610        61604       77742        79923
wg11    2021-06-22 19:29:36  1841152      645049      1918894      724972
wg11    2021-06-22 19:31:11  115343       41809       2034237      766781
wg11    2021-06-22 19:38:14  2348811      1686887     4383048      2453668
where the first '* * * *' record denotes the 'client' Peer was terminated and the second '0 0 0 0' record denotes the 'client' Peer was initialised.

So the 3rd record shows the metrics produced by the first manually executed generatestats command, and the 4th record shows the results of the second generatestats command

Hopefully it can be seen that the RX Total (77742) matches the sum of the two previous 'period/interval' RX values (52132 + 25610)

The 'traffic' table columns addition and the forced update completed successfully. Here is the outcome after 2 generatestats commands:
Code:
wg11  2021-06-22 23:37:40  926720983    14048471    931345203   14669578
wg14  2021-06-22 23:37:40  214642863    8900188     214643507   8902410
wg12  2021-06-22 23:37:41  1348434      807895      1426063     875643
wg13  2021-06-22 23:37:41  7732         28621       8376        30843
wg11  2021-06-23 00:57:13  958440407    120837898   1889785610  135507476
wg14  2021-06-23 00:57:13  221868196    9625928     436511703   18528338
wg12  2021-06-23 00:57:14  398459       361677      1824522     1237320
wg13  2021-06-23 00:57:14  3492         12923       11868       43766

WireGuard ACTIVE Peer Status: Clients 4, Servers 1
The math part doesn't look as cut and dry as in the example above. However, I assume that some packets were in-flight and/or the mechanics of the algorithm may contribute. Everything looks though quite impressive.

Is there a reason for the server peer types to be left out?
 
Are you able to divert unbound to any of the wg clients in policy mode?
That raises a question that crossed my mind the other day. ...and I believe to know who may have an authoritative answer :)...

@Martineau - would you mind adding the wg option along VPN to unbound?

Code:
vpn = BIND unbound to VPN {vpnid [debug]} | [disable | debug show] e.g. vpn 1
 
Is there a reason for the server peer types to be left out?
wg_manager should report (i.e. writes records to the 'traffic' SQL table) for the individual Road-Warrior 'client' Peers connected to the 'server' Peer?

e.g.
Code:
e  = Exit Script [?]

E:Option ==> 3

    interface: wg21     Port:51820    10.50.1.1/24         VPN Tunnel Network    # RT-AC86U Server #1
        peer: ttkzrjxHIVyfn0B6xcHAGjX0QmODPWu/cYus001zjhmk=     10.50.1.2/32        # SGS8 "Device"    
         latest handshake: 17 minutes, 36 seconds ago
         transfer: 163.26 KiB received, 429.96 KiB sent        0 Days, 00:37:31 from 2021-06-23 08:24:35

     WireGuard ACTIVE Peer Status: Clients 0, Servers 1
and using multiple manual generatestats commands, results in Syslog records:
Code:
Jun 23 08:35:42 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: transfer: 49.43 KiB received, 178.50 KiB sent        0 Days, 00:11:07 from 2021-06-23 08:24:35
Jun 23 08:35:42 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: period : 49.43 KiB received, 178.50 KiB sent (Rx=50616;Tx=182784)
Jun 23 08:44:13 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: transfer: 72.43 KiB received, 258.58 KiB sent        0 Days, 00:19:38 from 2021-06-23 08:24:35
Jun 23 08:44:13 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: period : 23.00 KiB received, 80.08 KiB sent (Rx=23552;Tx=82002)
Jun 23 08:45:18 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: transfer: 73.42 KiB received, 264.94 KiB sent        0 Days, 00:20:43 from 2021-06-23 08:24:35
Jun 23 08:45:18 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: period : 1014 Bytes received, 6.36 KiB sent (Rx=1014;Tx=6513)
Jun 23 08:45:49 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: transfer: 163.23 KiB received, 412.68 KiB sent        0 Days, 00:21:14 from 2021-06-23 08:24:35
Jun 23 08:45:49 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: period : 89.81 KiB received, 147.74 KiB sent (Rx=91966;Tx=151285)
Jun 23 08:46:41 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: transfer: 163.26 KiB received, 415.95 KiB sent        0 Days, 00:22:06 from 2021-06-23 08:24:35
Jun 23 08:46:41 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: period : 30 Bytes received, 3.27 KiB sent (Rx=30;Tx=3349)
Jun 23 08:47:25 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: transfer: 163.26 KiB received, 422.88 KiB sent        0 Days, 00:22:50 from 2021-06-23 08:24:35
Jun 23 08:47:25 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: period : 0 Bytes received, 6.93 KiB sent (Rx=0;Tx=7096)
Jun 23 08:47:39 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: transfer: 163.26 KiB received, 423.31 KiB sent
Jun 23 08:52:54 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: transfer: 163.26 KiB received, 429.96 KiB sent        0 Days, 00:28:19 from 2021-06-23 08:24:35
Jun 23 08:52:54 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: period : 0 Bytes received, 6.65 KiB sent (Rx=0;Tx=6810)

<# disconnected Road-Warrior 'client' Peer SGS8>

Jun 23 08:53:15 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: transfer: 163.26 KiB received, 429.96 KiB sent        0 Days, 00:28:40 from 2021-06-23 08:24:35
Jun 23 08:53:15 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: period : 0 Bytes received, 0 Bytes sent (Rx=0;Tx=0)
Jun 23 08:55:26 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: transfer: 163.26 KiB received, 429.96 KiB sent
Jun 23 08:55:26 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: period : 0 Bytes received, 0 Bytes sent (Rx=0;Tx=0)

 <# then reconnected 20mins later.....>
    
Jun 23 09:16:37 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: transfer: 223.55 KiB received, 523.38 KiB sent        0 Days, 00:52:02 from 2021-06-23 08:24:35
Jun 23 09:16:37 RT-AC86U-6160 (wg_manager.sh): 25036 SGS8: period : 60.29 KiB received, 93.42 KiB sent (Rx=61737;Tx=95662)

.....but wg_manager currently does not auto-total them under their parent 'server' Peer, i.e. in the example above with only one Road-Warrior 'client' Peer connected, you can infer that 'SGS8' is actually 'wg21'

@ZebMcKayhan FYI, I will now explicitly report the stats to the console when manually requested to give GUI feedback that something actually happens (rather than silently only report them to Syslog)
Code:
e  = Exit Script [?]

E:Option ==> generatestats

    SGS8: transfer: 223.55 KiB received, 523.38 KiB sent        0 Days, 00:52:02 from 2021-06-23 08:24:35
    SGS8: period : 60.29 KiB received, 93.42 KiB sent (Rx=61737;Tx=95662)

     WireGuard ACTIVE Peer Status: Clients 0, Servers 1
 
@Martineau - would you mind adding the wg option along VPN to unbound?
Code:
vpn = BIND unbound to VPN {vpnid [debug]} | [disable | debug show] e.g. vpn 1
Whilst WireGuard may now have started to gain further adoption/momentum on the ASUS routers, I'm not sure if there is much demand for this esoteric unbound_manager feature to be enhanced in light of other methods to implement the desired functionality?

However, I'll certainly give it some consideration to allow either a 'wg1x' or a 'tun1x' interface to be explicitly specified (or intelligently auto-determined) when invoking the unbound_manager vpn bind command.
 
Last edited:
Yes, my "ordinary" LAN 192.168.1.x is routed out wg11 in policy mode.

if I remember correctly I changed this line in unbound.conf:
Code:
#outgoing-interface: xxx.xxx.xxx.xxx        # v1.08 Martineau Use VPN tunnel to hide Root server queries from ISP (or force WAN ONLY)
to
Code:
outgoing-interface: 192.168.1.1        # v1.08 Martineau Use VPN tunnel to hide Root server queries from ISP (or force WAN ONLY)

without really knowing the mechanisms behind this I figured that if the source address of the packages sent will be this, then "ip route" will make sure they end up at wg11. wheither or not it is right it is working. according to dnsleaks.com my resolving adress is in fact wg11 public adress.

using multiple VPN's for different country to get the full experience we should resolve roughly from the same site (or atleast same continent). there is an interesting thread here: Unbound DNS VPN Client w/policy rules

//Zeb
This changes alone does not work for me. I have edit a bit unbound_via_vc1 script from @Swinson to go along with this and it is working now. What I like is when I bounce wg11, unbound does not restart. With openvpn, bouncing openvpn client interface causes unbound to restart and all cache are cleared. Not sure why the longest single session I have with openvpn to NordVPN is about a week.

Update: There is no effect if I just edit outgoing-interface in unbound.conf.add alone. So I added ip rule 9809 and edit iptables OUTPUT to get unbound to route through wg11. I just notice something strange. From router, dig google.com give me connection timeout. Dig google.com @127.0.0.1 gives me almost immediate result. Somehow I can ping to any url and it is able to resolve the address. Problem is it takes more than 5 seconds to get the first echo reply. chronyc tracking is working but also takes a few seconds to display the output. chronyc sources take few seconds for each of the ntp server to display. Everything from router itself going out takes a lot more times.
Code:
admin@RT-AC86U-DBA8:/tmp/home/root# ip rule
0:      from all lookup local
9809:   from all fwmark 0xd1/0xd1 lookup 121
9810:   from all fwmark 0xd2 lookup 210
9820:   from all fwmark 0xdc lookup 220
9900:   from all fwmark 0x8000/0x8000 lookup main
9911:   from 192.168.1.111 lookup 121
9921:   from 192.168.1.92 lookup 122
9990:   from all fwmark 0x8000/0x8000 lookup main
9993:   from all fwmark 0x4000/0x4000 lookup ovpnc3
9995:   from all fwmark 0x1000/0x1000 lookup ovpnc1
10101:  from 10.50.2.0/24 lookup 121
10501:  from 10.22.0.0/24 lookup ovpnc3
10502:  from 192.168.1.2 lookup ovpnc3
10503:  from 192.168.1.11 lookup ovpnc3
10504:  from 192.168.1.21 lookup ovpnc3
32766:  from all lookup main
32767:  from all lookup default
admin@RT-AC86U-DBA8:/tmp/home/root# iptables -nvL OUTPUT -t mangle --line
Chain OUTPUT (policy ACCEPT 1037K packets, 424M bytes)
num   pkts bytes target     prot opt in     out     source               destination       
1       87  5714 MARK       udp  --  *      *       0.0.0.0/0            202.188.18.188       udp dpt:53 /* unbound_rule */ MARK or 0x8000
2       75  4766 MARK       udp  --  *      *       0.0.0.0/0            8.8.8.8              udp dpt:53 /* unbound_rule */ MARK or 0x8000
3        0     0 MARK       tcp  --  *      *       0.0.0.0/0            202.188.18.188       tcp dpt:53 /* unbound_rule */ MARK or 0x8000
4        0     0 MARK       tcp  --  *      *       0.0.0.0/0            8.8.8.8              tcp dpt:53 /* unbound_rule */ MARK or 0x8000
5        0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53 /* unbound_rule */ MARK or 0xd1
6      327 22088 MARK       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 /* unbound_rule */ MARK or 0xd1
admin@RT-AC86U-DBA8:/tmp/home/root#
 
Last edited:
This changes alone does not work for me. I have edit a bit unbound_via_vc1 script from @Swinson to go along with this and it is working now. What I like is when I bounce wg11, unbound does not restart. With openvpn, bouncing openvpn client interface causes unbound to restart and all cache are cleared. Not sure why the longest single session I have with openvpn to NordVPN is about a week.

Update: There is no effect if I just edit outgoing-interface in unbound.conf.add alone. So I added ip rule 9809 and edit iptables OUTPUT to get unbound to route through wg11. I just notice something strange. From router, dig google.com give me connection timeout. Dig google.com @127.0.0.1 gives me almost immediate result. Somehow I can ping to any url and it is able to resolve the address. Problem is it takes more than 5 seconds to get the first echo reply. chronyc tracking is working but also takes a few seconds to display the output. chronyc sources take few seconds for each of the ntp server to display. Everything from router itself going out takes a lot more times.
Code:
admin@RT-AC86U-DBA8:/tmp/home/root# ip rule
0:      from all lookup local
9809:   from all fwmark 0xd1/0xd1 lookup 121
9810:   from all fwmark 0xd2 lookup 210
9820:   from all fwmark 0xdc lookup 220
9900:   from all fwmark 0x8000/0x8000 lookup main
9911:   from 192.168.1.111 lookup 121
9921:   from 192.168.1.92 lookup 122
9990:   from all fwmark 0x8000/0x8000 lookup main
9993:   from all fwmark 0x4000/0x4000 lookup ovpnc3
9995:   from all fwmark 0x1000/0x1000 lookup ovpnc1
10101:  from 10.50.2.0/24 lookup 121
10501:  from 10.22.0.0/24 lookup ovpnc3
10502:  from 192.168.1.2 lookup ovpnc3
10503:  from 192.168.1.11 lookup ovpnc3
10504:  from 192.168.1.21 lookup ovpnc3
32766:  from all lookup main
32767:  from all lookup default
admin@RT-AC86U-DBA8:/tmp/home/root# iptables -nvL OUTPUT -t mangle --line
Chain OUTPUT (policy ACCEPT 1037K packets, 424M bytes)
num   pkts bytes target     prot opt in     out     source               destination
1       87  5714 MARK       udp  --  *      *       0.0.0.0/0            202.188.18.188       udp dpt:53 /* unbound_rule */ MARK or 0x8000
2       75  4766 MARK       udp  --  *      *       0.0.0.0/0            8.8.8.8              udp dpt:53 /* unbound_rule */ MARK or 0x8000
3        0     0 MARK       tcp  --  *      *       0.0.0.0/0            202.188.18.188       tcp dpt:53 /* unbound_rule */ MARK or 0x8000
4        0     0 MARK       tcp  --  *      *       0.0.0.0/0            8.8.8.8              tcp dpt:53 /* unbound_rule */ MARK or 0x8000
5        0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53 /* unbound_rule */ MARK or 0xd1
6      327 22088 MARK       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 /* unbound_rule */ MARK or 0xd1
admin@RT-AC86U-DBA8:/tmp/home/root#
Strange, I don't need iptables rules for unbound to go out my wg11 in policy mode. The fact that your request takes seconds to complete must mean that something is not working properly (wg11 rp_filter?).

I believe I directly changed opt/var/lib/unbound/unbound.conf

You probably need to stop unbound as you edit the file.

Router Configuration recommended pre-reqs status:
[✔] Swapfile=2097148 kB
[✔] DNS Filter=ON
[✔] DNS Filter=ROUTER
[✔] WAN: Use local caching DNS server as system resolver=NO
[✔] Enable local NTP server=YES
[✔] Enable DNS Rebind protection=N
[✔] Enable DNSSEC support=NO
Options:
[✔] unbound CPU/Memory Performance tweaks
[✔] Firefox DNS-over-HTTPS (DoH) DISABLE/Blocker
[✔] Router Graphical GUI statistics TAB installed
[✔] unbound-control FAST response ENABLED
[✔] unbound requests via VPN Client (192.168.1.1) tunnel ENABLED
[✔] Safe Search ENABLED (209 domains e.g. redirect "www.google.com" to "for cesafesearch.google.com")

0: from all lookup local
9900: from 192.168.1.1/24 fwmark 0x8000 lookup main
9910: from all to 192.168.1.1/16 lookup main
9911: from 192.168.1.1/24 lookup 121
9921: from 192.168.6.0/24 lookup 122
32766: from all lookup main
32767: from all lookup default

Unbound_manager seems to put vpn client ip in outgoing interface (as far as I understand) thus it needs to be reset as the vpn client is reset. Using a local interface as 192.168.1.1 this would not need to change as vpn interface is reset (just a theory). But I guess it must be a router interface and there must be a rule to route this out vpn. Did you create a rule for the ip you put in Outgoing Interface? Was it a local ip to the router?


//Zeb
 
Last edited:
Strange, I don't need iptables rules for unbound to go out my wg11 in policy mode. The fact that your request takes seconds to complete must mean that something is not working properly (wg11 rp_filter?).

I believe I directly changed opt/var/lib/unbound/unbound.conf

You probably need to stop unbound as you edit the file.

Router Configuration recommended pre-reqs status:
[✔] Swapfile=2097148 kB
[✔] DNS Filter=ON
[✔] DNS Filter=ROUTER
[✔] WAN: Use local caching DNS server as system resolver=NO
[✔] Enable local NTP server=YES
[✔] Enable DNS Rebind protection=N
[✔] Enable DNSSEC support=NO
Options:
[✔] unbound CPU/Memory Performance tweaks
[✔] Firefox DNS-over-HTTPS (DoH) DISABLE/Blocker
[✔] Router Graphical GUI statistics TAB installed
[✔] unbound-control FAST response ENABLED
[✔] unbound requests via VPN Client (192.168.1.1) tunnel ENABLED
[✔] Safe Search ENABLED (209 domains e.g. redirect "www.google.com" to "for cesafesearch.google.com")

0: from all lookup local
9900: from 192.168.1.1/24 fwmark 0x8000 lookup main
9910: from all to 192.168.1.1/16 lookup main
9911: from 192.168.1.1/24 lookup 121
9921: from 192.168.6.0/24 lookup 122
32766: from all lookup main
32767: from all lookup default

Unbound_manager seems to put vpn client ip in outgoing interface (as far as I understand) thus it needs to be reset as the vpn client is reset. Using a local interface as 192.168.1.1 this would not need to change as vpn interface is reset (just a theory). But I guess it must be a router interface and there must be a rule to route this out vpn. Did you create a rule for the ip you put in Outgoing Interface? Was it a local ip to the router?


//Zeb
In wg11-up, I did not put the second line. The rp_filter by default I think is 0. I just try to add the second line but still the same.
Code:
ip rule add from 192.168.1.0/24 fwmark 0x8000 table main prio 9900
echo 2 > /proc/sys/net/ipv4/conf/eth0/rp_filter

Regarding unbound outgoing-interface, I think changes in unbound.conf.add should be the same. It will take effect after restart unbound.
Code:
A:Option ==> oq outgoing-interface

unbound-control 'outgoing-interface' '192.168.1.1'

Looking at the config file, I think you are right that unbound put vpn client ip in the outgoing-interface. Probably this is the reason unbound need to be restarted when ovpn client bounce.
How do you get this in unbound? I am not able to get this up without open vpn client. It seems tied to open vpn and not wg.
Code:
[✔] unbound requests via VPN Client  (192.168.1.1) tunnel ENABLED
Here is my attempt with wg11 up but ovpnc tun11 down:
Code:
A:Option ==> vpn

        VPN Client arg '' invalid, must be in range 1-5


A:Option ==> vpn 1


Do you want to route unbound requests through VPN Client '1' tunnel?

        Reply 'y' or press [Enter]  to skip
y


(unbound_manager): 2443 ***ERROR VPN Client '1' is NOT Connected?


I put back ip rule and mangle iptables OUTPUT. I think I hit the earlier issue again in ip rule sequence. After I move fwmark for dns lookup 121 to lower priority than main it seems all working good now.
Code:
9900:   from all fwmark 0x8000/0x8000 lookup main
9905:   from all fwmark 0xd1/0xd1 lookup 121
 
How do you get this in unbound?
Code:
[✔] unbound requests via VPN Client  (192.168.1.1) tunnel ENABLED
As far as I remember I just stopped unbound, edited (using nano) unbound.conf, started unbound again and there it was.

Using vpn command in unbound manager will attempt to alter this line so I have never done that.

The reverse path filter is a check performed on incoming packages to see if a reply would get routed out the same interface it came in. It seems to understand all ip rules but not the fact that iptables will mark the reply package. If you are marking packages to be routed out wg11 then it would be wg11 that gets reply packages checked through rp_filter and this interface that needs to have adjusted rp_filter and not eth0 as we set for packages marked for wan.

However I guess if that was your problem dns lookup would not work at all.

//Zeb
 
As far as I remember I just stopped unbound, edited (using nano) unbound.conf, started unbound again and there it was.

Using vpn command in unbound manager will attempt to alter this line so I have never done that.

The reverse path filter is a check performed on incoming packages to see if a reply would get routed out the same interface it came in. It seems to understand all ip rules but not the fact that iptables will mark the reply package. If you are marking packages to be routed out wg11 then it would be wg11 that gets reply packages checked through rp_filter and this interface that needs to have adjusted rp_filter and not eth0 as we set for packages marked for wan.

However I guess if that was your problem dns lookup would not work at all.

//Zeb
I follow the steps. stop unbound, edit unbound.conf then restart unbound. While I can see the state in unbound options, I checked my devices that routed to ovpnc, wg11, and direct to WAN all show WAN IP as my DNS. I have go back to last working setup. After make sure ip rule main priority remain higher than others, there is no more delay in router connection going out.
Code:
admin@RT-AC86U-DBA8:/tmp/home/root# unbound_manager advanced

+======================================================================+
|  Welcome to the unbound Manager/Installation script (Asuswrt-Merlin) |
|                                                                      |
|                      Version 3.22 by Martineau                       |
|                                                                      |
+======================================================================+


Warning unbound not running!! - Config last loaded info: # Version=v1.13 Martineau update (Date Loaded by unbound_manager Thu Jun 24 11:19:32 MYT 2021)

[1624529465] unbound-control[7528:0] error: connect: Connection refused for 127.0.0.1 port 953
i  = Update unbound and configuration ('/opt/var/lib/unbound/')
z  = Remove unbound/unbound_manager                                     v  = View ('/opt/var/lib/unbound/') unbound Configuration (vx=Edit;vh=help)
3  = Advanced Tools                                                     rl = Reload Configuration (Doesn't halt unbound) e.g. 'rl test1[.conf]' (Recovery use 'rl reset/user')
?  = About Configuration

rs = Restart (or Start) unbound (use 'rs nocache' to flush cache)

e  = Exit Script [?]

A:Option ==> rs

18:11:13 Checking 'unbound.conf' for valid Syntax.....
18:11:13 Requesting unbound (S61unbound) restart.....
 Starting unbound...              done. 
18:11:14 Checking status, please wait..... 
18:11:16 Restoring unbound cache from '/opt/share/unbound/configs/cache.txt' (2021-06-24 18:08:35) msg.cache=16/936 rrset.cache=73/3011
18:11:17 unbound OK



unbound (pid 7878) is running... uptime: 0 Days, 00:00:03 version: 1.13.1 # Version=v1.13 Martineau update (Date Loaded by unbound_manager Thu Jun 24 18:11:14 MYT 2021)

i  = Update unbound and configuration ('/opt/var/lib/unbound/')         l  = Show unbound LIVE (Loglevel=1) log entries (lx=Disable Logging)
z  = Remove unbound/unbound_manager                                     v  = View ('/opt/var/lib/unbound/') unbound Configuration (vx=Edit;vh=help)
3  = Advanced Tools                                                     rl = Reload Configuration (Doesn't halt unbound) e.g. 'rl test1[.conf]' (Recovery use 'rl reset/user')
?  = About Configuration                                                oq = Query unbound Configuration option e.g 'oq verbosity' (ox=Set) e.g. 'ox log-queries yes'

rs = Restart (or Start) unbound (use 'rs nocache' to flush cache)       s  = Show unbound Extended statistics (s=Summary Totals; sa=All; http://192.168.1.1:80/user1.asp)

e  = Exit Script [?]

A:Option ==> ?

        Version=3.22                                    (Change Log: https://github.com/MartineauUK/Unbound-Asuswrt-Merlin/commits/master/unbound_manager.sh)
        Local                                           md5=6b4a500c071bcbb3f4a6e9596a178d43
        Github                                          md5=6b4a500c071bcbb3f4a6e9596a178d43
        /jffs/addons/unbound/unbound_manager.md5        md5=6b4a500c071bcbb3f4a6e9596a178d43

        Router Configuration recommended pre-reqs status:

        [✔] Swapfile=2097148 kB
        [✔] DNS Filter=ON
        [✔] DNS Filter=ROUTER
        [✔] WAN: Use local caching DNS server as system resolver=NO
        [✔] Entware NTP server 'S77chronyd' is running
        [✔] Enable DNS Rebind protection=NO
        [✔] Enable DNSSEC support=NO

        Options: 

        [✔] unbound Logging
        [✔] unbound CPU/Memory Performance tweaks
        [✔] Firefox DNS-over-HTTPS (DoH) DISABLE/Blocker
        [✔] Router Graphical GUI statistics TAB installed
        [✔] unbound-control FAST response ENABLED
        [✔] unbound requests via VPN Client  (192.168.1.1) tunnel ENABLED
        [✔] DNS Firewall ENABLED

        unbound Memory/Cache:

        'key-cache-size:'       8388608 (8.00 MB)
        'msg-cache-size:'       8388608 (8.00 MB)       2% used 246085  (240.32 KB)
        'rrset-cache-size:'     16777216 (16.00 MB)     6% used 1087718 (1.04 MB)

        System Memory/Cache:

                     total       used       free     shared    buffers     cached
        Mem:        440324     401320      39004       1624      15320      76844
        -/+ buffers/cache:     309156     131168
        Swap:      2097148      10332    2086816

        About unbound: https://nlnetlabs.nl/projects/unbound/about/ , Manual https://nlnetlabs.nl/documentation/unbound/unbound.conf/



unbound (pid 7878) is running... uptime: 0 Days, 00:00:12 version: 1.13.1 # Version=v1.13 Martineau update (Date Loaded by unbound_manager Thu Jun 24 18:11:14 MYT 2021)

i  = Update unbound and configuration ('/opt/var/lib/unbound/')         l  = Show unbound LIVE (Loglevel=1) log entries (lx=Disable Logging)
z  = Remove unbound/unbound_manager                                     v  = View ('/opt/var/lib/unbound/') unbound Configuration (vx=Edit;vh=help)
3  = Advanced Tools                                                     rl = Reload Configuration (Doesn't halt unbound) e.g. 'rl test1[.conf]' (Recovery use 'rl reset/user')
?  = About Configuration                                                oq = Query unbound Configuration option e.g 'oq verbosity' (ox=Set) e.g. 'ox log-queries yes'

rs = Restart (or Start) unbound (use 'rs nocache' to flush cache)       s  = Show unbound Extended statistics (s=Summary Totals; sa=All; http://192.168.1.1:80/user1.asp)

e  = Exit Script [?]

A:Option ==>
 
I follow the steps. stop unbound, edit unbound.conf then restart unbound. While I can see the state in unbound options, I checked my devices that routed to ovpnc, wg11, and direct to WAN all show WAN IP as my DNS. I have go back to last working setup. After make sure ip rule main priority remain higher than others, there is no more delay in router connection going out.
ok, that is too bad.. did you make a "ip rule" for 192.168.1.1 to use wg11 route table? if so, I wonder what makes our system differ.

//Zeb
 
Update wg_manager.sh
Remove 'generatestats' debugging statements
CHANGE: If Road-Warrior 'client' Peer has been dormant for more than 30mins then don't needlessly report obviously redundant RX=0/RX=0 metrics
@Martineau, it gets prettier day by day. Thank you (again.)
 
What does this do?
Code:
# WAN KILL-Switch
KILLSWITCH

And why does "?" say it's DISABLED?
When there is no "#" in front of "KILLSWITCH"

Unbenannt.PNG
 
ok, that is too bad.. did you make a "ip rule" for 192.168.1.1 to use wg11 route table? if so, I wonder what makes our system differ.

//Zeb
Sorry for my late reply. Maybe that’s the difference. I did not make an ip rule for 192.168.1.1 to use wg11. I have the impression that router have to go through WAN interface during boot-up, otherwise NTP will not sync and unbound will not run. So in your case even router itself go through wg11? Will it work after reboot? Or you add the rule in wg11-up.sh? What I have now is router go through WAN interface and all other devices DNS leak test show wg11 public ip.
 
Sorry for my late reply. Maybe that’s the difference. I did not make an ip rule for 192.168.1.1 to use wg11. I have the impression that router have to go through WAN interface during boot-up, otherwise NTP will not sync and unbound will not run. So in your case even router itself go through wg11? Will it work after reboot? Or you add the rule in wg11-up.sh? What I have now is router go through WAN interface and all other devices DNS leak test show wg11 public ip.

I have no problem at reboot.
I created the rule inside wgm "peer wg11 rule add ......" so it will not be active before the tunnel is up.

Router itself (local process) has a number of source adresses and the selection process is quite complicated, usually determined by most likely route based on destination. Usually this means default routing in mains table for unknown destinations. From what I have seen is that processes we have not specifically requested source ip (as we now have with Unbound) will still use wan as this is main default)

Even if the rule is there both unbound and transmission will use wan unless we request specific source adress. This tells me that the NTP process will be unaffected by this. We are not making illegal routes, the source 198.168.1.1 will always be routable.

Interesting reading:
Source adress selection

Guess the best way to find out is to test for
yourself.

//Zeb

Edit: just remembered I had some wierd issues in the beginning. Unbound was working but some unbound cron job were not there (the stat update at 59 for example)
I added a script wg11-route-up.sh which executes after the vpn connection is set up but before rules are added. I simply put "sleep 2" in there to give the connection time to establish before the rules gets appended. No problem since then...
 
Last edited:
I have no problem at reboot.
I created the rule inside wgm "peer wg11 rule add ......" so it will not be active before the tunnel is up.

Router itself (local process) has a number of source adresses and the selection process is quite complicated, usually determined by most likely route based on destination. Usually this means default routing in mains table for unknown destinations. From what I have seen is that processes we have not specifically requested source ip (as we now have with Unbound) will still use wan as this is main default)

Even if the rule is there both unbound and transmission will use wan unless we request specific source adress. This tells me that the NTP process will be unaffected by this. We are not making illegal routes, the source 198.168.1.1 will always be routable.

Interesting reading:
Source adress selection

Guess the best way to find out is to test for
yourself.

//Zeb

Edit: just remembered I had some wierd issues in the beginning. Unbound was working but some unbound cron job were not there (the stat update at 59 for example)
I added a script wg11-route-up.sh which executes after the vpn connection is set up but before rules are added. I simply put "sleep 2" in there to give the connection time to establish before the rules gets appended. No problem since then...
Good idea to create the rule inside wgm. Thank you for your explanation. I will probably try this again later. I am into the second day of comparing stability/uptime between openvpn and wg connection.
Another question on the script. What is the difference between wg11-up.sh and wg11-route-up.sh? And when to use it? I only have wg11-up.sh.
 
Good idea to create the rule inside wgm. Thank you for your explanation. I will probably try this again later. I am into the second day of comparing stability/uptime between openvpn and wg connection.
Another question on the script. What is the difference between wg11-up.sh and wg11-route-up.sh? And when to use it? I only have wg11-up.sh.
If you create "wg11-route-up.sh" wgm will run it after the interface is created but before the routing tables and rules are created.

If you create "wg11-up.sh" wgm will run it after all is setup (well, most anyway).

The main difference is when they are executed.
Guess the main reason would be to add custom routes before the rules kicks in.

Good luck and please report back about your results!

//Zeb
 
Last edited:

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top