Aegis Aegis (simple yet effective protection)

[Panner]

Manually updated and got:

Warnings

• iptables: WAN network range bypass rules are missing!
• iptables: VPN network range bypass rules are missing!

When followed by restart the warnings disappear

 

[HELLO_wORLD]

Ok, so you don’t have the same result if it is launched by cron at 3:15 or manually?
You have the error described by @D3FenD3r at 3:15, but the bypass warning when manually launched?

 

[Panner]

Sorry I cannot remember what the error said on mine for the 3:15 cron update and unfortunately did not take a copy of the status page

Just tested with a cron initiated update (have also done a reboot earlier this morning). Then did a manual update (without clean)
Status details (differences in red) following cron update and a subsequent manual update:

Hope it helps

Status – Following Cron initiated update:

• Something is not right!

Errors

• iptables: engine chains are not right!

Detailed status

• Active WAN interface is 'brwan'.
• Active VPN tunnel is 'tun21'.
• Blocklist generation time: 2020-12-14 12:35:10
• Whitelist generation time: 2020-12-14 12:35:12
• 'firewall-start.sh' is set for aegis.
• 'post-mount.sh' is set for aegis.
• ipset: blocklist is set.
• ipset: whitelist is set.
• iptables: VPN tunnel IFO rules are set.
• iptables: WAN interface IFO rules are set.

Last Aegis engine launch report

• engine was launched from: aegis script @ 2020-12-14 12:35:15
• WAN interface was 'brwan'.
• VPN tunnel was 'tun21'.
• ipset: blocklist was set from file.
• ipset: whitelist was already set and identical to file.
• iptables: engine inbound chain was already set.
• iptables: engine outbound chain was already set.
• iptables: some irrelevant bypass rules had to be removed.
• iptables: inbound WAN network range bypass rules were kept.
• iptables: outbound WAN network range bypass rules were kept.
• iptables: inbound VPN network range bypass rules were kept.
• iptables: outbound VPN network range bypass rules were kept.
• iptables: inbound whitelist rules were kept.
• iptables: outbound whitelist rules were kept.
• iptables: WAN interface IFO rules were kept.
• iptables: VPN tunnel IFO rules were kept.

Debug

• device info: R9000 R9000 V1.0.4.46HF
• aegis info: aegis 1.4.3-ext
• status codes: 1551#16#0#brwan#192.168.1.0/24#tun21#10.8.8.0/24#619651040#1
• file codes: 42012607/brwan/tun21
• iptables engine rules:
  • -N aegis_dst
  • -N aegis_src
  • -A INPUT -i brwan -m set --match-set aegis_bl src -m comment --comment "incoming in aegis blacklist" -j aegis_src
  • -A INPUT -i tun21 -m set --match-set aegis_bl src -m comment --comment "incoming in aegis blacklist" -j aegis_src
  • -A FORWARD -i brwan -m set --match-set aegis_bl src -m comment --comment "incoming in aegis blacklist" -j aegis_src
  • -A FORWARD -i tun21 -m set --match-set aegis_bl src -m comment --comment "incoming in aegis blacklist" -j aegis_src
  • -A FORWARD -o brwan -m set --match-set aegis_bl dst -m comment --comment "outgoing in aegis blacklist" -j aegis_dst
  • -A FORWARD -o tun21 -m set --match-set aegis_bl dst -m comment --comment "outgoing in aegis blacklist" -j aegis_dst
  • -A OUTPUT -o brwan -m set --match-set aegis_bl dst -m comment --comment "outgoing in aegis blacklist" -j aegis_dst
  • -A OUTPUT -o tun21 -m set --match-set aegis_bl dst -m comment --comment "outgoing in aegis blacklist" -j aegis_dst
  • -A aegis_dst -m set --match-set aegis_wl dst -m comment --comment "in aegis whitelist" -j RETURN
  • -A aegis_src -m set --match-set aegis_wl src -m comment --comment "in aegis whitelist" -j RETURN
• ipset engine sets:
  • blocklist:
    • Name: aegis_bl
    • Type: hash:net
    • Revision: 7
    • Header: family inet hashsize 16384 maxelem 51629 bucketsize 12 initval 0xb18a54cc
    • Size in memory: 1196760
    • References: 8
    • Number of entries: 51629
  • whitelist:
    • Name: aegis_wl
    • Type: hash:net
    • Revision: 7
    • Header: family inet hashsize 1024 maxelem 1 bucketsize 12 initval 0xd7655dd2
    • Size in memory: 420
    • References: 2
    • Number of entries: 1

Status following manual update:

• Aegis is set and active for WAN interface (brwan) and VPN tunnel (tun21).
• Filtering 619651040 IP adresses.
• Bypassing 1 IP adresses.

Detailed status

• Active WAN interface is 'brwan'.
• Active VPN tunnel is 'tun21'.
• Blocklist generation time: 2020-12-14 12:37:39
• Whitelist generation time: 2020-12-14 12:37:39
• 'firewall-start.sh' is set for aegis.
• 'post-mount.sh' is set for aegis.
• ipset: blocklist is set.
• ipset: whitelist is set.
• iptables: engine chains are set.
• iptables: WAN network range bypass rules are set.
• iptables: VPN network range bypass rules are set.
• iptables: whitelist rules are set.
• iptables: aegis logging is on.
• iptables: VPN tunnel IFO rules are set.
• iptables: WAN interface IFO rules are set.

Last Aegis engine launch report

• engine was launched from: aegis script @ 2020-12-14 12:37:40
• WAN interface was 'brwan'.
• VPN tunnel was 'tun21'.
• ipset: blocklist was already set and identical to file.
• ipset: whitelist was already set and identical to file.
• iptables: engine inbound chain was already set.
• iptables: engine outbound chain was already set.
• iptables: inbound WAN network range bypass rules were set.
• iptables: outbound WAN network range bypass rules were set.
• iptables: inbound VPN network range bypass rules were set.
• iptables: outbound VPN network range bypass rules were set.
• iptables: inbound whitelist rules were set.
• iptables: outbound whitelist rules were set.
• iptables: inbound logging rules were set.
• iptables: outbound logging rules were set.
• iptables: WAN interface IFO rules were kept.
• iptables: VPN tunnel IFO rules were kept.

Debug

• device info: R9000 R9000 V1.0.4.46HF
• aegis info: aegis 1.4.3-ext
• status codes: 2047#0#0#brwan#192.168.1.0/24#tun21#10.8.8.0/24#619651040#1
• file codes: 46135223/brwan/tun21
• iptables engine rules:
  • -N aegis_dst
  • -N aegis_src
  • -A INPUT -i brwan -m set --match-set aegis_bl src -m comment --comment "incoming in aegis blacklist" -j aegis_src
  • -A INPUT -i tun21 -m set --match-set aegis_bl src -m comment --comment "incoming in aegis blacklist" -j aegis_src
  • -A FORWARD -i brwan -m set --match-set aegis_bl src -m comment --comment "incoming in aegis blacklist" -j aegis_src
  • -A FORWARD -i tun21 -m set --match-set aegis_bl src -m comment --comment "incoming in aegis blacklist" -j aegis_src
  • -A FORWARD -o brwan -m set --match-set aegis_bl dst -m comment --comment "outgoing in aegis blacklist" -j aegis_dst
  • -A FORWARD -o tun21 -m set --match-set aegis_bl dst -m comment --comment "outgoing in aegis blacklist" -j aegis_dst
  • -A OUTPUT -o brwan -m set --match-set aegis_bl dst -m comment --comment "outgoing in aegis blacklist" -j aegis_dst
  • -A OUTPUT -o tun21 -m set --match-set aegis_bl dst -m comment --comment "outgoing in aegis blacklist" -j aegis_dst
  • -A aegis_dst -d 192.168.1.0/24 -o brwan -m comment --comment "aegis inet bypass" -j RETURN
  • -A aegis_dst -d 10.8.8.0/24 -o tun21 -m comment --comment "aegis inet bypass" -j RETURN
  • -A aegis_dst -m set --match-set aegis_wl dst -m comment --comment "in aegis whitelist" -j RETURN
  • -A aegis_dst -j LOG --log-prefix "[aegis] "
  • -A aegis_dst -m comment --comment "aegis reject outgoing" -j REJECT --reject-with icmp-admin-prohibited
  • -A aegis_src -s 192.168.1.0/24 -i brwan -m comment --comment "aegis inet bypass" -j RETURN
  • -A aegis_src -s 10.8.8.0/24 -i tun21 -m comment --comment "aegis inet bypass" -j RETURN
  • -A aegis_src -m set --match-set aegis_wl src -m comment --comment "in aegis whitelist" -j RETURN
  • -A aegis_src -j LOG --log-prefix "[aegis] "
  • -A aegis_src -m comment --comment "aegis drop incoming" -j DROP
• ipset engine sets:
  • blocklist:
    • Name: aegis_bl
    • Type: hash:net
    • Revision: 7
    • Header: family inet hashsize 16384 maxelem 51629 bucketsize 12 initval 0xb18a54cc
    • Size in memory: 1196760
    • References: 8
    • Number of entries: 51629
  • whitelist:
    • Name: aegis_wl
    • Type: hash:net
    • Revision: 7
    • Header: family inet hashsize 1024 maxelem 1 bucketsize 12 initval 0xd7655dd2
    • Size in memory: 420
    • References: 2
    • Number of entries: 1

 

[HELLO_wORLD]

Thank you for the report.
It appears that when launched from cron, it does not finish creating the rules. Likely an environment difference (script launched from cron is not the same as launched from shell).
I will look into that when I can, hopefully this week, and might ask you to beta test, as I strangely don’t encounter this error with my cron (maybe specific to R9000).

 

[Panner]

Happy to help

3:15 am cron this morning still produced the error.

Tested temporary fix this morning - added a second cron 5 minutes later replacing "update" with "restart"

Status report shows all OK - no errors

 

[HELLO_wORLD]

1.4.4

Should fix the bug encountered.

New: logging is now taking place in its own file: /var/log/log-aegis
A little daemon is created to keep it updated and stick around 5000 to 6000 lines. The file never changes node id, making it possible to follow (tail -f)

Web Companion was updated to use new log file.

 

[Panner]

I can confirm the fix worked on my R9000. The 3:15 am cron update completed all steps and no errors or warnings

Well done HELLO_wORLD

 

[D3FenD3r]

an idea could be the blocking of ip by states
would be welcome ..... if possible

 

[HELLO_wORLD]

an idea could be the blocking of ip by states
would be welcome ..... if possible

You just have to find a database with ip ranges associated to geographic areas.
There are already many lists for countries, for example:

GitHub - herrbischoff/country-ip-blocks: CIDR country-level IP data, straight from the Regional Internet Registries, updated hourly. This is a read-only mirror.

CIDR country-level IP data, straight from the Regional Internet Registries, updated hourly. This is a read-only mirror. - herrbischoff/country-ip-blocks

github.com

From that example, if you want to block IPs related to Canada, just add the link to the raw list file in the aegis sources: https://raw.githubusercontent.com/herrbischoff/country-ip-blocks/master/ipv4/ca.cidr
Or from China: https://raw.githubusercontent.com/herrbischoff/country-ip-blocks/master/ipv4/cn.cidr

 

[D3FenD3r]

I was thinking something via web page
but even so it can go ................

grazie

 

[NetBytes]

I have been using aegis successfully since it's inception.
After upgrading to Voxel's .81 I have a new issue now.
Running 'aegis update' from telnet command line or cron causes it to hang after it says
"- Starting aegis engine, forcing reload of blocklist..."

Running the Aegis GUI update works correctly and completes.
What will help diagnose what's wrong?

 

[HELLO_wORLD]

I have been using aegis successfully since it's inception.
After upgrading to Voxel's .81 I have a new issue now.
Running 'aegis update' from telnet command line or cron causes it to hang after it says
"- Starting aegis engine, forcing reload of blocklist..."

Running the Aegis GUI update works correctly and completes.
What will help diagnose what's wrong?

Ok, do you have this problem systematically? Once you have updated from GUI, if you run update again from telnet, do you have the same problem?
To diagnose better, could you run aegis status -vvv before you run the update from telnet, and after (I suppose you force quit with ctrl-c)?

 

[HELLO_wORLD]

While waiting to resolve the bug @NetBytes is encountering, I am working on Aegis, with some changes in the commands.
@kamoj brought my attention on the fact that the commands might not be user friendly.

I skip the technical details, but it is not easy to find the right balance between simplicity, efficiency (cpu and memory performance), and user friendliness. With time, Aegis evolved, and rethinking the commands makes sense.

I have some ideas...

 

[HELLO_wORLD]

The bug suddenly appeared for me, randomly...

Found the culprit:
Some [ CONDITION ] && TRUE || FALSE is behaving weirdly with last firmware. Sometimes it works, sometimes not... Must be environment related as from web GUI is works but not from ash.
I replaced the one that was blocking by if [ CONDITION ]; then TRUE; else FALSE; fi and it works all the time.
So next version (this weekend) will correct the bug mentioned by @NetBytes and bring some optimizations and new command syntax.

 

[HELLO_wORLD]

1.4.5

Fixed bug experienced by @NetBytes
Log daemon is now making sure log-aegis entries with equal timestamp have a count, making them truely unique and ordered.
This allows a lighter process for aegis_web to get the logs, so less CPU impact.
Several other optimizations.

More important, commands are completely revamped, so please read this:
Now, there is only one command to start aegis. Aegis is a shield, so the command is:

aegis up

This will start or restart the engine.
This command can have options:
-refresh to update sets from the lists in aegis.sources and the custom lists (blacklist and whitelist) before (re)starting the engine.
-net-wan to restart router firewall as well as aegis engine.
-log-enable to enable logging while (re)starting.
-log-disable to disable logging while (re)starting.
-wan-no-bypass to not set the WAN network range bypass.
-vpn-no-bypass to not set the VPN network range bypass.

To stop aegis (put the shield down):

aegis down

No specific options for aegis down.

aegis refresh

This will update sets from the lists in aegis.sources and the custom lists (blacklist and whitelist).
If aegis is running, it will update the sets and restart it with updated sets.
If aegis is not running, it won’t start it.
This command has no specific options.

aegis log -enable

This will enable logging.
If aegis is running, it will restart it with logging enabled.
If aegis is not running, it won’t start it, but logging will be enabled from now on (until disabled).
This command has no specific options.

aegis log -disable

This will disable logging.
If aegis is running, it will restart it with logging disabled.
If aegis is not running, it won’t start it, but logging will be disabled from now on (until enabled).
This command has no specific options.

log -show

This will display the log report.
It has one option:
-lines=N will display N last lines (N being the number of lines to show).

aegis clean

stops aegis engine and allow further removal with option.
This command can have options:
-rm-config removes aegis configuration file.
-rm-symlink removes the symlink /usr/bin/aegis.
-rm-web removes Web Companion.
-rm-log removes log file.

Then what did not change is: status, help, info, upgrade, web -install and web -remove and the global options -v -vv -vvv -q.

So to upgrade, please:
1) aegis clean
2) aegis upgrade
3) aegis up
Or upgrade from the web companion.

Last point: don’t forget to update your crons if you are using it as aegis update does not exist anymore.
Most logical cron command would be aegis refresh or aegis up -refresh

 

[NetBytes]

Updated and running.
Cron job updated, will see success/fail tomorrow after job runs overnight.
Thank you!

Edit: Overnight cron job is a success! Looks like you found the issue. Thanks Again!

 

[Jauger]

Got a little bug with the web ui log....when I first go to it it runs as normal "block incoming from wan".........after about a minute or so it flips all logs as "blocked incoming packet from vpn"....I can post my debug if you need it

 

[HELLO_wORLD]

Got a little bug with the web ui log....when I first go to it it runs as normal "block incoming from wan".........after about a minute or so it flips all logs as "blocked incoming packet from vpn"....I can post my debug if you need it

Yes please, and if you could also include the result of this command:

cat /tmp/aegis_web

First while the log in web is showing correct information, then a second time when it is showing the wrong one (vpn instead of wan).

 

[Jauger]

Normal

/var/log/log-aegis 300 1608442662 67663000 brwan -

Flipped to VPN

/var/log/log-aegis 300 1608442662

 

[Jauger]

Oh yeah...my Debug

• device info: R9000 R9000 V1.0.4.46HF
• aegis info: aegis 1.4.5-int
• status codes: 1437#0#0#brwan#73.59.157.0/24###1092310946#60#0
• file codes: 8980023/brwan/
• iptables engine rules:
  • -N aegis_dst
  • -N aegis_src
  • -A INPUT -i brwan -m set --match-set aegis_bl src -m comment --comment "incoming in aegis blacklist" -j aegis_src
  • -A FORWARD -i brwan -m set --match-set aegis_bl src -m comment --comment "incoming in aegis blacklist" -j aegis_src
  • -A FORWARD -o brwan -m set --match-set aegis_bl dst -m comment --comment "outgoing in aegis blacklist" -j aegis_dst
  • -A OUTPUT -o brwan -m set --match-set aegis_bl dst -m comment --comment "outgoing in aegis blacklist" -j aegis_dst
  • -A aegis_dst -m set --match-set aegis_wl dst -m comment --comment "in aegis whitelist" -j RETURN
  • -A aegis_dst -j LOG --log-prefix "[aegis] "
  • -A aegis_dst -m comment --comment "aegis reject outgoing" -j REJECT --reject-with icmp-admin-prohibited
  • -A aegis_src -m set --match-set aegis_wl src -m comment --comment "in aegis whitelist" -j RETURN
  • -A aegis_src -j LOG --log-prefix "[aegis] "
  • -A aegis_src -m comment --comment "aegis drop incoming" -j DROP
• ipset engine sets:
  • blocklist:
    • Name: aegis_bl
    • Type: hash:net
    • Revision: 7
    • Header: family inet hashsize 131072 maxelem 298748 bucketsize 12 initval 0x1878bb6e
    • Size in memory: 7272144
    • References: 4
    • Number of entries: 298748
  • whitelist:
    • Name: aegis_wl
    • Type: hash:net
    • Revision: 7
    • Header: family inet hashsize 1024 maxelem 60 bucketsize 12 initval 0x62d37e0e
    • Size in memory: 2740
    • References: 2
    • Number of entries: 60