firewall & pinguin: iptables where do thou go?

it is said that when using “ip-sets” iptables and nftables achieve almost same performance (amounts of ips possible to block, without server becoming slow/unresponsive)

Redhat and nftables on DDoS “so the only thing to fall back to is establishing a blacklist for all the different source IP addresses” (src) (which is exactly what iptables + cron + autoban.sh a simple bash script does)

the #3rd concept: bpfilter

2018-02: “The Linux kernel currently supports two separate network packet-filtering mechanisms: iptables and nftables.
For the last few years, it has been generally assumed that nftables would eventually replace the older iptables implementation;
few people expected that the kernel developers would, instead, add a third packet filter.
But that would appear to be what is happening with the newly announced bpfilter mechanism.
Bpfilter may eventually replace both iptables and nftables, but there are a lot of questions that will need to be answered first.” (src: https://lwn.net/Articles/747551/)
even faster X-D https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/

and: will already established iptables scripts keep on working?

We like iptables after all, this tool has been serving us (and will likely keep serving still for a while in many deployments) to filter out traffic on both per-packet and per-flow basis, log suspicious traffic activity, perform NAT and many other things.

It comes with more than a hundred of extensions that have been contributed along the last 15 years!.

Nevertheless, the iptables framework suffers from limitations that cannot be easily worked around:

  • Avoid code duplication and inconsistencies:
    • Many of the iptables extensions are protocol specific
    • so there is no a consolidated way to match packet fields
    • instead there is one extension for each protocol that it supports
    • this bloats the codebase with very similar code to perform a similar task: payload matching
  • Faster packet classification through enhanced generic set and map infrastructure
  • Simplified dual stack IPv4/IPv6 administration, through the new inet family that allows you to register base chains that see both IPv4 and IPv6 traffic
  • Better dynamic ruleset updates support
  • Provide a Netlink API for third party applications, just as other Linux Networking and Netfilter subsystem do
  • Address syntax inconsistencies and provide nicer and more compact syntax (aha aha X-D)

These, among other things not listed here, triggered the nftables development which was originally presented to the Netfilter community in the 6th Netfilter Workshop in Paris (2008, France).” (src: wiki.nftables.org)

updated: 2020-05:

how do you do your firewalling?

let people know!

iptables manpage: iptables.man.txt

hostnamectl; # tested on
         Icon name: computer-desktop
           Chassis: desktop
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.19.0-9-amd64
      Architecture: x86-64

iptables --version;
iptables v1.8.2 (nf_tables)  # <- aha!

# install tools
# needed: installs a great text editor (also can use nano or emacs)
apt install vim
# optional: installs legacy ifconfig
apt install net-tools

# ===== view/list network interfaces (NICs, LAN Network Cards and other network devices) =====
ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:1e:06:45:0f:2d brd ff:ff:ff:ff:ff:ff
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1e:06:45:0f:2e brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.223/24 brd 192.168.0.255 scope global enp3s0
       valid_lft forever preferred_lft forever
    inet6 fe80::21e:6ff:fe45:f2e/64 scope link 
       valid_lft forever preferred_lft forever

# those are the two available network interfaces

# ===== view/list rules list =====
# will be processed from top to bottom, first rule that matches is applied, rest ignored
iptables -vnL --line-numbers
# num (--line-numbers), show rule number, can be used to modify/delete the rule
Chain INPUT (policy DROP 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
2        0     0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02
3      141  9680 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
4        0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 8
5        0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 11
6        0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 3
7        0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22
8        0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 80,443

Chain FORWARD (policy DROP 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 87 packets, 10512 bytes)
num   pkts bytes target     prot opt in     out     source               destination       

# maybe it is convenient to define an alias in /etc/bash.bashrc
alias iptablesll="iptables -vnL --line-numbers"
# different server, alternatively
# show all iptable rules
iptables -L -n -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blacklist src
11573 1115K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
  755 60068 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80
38863 2528K f2b-sshd   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22
39100 2565K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    1    32 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
   12   680 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
  151  8295 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22 ctstate NEW,ESTABLISHED
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 ctstate NEW,ESTABLISHED
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 ctstate NEW,ESTABLISHED
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 80,443 ctstate NEW,ESTABLISHED
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:2222 ctstate NEW,ESTABLISHED

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blacklist src
   69  3310 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 105 packets, 7373 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0           
45705  110M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate ESTABLISHED
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp spt:22 ctstate ESTABLISHED
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp spt:80 ctstate ESTABLISHED
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp spt:443 ctstate ESTABLISHED
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 80,443 ctstate ESTABLISHED
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp spt:2222 ctstate ESTABLISHED

Chain f2b-sshd (1 references)
 pkts bytes target     prot opt in     out     source               destination         
38863 2528K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           


# same same but different
iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N f2b-sshd
-A INPUT -m set --match-set blacklist src -j DROP
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 22 -j f2b-sshd
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m multiport --dports 80,443 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2222 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A FORWARD -m set --match-set blacklist src -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp -m tcp --sport 22 -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp -m tcp --sport 80 -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp -m tcp --sport 443 -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp -m multiport --dports 80,443 -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp -m tcp --sport 2222 -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A f2b-sshd -j RETURN


# -S, --list-rules [chain]
# Print all rules in the selected chain. If no chain is selected, all chains are printed like iptables-save. Like every other iptables command, it applies to the specified table (filter is
the default).

iptables -t nat -n -L
# Please note that it is often used with the -n option, in order to avoid long reverse DNS lookups.
# It is legal to specify the -Z (zero) option as well, in which case the chain(s) will be atomically listed and zeroed.
# The exact output is affected by the other arguments given. The exact rules are suppressed until you use

 -t, --table table
              This  option specifies the packet matching table which the command should operate on.  If the kernel is configured with automatic module loading, an attempt will be made to load the appro‐
              priate module for that table if it is not already there.

The tables are as follows:

filter:
This is the default table (if no -t option is passed). It contains the built-in chains INPUT (for packets destined to local sockets), FORWARD (for packets being routed through the
box), and OUTPUT (for locally-generated packets).

nat:
This table is consulted when a packet that creates a new connection is encountered. It consists of three built-ins: PREROUTING (for altering packets as soon as they come in), OUTPUT
(for altering locally-generated packets before routing), and POSTROUTING (for altering packets as they are about to go out). IPv6 NAT support is available since kernel 3.7.

mangle:
This table is used for specialized packet alteration. Until kernel 2.4.17 it had two built-in chains: PREROUTING (for altering incoming packets before routing) and OUTPUT (for alter‐
ing locally-generated packets before routing). Since kernel 2.4.18, three other built-in chains are also supported: INPUT (for packets coming into the box itself), FORWARD (for alter‐
ing packets being routed through the box), and POSTROUTING (for altering packets as they are about to go out).

raw:
This table is used mainly for configuring exemptions from connection tracking in combination with the NOTRACK target. It registers at the netfilter hooks with higher priority and is
thus called before ip_conntrack, or any other IP tables. It provides the following built-in chains: PREROUTING (for packets arriving via any network interface) OUTPUT (for packets
generated by local processes)

security:
This table is used for Mandatory Access Control (MAC) networking rules, such as those enabled by the SECMARK and CONNSECMARK targets. Mandatory Access Control is implemented by Linux
Security Modules such as SELinux. The security table is called after the filter table, allowing any Discretionary Access Control (DAC) rules in the filter table to take effect before
MAC rules. This table provides the following built-in chains: INPUT (for packets coming into the box itself), OUTPUT (for altering locally-generated packets before routing), and FOR‐
WARD (for altering packets being routed through the box).

warning: those scripts are probably far from perfect!

check the source test and improve 🙂 and share

firewall init script:

one possible firewall this init script:

https://dwaves.de/downloads/firewall/mrfirewall/firewall.init.sh.txt

this is straight from the official Debian wiki “Example: Basic gateway machine firewall”

and guess what iptables v1.8.2 has to say about “–log-level DEBUG”

“log level “DEBUG” unkown” ha! nftables sucks!

even the official Debian wiki is now wrong or outdated documentation!

what a mess!

autoban bad ssh brute forcing IPs:

firewall scripts that search the logs and ban automatically:

when log files give

  1. meaningful error message (that an auth / login failed, not because of bad connection, but because of bad password)
  2. correct/full IP address of an attacker (bruteforce ssh, imap etc.) (even when inside an vm! X-D)

this script manages to block hundreds (more than 5000 is getting slow/depending on one’s cpu and hardware)

system goes like this:

  1. crontab: every night at 1:00 run reset.sh
    • this will unblock all previous blocked ips and start the process of detecting and blocking misbehaving IPs all over
    • will reset the the firewall (all existing connections will be dropped)
  2. crontab: every 30min run autoban.sh
    • this scans the log for misbehaving IPs and puts them on a list

let’s start:

su - root; # become root
mkdir -p /scripts/firewall
cd /scripts/firewall;

this will reset the auto-banned ip list:

wget https://dwaves.de/downloads/firewall/reset.sh.txt

mv -v reset.sh.txt /scripts/firewall/reset.sh

cat /scripts/firewall/reset.sh
echo "===== will reset iptables rules to default basically unblocking all blocked ips to date ====="
/sbin/service iptables restart

# general firewall settings and allowed ports and blacklist.txt
/scripts/firewall/rules.sh

# ban ips that missbehaved
/scripts/firewall/autoban.sh

this is just a “tool script” used by autoban.sh to block a single ip. can also be called manually from terminal

wget https://dwaves.de/downloads/firewall/ban_ip.sh.txt

mv -v ban_ip.sh.txt /scripts/firewall/

cat /scripts/firewall/ban_ip.sh
echo "banning ip until firewall restart: "$1
iptables -I INPUT -s $1 -j DROP

this is just a “tool script” to unblock/unban a single ip. can also be called manually from terminal

wget https://dwaves.de/downloads/firewall/unban_ip.sh.txt

mv -v unban_ip.sh.txt /scripts/firewall/

cat /scripts/firewall/unban_ip.sh
echo "unbanning ip: "$1
iptables -D INPUT -s $1 -j DROP

this will parse logs and extract evil IPs into lists to block until midnight

wget https://dwaves.de/downloads/firewall/autoban.sh.txt

mv -v ban_ip.sh.txt /scripts/firewall/

cat /scripts/firewall/autoban.sh

#!/bin/bash
# generate a list of ips worth blocking, by searching /var/log/secure for:

# 1) "Invalid user"
# 2) "preauth"
# 3) "Did not receive identification string from"

# and block them via iptables (because fail2ban SUCKS! (it is too complicated and fails often to generate iptables rules, also not compatible with newer nftables)

# reset of iptables will be done via /scripts/reset.sh every 3 days (crontab)
# empty file
echo "" > /var/log/secure_invalid_preauth.log

# download exim failed login attempts
# rsync -r -v --progress -e "ssh -i /home/user/.ssh/id_rsa" user@192.168.56.XXX:/var/log/exim_failed_logins.log /var/log

# filter out according to patterns and save to /var/log/secure_invalid_preauth.log
cat /var/log/secure | grep -e "Invalid user" -e "preauth" -e "Did not receive identification string from" > /var/log/secure_invalid_preauth.log
# cat /var/log/messages | grep -e "SRC" > /var/log/secure_bad_packages.log

# prepare ip extraction patterns
octet='\<(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]?)\>'
ip="$octet\\.$octet\\.$octet\\.$octet"

# extract ips and filter duplicates ( sort | uniq )
grep -Eo "$ip" /var/log/secure_invalid_preauth.log | paste - | sort | uniq > /var/log/secure_invalid_preauth.log2

# add ips from failed exim logins downloaded from webserver dwaves.org
grep -Eo "$ip" /var/log/exim_failed_logins.log | paste - | sort | uniq >> /var/log/secure_invalid_preauth.log2

echo "=== delete all IPs from the ban-list (user's fixed ips, dns servers/router/proxys etc.) ==="

# delete this ip
sed -i '/123.123.123.123/d' /var/log/secure_invalid_preauth.log2
# delete this subnet
sed -i '/123.123./d' /var/log/secure_invalid_preauth.log2
# delete this subnet
sed -i '/123.123./d' /var/log/secure_invalid_preauth.log2
# delete this ip
sed -i '/^123.123.123.123/d' /var/log/secure_invalid_preauth.log2

# check if ip already banned
iptables -nL > /var/log/secure_invalid_preauth.log3

# iterate over every line of a file and 
while read line; do
	if grep -q $line "/var/log/secure_invalid_preauth.log3";
		then
			echo $line" is already banned.";
		else
			echo "banning "$line;
			echo "banning "$line >> /var/log/firewall_autoban.log;
			/scripts/firewall/ban_ip.sh $line
		# i hate fail2ban it does not work properly, i just will create an iptable rule directly
  		# fail2ban-client set sshd banip $line;
	fi

done </var/log/secure_invalid_preauth.log2
echo "done"

# unban user's fixed ips (just to be sure X-D)
/scripts/firewall/unban_ip.sh 123.123.123.123
/scripts/firewall/unban_ip.sh 124.124.124.124

to make all of those scripts work don’t forget to mark them as runnable:

chmod +x /scripts/firewall/*.sh

there is also a blacklist feature: (those are the ip’s/subnets that even after a reset, will get immediately re-banned X-D)

wget https://dwaves.de/downloads/firewall/blacklist.readme.txt

mv -v blacklist.readme.txt /scripts/firewall/blacklist.readme.txt

cat /scripts/firewall/blacklist.readme.txt

after editing blacklist.txt

manual run  /scripts/firewall/rules.sh

which will automatically apply the rules and save the config

to be loaded on next startup or on every firewall reset (every 3 days crontab)

then run:

/scripts/firewall/autoban.sh

cat blacklist.txt
123.123.123.0/24
123.123.123.123
...

how to ban / block a single ip:

iptables -L -n|less
# Howto block an malicious IP address/subnet?
# $1 = ip.to.bock.123 or 123.123.123.0/24 subnet to block
cat /scripts/firewall_ban_ip.sh
echo "========== banning until reboot this host: ==========="
host $1
/sbin/iptables -I INPUT -s $1 -j DROP

cat /scripts/firewall_unban_ip.sh
echo "========== UNbanning this host: ==========="
host $1

echo "unbanning ip: "$1
/sbin/iptables -D INPUT -s $1 -j DROP

# in case autoban.sh is used
echo "remove ip from var log secure"
sed -i '/$1/d' /var/log/secure
sed -i '/$1/d' /var/log/secure_invalid_preauth.log

# in both cases save your results
service iptables save

links:

https://wiki.debian.org/DebianFirewall

admin