Firewall detects many attacks

Today I was checking my logs and noticed this entry. As we can see, within about 600ms, an attacker was trying to connect to many different ports (20480, 20736, 36895, 37151, 22528, 16671, 14340, 20992, 4135, 64288, 45090, 21248, 21504, 31775, 39455, 42254, 47115.)

Note: I hid the destination URL (x.x.x.x) on purpose. However, I did not hide the source!

Jul 23 11:20:34 finball1 kernel: [1661019.650298] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=52074 PROTO=TCP SPT=26091 DPT=20480 WINDOW=131 RES=0x00 SYN URGP=0
Jul 23 11:20:34 finball1 kernel: [1661019.654113] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=59130 PROTO=TCP SPT=26091 DPT=20736 WINDOW=131 RES=0x00 SYN URGP=0
Jul 23 11:20:34 finball1 kernel: [1661019.657914] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=38761 PROTO=TCP SPT=26091 DPT=36895 WINDOW=131 RES=0x00 SYN URGP=0
Jul 23 11:20:34 finball1 kernel: [1661019.661723] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=16890 PROTO=TCP SPT=26091 DPT=37151 WINDOW=131 RES=0x00 SYN URGP=0
Jul 23 11:20:34 finball1 kernel: [1661019.665596] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=46093 PROTO=TCP SPT=26091 DPT=22528 WINDOW=131 RES=0x00 SYN URGP=0
Jul 23 11:20:34 finball1 kernel: [1661019.669753] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=6736  PROTO=TCP SPT=26091 DPT=16671 WINDOW=131 RES=0x00 SYN URGP=0
Jul 23 11:20:34 finball1 kernel: [1661019.674068] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=46231 PROTO=TCP SPT=26091 DPT=14340 WINDOW=131 RES=0x00 SYN URGP=0
Jul 23 11:20:34 finball1 kernel: [1661019.678177] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=41886 PROTO=TCP SPT=26091 DPT=20992 WINDOW=131 RES=0x00 SYN URGP=0
Jul 23 11:20:34 finball1 kernel: [1661019.682235] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=27989 PROTO=TCP SPT=26091 DPT=4135  WINDOW=131 RES=0x00 SYN URGP=0
Jul 23 11:20:34 finball1 kernel: [1661019.686393] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=48924 PROTO=TCP SPT=26091 DPT=64288 WINDOW=131 RES=0x00 SYN URGP=0
Jul 23 11:20:34 finball1 kernel: [1661019.690583] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=54951 PROTO=TCP SPT=26091 DPT=45090 WINDOW=131 RES=0x00 SYN URGP=0
Jul 23 11:20:34 finball1 kernel: [1661019.694452] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=16235 PROTO=TCP SPT=26091 DPT=21248 WINDOW=131 RES=0x00 SYN URGP=0
Jul 23 11:20:34 finball1 kernel: [1661019.698426] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=60093 PROTO=TCP SPT=26091 DPT=21504 WINDOW=131 RES=0x00 SYN URGP=0
Jul 23 11:20:34 finball1 kernel: [1661019.702404] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=9954  PROTO=TCP SPT=26091 DPT=31775 WINDOW=131 RES=0x00 SYN URGP=0
Jul 23 11:20:34 finball1 kernel: [1661019.706673] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=33443 PROTO=TCP SPT=26091 DPT=39455 WINDOW=131 RES=0x00 SYN URGP=0
Jul 23 11:20:34 finball1 kernel: [1661019.711026] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=52979 PROTO=TCP SPT=26091 DPT=42254 WINDOW=131 RES=0x00 SYN URGP=0
Jul 23 11:20:34 finball1 kernel: [1661019.715426] [iptables] unknown: IN=eth0 OUT= SRC=87.62.140.210 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=57 ID=9032  PROTO=TCP SPT=26091 DPT=47115 WINDOW=131 RES=0x00 SYN URGP=0

The result?

The attacker got blocked within another 300ms or so:

2018/07/23 11:20:35 finball1 snapfirewall[24866]: trace: received messenger message [*/BLOCK period=year;reason=fail2ban: iptables;uri=all://87.62.140.210] for finball1 (in function "process_message()") (snapfirewall.cpp:1994)

Our Snap! Websites system automatically adds all the necessary rules to block potential intruders in this way. This prevents hackers from testing much really fast because if you let them do that, they can scan your entire system and possibly find a weakness. By blocking them quickly, it makes it much harder for them to find a weakness. Note that we receive similar hits from many other IP addresses, only often times these accesses are not repeated over and over like this.

Notice the period parameter. it is set to "year". That means that IP address will be blocked "forever" (as far as the Internet goes, this is a really long time!)

The reason parameter explains where the data was found and by whom. Here it is fail2ban and the source of the logs was iptables.

Most of our front end facing tools check various parameters to determine whether an access is considered valid or not. For example, we do not allow proxying to our systems. When that happens, the remote IP address for that access gets banned as well.

The following is a ban in link with a proxy. This means this hacker tried to access a computer through our Apache server. In other words, they were trying to hide their IP address from the destination Apache server. Note that in our case here, we block the IP address very quickly too (with 1 second) and pretty much permanently.

2018/07/23 09:54:10 finball3 snapfirewall[5639]: trace: received messenger message [*/BLOCK period=year;reason=fail2ban: apache proxy;uri=all://93.171.65.149] for finball3 (in function "process_message()") (snapfirewall.cpp:1994)

One of our machine has what I call "a known IP address," an address that some hacker discovered to host tools that they could readily penetrate and play with. In other words, they could use that computer for free. Very lucky we got that computer IP address and are getting a lot of bad guys doing such accesses all the time. This is wonderful for us to train our systems and make sure that the firewall is functional (it is although with such permanent attacks it's a tad bit big... but it allowed us to better setup fail2ban so it would not itself fail all the time!)

Note that even though we block IP addresses that way, we still get a large amount of banned IPs. To give you an idea, we banned 429 IPs on Jul 24, 2018 and 328 on Jul 25, 2018. However, Jul 23, 2018 we got a whooping 797 and on Jul 26, 2018 (which is not over yet) we got a huge spike at 1956 BLOCKs in about 16 hours. This gives you an idea of how bad this computer IP address is! (And I hope we can keep it forever! It's an excellent test bed. In comparison, the other IPs we have get one or two a day. Not much going on...)

This following algorithm shows how one of our tools triggers a block of an IP address.This image shows the basic algorithm. As we can see, a tool sends a BLOCK message to snapcommunicator on the computer that tool is running. The message is expected to be a broadcast message so it gets propagated to other computers (here I show Server 2 and Server 3.)

The snapcommunicator service also forwards the BLOCK message to all the daemons that said they understood that message. At this time this is snapfirewall.

If the IP address is already banned, then it does nothing with iplock/iptables unless the port information is different. It will always register the new date in the database, though. This is important since the ban period starts over on this new hit.

Assuming the IP must now be blocked or the block modified, the iplock tool is run with the new information. The iplock tool is used to block and unblock IP addresses in iptables. It can become root which is why it is separate from the daemon which runs as snapwebsites instead (for obvious security reasons.) The iplock is very limited in what it can do to make the process as secure as possible. It offers to add and remove IP addresses in a specific (i.e. named) firewall CHAIN.

The iptables tool is the standard firewall tool offered with Linux. It is expected to already be installed. Note that when you install the snapfirewall, one of the first thing that happens is the installation of a large set of rules that will immediately block access to most of your daemons. For this reason alone, it should always be a priority to very quickly install snapfirewall on your Snap! servers. Otherwise it could happen that a hacker gets a hold of one of your services. Our rules are setup on a reboot before any of our services start (pretty much any services, actually!) So as soon as snapfirewall is installed, your server is quite safe.

Note that you will be asked to enter the administrator IP address. This is very important because the firewall will be setup in a way that allows the administrator to access many parts of the service. This should NOT be the IP address of an Internet Cafe. Please. It is expected to be a static IP. If you don't have a static IP address, you should be able to tweak the setup to match any one IP you may get, but in such a situation, please make sure you have a form of backup (i.e. a way to access the computer remotely without using SSH or Apache.)

Snap! Websites
An Open Source CMS System in C++

Contact Us Directly