Block agressive scraper

New IPs addresses arrived:

root@moma:/var/log/varnish# awk '/GET/ && !/128\.93\./ {print $1}' varnishncsa.log{,.1} | sort -n | uniq -c | sort -k1 -n | awk 'BEGIN{count=0}$1>10000 && $2 ~ /^(124|119|170|150)/{count+=$1;print}END{printf "total: %s\n", count}'
  10513 150.109.20.64
  10920 150.109.21.76
  18081 150.109.11.222
  18674 170.106.177.16
  18934 170.106.115.155
  18967 170.106.169.165
  18975 170.106.142.139
  19018 170.106.84.19
  19024 170.106.187.99
  19133 170.106.148.16
  19176 170.106.109.24
  19215 150.109.13.249
  19273 170.106.172.189
  19290 150.109.24.245
  19314 124.156.196.210
  19339 170.106.197.128
  19347 150.109.25.235
  19380 150.109.17.45
  19383 124.156.200.172
  19393 170.106.187.237
  19414 170.106.151.66
  19484 170.106.108.10
  19503 124.156.205.228
  19579 150.109.12.106
  19663 150.109.23.33
  19688 170.106.107.202
  19768 124.156.207.130
  19844 119.28.105.111
  19883 170.106.192.82
total: 542175

root@moma:/var/log/varnish# awk '/GET/ && !/128\.93\./ {print $1}' varnishncsa.log{,.1} | sort -n | uniq -c | sort -k2 -n | awk 'BEGIN{count=0}$1>10000 && $2 ~ /^(124|119|170|150|49|103|218|129)/{count+=$1;print}END{printf "total: %s\n", count}'         
  23725 49.51.70.245
  24042 49.51.206.134
  24102 49.51.70.229
  16422 103.177.248.124
  24734 119.28.105.111
  23911 124.156.196.210
  24139 124.156.205.228
  24205 124.156.200.172
  24230 124.156.207.130
  13701 129.226.146.19
  13853 129.226.151.215
  23620 129.226.192.111
  23837 129.226.91.207
  23843 129.226.158.117
  23893 129.226.92.4
  23981 129.226.192.224
  24057 129.226.154.196
  24171 129.226.150.55
  24256 129.226.193.30
  24549 129.226.209.118
  24575 129.226.92.236
  24600 129.226.156.129
  13478 150.109.20.64
  13824 150.109.21.76
  22708 150.109.11.222
  23831 150.109.13.249
  23839 150.109.24.245
  23895 150.109.25.235
  23912 150.109.17.45
  24294 150.109.12.106
  24312 150.109.23.33
  23434 170.106.177.16
  23574 170.106.187.99
  23583 170.106.169.165
  23618 170.106.84.19
  23637 170.106.115.155
  23775 170.106.142.139
  23782 170.106.172.189
  23825 170.106.109.24
  23828 170.106.148.16
  23866 170.106.197.128
  23934 170.106.187.237
  24033 170.106.108.10
  24123 170.106.151.66
  24235 170.106.107.202
  24604 170.106.192.82
  18822 218.107.211.73
total: 1073212

ᐅ whois 49.51.70.245 | grep -m 2 "country\|netname"
netname:        TencentCloud
country:        US
ᐅ whois 103.177.248.124 | grep -m 2 "country\|netname"
netname:        HOSTUPAB-248
country:        HK
ᐅ whois 119.28.105.111 | grep -m 2 "country\|netname"
netname:        TencentCloud
country:        SG
ᐅ whois 124.156.196.210 | grep -m 2 "country\|netname"
netname:        ACEVILLEPTELTD-SG
country:        IN
ᐅ whois 129.226.146.19 | grep -m 2 "country\|netname"
netname:        ACE-SG
country:        SG
ᐅ whois 150.109.20.64 | grep -m 2 "country\|netname"
netname:        ACEVILLEPTELTD-SG
country:        SG
ᐅ whois 170.106.177.16 | grep -m 2 "country\|netname"
netname:        ACEVILLEPTELTD-SG
country:        US
ᐅ whois 218.107.211.73 | grep -m 2 "country\|netname"
netname:        CNCGROUP-FJ-XIAMEN-MAN
country:        CN

There are still alerts TargetDown on pod web-archive and slowdowns on the webapp.
Here are the addresses with the most requests since yesterday:

root@moma:/var/log/varnish# awk '/GET/ && !/128\.93\./ {print $1}' varnishncsa.log{,.1} | sort -n | uniq -c | sort -k2 -n | awk 'BEGIN{count=0}$1>10000{count+=$1;print}END{printf "total: %s\n", count}'
  38283 3.224.220.101
  35188 23.22.35.162
  52680 49.51.70.245
  52690 49.51.206.134
  53121 49.51.70.229
  37973 52.70.240.171
  12013 66.249.79.194
  52663 119.28.105.111
  52045 124.156.207.130
  52105 124.156.196.210
  52196 124.156.205.228
  52609 124.156.200.172
  31819 129.226.146.19
  32420 129.226.151.215
  51538 129.226.158.117
  51788 129.226.91.207
  52156 129.226.209.118
  52190 129.226.92.4
  52196 129.226.154.196
  52271 129.226.192.111
  52444 129.226.193.30
  52521 129.226.192.224
  52641 129.226.156.129
  52705 129.226.92.236
  52800 129.226.150.55
  31214 150.109.20.64
  31972 150.109.21.76
  51653 150.109.17.45
  51803 150.109.11.222
  52144 150.109.13.249
  52630 150.109.25.235
  52750 150.109.23.33
  52754 150.109.12.106
  52772 150.109.24.245
  51368 170.106.84.19
  51775 170.106.177.16
  51818 170.106.108.10
  51983 170.106.142.139
  52024 170.106.109.24
  52043 170.106.187.99
  52058 170.106.151.66
  52133 170.106.169.165
  52153 170.106.192.82
  52182 170.106.148.16
  52296 170.106.197.128
  52402 170.106.115.155
  52407 170.106.107.202
  52442 170.106.172.189
  52468 170.106.187.237
total: 2394299

I updated the firewall INPUT chain:

root@moma:/var/log/varnish# iptables -nvL INPUT
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  538 31760 REJECT     all  --  *      *       49.51.64.0/18        0.0.0.0/0            reject-with icmp-port-unreachable
  139  7876 REJECT     all  --  *      *       119.28.105.111       0.0.0.0/0            reject-with icmp-port-unreachable
  988 63612 REJECT     all  --  *      *       124.156.0.0/16       0.0.0.0/0            reject-with icmp-port-unreachable
 2805  177K REJECT     all  --  *      *       129.226.0.0/16       0.0.0.0/0            reject-with icmp-port-unreachable
 1890  118K REJECT     all  --  *      *       150.109.0.0/16       0.0.0.0/0            reject-with icmp-port-unreachable
10362  631K REJECT     all  --  *      *       170.106.0.0/16       0.0.0.0/0            reject-with icmp-port-unreachable
6198K  372M REJECT     all  --  *      *       43.128.0.0/10        0.0.0.0/0            reject-with icmp-port-unreachable
64471 3868K DROP       all  --  *      *       101.32.240.178       0.0.0.0/0           
65354 3920K DROP       all  --  *      *       101.32.115.96        0.0.0.0/0           
64511 3870K DROP       all  --  *      *       101.32.254.77        0.0.0.0/0           
64438 3871K DROP       all  --  *      *       101.32.242.16        0.0.0.0/0           
64287 3856K DROP       all  --  *      *       101.32.244.173       0.0.0.0/0           
64896 3894K DROP       all  --  *      *       101.32.243.24        0.0.0.0/0           
2715K  163M REJECT     all  --  *      *       101.47.7.72          0.0.0.0/0            reject-with icmp-port-unreachable

IP 52.70.240.171 is Amazonbot:

root@moma:/var/log/varnish# awk '/52\.70\.240\.171/{first=$25}END{print substr(first,2,9)}' varnishncsa.log 
Amazonbot

IP 66.249.79.194 is Googlebot:

root@moma:/var/log/varnish# awk '/66\.249\.79\.194/{first=$NF}END{print substr(first,1,11)}' varnishncsa.log 
GoogleOther

IP 23.22.35.162 is Amazonbot:

root@moma:/var/log/varnish# awk '/23\.22\.35\.162/{first=$25}END{print substr(first,2,9)}' varnishncsa.log 
Amazonbot

IP 3.224.220.101 is Amazonbot:

root@moma:/var/log/varnish# awk '/3\.224\.220\.101/{first=$25}END{print substr(first,2,9)}' varnishncsa.log 
Amazonbot

ClaudeBot uses multiple IPs:

root@moma:/var/log/varnish# awk '/ClaudeBot\/1\.0/{print $1}' varnishncsa.log{,.1} | sort -n | uniq -c | sort -k2 -n | awk 'BEGIN{count=0}{count+=$1}END{printf "total: %s\n", count}'
total: 195786

ChatGPT-User is a lazy bot:

root@moma:/var/log/varnish# awk '/ChatGPT-User\/1\.0/{print $1}' varnishncsa.log{,.1} | sort -n | uniq -c | sort -k2 -n 
      1 40.84.221.231
      1 40.84.221.233

Bingbot is a gentle bot:

root@moma:/var/log/varnish# awk '/bingbot\/2\.0/{print $1}' varnishncsa.log{,.1} | sort -n | uniq -c | sort -k2 -n | awk 'BEGIN{count=0}{count+=$1}END{printf "total: %s\n", count}'
total: 24095

The bot with the most requests is Amazonbot.
The blocked address ranges made many many more requests than Amazonbot.
I will continue to monitor incoming traffic.

added 4h of time spent

Every time we've experienced slowdowns and downtimes, there were a huge number of writing connections as we can see in this graph:

Here is the total numbers of requests since 28/Jul/2024:00:00:11:

root@moma:/var/log/varnish# awk '{print $1}' varnishncsa.log{,.1} | sort -n | uniq -c | sort -k1 -n | awk 'BEGIN{count=0}{count+=$1}END{printf "total: %s\n", count}'
total: 4830378

Here is the number of Svix requests during the same period:

root@moma:/var/log/varnish# awk '/POST/&&/Svix-Webhooks/{print $1}' varnishncsa.log{,.1} | sort -n | uniq -c | sort -k1 -n | awk 'BEGIN{count=0}{count+=$1;print}END{printf "total: %s\n", count}'
3548622 128.93.166.2
total: 3548622

Here is the number of the other POST requests (output only displays IP addresses with more than 2k requests):

root@moma:/var/log/varnish# awk '/POST/&&!/Svix-Webhooks/{print $1}' varnishncsa.log{,.1} | sort -n | uniq -c | sort -k1 -n | awk 'BEGIN{count=0}{count+=$1;if($1>2000)print}END{printf "total: %s\n", count}'
   2282 128.93.166.14
total: 2878

Here is the number of the GET requests (output only displays IP addresses with more than 10k requests):

root@moma:/var/log/varnish# awk '/GET/{print $1}' varnishncsa.log{,.1} | sort -n | uniq -c | sort -k1 -n | awk 'BEGIN{count=0}{count+=$1;if($1>10000)print}END{printf "total: %s\n", count}'
  10666 66.249.76.194
  13380 66.249.79.194
  15276 101.33.123.172
  18484 108.24.114.237
  23277 64.124.8.200
  26944 74.80.208.146
  47465 128.93.166.14
  55838 23.22.35.162
  61411 3.224.220.101
  61566 52.70.240.171
total: 1276591

Svix seems to be the factor with the greatest impact.

Here is the last alert and the last unavailability we've encountered:

start at 9:02 p.m.

end at 9:21 p.m.

Here is the graph of the web-archive ingress (web-cassandra-ingress-default web-app1-ingress-default web-archive-ingress-default web-archive-ingress-webhooks):

Total number of requests between 21h00 and 21h29:

root@moma:/var/log/varnish# awk '/28\/Jul\/2024:19:[012]/{print $1}' varnishncsa.log.1 | sort -n | uniq -c | sort -k1 -n | awk 'BEGIN{count=0}{count+=$1;if($1>1000)print}END{printf "total: %s\n", count}'
  99382 128.93.166.2
total: 110486

Total number of Svix requests during the same period:

root@moma:/var/log/varnish# awk '/Svix-Webhooks/&&/28\/Jul\/2024:19:[012]/{print $1}' varnishncsa.log.1 | sort -n | uniq -c | sort -k1 -n | awk 'BEGIN{count=0}{count+=$1;if($1>1000)print}END{printf "total: %s\n", count}'
  99291 128.93.166.2
total: 99291

There were alerts on web-archive all night long.
There is a new scraper from Singapore:

root@moma:/var/log/varnish# awk '!/Svix-Webhooks/{print $1}' varnishncsa.log | sort -n | uniq -c | sort -k1 -n | awk 'BEGIN{count=0}{count+=$1;if($1>10000)print}END{printf "total: %s\n", count}'
  10158 23.22.35.162
  10188 128.93.166.14
  11160 52.70.240.171
  11267 3.224.220.101
 545227 101.47.8.29
total: 1126636

ᐅ whois 101.47.8.29 | grep -m 2 "country\|netname"
netname:        BYTEPLUS-SG
country:        SG

545k requests since midnight, IP address 101.47.8.29 is now blocked (I really wants to block the all range as soon as Singapore addresses are involved).

Note that the highest number of requests is always made by svix:

root@moma:/var/log/varnish# awk '/Svix-Webhooks/{print $1}' varnishncsa.log | sort -n | uniq -c | sort -k1 -n | awk 'BEGIN{count=0}{count+=$1;if($1>1000)print}END{printf "total: %s\n", count}'
 819790 128.93.166.2
total: 819790

800k since midnight, we should disable webhooks/svix for SCN statuses.

Here is the current firewall INPUT chain on moma:

root@moma:/var/log/varnish# iptables -nvL INPUT
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
45325 2736K REJECT     all  --  *      *       101.47.8.29          0.0.0.0/0            reject-with icmp-port-unreachable
   25  1500 REJECT     all  --  *      *       218.107.211.73       0.0.0.0/0            reject-with icmp-port-unreachable
 140K 8400K REJECT     all  --  *      *       49.51.206.134        0.0.0.0/0            reject-with icmp-port-unreachable
 431K   26M REJECT     all  --  *      *       49.51.64.0/18        0.0.0.0/0            reject-with icmp-port-unreachable
72639 4357K REJECT     all  --  *      *       119.28.105.111       0.0.0.0/0            reject-with icmp-port-unreachable
 441K   26M REJECT     all  --  *      *       124.156.0.0/16       0.0.0.0/0            reject-with icmp-port-unreachable
 940K   56M REJECT     all  --  *      *       129.226.0.0/16       0.0.0.0/0            reject-with icmp-port-unreachable
 653K   39M REJECT     all  --  *      *       150.109.0.0/16       0.0.0.0/0            reject-with icmp-port-unreachable
3248K  195M REJECT     all  --  *      *       170.106.0.0/16       0.0.0.0/0            reject-with icmp-port-unreachable
  29M 1720M REJECT     all  --  *      *       43.128.0.0/10        0.0.0.0/0            reject-with icmp-port-unreachable
 281K   17M DROP       all  --  *      *       101.32.240.178       0.0.0.0/0           
 282K   17M DROP       all  --  *      *       101.32.115.96        0.0.0.0/0           
 281K   17M DROP       all  --  *      *       101.32.254.77        0.0.0.0/0           
 281K   17M DROP       all  --  *      *       101.32.242.16        0.0.0.0/0           
 279K   17M DROP       all  --  *      *       101.32.244.173       0.0.0.0/0           
 280K   17M DROP       all  --  *      *       101.32.243.24        0.0.0.0/0           
2715K  163M REJECT     all  --  *      *       101.47.7.72          0.0.0.0/0            reject-with icmp-port-unreachable

added 1h of time spent

mentioned in commit swh/infra/ci-cd/swh-charts@5ecb615a

mentioned in commit swh/infra/ci-cd/swh-charts@d1aea483

I have deactivated swh-webhooks and activated the cronjob "refresh scn statuses".
Svix-server continued to send a lots of requests to web-archive.
Hence I have deactivated the svix-server.
There is a syncwindow on argocd, I just synced svix-server deployment with prune option and I have left the service and the network policies untouched.
There are no more alerts but web-archive have still some slowdowns.

added 3h of time spent

There are no more alerts but web-archive have still some slowdowns.

The archive got a bit stressed so i increased slightly the resources and activated the autoscaling (from 6 to 8 pods vs the current hard-coded 4). We'll see if that relieves it a bit.

[1] https://grafana.softwareheritage.org/goto/-yUSR59Iz?orgId=1

Who made a high number of requests since 11 a.m. (last firewall update):

root@moma:/var/log/varnish# awk '/02\/Aug\/2024:09/{split($1,a,".");print a[1]"."a[2]}' varnishncsa.log | sort -n | uniq -c | awk 'BEGIN{count=0}$1>1000{count+=$1;print}END{printf "total: %s\n", count}'
  20338 101.47
   2202 110.249
   1929 111.225
  17291 128.93
   1634 152.39
   1771 161.123
total: 45165

101.47 looks like a known scraper IP already blocked:

root@moma:/var/log/varnish# awk '/02\/Aug\/2024:09/&&$1~/^101\.47/{print $1}' varnishncsa.log | sort -n | uniq -c | awk 'BEGIN{count=0}$1>1000{count+=$1;print}END{printf "total: %s\n", count}'
  20338 101.47.11.243
total: 20338

whois 101.47.11.243 | grep -m 2 "country\|netname"
netname:        BYTEPLUS-SG
country:        SG

101.47.11.243 is now blocked.

added 40m of time spent

Block agressive scraper

Designs

Child items ...

Activity