CRIN Trac: Ticket #94: Ongoing high load caused by a bot

attachment set

chris — Thu, 15 Dec 2016 19:28:13 GMT

attachment set to memory-day.png

attachment set

chris — Thu, 15 Dec 2016 19:28:25 GMT

attachment set to load-day.png

attachment set

chris — Thu, 15 Dec 2016 19:28:36 GMT

attachment set to cpu-day.png

attachment set

chris — Thu, 15 Dec 2016 19:29:10 GMT

attachment set to multips_memory-day.png

attachment set

chris — Thu, 15 Dec 2016 19:29:21 GMT

attachment set to multips-day.png

attachment set

chris — Thu, 15 Dec 2016 19:29:33 GMT

attachment set to phpfpm_status-day.png

attachment set

chris — Thu, 15 Dec 2016 19:29:45 GMT

attachment set to phpfpm_memory-day.png

attachment set

chris — Thu, 15 Dec 2016 19:30:13 GMT

attachment set to phpfpm_memory-day.2.png

attachment set

chris — Thu, 15 Dec 2016 19:30:54 GMT

attachment set to nginx_status-day.png

attachment set

chris — Thu, 15 Dec 2016 19:31:42 GMT

attachment set to http_loadtime-day.png

attachment set

chris — Thu, 15 Dec 2016 19:31:57 GMT

attachment set to fw_packets-day.png

attachment set

chris — Thu, 15 Dec 2016 19:32:09 GMT

attachment set to if_eth0-day.png

attachment set

chris — Thu, 15 Dec 2016 19:32:23 GMT

attachment set to fw_conntrack-day.png

attachment set

chris — Thu, 15 Dec 2016 19:32:36 GMT

attachment set to memcached_rates-day.png

hours changed; totalhours set

chris — Thu, 15 Dec 2016 19:45:01 GMT

hours changed from 0 to 0.5
totalhours set to 0.5

The PHP server, Crin2 is really suffering:

top - 19:23:57 up 228 days, 6 min,  2 users,  load average: 7.24, 8.24, 8.42
Tasks: 181 total,  11 running, 170 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.3 us,  0.2 sy,  0.0 ni, 92.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   8195036 total,  7548024 used,   647012 free,    48340 buffers
KiB Swap:  5468156 total,   139116 used,  5329040 free.  1132416 cached Mem
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
28407 www-data  20   0  622320 260800  43776 R 100.0  3.2  26:28.39 php5-fpm
28965 www-data  20   0  781076 422372  46692 R 100.0  5.2  25:13.99 php5-fpm
28408 www-data  20   0  503676 145892  46688 R  95.7  1.8  25:59.46 php5-fpm
28411 www-data  20   0  506228 145824  43980 R  95.7  1.8  25:10.23 php5-fpm
28415 www-data  20   0  528292 173916  52080 R  95.7  2.1  24:48.56 php5-fpm
28975 www-data  20   0  497376 135620  42848 R  95.7  1.7  24:58.28 php5-fpm
28976 www-data  20   0  644020 282436  43636 R  95.7  3.4  25:07.01 php5-fpm
28405 www-data  20   0  614152 253392  43880 R  89.3  3.1  26:19.49 php5-fpm
28966 www-data  20   0  470168 108520  42912 R  89.3  1.3  23:57.28 php5-fpm
28974 www-data  20   0  631000 270404  43256 R  89.3  3.3  24:34.78 php5-fpm
...

Some graphs of the ongoing high load:

And the bot responsible:

Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)

The number of requests:

grep "SemrushBot/" /var/log/nginx/crin.org.ssl_access.log | wc -l
15866

And an example request:

46.229.168.66 - - [15/Dec/2016:19:36:41 +0000] "GET /en/library/custom-search-legal?f%5B0%5D=field_date%3Avalue%3A%5B2008-01-01T00%3A00%3A00Z%20TO%202009-01-01T00%3A00%3A00Z%5D&f%5B1%5D=field_date%3Avalue%3A%5B2006-01-01T00%3A00%3A00Z%20TO%202007-01-01T00%3A00%3A00Z%5D&qt-countr-tabs=3 HTTP/1.1" 200 19601 "-" "Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)"

So we could simply block this bot, or rate limit it, notes that the reason it is generating a high load is because it is requesting searches.

At the moment we have this in /etc/nginx/nginx.conf:

limit_req_zone  $binary_remote_addr  zone=one:10m   rate=6r/s;

And I can't see a simple way right now to add a lower rate for one IP address so using the ip tables script to block it:

ipdrop 46.229.168.66

And I'll check back later to see the results...

chris — Thu, 15 Dec 2016 19:46:32 GMT

It is using more than one IP:

46.229.168.71 - - [15/Dec/2016:19:45:46 +0000] "GET /en/library/custom-search-legal?f%5B0%5D=field_date%3Avalue%3A%5B2013-01-01T00%3A00%3A00Z%20TO%202014-01-01T00%3A00%3A00Z%5D&f%5B1%5D=field_date%3Avalue%3A%5B1997-01-01T00%3A00%3A00Z%20TO%201998-01-01T00%3A00%3A00Z%5D&field_country=All&field_country_1=All&field_crc=All&field_instruments=All&field_monitoring_body=All&field_scope=All&field_themes=All&promo=1&search_api_language=current HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)"

So:

ipdrop 46.229.168.71

chris — Thu, 15 Dec 2016 19:48:49 GMT

And a few more:

ipdrop 46.229.168.67
ipdrop 46.229.168.72
ipdrop 46.229.168.73
ipdrop 46.229.168.69
ipdrop 46.229.168.70
ipdrop 46.229.168.74
ipdrop 46.229.168.65
ipdrop 46.229.168.68