High load and slow responses from Crin2

[chris]

Trac threw this error:

Warning:

• Error with navigation contributor "TimingEstimationAndBillingPage?"
• Error with navigation contributor "AdminModule?"
• Error with navigation contributor "SearchModule?"
• Failed to sync with repository "(default)": database is locked; repository information may be out of date. Look in the Trac log for more information including mitigation strategies.

And in /var/log/php5-fpm.log we have:

[27-Mar-2017 10:20:42] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 10:30:10] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:54:51] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:56:35] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:57:08] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:57:58] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 12:01:48] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 12:02:26] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it

From an initial look at the Nginx logs the problem seesm to be caused but too much agressive crawling of the site by bots.

[chris]

A lot of requests were coming form ​a Russian IP address which appears to be crawling the site but getting a lote of 301's due to the Nginx rate limiting:

 grep "95.213.191.13" /var/log/nginx/crin.org.access.log | wc -l
1451

grep "95.213.191.13" /var/log/nginx/crin.org.access.log | awk '{ print $9 }' | grep 200 | wc -l
491
grep "95.213.191.13" /var/log/nginx/crin.org.access.log | awk '{ print $9 }' | grep 301 | wc -l
943

I have blocked that IP address.

Another one causing problems:

grep 163.172.71.23  /var/log/nginx/crin.org.access.log | wc -l
1961

Again this bot is getting more 301 responses than 200 responses:

grep 163.172.71.23  /var/log/nginx/crin.org.access.log | awk '{ print $9 }' | grep 200 | wc -l
807
grep 163.172.71.23  /var/log/nginx/crin.org.access.log | awk '{ print $9 }' | grep 301 | wc -l
1149

This ​French IP address has also been blocked.

The load spike now appears to have passed.