#105 closed defect (fixed)

High load and slow responses from Crin2

Reported by: chris Owned by: chris
Priority: major Milestone: Maintenance
Component: crin2 Version:
Keywords: Cc:
Estimated Number of Hours: 0 Add Hours to Ticket: 0
Billable?: yes Total Hours: 0.5

Description

Trac threw this error:

Warning:

  • Error with navigation contributor "TimingEstimationAndBillingPage?"
  • Error with navigation contributor "AdminModule?"
  • Error with navigation contributor "SearchModule?"
  • Failed to sync with repository "(default)": database is locked; repository information may be out of date. Look in the Trac log for more information including mitigation strategies.

And in /var/log/php5-fpm.log we have:

[27-Mar-2017 10:20:42] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 10:30:10] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:54:51] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:56:35] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:57:08] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:57:58] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 12:01:48] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 12:02:26] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it

From an initial look at the Nginx logs the problem seesm to be caused but too much agressive crawling of the site by bots.

Change History (1)

comment:1 Changed 18 months ago by chris

  • Add Hours to Ticket changed from 0 to 0.5
  • Resolution set to fixed
  • Status changed from new to closed
  • Total Hours set to 0.5

A lot of requests were coming form a Russian IP address which appears to be crawling the site but getting a lote of 301's due to the Nginx rate limiting:

 grep "95.213.191.13" /var/log/nginx/crin.org.access.log | wc -l
1451
grep "95.213.191.13" /var/log/nginx/crin.org.access.log | awk '{ print $9 }' | grep 200 | wc -l
491
grep "95.213.191.13" /var/log/nginx/crin.org.access.log | awk '{ print $9 }' | grep 301 | wc -l
943

I have blocked that IP address.

Another one causing problems:

grep 163.172.71.23  /var/log/nginx/crin.org.access.log | wc -l
1961

Again this bot is getting more 301 responses than 200 responses:

grep 163.172.71.23  /var/log/nginx/crin.org.access.log | awk '{ print $9 }' | grep 200 | wc -l
807
grep 163.172.71.23  /var/log/nginx/crin.org.access.log | awk '{ print $9 }' | grep 301 | wc -l
1149

This French IP address has also been blocked.

The load spike now appears to have passed.

Note: See TracTickets for help on using tickets.