<?xml version="1.0"?>
<rss xmlns:dc="http://purl.org/dc/elements/1.1/" version="2.0">
  <channel>
    <title>CRIN Trac: Ticket #105: High load and slow responses from Crin2</title>
    <link>https://trac.crin.org/trac/ticket/105</link>
    <description>&lt;p&gt;
Trac threw this error:
&lt;/p&gt;
&lt;blockquote class="citation"&gt;
&lt;p&gt;
Warning:
&lt;/p&gt;
&lt;ul&gt;&lt;li&gt;Error with navigation contributor "&lt;a class="missing wiki"&gt;TimingEstimationAndBillingPage?&lt;/a&gt;"
&lt;/li&gt;&lt;li&gt;Error with navigation contributor "&lt;a class="missing wiki"&gt;AdminModule?&lt;/a&gt;"
&lt;/li&gt;&lt;li&gt;Error with navigation contributor "&lt;a class="missing wiki"&gt;SearchModule?&lt;/a&gt;"
&lt;/li&gt;&lt;li&gt;Failed to sync with repository "(default)": database is locked; repository information may be out of date. Look in the Trac log for more information including mitigation strategies.
&lt;/li&gt;&lt;/ul&gt;&lt;/blockquote&gt;
&lt;p&gt;
And in &lt;tt&gt;/var/log/php5-fpm.log&lt;/tt&gt; we have:
&lt;/p&gt;
&lt;pre class="wiki"&gt;[27-Mar-2017 10:20:42] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 10:30:10] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:54:51] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:56:35] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:57:08] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:57:58] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 12:01:48] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 12:02:26] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
&lt;/pre&gt;&lt;p&gt;
From an initial look at the Nginx logs the problem seesm to be caused but too much agressive crawling of the site by bots.
&lt;/p&gt;
</description>
    <language>en-us</language>
    <image>
      <title>CRIN Trac</title>
      <url>https://trac.crin.org/trac/chrome/site/logo.gif</url>
      <link>https://trac.crin.org/trac/ticket/105</link>
    </image>
    <generator>Trac 1.0.2</generator>
    <item>
      
        <dc:creator>chris</dc:creator>

      <pubDate>Mon, 27 Mar 2017 12:32:41 GMT</pubDate>
      <title>hours, status changed; resolution, totalhours set</title>
      <link>https://trac.crin.org/trac/ticket/105#comment:1</link>
      <guid isPermaLink="false">https://trac.crin.org/trac/ticket/105#comment:1</guid>
      <description>
          &lt;ul&gt;
            &lt;li&gt;&lt;strong&gt;hours&lt;/strong&gt;
                changed from &lt;em&gt;0&lt;/em&gt; to &lt;em&gt;0.5&lt;/em&gt;
            &lt;/li&gt;
            &lt;li&gt;&lt;strong&gt;status&lt;/strong&gt;
                changed from &lt;em&gt;new&lt;/em&gt; to &lt;em&gt;closed&lt;/em&gt;
            &lt;/li&gt;
            &lt;li&gt;&lt;strong&gt;resolution&lt;/strong&gt;
                set to &lt;em&gt;fixed&lt;/em&gt;
            &lt;/li&gt;
            &lt;li&gt;&lt;strong&gt;totalhours&lt;/strong&gt;
                set to &lt;em&gt;0.5&lt;/em&gt;
            &lt;/li&gt;
          &lt;/ul&gt;
        &lt;p&gt;
A lot of requests were coming form &lt;a class="ext-link" href="https://apps.db.ripe.net/search/query.html?searchtext=95.213.191.13"&gt;&lt;span class="icon"&gt;​&lt;/span&gt;a Russian IP address&lt;/a&gt; which appears to be crawling the site but getting a lote of 301's due to the Nginx rate limiting:
&lt;/p&gt;
&lt;pre class="wiki"&gt; grep "95.213.191.13" /var/log/nginx/crin.org.access.log | wc -l
1451
&lt;/pre&gt;&lt;pre class="wiki"&gt;grep "95.213.191.13" /var/log/nginx/crin.org.access.log | awk '{ print $9 }' | grep 200 | wc -l
491
grep "95.213.191.13" /var/log/nginx/crin.org.access.log | awk '{ print $9 }' | grep 301 | wc -l
943
&lt;/pre&gt;&lt;p&gt;
I have blocked that IP address.
&lt;/p&gt;
&lt;p&gt;
Another one causing problems:
&lt;/p&gt;
&lt;pre class="wiki"&gt;grep 163.172.71.23  /var/log/nginx/crin.org.access.log | wc -l
1961
&lt;/pre&gt;&lt;p&gt;
Again this bot is getting more 301 responses than 200 responses:
&lt;/p&gt;
&lt;pre class="wiki"&gt;grep 163.172.71.23  /var/log/nginx/crin.org.access.log | awk '{ print $9 }' | grep 200 | wc -l
807
grep 163.172.71.23  /var/log/nginx/crin.org.access.log | awk '{ print $9 }' | grep 301 | wc -l
1149
&lt;/pre&gt;&lt;p&gt;
This &lt;a class="ext-link" href="https://apps.db.ripe.net/search/query.html?searchtext=163.172.71.23"&gt;&lt;span class="icon"&gt;​&lt;/span&gt;French&lt;/a&gt; IP address has also been blocked.
&lt;/p&gt;
&lt;p&gt;
The load spike now appears to have passed.
&lt;/p&gt;
      </description>
      <category>Ticket</category>
    </item>
 </channel>
</rss>