https://trac.crin.org/trac/ticket/82 <p> Yesterday evening there 1k 502 errors caused by one user agent from multiple IP addresses. This isn't an uncommon event, I usually don't write them up, but I thought it might be useful for Mathew if I do with this one. </p> en-us https://trac.crin.org/trac/chrome/site/logo.gif https://trac.crin.org/trac/ticket/82 Trac 1.0.2 chris Sat, 24 Sep 2016 09:48:43 GMT https://trac.crin.org/trac/ticket/82 https://trac.crin.org/trac/ticket/82 <ul> <li><strong>attachment</strong> set to <em>fw_conntrack-day.png</em> </li> </ul> Ticket chris Sat, 24 Sep 2016 09:48:57 GMT https://trac.crin.org/trac/ticket/82 https://trac.crin.org/trac/ticket/82 <ul> <li><strong>attachment</strong> set to <em>fw_forwarded_local-day.png</em> </li> </ul> Ticket chris Sat, 24 Sep 2016 09:49:13 GMT https://trac.crin.org/trac/ticket/82 https://trac.crin.org/trac/ticket/82 <ul> <li><strong>attachment</strong> set to <em>fw_packets-day.png</em> </li> </ul> Ticket chris Sat, 24 Sep 2016 09:49:27 GMT https://trac.crin.org/trac/ticket/82 https://trac.crin.org/trac/ticket/82 <ul> <li><strong>attachment</strong> set to <em>if_eth0-day.png</em> </li> </ul> Ticket chris Sat, 24 Sep 2016 09:49:42 GMT https://trac.crin.org/trac/ticket/82 https://trac.crin.org/trac/ticket/82 <ul> <li><strong>attachment</strong> set to <em>load-day.png</em> </li> </ul> Ticket chris Sat, 24 Sep 2016 09:49:58 GMT https://trac.crin.org/trac/ticket/82 https://trac.crin.org/trac/ticket/82 <ul> <li><strong>attachment</strong> set to <em>multips_memory-day.png</em> </li> </ul> Ticket chris Sat, 24 Sep 2016 09:50:23 GMT https://trac.crin.org/trac/ticket/82 https://trac.crin.org/trac/ticket/82 <ul> <li><strong>attachment</strong> set to <em>memory-day.png</em> </li> </ul> Ticket chris Sat, 24 Sep 2016 09:50:41 GMT https://trac.crin.org/trac/ticket/82 https://trac.crin.org/trac/ticket/82 <ul> <li><strong>attachment</strong> set to <em>nginx_request-day.png</em> </li> </ul> Ticket chris Sat, 24 Sep 2016 09:50:58 GMT https://trac.crin.org/trac/ticket/82 https://trac.crin.org/trac/ticket/82 <ul> <li><strong>attachment</strong> set to <em>nginx_status-day.png</em> </li> </ul> Ticket chris Sat, 24 Sep 2016 09:51:20 GMT https://trac.crin.org/trac/ticket/82 https://trac.crin.org/trac/ticket/82 <ul> <li><strong>attachment</strong> set to <em>phpfpm_average-day.png</em> </li> </ul> Ticket chris Sat, 24 Sep 2016 09:51:35 GMT https://trac.crin.org/trac/ticket/82 https://trac.crin.org/trac/ticket/82 <ul> <li><strong>attachment</strong> set to <em>phpfpm_connections-day.png</em> </li> </ul> Ticket chris Sat, 24 Sep 2016 09:51:53 GMT https://trac.crin.org/trac/ticket/82 https://trac.crin.org/trac/ticket/82 <ul> <li><strong>attachment</strong> set to <em>phpfpm_memory-day.png</em> </li> </ul> Ticket chris Sat, 24 Sep 2016 10:12:50 GMT https://trac.crin.org/trac/ticket/82#comment:1 https://trac.crin.org/trac/ticket/82#comment:1 <ul> <li><strong>hours</strong> changed from <em>0</em> to <em>0.82</em> </li> <li><strong>totalhours</strong> set to <em>0.82</em> </li> </ul> <p> The spike is very clear on <a class="ext-link" href="https://munin.crin.org/munin/crin.org/crin2.crin.org/index.html"><span class="icon">​</span>the munin graphs</a>: </p> <p> <a style="padding:0; border:none" href="https://trac.crin.org/trac/attachment/ticket/82/fw_conntrack-day.png"><img src="https://trac.crin.org/trac/raw-attachment/ticket/82/fw_conntrack-day.png" /></a> <a style="padding:0; border:none" href="https://trac.crin.org/trac/attachment/ticket/82/fw_forwarded_local-day.png"><img src="https://trac.crin.org/trac/raw-attachment/ticket/82/fw_forwarded_local-day.png" /></a> <a style="padding:0; border:none" href="https://trac.crin.org/trac/attachment/ticket/82/fw_packets-day.png"><img src="https://trac.crin.org/trac/raw-attachment/ticket/82/fw_packets-day.png" /></a> <a style="padding:0; border:none" href="https://trac.crin.org/trac/attachment/ticket/82/if_eth0-day.png"><img src="https://trac.crin.org/trac/raw-attachment/ticket/82/if_eth0-day.png" /></a> <a style="padding:0; border:none" href="https://trac.crin.org/trac/attachment/ticket/82/load-day.png"><img src="https://trac.crin.org/trac/raw-attachment/ticket/82/load-day.png" /></a> <a style="padding:0; border:none" href="https://trac.crin.org/trac/attachment/ticket/82/multips_memory-day.png"><img src="https://trac.crin.org/trac/raw-attachment/ticket/82/multips_memory-day.png" /></a> <a style="padding:0; border:none" href="https://trac.crin.org/trac/attachment/ticket/82/memory-day.png"><img src="https://trac.crin.org/trac/raw-attachment/ticket/82/memory-day.png" /></a> <a style="padding:0; border:none" href="https://trac.crin.org/trac/attachment/ticket/82/nginx_request-day.png"><img src="https://trac.crin.org/trac/raw-attachment/ticket/82/nginx_request-day.png" /></a> <a style="padding:0; border:none" href="https://trac.crin.org/trac/attachment/ticket/82/nginx_status-day.png"><img src="https://trac.crin.org/trac/raw-attachment/ticket/82/nginx_status-day.png" /></a> <a style="padding:0; border:none" href="https://trac.crin.org/trac/attachment/ticket/82/phpfpm_average-day.png"><img src="https://trac.crin.org/trac/raw-attachment/ticket/82/phpfpm_average-day.png" /></a> <a style="padding:0; border:none" href="https://trac.crin.org/trac/attachment/ticket/82/phpfpm_connections-day.png"><img src="https://trac.crin.org/trac/raw-attachment/ticket/82/phpfpm_connections-day.png" /></a> <a style="padding:0; border:none" href="https://trac.crin.org/trac/attachment/ticket/82/phpfpm_memory-day.png"><img src="https://trac.crin.org/trac/raw-attachment/ticket/82/phpfpm_memory-day.png" /></a> </p> <p> The number of requests made, based on the UA string: </p> <pre class="wiki">grep "Pcore-HTTP/v0.24.5" *log.1 | wc -l 10455 </pre><p> The unique IP addresses: </p> <pre class="wiki">grep "Pcore-HTTP/v0.24.5" *log.1 | awk '{print $1}' | sort -u crin.org.access.log.1:104.144.17.249 crin.org.access.log.1:104.144.228.196 crin.org.access.log.1:104.151.100.125 crin.org.access.log.1:104.151.121.80 crin.org.access.log.1:104.151.127.209 crin.org.access.log.1:104.151.28.124 crin.org.access.log.1:104.151.74.125 crin.org.access.log.1:104.168.13.147 crin.org.access.log.1:104.168.13.149 crin.org.access.log.1:104.168.13.152 crin.org.access.log.1:104.168.14.171 crin.org.access.log.1:104.168.2.225 crin.org.access.log.1:104.168.2.226 crin.org.access.log.1:104.168.2.227 crin.org.access.log.1:104.168.2.228 crin.org.access.log.1:104.168.23.238 crin.org.access.log.1:104.203.19.124 crin.org.access.log.1:104.223.19.131 crin.org.access.log.1:104.227.202.147 crin.org.access.log.1:104.227.203.135 crin.org.access.log.1:104.227.205.77 crin.org.access.log.1:107.172.100.5 crin.org.access.log.1:107.172.100.6 crin.org.access.log.1:107.172.100.7 crin.org.access.log.1:107.172.100.8 crin.org.access.log.1:107.172.100.9 crin.org.access.log.1:107.172.234.96 crin.org.access.log.1:107.172.235.204 crin.org.access.log.1:107.172.96.4 crin.org.access.log.1:107.172.96.5 crin.org.access.log.1:107.172.96.6 crin.org.access.log.1:107.172.96.7 crin.org.access.log.1:107.172.96.8 crin.org.access.log.1:107.172.97.21 crin.org.access.log.1:107.172.97.8 crin.org.access.log.1:107.172.97.9 crin.org.access.log.1:107.172.98.4 crin.org.access.log.1:107.172.98.6 crin.org.access.log.1:107.172.98.7 crin.org.access.log.1:107.172.98.8 crin.org.access.log.1:107.172.98.9 crin.org.access.log.1:107.172.99.5 crin.org.access.log.1:107.172.99.6 crin.org.access.log.1:107.172.99.7 crin.org.access.log.1:107.172.99.8 crin.org.access.log.1:107.172.99.9 crin.org.access.log.1:107.174.252.21 crin.org.access.log.1:107.174.252.22 crin.org.access.log.1:107.174.252.5 crin.org.access.log.1:107.174.252.7 crin.org.access.log.1:107.174.252.8 crin.org.access.log.1:107.174.252.9 crin.org.access.log.1:107.174.253.2 crin.org.access.log.1:107.174.253.4 crin.org.access.log.1:107.174.253.6 crin.org.access.log.1:107.174.253.7 crin.org.access.log.1:107.174.253.8 crin.org.access.log.1:107.174.253.9 crin.org.access.log.1:107.174.254.24 crin.org.access.log.1:107.174.254.8 crin.org.access.log.1:107.174.254.9 crin.org.access.log.1:107.174.255.21 crin.org.access.log.1:107.174.255.5 crin.org.access.log.1:107.174.255.6 crin.org.access.log.1:107.174.255.7 crin.org.access.log.1:107.174.255.8 crin.org.access.log.1:107.174.255.9 crin.org.access.log.1:107.175.227.185 crin.org.access.log.1:107.175.236.140 crin.org.access.log.1:107.183.10.222 crin.org.access.log.1:107.183.117.80 crin.org.access.log.1:107.183.160.204 crin.org.access.log.1:107.183.168.56 crin.org.access.log.1:107.183.22.80 crin.org.access.log.1:107.183.7.204 crin.org.access.log.1:108.174.50.153 crin.org.access.log.1:108.174.59.115 crin.org.access.log.1:138.128.127.124 crin.org.access.log.1:138.128.34.83 crin.org.access.log.1:155.94.136.98 crin.org.access.log.1:172.245.250.247 crin.org.access.log.1:172.245.46.135 crin.org.access.log.1:172.245.56.122 crin.org.access.log.1:192.210.148.14 crin.org.access.log.1:192.210.148.74 crin.org.access.log.1:192.210.164.28 crin.org.access.log.1:192.210.195.20 crin.org.access.log.1:192.227.244.85 crin.org.access.log.1:192.227.248.213 crin.org.access.log.1:192.241.76.22 crin.org.access.log.1:192.3.108.203 crin.org.access.log.1:192.3.244.197 crin.org.access.log.1:192.3.246.170 crin.org.access.log.1:192.3.9.215 crin.org.access.log.1:198.12.121.3 crin.org.access.log.1:198.12.121.6 crin.org.access.log.1:198.12.121.7 crin.org.access.log.1:198.12.121.8 crin.org.access.log.1:198.12.121.9 crin.org.access.log.1:198.12.122.174 crin.org.access.log.1:198.12.122.175 crin.org.access.log.1:198.12.72.107 crin.org.access.log.1:198.20.167.233 crin.org.access.log.1:198.23.226.136 crin.org.access.log.1:198.23.227.221 crin.org.access.log.1:198.23.243.142 crin.org.access.log.1:198.23.243.201 crin.org.access.log.1:198.23.247.170 crin.org.access.log.1:198.46.244.71 crin.org.access.log.1:198.46.246.69 crin.org.access.log.1:216.45.57.55 crin.org.access.log.1:23.244.123.125 crin.org.access.log.1:23.244.44.66 crin.org.access.log.1:23.244.54.240 crin.org.access.log.1:23.245.124.126 crin.org.access.log.1:23.245.195.80 crin.org.access.log.1:23.245.254.238 crin.org.access.log.1:23.250.118.104 crin.org.access.log.1:23.250.119.67 crin.org.access.log.1:23.250.120.204 crin.org.access.log.1:23.250.121.194 crin.org.access.log.1:23.89.122.125 crin.org.access.log.1:23.89.241.126 crin.org.access.log.1:23.89.99.23 crin.org.access.log.1:23.94.100.213 crin.org.access.log.1:23.94.100.214 crin.org.access.log.1:23.94.100.215 crin.org.access.log.1:23.94.100.216 crin.org.access.log.1:23.94.100.217 crin.org.access.log.1:23.94.104.147 crin.org.access.log.1:23.94.104.148 crin.org.access.log.1:23.94.104.149 crin.org.access.log.1:23.94.104.150 crin.org.access.log.1:23.94.104.151 crin.org.access.log.1:23.94.108.114 crin.org.access.log.1:23.94.133.224 crin.org.access.log.1:23.94.222.55 crin.org.access.log.1:23.94.228.70 crin.org.access.log.1:23.94.47.119 crin.org.access.log.1:23.94.8.206 crin.org.access.log.1:23.95.29.203 crin.org.access.log.1:23.95.55.148 crin.org.access.log.1:23.95.57.116 crin.org.access.log.1:23.95.57.117 crin.org.access.log.1:75.127.0.149 crin.org.access.log.1:75.127.0.151 crin.org.access.log.1:75.127.0.152 crin.org.access.log.1:75.127.0.153 crin.org.access.log.1:75.127.0.154 crin.org.access.log.1:96.8.112.38 crin.org.ssl_access.log.1:104.144.17.249 crin.org.ssl_access.log.1:104.144.228.196 crin.org.ssl_access.log.1:104.151.100.125 crin.org.ssl_access.log.1:104.151.121.80 crin.org.ssl_access.log.1:104.151.127.209 crin.org.ssl_access.log.1:104.151.28.124 crin.org.ssl_access.log.1:104.151.74.125 crin.org.ssl_access.log.1:104.168.13.147 crin.org.ssl_access.log.1:104.168.13.149 crin.org.ssl_access.log.1:104.168.14.171 crin.org.ssl_access.log.1:104.168.2.225 crin.org.ssl_access.log.1:104.168.2.226 crin.org.ssl_access.log.1:104.168.2.227 crin.org.ssl_access.log.1:104.168.2.228 crin.org.ssl_access.log.1:104.168.23.238 crin.org.ssl_access.log.1:104.203.19.124 crin.org.ssl_access.log.1:104.223.19.131 crin.org.ssl_access.log.1:104.227.202.147 crin.org.ssl_access.log.1:104.227.203.135 crin.org.ssl_access.log.1:104.227.205.77 crin.org.ssl_access.log.1:107.172.100.6 crin.org.ssl_access.log.1:107.172.100.7 crin.org.ssl_access.log.1:107.172.100.8 crin.org.ssl_access.log.1:107.172.100.9 crin.org.ssl_access.log.1:107.172.234.96 crin.org.ssl_access.log.1:107.172.235.204 crin.org.ssl_access.log.1:107.172.96.4 crin.org.ssl_access.log.1:107.172.96.5 crin.org.ssl_access.log.1:107.172.96.6 crin.org.ssl_access.log.1:107.172.96.7 crin.org.ssl_access.log.1:107.172.96.8 crin.org.ssl_access.log.1:107.172.97.21 crin.org.ssl_access.log.1:107.172.97.8 crin.org.ssl_access.log.1:107.172.97.9 crin.org.ssl_access.log.1:107.172.98.4 crin.org.ssl_access.log.1:107.172.98.6 crin.org.ssl_access.log.1:107.172.98.7 crin.org.ssl_access.log.1:107.172.98.8 crin.org.ssl_access.log.1:107.172.98.9 crin.org.ssl_access.log.1:107.172.99.5 crin.org.ssl_access.log.1:107.172.99.6 crin.org.ssl_access.log.1:107.172.99.7 crin.org.ssl_access.log.1:107.172.99.8 crin.org.ssl_access.log.1:107.172.99.9 crin.org.ssl_access.log.1:107.174.252.21 crin.org.ssl_access.log.1:107.174.252.22 crin.org.ssl_access.log.1:107.174.252.5 crin.org.ssl_access.log.1:107.174.252.7 crin.org.ssl_access.log.1:107.174.252.8 crin.org.ssl_access.log.1:107.174.252.9 crin.org.ssl_access.log.1:107.174.253.2 crin.org.ssl_access.log.1:107.174.253.4 crin.org.ssl_access.log.1:107.174.253.6 crin.org.ssl_access.log.1:107.174.253.7 crin.org.ssl_access.log.1:107.174.253.8 crin.org.ssl_access.log.1:107.174.253.9 crin.org.ssl_access.log.1:107.174.254.24 crin.org.ssl_access.log.1:107.174.254.8 crin.org.ssl_access.log.1:107.174.254.9 crin.org.ssl_access.log.1:107.174.255.21 crin.org.ssl_access.log.1:107.174.255.5 crin.org.ssl_access.log.1:107.174.255.6 crin.org.ssl_access.log.1:107.174.255.7 crin.org.ssl_access.log.1:107.174.255.8 crin.org.ssl_access.log.1:107.174.255.9 crin.org.ssl_access.log.1:107.175.227.185 crin.org.ssl_access.log.1:107.175.236.140 crin.org.ssl_access.log.1:107.183.10.222 crin.org.ssl_access.log.1:107.183.117.80 crin.org.ssl_access.log.1:107.183.160.204 crin.org.ssl_access.log.1:107.183.168.56 crin.org.ssl_access.log.1:107.183.22.80 crin.org.ssl_access.log.1:107.183.7.204 crin.org.ssl_access.log.1:108.174.50.153 crin.org.ssl_access.log.1:108.174.59.115 crin.org.ssl_access.log.1:138.128.127.124 crin.org.ssl_access.log.1:138.128.34.83 crin.org.ssl_access.log.1:155.94.136.98 crin.org.ssl_access.log.1:172.245.250.247 crin.org.ssl_access.log.1:172.245.46.135 crin.org.ssl_access.log.1:172.245.56.122 crin.org.ssl_access.log.1:192.210.148.14 crin.org.ssl_access.log.1:192.210.148.74 crin.org.ssl_access.log.1:192.210.164.28 crin.org.ssl_access.log.1:192.210.195.20 crin.org.ssl_access.log.1:192.227.244.85 crin.org.ssl_access.log.1:192.227.248.213 crin.org.ssl_access.log.1:192.241.76.22 crin.org.ssl_access.log.1:192.3.108.203 crin.org.ssl_access.log.1:192.3.244.197 crin.org.ssl_access.log.1:192.3.246.170 crin.org.ssl_access.log.1:192.3.9.215 crin.org.ssl_access.log.1:198.12.121.3 crin.org.ssl_access.log.1:198.12.121.6 crin.org.ssl_access.log.1:198.12.121.7 crin.org.ssl_access.log.1:198.12.121.8 crin.org.ssl_access.log.1:198.12.121.9 crin.org.ssl_access.log.1:198.12.122.174 crin.org.ssl_access.log.1:198.12.122.175 crin.org.ssl_access.log.1:198.12.72.107 crin.org.ssl_access.log.1:198.20.167.233 crin.org.ssl_access.log.1:198.23.226.136 crin.org.ssl_access.log.1:198.23.227.221 crin.org.ssl_access.log.1:198.23.243.142 crin.org.ssl_access.log.1:198.23.243.201 crin.org.ssl_access.log.1:198.23.247.170 crin.org.ssl_access.log.1:198.46.244.71 crin.org.ssl_access.log.1:198.46.246.69 crin.org.ssl_access.log.1:216.45.57.55 crin.org.ssl_access.log.1:23.244.123.125 crin.org.ssl_access.log.1:23.244.44.66 crin.org.ssl_access.log.1:23.244.54.240 crin.org.ssl_access.log.1:23.245.124.126 crin.org.ssl_access.log.1:23.245.195.80 crin.org.ssl_access.log.1:23.245.254.238 crin.org.ssl_access.log.1:23.250.118.104 crin.org.ssl_access.log.1:23.250.119.67 crin.org.ssl_access.log.1:23.250.120.204 crin.org.ssl_access.log.1:23.250.121.194 crin.org.ssl_access.log.1:23.89.122.125 crin.org.ssl_access.log.1:23.89.241.126 crin.org.ssl_access.log.1:23.89.99.23 crin.org.ssl_access.log.1:23.94.100.213 crin.org.ssl_access.log.1:23.94.100.214 crin.org.ssl_access.log.1:23.94.100.215 crin.org.ssl_access.log.1:23.94.100.216 crin.org.ssl_access.log.1:23.94.100.217 crin.org.ssl_access.log.1:23.94.104.147 crin.org.ssl_access.log.1:23.94.104.148 crin.org.ssl_access.log.1:23.94.104.149 crin.org.ssl_access.log.1:23.94.104.150 crin.org.ssl_access.log.1:23.94.104.151 crin.org.ssl_access.log.1:23.94.108.114 crin.org.ssl_access.log.1:23.94.133.224 crin.org.ssl_access.log.1:23.94.222.55 crin.org.ssl_access.log.1:23.94.228.70 crin.org.ssl_access.log.1:23.94.47.119 crin.org.ssl_access.log.1:23.95.29.203 crin.org.ssl_access.log.1:23.95.55.148 crin.org.ssl_access.log.1:23.95.57.116 crin.org.ssl_access.log.1:23.95.57.117 crin.org.ssl_access.log.1:75.127.0.149 crin.org.ssl_access.log.1:75.127.0.151 crin.org.ssl_access.log.1:75.127.0.152 crin.org.ssl_access.log.1:75.127.0.153 crin.org.ssl_access.log.1:75.127.0.154 crin.org.ssl_access.log.1:96.8.112.38 </pre><p> the total number of IP's: </p> <pre class="wiki">grep "Pcore-HTTP/v0.24.5" *log.1 | awk '{print $1}' | sort -u | wc -l 297 </pre><p> This UA has been <a class="ext-link" href="https://community.centminmod.com/threads/blocking-bad-or-aggressive-bots.6433/page-3"><span class="icon">​</span>identified as a DDOS tool before</a>. </p> <p> If it wasn't for the Nginx rate limiting we do this would have brought the server down -- the rate limiting was added following a previous incident like this, see <a class="new ticket" href="https://trac.crin.org/trac/ticket/54" title="defect: 502 Errors (new)">ticket:54</a>. </p> <p> There are no doubt more defences we could consider putting in place, events like this usually happen several times a month. </p> Ticket