Context Navigation

#82 new defect

Attempted DDOS?

Reported by:	chris	Owned by:	chris
Priority:	minor	Milestone:	Maintenance
Component:	crin1	Version:
Keywords:		Cc:	mathew
Estimated Number of Hours:	0	Add Hours to Ticket:	0
Billable?:	yes	Total Hours:	0.82

Description

Yesterday evening there 1k 502 errors caused by one user agent from multiple IP addresses. This isn't an uncommon event, I usually don't write them up, but I thought it might be useful for Mathew if I do with this one.

Attachments (12)

fw_conntrack-day.png (28.9 KB) - added by chris 2 years ago.
fw_forwarded_local-day.png (31.0 KB) - added by chris 2 years ago.
fw_packets-day.png (23.2 KB) - added by chris 2 years ago.
if_eth0-day.png (20.1 KB) - added by chris 2 years ago.
load-day.png (26.4 KB) - added by chris 2 years ago.
multips_memory-day.png (25.2 KB) - added by chris 2 years ago.
memory-day.png (44.5 KB) - added by chris 2 years ago.
nginx_request-day.png (27.4 KB) - added by chris 2 years ago.
nginx_status-day.png (35.7 KB) - added by chris 2 years ago.
phpfpm_average-day.png (26.2 KB) - added by chris 2 years ago.
phpfpm_connections-day.png (19.9 KB) - added by chris 2 years ago.
phpfpm_memory-day.png (26.0 KB) - added by chris 2 years ago.

Download all attachments as: .zip

Change History (13)

Changed 2 years ago by chris

Attachment fw_conntrack-day.png added

Changed 2 years ago by chris

Attachment fw_forwarded_local-day.png added

Changed 2 years ago by chris

Attachment fw_packets-day.png added

Changed 2 years ago by chris

Attachment if_eth0-day.png added

Changed 2 years ago by chris

Attachment load-day.png added

Changed 2 years ago by chris

Attachment multips_memory-day.png added

Changed 2 years ago by chris

Attachment memory-day.png added

comment:1 Changed 2 years ago by chris

Add Hours to Ticket changed from 0 to 0.82
Total Hours set to 0.82

The spike is very clear on the munin graphs:

The number of requests made, based on the UA string:

grep "Pcore-HTTP/v0.24.5" *log.1 | wc -l
10455

The unique IP addresses:

grep "Pcore-HTTP/v0.24.5" *log.1 | awk '{print $1}' | sort -u
crin.org.access.log.1:104.144.17.249
crin.org.access.log.1:104.144.228.196
crin.org.access.log.1:104.151.100.125
crin.org.access.log.1:104.151.121.80
crin.org.access.log.1:104.151.127.209
crin.org.access.log.1:104.151.28.124
crin.org.access.log.1:104.151.74.125
crin.org.access.log.1:104.168.13.147
crin.org.access.log.1:104.168.13.149
crin.org.access.log.1:104.168.13.152
crin.org.access.log.1:104.168.14.171
crin.org.access.log.1:104.168.2.225
crin.org.access.log.1:104.168.2.226
crin.org.access.log.1:104.168.2.227
crin.org.access.log.1:104.168.2.228
crin.org.access.log.1:104.168.23.238
crin.org.access.log.1:104.203.19.124
crin.org.access.log.1:104.223.19.131
crin.org.access.log.1:104.227.202.147
crin.org.access.log.1:104.227.203.135
crin.org.access.log.1:104.227.205.77
crin.org.access.log.1:107.172.100.5
crin.org.access.log.1:107.172.100.6
crin.org.access.log.1:107.172.100.7
crin.org.access.log.1:107.172.100.8
crin.org.access.log.1:107.172.100.9
crin.org.access.log.1:107.172.234.96
crin.org.access.log.1:107.172.235.204
crin.org.access.log.1:107.172.96.4
crin.org.access.log.1:107.172.96.5
crin.org.access.log.1:107.172.96.6
crin.org.access.log.1:107.172.96.7
crin.org.access.log.1:107.172.96.8
crin.org.access.log.1:107.172.97.21
crin.org.access.log.1:107.172.97.8
crin.org.access.log.1:107.172.97.9
crin.org.access.log.1:107.172.98.4
crin.org.access.log.1:107.172.98.6
crin.org.access.log.1:107.172.98.7
crin.org.access.log.1:107.172.98.8
crin.org.access.log.1:107.172.98.9
crin.org.access.log.1:107.172.99.5
crin.org.access.log.1:107.172.99.6
crin.org.access.log.1:107.172.99.7
crin.org.access.log.1:107.172.99.8
crin.org.access.log.1:107.172.99.9
crin.org.access.log.1:107.174.252.21
crin.org.access.log.1:107.174.252.22
crin.org.access.log.1:107.174.252.5
crin.org.access.log.1:107.174.252.7
crin.org.access.log.1:107.174.252.8
crin.org.access.log.1:107.174.252.9
crin.org.access.log.1:107.174.253.2
crin.org.access.log.1:107.174.253.4
crin.org.access.log.1:107.174.253.6
crin.org.access.log.1:107.174.253.7
crin.org.access.log.1:107.174.253.8
crin.org.access.log.1:107.174.253.9
crin.org.access.log.1:107.174.254.24
crin.org.access.log.1:107.174.254.8
crin.org.access.log.1:107.174.254.9
crin.org.access.log.1:107.174.255.21
crin.org.access.log.1:107.174.255.5
crin.org.access.log.1:107.174.255.6
crin.org.access.log.1:107.174.255.7
crin.org.access.log.1:107.174.255.8
crin.org.access.log.1:107.174.255.9
crin.org.access.log.1:107.175.227.185
crin.org.access.log.1:107.175.236.140
crin.org.access.log.1:107.183.10.222
crin.org.access.log.1:107.183.117.80
crin.org.access.log.1:107.183.160.204
crin.org.access.log.1:107.183.168.56
crin.org.access.log.1:107.183.22.80
crin.org.access.log.1:107.183.7.204
crin.org.access.log.1:108.174.50.153
crin.org.access.log.1:108.174.59.115
crin.org.access.log.1:138.128.127.124
crin.org.access.log.1:138.128.34.83
crin.org.access.log.1:155.94.136.98
crin.org.access.log.1:172.245.250.247
crin.org.access.log.1:172.245.46.135
crin.org.access.log.1:172.245.56.122
crin.org.access.log.1:192.210.148.14
crin.org.access.log.1:192.210.148.74
crin.org.access.log.1:192.210.164.28
crin.org.access.log.1:192.210.195.20
crin.org.access.log.1:192.227.244.85
crin.org.access.log.1:192.227.248.213
crin.org.access.log.1:192.241.76.22
crin.org.access.log.1:192.3.108.203
crin.org.access.log.1:192.3.244.197
crin.org.access.log.1:192.3.246.170
crin.org.access.log.1:192.3.9.215
crin.org.access.log.1:198.12.121.3
crin.org.access.log.1:198.12.121.6
crin.org.access.log.1:198.12.121.7
crin.org.access.log.1:198.12.121.8
crin.org.access.log.1:198.12.121.9
crin.org.access.log.1:198.12.122.174
crin.org.access.log.1:198.12.122.175
crin.org.access.log.1:198.12.72.107
crin.org.access.log.1:198.20.167.233
crin.org.access.log.1:198.23.226.136
crin.org.access.log.1:198.23.227.221
crin.org.access.log.1:198.23.243.142
crin.org.access.log.1:198.23.243.201
crin.org.access.log.1:198.23.247.170
crin.org.access.log.1:198.46.244.71
crin.org.access.log.1:198.46.246.69
crin.org.access.log.1:216.45.57.55
crin.org.access.log.1:23.244.123.125
crin.org.access.log.1:23.244.44.66
crin.org.access.log.1:23.244.54.240
crin.org.access.log.1:23.245.124.126
crin.org.access.log.1:23.245.195.80
crin.org.access.log.1:23.245.254.238
crin.org.access.log.1:23.250.118.104
crin.org.access.log.1:23.250.119.67
crin.org.access.log.1:23.250.120.204
crin.org.access.log.1:23.250.121.194
crin.org.access.log.1:23.89.122.125
crin.org.access.log.1:23.89.241.126
crin.org.access.log.1:23.89.99.23
crin.org.access.log.1:23.94.100.213
crin.org.access.log.1:23.94.100.214
crin.org.access.log.1:23.94.100.215
crin.org.access.log.1:23.94.100.216
crin.org.access.log.1:23.94.100.217
crin.org.access.log.1:23.94.104.147
crin.org.access.log.1:23.94.104.148
crin.org.access.log.1:23.94.104.149
crin.org.access.log.1:23.94.104.150
crin.org.access.log.1:23.94.104.151
crin.org.access.log.1:23.94.108.114
crin.org.access.log.1:23.94.133.224
crin.org.access.log.1:23.94.222.55
crin.org.access.log.1:23.94.228.70
crin.org.access.log.1:23.94.47.119
crin.org.access.log.1:23.94.8.206
crin.org.access.log.1:23.95.29.203
crin.org.access.log.1:23.95.55.148
crin.org.access.log.1:23.95.57.116
crin.org.access.log.1:23.95.57.117
crin.org.access.log.1:75.127.0.149
crin.org.access.log.1:75.127.0.151
crin.org.access.log.1:75.127.0.152
crin.org.access.log.1:75.127.0.153
crin.org.access.log.1:75.127.0.154
crin.org.access.log.1:96.8.112.38
crin.org.ssl_access.log.1:104.144.17.249
crin.org.ssl_access.log.1:104.144.228.196
crin.org.ssl_access.log.1:104.151.100.125
crin.org.ssl_access.log.1:104.151.121.80
crin.org.ssl_access.log.1:104.151.127.209
crin.org.ssl_access.log.1:104.151.28.124
crin.org.ssl_access.log.1:104.151.74.125
crin.org.ssl_access.log.1:104.168.13.147
crin.org.ssl_access.log.1:104.168.13.149
crin.org.ssl_access.log.1:104.168.14.171
crin.org.ssl_access.log.1:104.168.2.225
crin.org.ssl_access.log.1:104.168.2.226
crin.org.ssl_access.log.1:104.168.2.227
crin.org.ssl_access.log.1:104.168.2.228
crin.org.ssl_access.log.1:104.168.23.238
crin.org.ssl_access.log.1:104.203.19.124
crin.org.ssl_access.log.1:104.223.19.131
crin.org.ssl_access.log.1:104.227.202.147
crin.org.ssl_access.log.1:104.227.203.135
crin.org.ssl_access.log.1:104.227.205.77
crin.org.ssl_access.log.1:107.172.100.6
crin.org.ssl_access.log.1:107.172.100.7
crin.org.ssl_access.log.1:107.172.100.8
crin.org.ssl_access.log.1:107.172.100.9
crin.org.ssl_access.log.1:107.172.234.96
crin.org.ssl_access.log.1:107.172.235.204
crin.org.ssl_access.log.1:107.172.96.4
crin.org.ssl_access.log.1:107.172.96.5
crin.org.ssl_access.log.1:107.172.96.6
crin.org.ssl_access.log.1:107.172.96.7
crin.org.ssl_access.log.1:107.172.96.8
crin.org.ssl_access.log.1:107.172.97.21
crin.org.ssl_access.log.1:107.172.97.8
crin.org.ssl_access.log.1:107.172.97.9
crin.org.ssl_access.log.1:107.172.98.4
crin.org.ssl_access.log.1:107.172.98.6
crin.org.ssl_access.log.1:107.172.98.7
crin.org.ssl_access.log.1:107.172.98.8
crin.org.ssl_access.log.1:107.172.98.9
crin.org.ssl_access.log.1:107.172.99.5
crin.org.ssl_access.log.1:107.172.99.6
crin.org.ssl_access.log.1:107.172.99.7
crin.org.ssl_access.log.1:107.172.99.8
crin.org.ssl_access.log.1:107.172.99.9
crin.org.ssl_access.log.1:107.174.252.21
crin.org.ssl_access.log.1:107.174.252.22
crin.org.ssl_access.log.1:107.174.252.5
crin.org.ssl_access.log.1:107.174.252.7
crin.org.ssl_access.log.1:107.174.252.8
crin.org.ssl_access.log.1:107.174.252.9
crin.org.ssl_access.log.1:107.174.253.2
crin.org.ssl_access.log.1:107.174.253.4
crin.org.ssl_access.log.1:107.174.253.6
crin.org.ssl_access.log.1:107.174.253.7
crin.org.ssl_access.log.1:107.174.253.8
crin.org.ssl_access.log.1:107.174.253.9
crin.org.ssl_access.log.1:107.174.254.24
crin.org.ssl_access.log.1:107.174.254.8
crin.org.ssl_access.log.1:107.174.254.9
crin.org.ssl_access.log.1:107.174.255.21
crin.org.ssl_access.log.1:107.174.255.5
crin.org.ssl_access.log.1:107.174.255.6
crin.org.ssl_access.log.1:107.174.255.7
crin.org.ssl_access.log.1:107.174.255.8
crin.org.ssl_access.log.1:107.174.255.9
crin.org.ssl_access.log.1:107.175.227.185
crin.org.ssl_access.log.1:107.175.236.140
crin.org.ssl_access.log.1:107.183.10.222
crin.org.ssl_access.log.1:107.183.117.80
crin.org.ssl_access.log.1:107.183.160.204
crin.org.ssl_access.log.1:107.183.168.56
crin.org.ssl_access.log.1:107.183.22.80
crin.org.ssl_access.log.1:107.183.7.204
crin.org.ssl_access.log.1:108.174.50.153
crin.org.ssl_access.log.1:108.174.59.115
crin.org.ssl_access.log.1:138.128.127.124
crin.org.ssl_access.log.1:138.128.34.83
crin.org.ssl_access.log.1:155.94.136.98
crin.org.ssl_access.log.1:172.245.250.247
crin.org.ssl_access.log.1:172.245.46.135
crin.org.ssl_access.log.1:172.245.56.122
crin.org.ssl_access.log.1:192.210.148.14
crin.org.ssl_access.log.1:192.210.148.74
crin.org.ssl_access.log.1:192.210.164.28
crin.org.ssl_access.log.1:192.210.195.20
crin.org.ssl_access.log.1:192.227.244.85
crin.org.ssl_access.log.1:192.227.248.213
crin.org.ssl_access.log.1:192.241.76.22
crin.org.ssl_access.log.1:192.3.108.203
crin.org.ssl_access.log.1:192.3.244.197
crin.org.ssl_access.log.1:192.3.246.170
crin.org.ssl_access.log.1:192.3.9.215
crin.org.ssl_access.log.1:198.12.121.3
crin.org.ssl_access.log.1:198.12.121.6
crin.org.ssl_access.log.1:198.12.121.7
crin.org.ssl_access.log.1:198.12.121.8
crin.org.ssl_access.log.1:198.12.121.9
crin.org.ssl_access.log.1:198.12.122.174
crin.org.ssl_access.log.1:198.12.122.175
crin.org.ssl_access.log.1:198.12.72.107
crin.org.ssl_access.log.1:198.20.167.233
crin.org.ssl_access.log.1:198.23.226.136
crin.org.ssl_access.log.1:198.23.227.221
crin.org.ssl_access.log.1:198.23.243.142
crin.org.ssl_access.log.1:198.23.243.201
crin.org.ssl_access.log.1:198.23.247.170
crin.org.ssl_access.log.1:198.46.244.71
crin.org.ssl_access.log.1:198.46.246.69
crin.org.ssl_access.log.1:216.45.57.55
crin.org.ssl_access.log.1:23.244.123.125
crin.org.ssl_access.log.1:23.244.44.66
crin.org.ssl_access.log.1:23.244.54.240
crin.org.ssl_access.log.1:23.245.124.126
crin.org.ssl_access.log.1:23.245.195.80
crin.org.ssl_access.log.1:23.245.254.238
crin.org.ssl_access.log.1:23.250.118.104
crin.org.ssl_access.log.1:23.250.119.67
crin.org.ssl_access.log.1:23.250.120.204
crin.org.ssl_access.log.1:23.250.121.194
crin.org.ssl_access.log.1:23.89.122.125
crin.org.ssl_access.log.1:23.89.241.126
crin.org.ssl_access.log.1:23.89.99.23
crin.org.ssl_access.log.1:23.94.100.213
crin.org.ssl_access.log.1:23.94.100.214
crin.org.ssl_access.log.1:23.94.100.215
crin.org.ssl_access.log.1:23.94.100.216
crin.org.ssl_access.log.1:23.94.100.217
crin.org.ssl_access.log.1:23.94.104.147
crin.org.ssl_access.log.1:23.94.104.148
crin.org.ssl_access.log.1:23.94.104.149
crin.org.ssl_access.log.1:23.94.104.150
crin.org.ssl_access.log.1:23.94.104.151
crin.org.ssl_access.log.1:23.94.108.114
crin.org.ssl_access.log.1:23.94.133.224
crin.org.ssl_access.log.1:23.94.222.55
crin.org.ssl_access.log.1:23.94.228.70
crin.org.ssl_access.log.1:23.94.47.119
crin.org.ssl_access.log.1:23.95.29.203
crin.org.ssl_access.log.1:23.95.55.148
crin.org.ssl_access.log.1:23.95.57.116
crin.org.ssl_access.log.1:23.95.57.117
crin.org.ssl_access.log.1:75.127.0.149
crin.org.ssl_access.log.1:75.127.0.151
crin.org.ssl_access.log.1:75.127.0.152
crin.org.ssl_access.log.1:75.127.0.153
crin.org.ssl_access.log.1:75.127.0.154
crin.org.ssl_access.log.1:96.8.112.38

the total number of IP's:

grep "Pcore-HTTP/v0.24.5" *log.1 | awk '{print $1}' | sort -u | wc -l
297

This UA has been identified as a DDOS tool before.

If it wasn't for the Nginx rate limiting we do this would have brought the server down -- the rate limiting was added following a previous incident like this, see ticket:54.

There are no doubt more defences we could consider putting in place, events like this usually happen several times a month.

Note: See TracTickets for help on using tickets.

Download in other formats: