CRIN Trac: Ticket Query

#119: MySQL Downtime

chris — Thu, 15 Jun 2017 10:31:52 GMT

Just after midnight last night MySQL appears to have started having some issues.

#17: Debian System Updates

chris — Tue, 09 Jun 2015 18:42:00 GMT

This ticket is document system updates done to the servers.

Sources for further information:

These updates are generally done using the wiki:AptitudeUpdateScript and this records all the changes in the /root/Changelog and then the contents of the Changelog are pasted into the ticket to document the upgrade.

#64: Let's Encrypt HTTPS Certs

chris — Fri, 26 Feb 2016 11:30:54 GMT

These certs, on Crin1, are due to expire soon:

https://cloud.crin.org/ 13/03/16
https://stats.crin.org/ 13/03/16
https://trac.crin.org.archived.website/ 29/04/16
https://wiki.crin.org/ 03/05/16

The one on Crin2 has a while to run:

https://www.crin.org/ 17/02/18

To save money I suggest that the certs on Crin1 are replaced with Let's Encrypt ones, for the above domains and in addition these sites which use CAcert:

https://munin.crin.org/ 07/05/17
https://phpmyadmin.crin.org/ 07/05/17

I suggest we still use CACert for the MySQL certs as these are valid for two years and Let's Encrypt ones are only valid for 3 months and it would be complicated to automate the renewal and deployment for these certs.

#65: Custom 502 and 503 pages for crin.org

chris — Sun, 28 Feb 2016 14:17:03 GMT

It would be more informative if there were custom 502 and 503 error pages for https://www.crin.org/ rather than the default Nginx ones.

I will write up some details below on the 502 and 503's we seeing getting, see also ticket:54.

I'm happy to create a static 502 and static 503 error page and will look at adding it to the git repo.

#76: Google Apps setup for crin.com

chris — Wed, 06 Jul 2016 19:50:38 GMT

This ticket is to track the time spent setting up Google Apps for crin.com.

#106: MySQL Error

chris — Mon, 27 Mar 2017 13:37:30 GMT

On Saturday morning the MySQL database on Crin1 was generating errors which prevented the site for working, this was fixed by running this command on the server:

mysqladmin flush-hosts

#113: MySQL Performance

chris — Tue, 25 Apr 2017 11:22:11 GMT

MySQL is set to use a lot of resources, see the RAM allocation here:

https://munin.crin.org/munin/crin.org/crin1.crin.org/multips_memory.html

And also a lot of queries are cached via memcache, which also has a lot of RAM allocated, see:

https://munin.crin.org/munin/crin.org/crin2.crin.org/multips_memory.html

What can we do to make it quicker?

See also the work done when the server was setup on ticket:9

#114: Update TLS certs for MySQL

chris — Fri, 05 May 2017 12:20:04 GMT

The TLS certs for MySQL connections between the servers are due to expire on 7th May 2017 so need to be updated.

Peter, Russell, I have added you just so you know this is happening, it'll involve restarting MySQL when the new keys and certs are in place, so there might be a few moments of site downtime, but I'll do this on the dev/staging sever first.

#116: Limit the number of MySQL connections from Crin4 to Crin1

chris — Wed, 17 May 2017 11:44:18 GMT

Both the dev, Crin4 and live Crin2, servers use the MySQL database on Crin1 and we want to limit the resources available to Crin4 so it reduced the impact on the live site.

Some documentation on this:

#127: 1984.is Server Rebuild

chris — Tue, 21 Nov 2017 19:26:16 GMT

Due to critical hardware failure at 1984.is all three servers needed rebuilding from scratch.

#82: Attempted DDOS?

chris — Sat, 24 Sep 2016 09:20:39 GMT

Yesterday evening there 1k 502 errors caused by one user agent from multiple IP addresses. This isn't an uncommon event, I usually don't write them up, but I thought it might be useful for Mathew if I do with this one.