CRIN Trac: Ticket #127: 1984.is Server Rebuild

hours changed; totalhours set

chris — Tue, 21 Nov 2017 19:26:16 GMT

hours changed from 0 to 5.0
totalhours set to 5

hours, totalhours changed

chris — Tue, 21 Nov 2017 20:06:19 GMT

hours changed from 0 to 0.63
totalhours changed from 5 to 5.63

Email isn't working for Trac but Munin and Piwik are now up:

And the https://crin.org/ is also up but there might be missing PHP modules and things like Solr -- some testing will be needed.

hours, totalhours changed

chris — Tue, 21 Nov 2017 20:38:55 GMT

hours changed from 0 to 0.25
totalhours changed from 5.63 to 5.88

phpmyadmin is up and running:

https://phpmyadmin.crin.org/

hours, totalhours changed

chris — Tue, 21 Nov 2017 21:21:23 GMT

hours changed from 0 to 0.3
totalhours changed from 5.88 to 6.18

Outgoing email might be working now, Exim has been installed and the config copied over.

There was a mistake in the backups, /usr/local wasn't backed up so various scripts (which are on this Trac site) and Munin plugins will need installing from scratch.

Non-root users haven't been added yet and the nothing has been done to get the dev server up and running.

hours, totalhours changed

chris — Wed, 22 Nov 2017 11:24:50 GMT

hours changed from 0 to 2
totalhours changed from 6.18 to 8.18

Missing WikiStart#Scripts have been installed on the servers, several using the versions in this repo.

User accounts have been added — people should be able to ssh in again.

The missing Munin plugins have been sorted out and it is possible that Trac email is working...

hours, totalhours changed; cc set

chris — Wed, 22 Nov 2017 12:03:58 GMT

cc peter added
hours changed from 0 to 0.65
totalhours changed from 8.18 to 8.83

Checking the /var/log/exim4/mainlog log confirms that the comment above did result in a email.

I have added Peter to this ticket in case there are any issues he wants to flag up.

I'll get the dev server up and running again in December.

hours, totalhours changed

chris — Mon, 27 Nov 2017 14:07:14 GMT

hours changed from 0 to 0.3
totalhours changed from 8.83 to 9.13

There is quite a mailq on crin2, looking at the install notes, ticket:6#comment:18, dpkg-reconfigure exim4-config was run and /etc/aliases was edited -- it looks like all the emails are undelivered email to root, so exim -qff was run to flush the queue.

Also /etc/exim4/exim4.conf.localmacros was updated, and update-exim4.conf was run and a test email was sent to root and everything seems to be working OK now.

hours, totalhours changed

chris — Mon, 27 Nov 2017 14:54:50 GMT

hours changed from 0 to 0.5
totalhours changed from 9.13 to 9.63

One of the, now working, emails:

PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php5/20131226/uploadprogress.so' - /usr/lib/php5/20131226/uploadprogress.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php5/20131226/memcached.so' - /usr/lib/php5/20131226/memcached.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php5/20131226/newrelic.so' - /usr/lib/php5/20131226/newrelic.so: cannot open shared object file: No such file or directory in Unknown on line 0

So following the install notes, ticket:6#comment:15 and ticket:117

pecl install uploadprogress
aptitude install php5-memcached
echo "deb https://apt.newrelic.com/debian/ newrelic non-free" > /etc/apt/sources.list.d/newrelic.list
wget -O- https://download.newrelic.com/548C16BF.gpg | sudo apt-key add -
apt-get update
apt-get install newrelic-php5 newrelic-sysmond
service php5-fpm restart

Then the New Relic key was copied for the backups and added to /etc/newrelic/nrsysmond.cfg and then it was started, /etc/init.d/newrelic-sysmond start.

I also whitelisted munijn.webarch.net so we can have some graphs to compare with the Iceland hosting when that is back online -- the load issues we had there with PHP seem to have gone away, see:

https://munin.crin.org/munin/crin.org/crin2.crin.org/#php