1984.is Server Rebuild

[chris]

Due to critical hardware failure at 1984.is all three servers needed rebuilding from scratch.

[chris]

Email isn't working for Trac but Munin and Piwik are now up:

• ​https://munin.crin.org/munin/
• ​https://stats.crin.org/

And the ​https://crin.org/ is also up but there might be missing PHP modules and things like Solr -- some testing will be needed.

[chris]

phpmyadmin is up and running:

• ​https://phpmyadmin.crin.org/

[chris]

Outgoing email might be working now, Exim has been installed and the config copied over.

There was a mistake in the backups, /usr/local wasn't backed up so various scripts (which are on this Trac site) and Munin plugins will need installing from scratch.

Non-root users haven't been added yet and the nothing has been done to get the dev server up and running.

[chris]

Missing WikiStart#Scripts have been installed on the servers, several using ​the versions in this repo.

User accounts have been added — people should be able to ssh in again.

The missing Munin plugins have been sorted out and it is possible that Trac email is working...

[chris]

Checking the /var/log/exim4/mainlog log confirms that the comment above did result in a email.

I have added Peter to this ticket in case there are any issues he wants to flag up.

I'll get the dev server up and running again in December.

[chris]

There is ​quite a mailq on crin2, looking at the install notes, ticket:6#comment:18, dpkg-reconfigure exim4-config was run and /etc/aliases was edited -- it looks like all the emails are undelivered email to root, so exim -qff was run to flush the queue.

Also /etc/exim4/exim4.conf.localmacros was updated, and update-exim4.conf was run and a test email was sent to root and everything seems to be working OK now.

[chris]

One of the, now working, emails:

PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php5/20131226/uploadprogress.so' - /usr/lib/php5/20131226/uploadprogress.so: cannot open shared object file: No such file or directory in Unknown on line 0                                                                                                                                         
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php5/20131226/memcached.so' - /usr/lib/php5/20131226/memcached.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php5/20131226/newrelic.so' - /usr/lib/php5/20131226/newrelic.so: cannot open shared object file: No such file or directory in Unknown on line 0

So following the install notes, ticket:6#comment:15 and ticket:117

pecl install uploadprogress
aptitude install php5-memcached
echo "deb https://apt.newrelic.com/debian/ newrelic non-free" > /etc/apt/sources.list.d/newrelic.list
wget -O- https://download.newrelic.com/548C16BF.gpg | sudo apt-key add -
apt-get update
apt-get install newrelic-php5 newrelic-sysmond
service php5-fpm restart

Then the New Relic key was copied for the backups and added to /etc/newrelic/nrsysmond.cfg and then it was started, /etc/init.d/newrelic-sysmond start.

I also whitelisted munijn.webarch.net so we can have some graphs to compare with the Iceland hosting when that is back online -- the load issues we had there with PHP seem to have gone away, see:

​https://munin.crin.org/munin/crin.org/crin2.crin.org/#php