CRIN Trac: Ticket Query

#52: 404 for top article - caused by 'und' in URL not 'en'

chris — Sat, 12 Dec 2015 16:06:20 GMT

The first articles linked to on the front page of https://www.crin.org/ is a 404 not found:

CIVIL SOCIETY UNDER THREAT: Old and new challenges for human and children's rights advocates in Eurasia 11 Dec 2015 In recent years, the space afforded to civil society to operate freely has been shrinking dramatically across the world,... https://www.crin.org/und/library/publications/civil-society-under-threat-old-and-new-challenges-human-and-childrens-rights

The article is actually here:

https://www.crin.org/en/library/publications/civil-society-under-threat-old-and-new-challenges-human-and-childrens-rights

The problem is the und rather then en in the URL?

#13: Access to db1.crin.org

chris — Wed, 13 May 2015 10:41:36 GMT

This ticket is for tracking the time taken to enable access to the db1.crin.org MySQL database server at GreenQloud for codepositive.com.

#22: Add Trac Account for Peter

chris — Wed, 17 Jun 2015 16:15:02 GMT

Add an account for Peter following wiki:Trac#CreateanTracaccount

#16: Add Trac account for Adam

chris — Thu, 04 Jun 2015 12:37:33 GMT

It turns out that Graham, for whom an account was created, see ticket:1#comment:25 was not responsible for the MediaWiki site, so he needs removing from the MediaWiki ticket, see ticket:5#comment:6 and an account for Adam needs to be created and he needs to be added to that ticket.

#21: Apache Solr HTTP 400 Bad Request Errors

chris — Wed, 17 Jun 2015 15:01:15 GMT

There are lots of errors like this in the dblog:

Type		Apache Solr
Date		Wednesday, June 17, 2015 - 15:53
User		Anonymous
Location	http://www.crin.org/fr/biblioth%C3%A8que/archives-des-actualit%C3%A9s?page=8
Message	HTTP 400; Bad Request: Bad Request
Severity	error

And

Type		Apache Solr
Date		Wednesday, June 17, 2015 - 15:53
User		Anonymous
Location	http://www.crin.org/fr/biblioth%C3%A8que/archives-des-actualit%C3%A9s?page=8
Message		HTTP Status: 400;
Message: 	Bad Request: Bad Request;
Response: 	<html><head><title>Apache Tomcat/7.0.56 (Debian) - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 400 - undefined field hash</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>undefined field hash</u></p><p><b>description</b> <u>The request sent by the client was syntactically incorrect.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.56 (Debian)</h3></body></html>;
Request: GET /solr/select?start=0&rows=5&fq=hash%3A7as1z6&fq=hash%3A7as1z6&q=id%3A7as1z6%2Fnode%2F266&qt=mlt&fl=entity_id%2Centity_type%2Clabel%2Cpath%2Curl%2Chash%2Csite&mlt.fl=path_alias%2Ctaxonomy_names&mlt.mintf=1&mlt.mindf=1&mlt.minwl=3&mlt.maxwl=15&mlt.maxqt=20&wt=json&json.nl=map HTTP/1.0 User-Agent: Drupal (+http://drupal.org/) Host: localhost:8080 ;
Caller: call_user_func_array() (line Inconnu of Inconnu)
Severity	error

Requests for pages such as the one in the example above generate both the above errors.

#40: Check Bitbucket documentation

chris — Wed, 09 Sep 2015 13:21:35 GMT

Chris needs to read and check the documentation here: https://bitbucket.org/crin/crin/wiki/Home and update the Trac documentation as need be.

#15: Create Trac accounts for Jenny and Gillian

chris — Tue, 02 Jun 2015 13:43:54 GMT

Jonas is leaving at the end of the week and Trac accounts are needed for Jenny and Gillian at CRIN.

#26: Create ssh and Trac account(s) for Code Positive

chris — Mon, 06 Jul 2015 15:52:34 GMT

Ticket to document the task and the time adding accounts ot the servers for Peter.

#28: Crin1 server email problem

chris — Tue, 14 Jul 2015 12:25:11 GMT

Web application on Crin1 are unable to send email, when they attempt to they log this error in the /var/log/exim4/paniclog:

2015-07-09 00:43:08 unable to set gid=115 or uid=0 (euid=0): forcing real = effective

Jenny / Gillian would you like me to fix this problem? It won't take long as the solution is here:

http://blog.gaiterjones.com/send-php-mail-ubuntu-upgrade/

#27: Crin2 certificate access

peter — Mon, 06 Jul 2015 16:29:00 GMT

When attempting to use Drush, when not root, we are not able to access the certificate.

SSL error: Unable to get certificate from '/etc/ssl/cacert/crin1_cert.pem'

How should we manage access to certificates? Should we create a developers group?

#62: Crin3 ethernet interface fails to come up after a reboot

chris — Mon, 15 Feb 2016 11:49:52 GMT

After updating the kernel on Crin3 I rebooted it and then couldn't connect to it apart from using VNC.

#23: Development and staging environment

chris — Wed, 17 Jun 2015 16:25:58 GMT

Email from Peter:

We would like to set up an environment that is as close as possible to the production environment as possible, so that we are able to trial and review any changes to the site before they go live.

We would ideally have two environments dev and stage on the same box.
Ideally we would use dev.crin.org and stage.crin.org as two document roots
We will script any deployment from git
the same script should be used on productin too
We will use drush to sync between production and staging/dev
we will need to have ssh access between the stage box and the prod box to be able to sync
I would also like to know how you manage updates on the boxes. In the long run it is pretty important that the dev box is kept in sync with the prod box.

#18: Drush

chris — Fri, 12 Jun 2015 16:34:39 GMT

Drush doesn't work on Crin2:

su - www-data -s /bin/bash
drush pm-updatestatus
Command pm-updatestatus needs a higher bootstrap level to run - you will need to invoke drush from a more functional Drupal [error]
environment to run this command.
The drush command 'pm-updatestatus' could not be executed. [error]
Drush was not able to start (bootstrap) the Drupal database. [error]
Hint: This may occur when Drush is trying to:
* bootstrap a site that has not been installed or does not have a configured database. In this case you can select another site
with a working database setup by specifying the URI to use with the --uri parameter on the command line. See `drush topic
docs-aliases` for details.
* connect the database through a socket. The socket file may be wrong or the php-cli may have no access to it in a jailed shell.
See http://drupal.org/node/1428638 for details.
Drush was attempting to connect to:
Drupal version : 7.24
Site URI : http://crin1.crin.org/
Database driver : mysql
Database hostname : crin1
Database port :
Database username : drupal
Database name : drupal
PHP executable : /usr/bin/php
PHP configuration : /etc/php5/cli/php.ini
PHP OS : Linux
Drush script : /usr/local/src/drush-master/drush.php
Drush version : 7.0-dev
Drush temp directory : /tmp
Drush configuration : /var/www/.drush/drushrc.php
Drush alias files :
Drupal root : /var/www/drupal
Site path : sites/default

#7: Enable Google Drive plugin for ownCloud

chris — Wed, 29 Apr 2015 09:42:55 GMT

Configure the external storage application on https://cloud.crin.org/ on the existing GreenCloud server for mounting Google Drive.

#24: Exim Mailqueue

chris — Fri, 19 Jun 2015 10:04:32 GMT

Yesterday a CRINmail newsletter was sent out (see also ticket:14#comment:5) and there are currently 245 emails stuck in the mail queue on Crin2, these are probably undeliverable addresses and there should be a way to mark them so and perhaps disable sending emails to these addresses in the future.

In addition as the queue is greater that 100 it is resulting in Munin sending a alert email every 5 mins, this threshold needs increasing to prevent these emails.

#78: HTTP_PROXY possible security risk

chris — Tue, 19 Jul 2016 10:21:46 GMT

See httpoxy.

#105: High load and slow responses from Crin2

chris — Mon, 27 Mar 2017 12:15:49 GMT

Trac threw this error:

Warning:

Error with navigation contributor "TimingEstimationAndBillingPage?"
Error with navigation contributor "AdminModule?"
Error with navigation contributor "SearchModule?"
Failed to sync with repository "(default)": database is locked; repository information may be out of date. Look in the Trac log for more information including mitigation strategies.

And in /var/log/php5-fpm.log we have:

[27-Mar-2017 10:20:42] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 10:30:10] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:54:51] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:56:35] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:57:08] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:57:58] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 12:01:48] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 12:02:26] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it

From an initial look at the Nginx logs the problem seesm to be caused but too much agressive crawling of the site by bots.

#75: Load spike and small downtime

chris — Sun, 03 Jul 2016 18:00:34 GMT

Since about 8:10am this morning there has been a noticeable increase in bandwidth usage and load on Crin2 and at 18:33, for 20 seconds, the Webarchitects monitoring server was unable to connect.

#130: Matomo 3.3.0

chris — Mon, 15 Jan 2018 09:59:07 GMT

A new version of Matomo (formally Piwik) is out, 3.3.0,

https://matomo.org/changelog/matomo-3-3-0/

#133: Matomo 3.5.0

chris — Wed, 09 May 2018 09:01:31 GMT

Upgrading to the latest version.

#34: Memcache Munin Stats

chris — Mon, 07 Sep 2015 09:48:01 GMT

Munin stats for memcache stopped at the and of August 2015 and I'm not sure why.

#50: Migrate to INNODB

chris — Wed, 25 Nov 2015 09:50:16 GMT

Email from Peter:

What is involved in changing any of the underlying server resources for Crin.org.

I think we generally have too many CPU's and not enough memory.

To improve the site performance, we need to move the database to INNODB. But we are also already using swap on the DB server, and INNODB wants the entire DB in ram. I think we should switch to at least 8GB ram, but can probably scale down to 4 cpus - and still get a performance increase.

#55: Migrating Crin2 server to new hardware

chris — Fri, 01 Jan 2016 18:21:49 GMT

In order for the Crin2 server to have it's RAM doubled it needs to be migrated to a different physical server, I would expect this would take between 15 mins and an hour at most, is there any preference when this is done? I'd suggest doing it this weekend before everybody is back at work on Monday would make sense? I have contacted 1984.is about it and they can script it to happen at what ever time is best.

#107: New cert for Fetechmail

chris — Tue, 04 Apr 2017 13:15:41 GMT

See wiki:Trac#email2trac

#51: Number of php5-fpm processes on Crin2

chris — Wed, 09 Dec 2015 12:34:55 GMT

Peter -- a couple of days ago the number of php5-fpm processes on Crin2 jumped from 9 to 39 and this uses upto 5G or RAM and there is only 4G, see the following Munin graphs:

Note: See TracTickets for help on using tickets.

Any idea what changed?

#32: PHP error logs not found

mori — Tue, 18 Aug 2015 14:41:56 GMT

I'm having issues on newprod, but I can't find PHP error logs. If they are logged, can you please let me know where they are? If they are not logged, can you start logging them on both CRIN2 and CRIN4? Thanks.

#44: PHP temp session files need deleting

chris — Fri, 18 Sep 2015 09:15:52 GMT

On Crin1 php session files are written to these directories:

/var/www/piwik/tmp
/var/www/mediawiki/w/tmp
/var/www/mediawiki/tmp
/var/www/owncloud/tmp

Something like tmpreaper needs installing and configuring to delete old files.

On Crin2 and Crin4 /tmp is used and that could also do with old files being deleted.

I have CC'd Mori in case there are any issues I need to be aware of regarding Drupal / Solr and temp files.

#25: Piwik 2.14.0

chris — Wed, 24 Jun 2015 08:46:52 GMT

A new version of Piwik will be out very soon (probably best to wait for the full release, there is a release candidate out today) which fixes critical security issues - upgrading should take no more than 15 mins, would you like your site upgraded?

Three security issues have been fixed. We are grateful for security researchers who responsibly disclosed these security issues to us: Abdullah Hussam Gazi (CSRF issue) and Dmitriy Shcherbatov (two XSS issues).

https://piwik.org/changelog/piwik-2-14-0/

#29: Piwik 2.14.1

chris — Fri, 17 Jul 2015 08:41:19 GMT

A new version of Piwik is out, from the changelog:

What’s new in Piwik 2.14.1?

In this release we have focused on fixing a few regressions reported in the last major release Piwik 2.14.0, as well as 27 other small improvements. As always we – the Piwik team – are very interested to hear your feedback!

27 tickets have been closed by 6 contributors.

#30: Piwik 2.14.2

chris — Tue, 28 Jul 2015 10:48:31 GMT

The Changelog:

What’s new in Piwik 2.14.2?

In this release we have focused on fixing a few regressions reported in the last major release Piwik 2.14.0, as well as 15 other small improvements. As always we – the Piwik team – are very interested to hear your feedback!

15 tickets have been closed by 6 contributors.

(I'll do this now and record the time in August)

#33: Piwik 2.14.3

chris — Mon, 24 Aug 2015 11:06:12 GMT

From the changelog:

August 5, 2015

Piwik 2.14.3 is a new minor release of Piwik.

What’s new?

This release addresses a regression introduced in 2.14.2 that prevents users from installing Piwik in some cases.

#46: Piwik 2.15.0

chris — Fri, 16 Oct 2015 09:08:19 GMT

The Piwik 2.15.0 changelog:

We are proud to announce Piwik 2.15.0 Release Candidate: a new major release of Piwik!

What’s new?

Piwik 2.15.0 is our Long Term Support release for Piwik 2.X. This release aims to be outstanding from a performance, security, and reliability point of view. It also includes a beautiful new design which lets you focus on your data, and a new awesome search bar to lets you easily navigate to any menu, website or segment in your Piwik dashboards. For administrators and power user, this release includes new diagnostic tools, logging messages, a new Log viewer plugin, and better overall performance and scalability. There are dozens other improvements, see the full list below. Piwik 2.15.0 is built to last.

Security release

This release is rated critical.

We are grateful for Security researchers who disclosed security issues privately to the Piwik Security Response team: Elamaran Venkatraman, Egidio Romano and Dmitriy Shcherbatov. The following vulnerabilities were fixed: XSS, CSRF, possible file inclusion in older PHP versions (low impact), possible Object Injection Vulnerability (low impact).

Database upgrade

This release does not contain any major database upgrade.

Platform Changes

Piwik is an open analytics platform. In an effort to help Piwik developers learn about improvements and changes in the core APIs, we document the changes since the last release.

In this 2.15.0 release there are API deprecations, New features, New APIs, New commmands, Internal API changes.

Read more in Platform Changelog for Developers to see all changes to the platform and APIs (you can also find it in the CHANGELOG.md in the root of your Piwik)

#61: Piwik 2.16.0

chris — Sun, 07 Feb 2016 10:41:42 GMT

From the changelog:

What’s new?

Piwik 2.16.0 is a release which includes more than 250 closed issues. To learn about the most important and visible changes, read our blog post: What’s new in Piwik 2.16.0?

This release is our Long Term Support release.

263 tickets have been closed by more than 30 contributors!

#71: Piwik 2.16.1

chris — Tue, 12 Apr 2016 10:33:41 GMT

Updated needed, see https://piwik.org/changelog/piwik-2-16-1/

#80: Piwik 2.16.2

chris — Tue, 16 Aug 2016 11:08:37 GMT

We are proud to announce Piwik 2.16.2: a new minor release of Piwik!

We have identified and fixed several XSS security issues in this release.

This release is rated critical.

https://piwik.org/changelog/piwik-2-16-2/

#83: Piwik 2.16.3

chris — Mon, 03 Oct 2016 10:00:18 GMT

The Changelog contains:

Security release

This release is rated critical.

The Piwik security engineering team has internally identified a critical security issue and has fixed it in Piwik 2.16.3. We recommend all users to upgrade to this latest version.

Database upgrade

Note: This release contains major database upgrades and upgrading your database will take a long time if you have a lot of data in your database.

Please make sure you read the Update Piwik guide for high traffic instances.

#84: Piwik 2.16.4

chris — Tue, 04 Oct 2016 11:54:50 GMT

The Changelog:

Piwik 2.16.4 fixes a regression introduced yesterday in Piwik 2.16.3. The Scheduled reports were not working for websites where the website’s timezone was UTC+N or UTC-N. This is now fixed.

#87: Piwik 2.16.5

chris — Wed, 05 Oct 2016 13:07:17 GMT

The Changelog:

Piwik 2.16.5 fixes a regression introduced yesterday in Piwik 2.16.3 and not fully fixed in Piwik 2.16.4. The Scheduled reports were not working for websites where the website’s timezone was UTC+N or UTC-N. This is now fixed.

#91: Piwik 2.17.0

chris — Thu, 27 Oct 2016 19:08:57 GMT

No security implications, see the changelog.

#95: Piwik 3.0.0

chris — Mon, 09 Jan 2017 11:16:07 GMT

New version of Piwik, see https://piwik.org/changelog/piwik-3-0-0/

#96: Piwik 3.0.1

chris — Wed, 11 Jan 2017 13:18:36 GMT

New version of Piwik: https://piwik.org/changelog/piwik-3-0-1/

#101: Piwik 3.0.2

chris — Thu, 02 Mar 2017 18:58:40 GMT

A new version of Piwik is out:

https://piwik.org/changelog/piwik-3-0-2/

#109: Piwik 3.0.3

chris — Wed, 05 Apr 2017 09:41:47 GMT

New version of Piwik, Piwik 3.0.3.

#124: Piwik 3.1.0

chris — Fri, 15 Sep 2017 10:16:09 GMT

Details: https://piwik.org/changelog/piwik-3-1-0/

#126: Piwik 3.2.0

chris — Mon, 16 Oct 2017 12:27:16 GMT

Changelog: https://piwik.org/changelog/piwik-3-2-0/

#128: Piwik 3.2.1

chris — Thu, 07 Dec 2017 12:07:06 GMT

Changelog: https://piwik.org/changelog/piwik-3-2-1/

#110: Piwik Accounts for Peter and Lisa

chris — Mon, 10 Apr 2017 13:53:26 GMT

Peter and Lisa from the-organization.com need Piwik access

#36: Piwik stats stopped on 31st August 2015

chris — Tue, 08 Sep 2015 11:14:19 GMT

The Piwik stats stopped on 31st August 2015.

The tracking code is missing, can the Drupal Piwik module be re-installed / re-enabled please?

In addition I believe the GA tracking code can be removed as the Piwik stats replace these.

#31: Preparation for deployment to new prod

mori — Mon, 10 Aug 2015 15:32:41 GMT

Hi Chris,

In order to run the CRIN production site under the new codebase, we need to go through a number of steps to set up and test the site. We need your help to accomplish this.

Overview: (please execute each step once we request you to do so through the comment)

Chris to create a new entry for the new prod site on CRIN2
Chris to create a new database for the new prod site on CRIN1
Code+ and CRIN to test the new prod site and sign off
Chris to update the nginx config so the new prod serves as the working prod

Rationale of the go-live process Because of the reasons listed below, deploying the new code to the existing prod results in an extended period of downtime:

repository too large: cloning takes 10+ minutes
changes to directory structure requires the registry to be rebuilt
the new deployment process needs testing

#35: Problem with the S3QL backups for Crin1

chris — Mon, 07 Sep 2015 10:55:30 GMT

Just tried mounting all the backup on Crin4 and there was a problem with the Crin1 backup:

umnt-s3ql crin1
/media/s3ql/crin1 is not on an S3QL file system
Can not check mounted file system.
mnt-s3ql crin1
Can not check mounted file system.
Filesystem check of s3c://s.qstack.advania.com:443/crin1 failed
Using 2 upload threads (memory limited).
Autodetected 65498 file descriptors available for cache entries
Using cached metadata.
File system damaged or not unmounted cleanly, run fsck!
Mounting s3c://s.qstack.advania.com:443/crin1 on /media/s3ql/crin1 failed
mnt-s3ql crin1 --force
Can not check mounted file system.
Filesystem check of s3c://s.qstack.advania.com:443/crin1 failed
Using 2 upload threads (memory limited).
Autodetected 65498 file descriptors available for cache entries
Using cached metadata.
File system damaged or not unmounted cleanly, run fsck!
Mounting s3c://s.qstack.advania.com:443/crin1 on /media/s3ql/crin1 failed

#67: S3QL 2.17 has been released

chris — Wed, 09 Mar 2016 19:07:56 GMT

This is worth noting:

From the changelog:

2016-03-08, S3QL 2.17

The internal file system revision has changed. File systems created with S3QL 2.17 or newer are not compatible with prior S3QL versions. To update an existing file system to the newest revision, use the s3qladm upgrade command.
https://groups.google.com/forum/?_escaped_fragment_=topic/s3ql/t36XMqiS5TU#!topic/s3ql/t36XMqiS5TU

Currently Debian unstable has 2.15:

https://packages.debian.org/stretch/s3ql

But it should update soon.

#39: S3QL backup problems

chris — Wed, 09 Sep 2015 11:36:18 GMT

Often backups fail during the rsync with "Transport endpoint is not connected".

The S3QL Crin3 backup script mounts on /tmp but the other scripts don't so where there is some issue the unmount don't work as the S3QL system is mounted in an unknown location, see ticket:35. [SOLVED]

There are also some unneeded scripts in /usr/local/bin on Crin3 which need removing to make things less confusing. [SOLVED]

#37: Site slowdown on 8th Sept 2015

chris — Tue, 08 Sep 2015 13:15:07 GMT

The site is really slow and the number of php5-fpm processes has gone up dramatically and so has the memory usage.

#59: Trac Spam

chris — Wed, 13 Jan 2016 20:38:11 GMT

We might have to whitelist email addresses which can open tickets or something, see ticket:58

#98: Trac account for Russell

chris — Mon, 30 Jan 2017 11:43:44 GMT

Create a Trac account for the new Drupal developer, Russell.

#123: Update Email Settings

chris — Mon, 24 Jul 2017 15:23:37 GMT

This Trac site uses a @webarch.net email account and that domain has been migrated to a new server and a new account needs setting up and the fetchmail settings need updating.

#74: Visit to CRIN office

chris — Sun, 05 Jun 2016 19:04:12 GMT

This ticket is to record the time spent visting the CRIN office on Friday 3rd June 2016.

#58: You have 1 new fax, document 00000576492

incoming@… — Wed, 13 Jan 2016 20:10:03 GMT

A new fax document for you.
Please, download fax document attached to this email.
Quality:            200 DPI
Sender:             Theodore Thompson
Pages:              12
Scanned in:         46 seconds
Document name:      scan.00000576492.doc
File size:          240 Kb
Scanned:            Wed, 13 Jan 2016 08:29:25 +0300
Thanks for using Interfax service!

scan.00000576492.zip

#38: [URGENT] crin.org down

mori — Tue, 08 Sep 2015 15:54:35 GMT

Hi Chris,

Jenny just reached out to us by email reporting crin.org is down. The error message I see on crin.org is:

PDOException: SQLSTATE[28000] [1045] Access denied for user 'newprod'@'crin2' (using password: YES) in lock_may_be_available() (line 167 of /var/www/prod/docroot/includes/lock.inc).

It sounds like it may be something that's related to the issue you mentioned in #36 (https://trac.crin.org.archived.website/trac/ticket/36#comment:7 )

Can you please have a look at this as a matter of urgency?

Thanks,

Mori

#48: crin4 /tmp filling up

chris — Wed, 28 Oct 2015 10:44:53 GMT

A task is need to clear /tmp on Crin4.

#43: ownCloud 8.1.3-13.1

chris — Thu, 17 Sep 2015 11:52:00 GMT

Email from Crin1:

Subject: 4 Debian package update(s) for crin1
apticron report [Thu, 17 Sep 2015 11:40:12 +0000]
========================================================================
apticron has detected that some packages need upgrading on:
        crin1
        [ 93.95.228.179 ]
The following packages are currently pending an upgrade:
        owncloud 8.1.3-13.1
        owncloud-config-apache 8.1.3-13.1
        owncloud-server 8.1.3-13.1
        php5-ldap 5.6.13+dfsg-0+deb8u1
========================================================================
Package Details:
Reading changelogs...
--- Changes for owncloud (owncloud owncloud-config-apache owncloud-server) ---
owncloud (8.1.3-13.1) stable; urgency=low
  * version number update by debtransform
 -- debtransform <build@opensuse.org>  Wed, 16 Sep 2015 14:39:25 +0000
owncloud (8.1.3-1) stable; urgency=low
  * Update to version 8.1.3
 -- Juergen Weigert <jw@owncloud.com>  Fri, 11 Sep 2015 11:48:14 +0200
owncloud (8.1.2-1) stable; urgency=low
  * Update to version 8.1.2
 -- Juergen Weigert <jw@owncloud.com>  Wed, 02 Sep 2015 12:39:29 +0200
owncloud (8.1.2~RC1-1) stable; urgency=low
  * Update to version 8.1.2~RC1
 -- Juergen Weigert <jw@owncloud.com>  Sun, 30 Aug 2015 15:20:03 +0200
owncloud (8.1.1-3) stable; urgency=low
  * Possible fix for https://github.com/owncloud/core/issues/18222
 -- Juergen Weigert <jw@owncloud.com>  Tue, 25 Aug 2015 17:42:12 +0200
owncloud (8.1.1-2) stable; urgency=low
  * Make occ executable. https://github.com/owncloud/core/issues/18508
  * Using DEBTRANSFORM-RELEASE to automate incrementing build release number.
  * Fix owncloud-server: php-script-but-no-phpX-cli-dep var/www/owncloud/occ
 -- Juergen Weigert <jw@owncloud.com>  Tue, 25 Aug 2015 14:25:48 +0200

#47: ownCloud 8.1.4.2

chris — Tue, 20 Oct 2015 13:58:02 GMT

From the email list:

Dear ownCloud users!

We're proud to make available the latest ownCloud Server 8.2, which delivers user interface improvements and significant new features for system administrators. This release is deemed stable for home users and small servers. Enterprise users are encouraged to wait for the release of the ownCloud Enterprise subscriptions before upgrading.

A blog post with instructions on upgrading will appear tomorrow on https://owncloud.org/news

Get the latest version on https://owncloud.org/install

You can find links and installation instructions on https://owncloud.org/release-channels and see what's new on https://owncloud.org/changelog

Note that if you use packages from the release channels, it is your responsibility to change your package repositories to a newer version before a release is no longer supported. If you still use ownCloud 6, you are strongly urged to upgrade to the stable 7 series or newer as soon as possible.

They all have received critical stability and security updates and it is highly recommended to upgrade.

Enjoy the latest ownCloud releases!

#112: phpMyAdmin HTTP Authentication

chris — Tue, 25 Apr 2017 10:51:50 GMT

There is a problem with the HTTP Authentication at https://phpmyadmin.crin.org/

#69: procmail, email2trac and TLS cert

chris — Thu, 07 Apr 2016 11:08:08 GMT

Each year the fingerpint of the server the Trac email accounts collects email from using fetchmail needs changing when the cert if updated, wiki:Trac#email2trac

#70: s3ql backup error

chris — Mon, 11 Apr 2016 10:08:41 GMT

Email from Crin3:

Starting fsck of s3c://s.qstack.advania.com:443/crin4/
Using cached metadata.
WARNING: Remote metadata is outdated.
ERROR: File system revision too old, please run `s3qladm upgrade` first.

#63: {Spam?} Best watches in the world>>

info@… — Wed, 24 Feb 2016 10:20:04 GMT

I need to sort out some spam filters, but not that the mailserver did detect this as spem and altered the subject line to add {Spam?}

#68: {Spam?} Best watches.

Alasteir@… — Mon, 14 Mar 2016 00:35:03 GMT

 Order watches here-
... spam removed...

#127: 1984.is Server Rebuild

chris — Tue, 21 Nov 2017 19:26:16 GMT

Due to critical hardware failure at 1984.is all three servers needed rebuilding from scratch.

#115: 500 Errors for crin.org

chris — Tue, 16 May 2017 17:00:33 GMT

There have been a lot of 500 errors today and they appear to be related to lines like this in /var/log/php/error.log:

[16-May-2017 17:33:08 Europe/London] PHP Fatal error:  Call to undefined function entity_translation_language_load() in /var/www/prod/docroot/includes/menu.inc on line 593

The section on the file in question:

 538
 539 /**
 540  * Loads objects into the map as defined in the $item['load_functions'].
 541  *
 542  * @param $item
 543  *   A menu router or menu link item
 544  * @param $map
 545  *   An array of path arguments; for example, array('node', '5').
 546  *
 547  * @return
 548  *   Returns TRUE for success, FALSE if an object cannot be loaded.
 549  *   Names of object loading functions are placed in $item['load_functions'].
 550  *   Loaded objects are placed in $map[]; keys are the same as keys in the
 551  *   $item['load_functions'] array.
 553  */
 554 function _menu_load_objects(&$item, &$map) {
 555   if ($load_functions = $item['load_functions']) {
 556     // If someone calls this function twice, then unserialize will fail.
 557     if (!is_array($load_functions)) {
 558       $load_functions = unserialize($load_functions);
 559     }
 560     $path_map = $map;
 561     foreach ($load_functions as $index => $function) {
 562       if ($function) {
 563         $value = isset($path_map[$index]) ? $path_map[$index] : '';
 564         if (is_array($function)) {
 565           // Set up arguments for the load function. These were pulled from
 566           // 'load arguments' in the hook_menu() entry, but they need
 567           // some processing. In this case the $function is the key to the
 568           // load_function array, and the value is the list of arguments.
 569           list($function, $args) = each($function);
 570           $load_functions[$index] = $function;
 571
 572           // Some arguments are placeholders for dynamic items to process.
 573           foreach ($args as $i => $arg) {
 574             if ($arg === '%index') {
 575               // Pass on argument index to the load function, so multiple
 576               // occurrences of the same placeholder can be identified.
 577               $args[$i] = $index;
 578             }
 579             if ($arg === '%map') {
 580               // Pass on menu map by reference. The accepting function must
 581               // also declare this as a reference if it wants to modify
 582               // the map.
 583               $args[$i] = &$map;
 584             }
 585             if (is_int($arg)) {
 586               $args[$i] = isset($path_map[$arg]) ? $path_map[$arg] : '';
 587             }
 588           }
 589           array_unshift($args, $value);
 590           $return = call_user_func_array($function, $args);
 591         }
 592         else {
 593           $return = $function($value);
 594         }
 595         // If callback returned an error or there is no callback, trigger 404.
 596         if ($return === FALSE) {
 597           $item['access'] = FALSE;
 598           $map = FALSE;
 599           return FALSE;
 600         }
 601         $map[$index] = $return;
 602       }
 603     }
 604     $item['load_functions'] = $load_functions;
 605   }
 606   return TRUE;
 607 }
 608

Peter, is the the cause of the 500 errors or is it a symptom, or is it unrelated? Do you have an idea what is going on here?

#54: 502 Errors

chris — Tue, 22 Dec 2015 11:06:04 GMT

Last night the wiki:NginxLogWatch script found a lot of 502 errors:

Dec 22     288 502, 0 503 and 0 504 errors from crin2.crin.org
Dec 22    4283 502, 0 503 and 0 504 errors from crin2.crin.org

I'm not sure why yet.

#60: Advania S3 Account

chris — Thu, 04 Feb 2016 11:36:22 GMT

This ticket is to track the time spent answering Andrew's questions regarding the Advania S3 account which is used for backups by Crin3.

#82: Attempted DDOS?

chris — Sat, 24 Sep 2016 09:20:39 GMT

Yesterday evening there 1k 502 errors caused by one user agent from multiple IP addresses. This isn't an uncommon event, I usually don't write them up, but I thought it might be useful for Mathew if I do with this one.

#111: Backups to Webarchitects server

chris — Thu, 13 Apr 2017 10:11:54 GMT

This is a ticket to setup backups to store1.webarch.net on.

#14: Blank CRINmail newsletters being sent out

chris — Sat, 16 May 2015 21:12:30 GMT

A newsletter sent via Drupal contained no content, Jenny reported:

We have tried twice today but both times when we send to all it sent out a blank email, with just the subject line. We have tried sending it to a few individual addresses and this works. We have sent other listservs out this week and they have worked as normal.

This is one of the emails:

From: CRIN [mailto:info@crin.org]
Sent: Friday, May 15, 2015 7:04 AM
To: Gertrud Lenzer
Subject: Children in Court CRINmail 48: Special edition on LGBTQI rights
--
*Child Rights International Network - CRIN*
Unit W125-127, Westminster Business Square
1-45 Durham Street
London SE11 5JH
United Kingdom
T: +44 20 7401 2257
 www.crin.org
@CRINwire <https://twitter.com/CRINwire>

#53: Blank email: CRINmail 1459

chris — Thu, 17 Dec 2015 13:04:49 GMT

I was sent this last night, with no email body:

To: chris@webarchitects.co.uk
Subject: CRINmail 1459
X-PHP-Originating-Script: 1005:mimemail.module
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8Bit
X-Mailer: Drupal
Sender: CRIN <info@crin.org>
From: CRIN <info@crin.org>
List-Unsubscribe: <https://www.crin.org/en/newsletter/confirm/remove/XXXX>
Precedence: bulk
Message-Id: <E1a9N9t-0001cP-Qt@crin2.crin.org>
Date: Thu, 17 Dec 2015 01:15:05 +0000

This looks like another case of the issue previously raised here: ticket:14

#89: Broken Links

matthew — Mon, 24 Oct 2016 12:11:44 GMT

https://www.crin.org/en/library/publications/crin-newsletter-15-mainstreaming-child-rights https://www.crin.org/en/library/publications/crin-newsletter-16-children-and-young-peoples-participation

were reported as broken

#73: CRIN2 ran out of memory

chris — Sun, 01 May 2016 19:16:38 GMT

The site has been down since Sun, 01 May 2016 18:05:44, it is now 8:15. I'm rebooting the server, it ran out of memory.

#41: CRINmail multipart email contain CSS in the plain text version

chris — Thu, 10 Sep 2015 10:07:17 GMT

CRINmail, which are sent by Drupal, have a HTML and plain text version, the plain text version contains CSS at the top, which takes a while to scroll through and makes it less likely that people with mail user agents set to display the plain text version first will read the newsletter, for example:

Return-Path: <info@crin.org>
Received: from mail.webarch.net ([unix socket])
        by mail.webarch.net (Cyrus v2.3.7-Invoca-RPM-2.3.7-16.el5_11) with LMTPA;
        Thu, 10 Sep 2015 00:19:23 +0100
X-Sieve: CMU Sieve 2.3
Received: from elderberry.rat.burntout.org (mx.webarch.net [81.95.52.71])
        (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits))
        (Client did not present a certificate)
        by mail.webarch.net (Postfix) with ESMTPS id A7BFB100072
        for <chris@webarchitects.co.uk>; Thu, 10 Sep 2015 00:19:23 +0100 (BST)
Received: from crin2.crin.org ([93.95.228.180])
        by elderberry.rat.burntout.org with esmtps (TLS1.2:RSA_AES_128_CBC_SHA1:128)
        (Exim 4.80)
        (envelope-from <info@crin.org>)
        id 1ZZoe5-00056h-Dl
        for chris@webarchitects.co.uk; Thu, 10 Sep 2015 00:19:23 +0100
Received: from www-data by crin2.crin.org with local (Exim 4.84)
        (envelope-from <info@crin.org>)
        id 1ZZoe4-0006TK-78
        for chris@webarchitects.co.uk; Wed, 09 Sep 2015 23:19:16 +0000
To: chris@webarchitects.co.uk
Subject: CRINmail 1445
X-PHP-Originating-Script: 1005:mimemail.module
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="8a276e6c6a6f9459ea8c98217bf4e986"
Content-Transfer-Encoding: 8Bit
X-Mailer: Drupal
Sender: CRIN <info@crin.org>
From: CRIN <info@crin.org>
List-Unsubscribe: <http://www.crin.org/en/newsletter/confirm/remove/XXX>
Precedence: bulk
Message-Id: <E1ZZoe4-0006TK-78@crin2.crin.org>
Date: Wed, 09 Sep 2015 23:19:16 +0000
[-- Attachment #1 --]
[-- Type: multipart/alternative, Encoding: 8bit, Size: 73K --]
Content-Type: multipart/alternative;
    boundary="f5f8c71876cf7825d95e748d55f23268"
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
-------- CRINMAIL 1445 -------------------------------------------------------
#nl_wrapper {background-color:#EFEFEF;}#nl_wrapper table,
#nl_wrapper td  {
   font-family:Verdana, Arial, Helvetica;
   line-height:150%;
   font-size:10px;
   margin: 0;
}
#nl_wrapper td img{
   margin: 0;
}
#nl_wrapper .headerTop {
background-color:#FFCC66; border-top:0px solid #000000; border-bottom:1px
solid #FFFFFF; text-align:center;
}
#nl_wrapper .adminText {
font-size:10px; color:#996600; line-height:200%; font-family:Verdana, Arial,
Helvetica; text-decoration:none;
}
#nl_wrapper .headerBar {
background-color:#FFFFFF; border-top:0px solid #333333; border-bottom:10px
solid #FFFFFF;
}
#nl_wrapper .title {
font-size:20px; font-weight:bold; color:#CC6600; font-family:Verdana, Arial,
Helvetica; line-height:150%;
}
#nl_wrapper .subTitle {
font-size:11px; font-weight:normal; color:#666666; font-style:italic;
font-family:arial;
}
#nl_wrapper td {
font-size:11px; color:#000000; line-height:150%; font-family:Verdana, Arial,
Helvetica;
}
#nl_wrapper .sideColumn {
background-color:#FFFFFF; border-left:1px dashed #CCCCCC; text-align:left;
}
#nl_wrapper .sideColumnText {
font-size:11px; font-weight:normal; color:#AAAAAA; font-family:Verdana,
Arial, Helvetica; line-height:150%;
}
#nl_wrapper .sideColumnTitle {
font-size:15px; font-weight:bold; color:#333333; font-family:Verdana, Arial,
Helvetica; line-height:150%;
}
#nl_wrapper .footerRow {
background-color:#FFFFCC; border-top:10px solid #FFFFFF;
}
#nl_wrapper .footerText {
font-size:10px; color:#996600; line-height:100%; font-family:Verdana, Arial,
Helvetica;
}
#nl_wrapper h1,
#nl_wrapper h2  {
   color:#888888;font-size:20px;font-weight:bold;font-family:Arial,
Helvetica;margin:0 0 10px 0; line-height:150%;
}
#nl_wrapper h2  { font-size:15px; margin:23px 0 10px 0; padding-top:8px;
border-top:1px solid #EFEFEF;}
#nl_wrapper p   {margin:10px 0;
font-size:11px;color:#333333;font-family:Verdana, Arial, Helvetica;
line-height:150%;}
#nl_wrapper a,
#nl_wrapper table td p a,
#nl_wrapper #crintable a,
#nl_wrapper #crintable a:visited { color:#1155cc; /* color:#003a6a;*/ }
#nl_wrapper a:hover,
#nl_wrapper table td p a:hover,
#nl_wrapper #crintable a:hover{color:#000;}
#nl_wrapper ul,
#nl_wrapper li,
#nl_wrapper
#crintable ul   {margin:0; padding:0; list-style:none; list-style-image:none;
list-style-type:none}
#nl_wrapper ul li,
#nl_wrapper #crintable ul li,
#nl_wrapper #CMrightcol ul li
{font-size:11px;color:#333333;background-image:url(http://crin.org/i/crinmail/bg_bullet.gif);
background-repeat:no-repeat; background-position:2px
center;font-family:Verdana, Arial, Helvetica;  margin:0; list-style:none;
list-style-type:none; padding:0 0 0 20px; margin:13px 0;}
#nl_wrapper div.FB,                                                                                     #nl_wrapper div.FB p    { margin:10px 0}
#nl_wrapper div.FB h2   {color:#5e768b; font-size:16px; margin:0;}
#nl_wrapper #crintable table td {
vertical-align: top;
}
#nl_wrapper h3.h3image{
   background-repeat: no-repeat;
   background-position: center right;
}
   http://crin.org
09 September 2015
subscribe http://crin.org/node/41907 | subscribe http://crin.org/node/293 |
submit information http://crin.org/en/home/about/contact-us
======== *CRINMAIL 1445* =====================================================
-------- IN THIS ISSUE: ------------------------------------------------------
*The year in children's rights:*
CRIN annual report 2015 #AR
*Latest news and reports* #news
- Discrimination on school grounds #dis
- Criminalising children for sexting #crim
- Access to bathrooms denied #ac
- States leave migrant children unprotected #sta
- Issues neglected by UN committees #iss
- UN calls for submissions #un
- Contribute to evaluation of CRIN’s work #cont
*Access to justice for children in Ireland* #a2j
*Upcoming events* #events
*Employment* #jbs
/Problems viewing this CRINmail? Click here
https://www.crin.org/en/home/what-we-do/crinmail/crinmail-1445. /
*/CRIN announcement/*
/We are aware that some of our readers may be receiving blank CRINmails. This
is due to our website experiencing some technical issues, which we are
working to resolve. Should you receive a blank or incomplete CRINmail, you
can also access the latest edition via our website here
https://www.crin.org/en/home/what-we-do/crinmail. /

Also note that the list unsub header contains a HTTP URL when this would be better as a HTTPS one for privacy reasons:

List-Unsubscribe: <http://www.crin.org/en/newsletter/confirm/remove/XXX>

#90: Cost of backup is estimated

matthew — Tue, 25 Oct 2016 07:06:20 GMT

It would be useful to estimate the expected cost of back up, by estimated the size and correlating this with the advertised price page.

#79: Create accounts for Matthew

chris — Thu, 21 Jul 2016 10:15:23 GMT

Create Trac and other accounts for Matthew.

#65: Custom 502 and 503 pages for crin.org

chris — Sun, 28 Feb 2016 14:17:03 GMT

It would be more informative if there were custom 502 and 503 error pages for https://www.crin.org/ rather than the default Nginx ones.

I will write up some details below on the 502 and 503's we seeing getting, see also ticket:54.

I'm happy to create a static 502 and static 503 error page and will look at adding it to the git repo.

#122: DB sync failed

russell — Fri, 23 Jun 2017 14:38:12 GMT

Hi Chris,

Our dev. site (https://dev.crin.org) looked like it got rebuilt from a very old backup.

I've just tried to resync dev. from live, and the DB sync has failed:

crin4:/var/www/dev/docroot/drush sql-sync @crin.prodbox-prod @crin.dev

.. with error 'Query failed'

This has always worked until today.

Any idea what's gone wrong?

Thanks, Russell

#97: Debian Stretch

chris — Mon, 30 Jan 2017 10:59:00 GMT

Debian Stretch will probably be released in May 2017 or so, we need a plan to migrate to it, specifically:

PHP 7.0
MariaDB 10

This has been raised with Russell on Bitbucket.

#17: Debian System Updates

chris — Tue, 09 Jun 2015 18:42:00 GMT

This ticket is document system updates done to the servers.

Sources for further information:

These updates are generally done using the wiki:AptitudeUpdateScript and this records all the changes in the /root/Changelog and then the contents of the Changelog are pasted into the ticket to document the upgrade.

#86: Discussion

chris — Wed, 05 Oct 2016 12:14:15 GMT

This ticket is to keep track of time spent discussing issues with Matthew via email and IRC.

#56: Drupal Caching

chris — Sun, 03 Jan 2016 11:46:35 GMT

This is a ticket to discuss and then implement some additional caching for the https://www.crin.org/ Drupal site.

Possible options:

Generating static content from Drupal and serving these to non-logged in users, for example using Boost or something like it.
Caching PHP generated pages using Nginx.
Using a caching server such as Varnish.

My preference would for 1. and 2., I can start doing some research on how best to do 2. I'd rather not do 3. if possible due to the complication it would add: php5-fpm -> varnish -> nginx.

#77: Drupal contrib - Highly Critical - Remote code execution PSA-2016-001

chris — Wed, 13 Jul 2016 09:47:24 GMT

At 16:00 UTC (5pm BST) fixes for multiple "highly critical remote code execution vulnerabilities" in unspecified Drupal 7 modules will be released and it is expected that "exploits are expected to be developed within hours/days", see:

https://www.drupal.org/node/2764899

Peter: do you have this in hand? I notice that the dev site is still not running properly:

https://dev.crin.org/

CSS is linked like this:

@import url("https:///sites/all/modules/contrib/domain/domain_nav/domain_nav.css?o4jry7");
@import url("https:///modules/field/theme/field.css?o4jry7");

JS is linked like this:

<script src="https:///sites/default/files/js/js_zzcIWOou_jnX0ZWAIA4sb6Xy_p5a8FZNA0GySvuWjPU.js"></script>

And images are linked like this:

<img typeof="foaf:Image" src="https:///sites/default/files/styles/390x387/public/1.1.jpg?itok=p_PWQReL" width="390" height="387" alt="" />

Is there anything I can do to help?

#132: Drupal core - Highly critical - Remote Code Execution - SA-CORE-2018-002

chris — Wed, 28 Mar 2018 20:06:22 GMT

https://www.drupal.org/sa-core-2018-002

#49: Frozen emails on crin4

chris — Thu, 05 Nov 2015 19:22:51 GMT

Munin is sending alert emails every 5 mins because of the number of frozen emails on Crin4.

#108: Go live

russell — Tue, 04 Apr 2017 16:01:05 GMT

Hi Chris,

I've got a round of work ready to go onto Crin live. All tested and signed off on stage. The code element is very light. I don't expect any problems, but just letting you know. I'm planning to get into this as the working day is finishing today.

Best, Russell

#42: How CRINmail deals with email addresses that no longer work

chris — Thu, 10 Sep 2015 10:42:07 GMT

When the CRINmail newsletter is sent out via Drupal it has the following from envelope:

Return-Path: <info@crin.org>

This results in emails that are returned as undeliverable going to that address and the Drupal site doesn't know when a recipient address no longer works so is unable to stop sending emails to address that bounce email back.

One symptom of this problem are the forozen emails that build up after each CRINmail run, see the blue on Munin graphs here:

https://munin.crin.org/munin/crin.org/crin2.crin.org/exim_mailqueue.html

What should happen is that the from envelope (this is different from the From: header users see, that can remain as it is) should be an address that Drupal can check (probably via IMAP but we could set up one of the servers to accept SMPT email if needs be) and then do some thing sensible with the bounces.

See for example the discussion here:

https://stackoverflow.com/q/1011714

#117: Install Newrelic on Crin2

chris — Wed, 17 May 2017 11:51:53 GMT

Peter has requested that Newrelic be installed on Crin2.

Note that last time it was installed we had to remove it due to the resources it was using, see this thread ticket:73#comment:24, so I'd advise against leaving it on the server all the time.

#99: Let's Encrypt Cert for CRIN.org

chris — Mon, 30 Jan 2017 14:32:43 GMT

The Gandi cert expires in a couple of weeks, in 17th Feb 2017, see:

https://www.ssllabs.com/ssltest/analyze.html?d=www.crin.org&latest

It would make sense to switch to use https://letsencrypt.org/ and the client implementation I suggest we use is this one https://github.com/Neilpang/acme.sh

#131: Let's Encrypt Certificates

chris — Tue, 20 Feb 2018 21:17:52 GMT

I have installed a new script for getting HTTPS certs on both servers:

https://github.com/Neilpang/acme.sh

And there are now new certs for these domains:

The Trac cert on this server is also used for, wiki.crin.org, phpmyadmin.crin.org, munin.crin.org and stats.crin.org.

#64: Let's Encrypt HTTPS Certs

chris — Fri, 26 Feb 2016 11:30:54 GMT

These certs, on Crin1, are due to expire soon:

https://cloud.crin.org/ 13/03/16
https://stats.crin.org/ 13/03/16
https://trac.crin.org.archived.website/ 29/04/16
https://wiki.crin.org/ 03/05/16

The one on Crin2 has a while to run:

https://www.crin.org/ 17/02/18

To save money I suggest that the certs on Crin1 are replaced with Let's Encrypt ones, for the above domains and in addition these sites which use CAcert:

https://munin.crin.org/ 07/05/17
https://phpmyadmin.crin.org/ 07/05/17

I suggest we still use CACert for the MySQL certs as these are valid for two years and Let's Encrypt ones are only valid for 3 months and it would be complicated to automate the renewal and deployment for these certs.

#116: Limit the number of MySQL connections from Crin4 to Crin1

chris — Wed, 17 May 2017 11:44:18 GMT

Both the dev, Crin4 and live Crin2, servers use the MySQL database on Crin1 and we want to limit the resources available to Crin4 so it reduced the impact on the live site.

Some documentation on this:

#45: MediaWiki 1.23.11

chris — Fri, 16 Oct 2015 08:44:21 GMT

Email on the announcements list:

Tomorrow we will be issuing a security release to all supported
branches of MediaWiki.
The new releases will be:
1.25.3
1.24.4
1.23.11
Fixes will be available in these respective release branches, the
unreleased 1.26.x branch, and master. Tarballs will be available
for the above mentioned point releases as well.
This security release will encompass core only, no bundled extensions
are affected.

#119: MySQL Downtime

chris — Thu, 15 Jun 2017 10:31:52 GMT

Just after midnight last night MySQL appears to have started having some issues.

#106: MySQL Error

chris — Mon, 27 Mar 2017 13:37:30 GMT

On Saturday morning the MySQL database on Crin1 was generating errors which prevented the site for working, this was fixed by running this command on the server:

mysqladmin flush-hosts

#113: MySQL Performance

chris — Tue, 25 Apr 2017 11:22:11 GMT

MySQL is set to use a lot of resources, see the RAM allocation here:

https://munin.crin.org/munin/crin.org/crin1.crin.org/multips_memory.html

And also a lot of queries are cached via memcache, which also has a lot of RAM allocated, see:

https://munin.crin.org/munin/crin.org/crin2.crin.org/multips_memory.html

What can we do to make it quicker?

See also the work done when the server was setup on ticket:9

#94: Ongoing high load caused by a bot

chris — Thu, 15 Dec 2016 19:23:12 GMT

Ticket to working out what to do with this issue...