CRIN Trac: Ticket Query

#13: Access to db1.crin.org

chris — Wed, 13 May 2015 10:41:36 GMT

This ticket is for tracking the time taken to enable access to the db1.crin.org MySQL database server at GreenQloud for codepositive.com.

#18: Drush

chris — Fri, 12 Jun 2015 16:34:39 GMT

Drush doesn't work on Crin2:

su - www-data -s /bin/bash
drush pm-updatestatus
Command pm-updatestatus needs a higher bootstrap level to run - you will need to invoke drush from a more functional Drupal [error]
environment to run this command.
The drush command 'pm-updatestatus' could not be executed. [error]
Drush was not able to start (bootstrap) the Drupal database. [error]
Hint: This may occur when Drush is trying to:
* bootstrap a site that has not been installed or does not have a configured database. In this case you can select another site
with a working database setup by specifying the URI to use with the --uri parameter on the command line. See `drush topic
docs-aliases` for details.
* connect the database through a socket. The socket file may be wrong or the php-cli may have no access to it in a jailed shell.
See http://drupal.org/node/1428638 for details.
Drush was attempting to connect to:
Drupal version : 7.24
Site URI : http://crin1.crin.org/
Database driver : mysql
Database hostname : crin1
Database port :
Database username : drupal
Database name : drupal
PHP executable : /usr/bin/php
PHP configuration : /etc/php5/cli/php.ini
PHP OS : Linux
Drush script : /usr/local/src/drush-master/drush.php
Drush version : 7.0-dev
Drush temp directory : /tmp
Drush configuration : /var/www/.drush/drushrc.php
Drush alias files :
Drupal root : /var/www/drupal
Site path : sites/default

#21: Apache Solr HTTP 400 Bad Request Errors

chris — Wed, 17 Jun 2015 15:01:15 GMT

There are lots of errors like this in the dblog:

Type		Apache Solr
Date		Wednesday, June 17, 2015 - 15:53
User		Anonymous
Location	http://www.crin.org/fr/biblioth%C3%A8que/archives-des-actualit%C3%A9s?page=8
Message	HTTP 400; Bad Request: Bad Request
Severity	error

And

Type		Apache Solr
Date		Wednesday, June 17, 2015 - 15:53
User		Anonymous
Location	http://www.crin.org/fr/biblioth%C3%A8que/archives-des-actualit%C3%A9s?page=8
Message		HTTP Status: 400;
Message: 	Bad Request: Bad Request;
Response: 	<html><head><title>Apache Tomcat/7.0.56 (Debian) - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 400 - undefined field hash</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>undefined field hash</u></p><p><b>description</b> <u>The request sent by the client was syntactically incorrect.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.56 (Debian)</h3></body></html>;
Request: GET /solr/select?start=0&rows=5&fq=hash%3A7as1z6&fq=hash%3A7as1z6&q=id%3A7as1z6%2Fnode%2F266&qt=mlt&fl=entity_id%2Centity_type%2Clabel%2Cpath%2Curl%2Chash%2Csite&mlt.fl=path_alias%2Ctaxonomy_names&mlt.mintf=1&mlt.mindf=1&mlt.minwl=3&mlt.maxwl=15&mlt.maxqt=20&wt=json&json.nl=map HTTP/1.0 User-Agent: Drupal (+http://drupal.org/) Host: localhost:8080 ;
Caller: call_user_func_array() (line Inconnu of Inconnu)
Severity	error

Requests for pages such as the one in the example above generate both the above errors.

#22: Add Trac Account for Peter

chris — Wed, 17 Jun 2015 16:15:02 GMT

Add an account for Peter following wiki:Trac#CreateanTracaccount

#23: Development and staging environment

chris — Wed, 17 Jun 2015 16:25:58 GMT

Email from Peter:

We would like to set up an environment that is as close as possible to the production environment as possible, so that we are able to trial and review any changes to the site before they go live.

We would ideally have two environments dev and stage on the same box.
Ideally we would use dev.crin.org and stage.crin.org as two document roots
We will script any deployment from git
the same script should be used on productin too
We will use drush to sync between production and staging/dev
we will need to have ssh access between the stage box and the prod box to be able to sync
I would also like to know how you manage updates on the boxes. In the long run it is pretty important that the dev box is kept in sync with the prod box.

#25: Piwik 2.14.0

chris — Wed, 24 Jun 2015 08:46:52 GMT

A new version of Piwik will be out very soon (probably best to wait for the full release, there is a release candidate out today) which fixes critical security issues - upgrading should take no more than 15 mins, would you like your site upgraded?

Three security issues have been fixed. We are grateful for security researchers who responsibly disclosed these security issues to us: Abdullah Hussam Gazi (CSRF issue) and Dmitriy Shcherbatov (two XSS issues).

https://piwik.org/changelog/piwik-2-14-0/

#27: Crin2 certificate access

peter — Mon, 06 Jul 2015 16:29:00 GMT

When attempting to use Drush, when not root, we are not able to access the certificate.

SSL error: Unable to get certificate from '/etc/ssl/cacert/crin1_cert.pem'

How should we manage access to certificates? Should we create a developers group?

#28: Crin1 server email problem

chris — Tue, 14 Jul 2015 12:25:11 GMT

Web application on Crin1 are unable to send email, when they attempt to they log this error in the /var/log/exim4/paniclog:

2015-07-09 00:43:08 unable to set gid=115 or uid=0 (euid=0): forcing real = effective

Jenny / Gillian would you like me to fix this problem? It won't take long as the solution is here:

http://blog.gaiterjones.com/send-php-mail-ubuntu-upgrade/

#29: Piwik 2.14.1

chris — Fri, 17 Jul 2015 08:41:19 GMT

A new version of Piwik is out, from the changelog:

What’s new in Piwik 2.14.1?

In this release we have focused on fixing a few regressions reported in the last major release Piwik 2.14.0, as well as 27 other small improvements. As always we – the Piwik team – are very interested to hear your feedback!

27 tickets have been closed by 6 contributors.

#33: Piwik 2.14.3

chris — Mon, 24 Aug 2015 11:06:12 GMT

From the changelog:

August 5, 2015

Piwik 2.14.3 is a new minor release of Piwik.

What’s new?

This release addresses a regression introduced in 2.14.2 that prevents users from installing Piwik in some cases.

#34: Memcache Munin Stats

chris — Mon, 07 Sep 2015 09:48:01 GMT

Munin stats for memcache stopped at the and of August 2015 and I'm not sure why.

#35: Problem with the S3QL backups for Crin1

chris — Mon, 07 Sep 2015 10:55:30 GMT

Just tried mounting all the backup on Crin4 and there was a problem with the Crin1 backup:

umnt-s3ql crin1
/media/s3ql/crin1 is not on an S3QL file system
Can not check mounted file system.
mnt-s3ql crin1
Can not check mounted file system.
Filesystem check of s3c://s.qstack.advania.com:443/crin1 failed
Using 2 upload threads (memory limited).
Autodetected 65498 file descriptors available for cache entries
Using cached metadata.
File system damaged or not unmounted cleanly, run fsck!
Mounting s3c://s.qstack.advania.com:443/crin1 on /media/s3ql/crin1 failed
mnt-s3ql crin1 --force
Can not check mounted file system.
Filesystem check of s3c://s.qstack.advania.com:443/crin1 failed
Using 2 upload threads (memory limited).
Autodetected 65498 file descriptors available for cache entries
Using cached metadata.
File system damaged or not unmounted cleanly, run fsck!
Mounting s3c://s.qstack.advania.com:443/crin1 on /media/s3ql/crin1 failed

#36: Piwik stats stopped on 31st August 2015

chris — Tue, 08 Sep 2015 11:14:19 GMT

The Piwik stats stopped on 31st August 2015.

The tracking code is missing, can the Drupal Piwik module be re-installed / re-enabled please?

In addition I believe the GA tracking code can be removed as the Piwik stats replace these.

#37: Site slowdown on 8th Sept 2015

chris — Tue, 08 Sep 2015 13:15:07 GMT

The site is really slow and the number of php5-fpm processes has gone up dramatically and so has the memory usage.

#38: [URGENT] crin.org down

mori — Tue, 08 Sep 2015 15:54:35 GMT

Hi Chris,

Jenny just reached out to us by email reporting crin.org is down. The error message I see on crin.org is:

PDOException: SQLSTATE[28000] [1045] Access denied for user 'newprod'@'crin2' (using password: YES) in lock_may_be_available() (line 167 of /var/www/prod/docroot/includes/lock.inc).

It sounds like it may be something that's related to the issue you mentioned in #36 (https://trac.crin.org.archived.website/trac/ticket/36#comment:7 )

Can you please have a look at this as a matter of urgency?

Thanks,

Mori

#39: S3QL backup problems

chris — Wed, 09 Sep 2015 11:36:18 GMT

Often backups fail during the rsync with "Transport endpoint is not connected".

The S3QL Crin3 backup script mounts on /tmp but the other scripts don't so where there is some issue the unmount don't work as the S3QL system is mounted in an unknown location, see ticket:35. [SOLVED]

There are also some unneeded scripts in /usr/local/bin on Crin3 which need removing to make things less confusing. [SOLVED]

#40: Check Bitbucket documentation

chris — Wed, 09 Sep 2015 13:21:35 GMT

Chris needs to read and check the documentation here: https://bitbucket.org/crin/crin/wiki/Home and update the Trac documentation as need be.

#43: ownCloud 8.1.3-13.1

chris — Thu, 17 Sep 2015 11:52:00 GMT

Email from Crin1:

Subject: 4 Debian package update(s) for crin1
apticron report [Thu, 17 Sep 2015 11:40:12 +0000]
========================================================================
apticron has detected that some packages need upgrading on:
        crin1
        [ 93.95.228.179 ]
The following packages are currently pending an upgrade:
        owncloud 8.1.3-13.1
        owncloud-config-apache 8.1.3-13.1
        owncloud-server 8.1.3-13.1
        php5-ldap 5.6.13+dfsg-0+deb8u1
========================================================================
Package Details:
Reading changelogs...
--- Changes for owncloud (owncloud owncloud-config-apache owncloud-server) ---
owncloud (8.1.3-13.1) stable; urgency=low
  * version number update by debtransform
 -- debtransform <build@opensuse.org>  Wed, 16 Sep 2015 14:39:25 +0000
owncloud (8.1.3-1) stable; urgency=low
  * Update to version 8.1.3
 -- Juergen Weigert <jw@owncloud.com>  Fri, 11 Sep 2015 11:48:14 +0200
owncloud (8.1.2-1) stable; urgency=low
  * Update to version 8.1.2
 -- Juergen Weigert <jw@owncloud.com>  Wed, 02 Sep 2015 12:39:29 +0200
owncloud (8.1.2~RC1-1) stable; urgency=low
  * Update to version 8.1.2~RC1
 -- Juergen Weigert <jw@owncloud.com>  Sun, 30 Aug 2015 15:20:03 +0200
owncloud (8.1.1-3) stable; urgency=low
  * Possible fix for https://github.com/owncloud/core/issues/18222
 -- Juergen Weigert <jw@owncloud.com>  Tue, 25 Aug 2015 17:42:12 +0200
owncloud (8.1.1-2) stable; urgency=low
  * Make occ executable. https://github.com/owncloud/core/issues/18508
  * Using DEBTRANSFORM-RELEASE to automate incrementing build release number.
  * Fix owncloud-server: php-script-but-no-phpX-cli-dep var/www/owncloud/occ
 -- Juergen Weigert <jw@owncloud.com>  Tue, 25 Aug 2015 14:25:48 +0200

#44: PHP temp session files need deleting

chris — Fri, 18 Sep 2015 09:15:52 GMT

On Crin1 php session files are written to these directories:

/var/www/piwik/tmp
/var/www/mediawiki/w/tmp
/var/www/mediawiki/tmp
/var/www/owncloud/tmp

Something like tmpreaper needs installing and configuring to delete old files.

On Crin2 and Crin4 /tmp is used and that could also do with old files being deleted.

I have CC'd Mori in case there are any issues I need to be aware of regarding Drupal / Solr and temp files.

#48: crin4 /tmp filling up

chris — Wed, 28 Oct 2015 10:44:53 GMT

A task is need to clear /tmp on Crin4.

#51: Number of php5-fpm processes on Crin2

chris — Wed, 09 Dec 2015 12:34:55 GMT

Peter -- a couple of days ago the number of php5-fpm processes on Crin2 jumped from 9 to 39 and this uses upto 5G or RAM and there is only 4G, see the following Munin graphs:

Note: See TracTickets for help on using tickets.

Any idea what changed?

#52: 404 for top article - caused by 'und' in URL not 'en'

chris — Sat, 12 Dec 2015 16:06:20 GMT

The first articles linked to on the front page of https://www.crin.org/ is a 404 not found:

CIVIL SOCIETY UNDER THREAT: Old and new challenges for human and children's rights advocates in Eurasia 11 Dec 2015 In recent years, the space afforded to civil society to operate freely has been shrinking dramatically across the world,... https://www.crin.org/und/library/publications/civil-society-under-threat-old-and-new-challenges-human-and-childrens-rights

The article is actually here:

https://www.crin.org/en/library/publications/civil-society-under-threat-old-and-new-challenges-human-and-childrens-rights

The problem is the und rather then en in the URL?

#58: You have 1 new fax, document 00000576492

incoming@… — Wed, 13 Jan 2016 20:10:03 GMT

A new fax document for you.
Please, download fax document attached to this email.
Quality:            200 DPI
Sender:             Theodore Thompson
Pages:              12
Scanned in:         46 seconds
Document name:      scan.00000576492.doc
File size:          240 Kb
Scanned:            Wed, 13 Jan 2016 08:29:25 +0300
Thanks for using Interfax service!

scan.00000576492.zip

#59: Trac Spam

chris — Wed, 13 Jan 2016 20:38:11 GMT

We might have to whitelist email addresses which can open tickets or something, see ticket:58

#62: Crin3 ethernet interface fails to come up after a reboot

chris — Mon, 15 Feb 2016 11:49:52 GMT

After updating the kernel on Crin3 I rebooted it and then couldn't connect to it apart from using VNC.

#63: {Spam?} Best watches in the world>>

info@… — Wed, 24 Feb 2016 10:20:04 GMT

I need to sort out some spam filters, but not that the mailserver did detect this as spem and altered the subject line to add {Spam?}

#67: S3QL 2.17 has been released

chris — Wed, 09 Mar 2016 19:07:56 GMT

This is worth noting:

From the changelog:

2016-03-08, S3QL 2.17

The internal file system revision has changed. File systems created with S3QL 2.17 or newer are not compatible with prior S3QL versions. To update an existing file system to the newest revision, use the s3qladm upgrade command.
https://groups.google.com/forum/?_escaped_fragment_=topic/s3ql/t36XMqiS5TU#!topic/s3ql/t36XMqiS5TU

Currently Debian unstable has 2.15:

https://packages.debian.org/stretch/s3ql

But it should update soon.

#68: {Spam?} Best watches.

Alasteir@… — Mon, 14 Mar 2016 00:35:03 GMT

 Order watches here-
... spam removed...

#69: procmail, email2trac and TLS cert

chris — Thu, 07 Apr 2016 11:08:08 GMT

Each year the fingerpint of the server the Trac email accounts collects email from using fetchmail needs changing when the cert if updated, wiki:Trac#email2trac

#70: s3ql backup error

chris — Mon, 11 Apr 2016 10:08:41 GMT

Email from Crin3:

Starting fsck of s3c://s.qstack.advania.com:443/crin4/
Using cached metadata.
WARNING: Remote metadata is outdated.
ERROR: File system revision too old, please run `s3qladm upgrade` first.

#71: Piwik 2.16.1

chris — Tue, 12 Apr 2016 10:33:41 GMT

Updated needed, see https://piwik.org/changelog/piwik-2-16-1/

#75: Load spike and small downtime

chris — Sun, 03 Jul 2016 18:00:34 GMT

Since about 8:10am this morning there has been a noticeable increase in bandwidth usage and load on Crin2 and at 18:33, for 20 seconds, the Webarchitects monitoring server was unable to connect.

#78: HTTP_PROXY possible security risk

chris — Tue, 19 Jul 2016 10:21:46 GMT

See httpoxy.

#80: Piwik 2.16.2

chris — Tue, 16 Aug 2016 11:08:37 GMT

We are proud to announce Piwik 2.16.2: a new minor release of Piwik!

We have identified and fixed several XSS security issues in this release.

This release is rated critical.

https://piwik.org/changelog/piwik-2-16-2/

#84: Piwik 2.16.4

chris — Tue, 04 Oct 2016 11:54:50 GMT

The Changelog:

Piwik 2.16.4 fixes a regression introduced yesterday in Piwik 2.16.3. The Scheduled reports were not working for websites where the website’s timezone was UTC+N or UTC-N. This is now fixed.

#87: Piwik 2.16.5

chris — Wed, 05 Oct 2016 13:07:17 GMT

The Changelog:

Piwik 2.16.5 fixes a regression introduced yesterday in Piwik 2.16.3 and not fully fixed in Piwik 2.16.4. The Scheduled reports were not working for websites where the website’s timezone was UTC+N or UTC-N. This is now fixed.

#91: Piwik 2.17.0

chris — Thu, 27 Oct 2016 19:08:57 GMT

No security implications, see the changelog.

#96: Piwik 3.0.1

chris — Wed, 11 Jan 2017 13:18:36 GMT

New version of Piwik: https://piwik.org/changelog/piwik-3-0-1/

#98: Trac account for Russell

chris — Mon, 30 Jan 2017 11:43:44 GMT

Create a Trac account for the new Drupal developer, Russell.

#101: Piwik 3.0.2

chris — Thu, 02 Mar 2017 18:58:40 GMT

A new version of Piwik is out:

https://piwik.org/changelog/piwik-3-0-2/

#105: High load and slow responses from Crin2

chris — Mon, 27 Mar 2017 12:15:49 GMT

Trac threw this error:

Warning:

Error with navigation contributor "TimingEstimationAndBillingPage?"
Error with navigation contributor "AdminModule?"
Error with navigation contributor "SearchModule?"
Failed to sync with repository "(default)": database is locked; repository information may be out of date. Look in the Trac log for more information including mitigation strategies.

And in /var/log/php5-fpm.log we have:

[27-Mar-2017 10:20:42] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 10:30:10] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:54:51] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:56:35] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:57:08] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:57:58] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 12:01:48] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 12:02:26] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it

From an initial look at the Nginx logs the problem seesm to be caused but too much agressive crawling of the site by bots.

#107: New cert for Fetechmail

chris — Tue, 04 Apr 2017 13:15:41 GMT

See wiki:Trac#email2trac

#109: Piwik 3.0.3

chris — Wed, 05 Apr 2017 09:41:47 GMT

New version of Piwik, Piwik 3.0.3.

#110: Piwik Accounts for Peter and Lisa

chris — Mon, 10 Apr 2017 13:53:26 GMT

Peter and Lisa from the-organization.com need Piwik access

#112: phpMyAdmin HTTP Authentication

chris — Tue, 25 Apr 2017 10:51:50 GMT

There is a problem with the HTTP Authentication at https://phpmyadmin.crin.org/

#123: Update Email Settings

chris — Mon, 24 Jul 2017 15:23:37 GMT

This Trac site uses a @webarch.net email account and that domain has been migrated to a new server and a new account needs setting up and the fetchmail settings need updating.

#124: Piwik 3.1.0

chris — Fri, 15 Sep 2017 10:16:09 GMT

Details: https://piwik.org/changelog/piwik-3-1-0/

#126: Piwik 3.2.0

chris — Mon, 16 Oct 2017 12:27:16 GMT

Changelog: https://piwik.org/changelog/piwik-3-2-0/

#128: Piwik 3.2.1

chris — Thu, 07 Dec 2017 12:07:06 GMT

Changelog: https://piwik.org/changelog/piwik-3-2-1/

#130: Matomo 3.3.0

chris — Mon, 15 Jan 2018 09:59:07 GMT

A new version of Matomo (formally Piwik) is out, 3.3.0,

https://matomo.org/changelog/matomo-3-3-0/

#133: Matomo 3.5.0

chris — Wed, 09 May 2018 09:01:31 GMT

Upgrading to the latest version.

#31: Preparation for deployment to new prod

mori — Mon, 10 Aug 2015 15:32:41 GMT

Hi Chris,

In order to run the CRIN production site under the new codebase, we need to go through a number of steps to set up and test the site. We need your help to accomplish this.

Overview: (please execute each step once we request you to do so through the comment)

Chris to create a new entry for the new prod site on CRIN2
Chris to create a new database for the new prod site on CRIN1
Code+ and CRIN to test the new prod site and sign off
Chris to update the nginx config so the new prod serves as the working prod

Rationale of the go-live process Because of the reasons listed below, deploying the new code to the existing prod results in an extended period of downtime:

repository too large: cloning takes 10+ minutes
changes to directory structure requires the registry to be rebuilt
the new deployment process needs testing

#50: Migrate to INNODB

chris — Wed, 25 Nov 2015 09:50:16 GMT

Email from Peter:

What is involved in changing any of the underlying server resources for Crin.org.

I think we generally have too many CPU's and not enough memory.

To improve the site performance, we need to move the database to INNODB. But we are also already using swap on the DB server, and INNODB wants the entire DB in ram. I think we should switch to at least 8GB ram, but can probably scale down to 4 cpus - and still get a performance increase.

#7: Enable Google Drive plugin for ownCloud

chris — Wed, 29 Apr 2015 09:42:55 GMT

Configure the external storage application on https://cloud.crin.org/ on the existing GreenCloud server for mounting Google Drive.

#15: Create Trac accounts for Jenny and Gillian

chris — Tue, 02 Jun 2015 13:43:54 GMT

Jonas is leaving at the end of the week and Trac accounts are needed for Jenny and Gillian at CRIN.

#16: Add Trac account for Adam

chris — Thu, 04 Jun 2015 12:37:33 GMT

It turns out that Graham, for whom an account was created, see ticket:1#comment:25 was not responsible for the MediaWiki site, so he needs removing from the MediaWiki ticket, see ticket:5#comment:6 and an account for Adam needs to be created and he needs to be added to that ticket.

#24: Exim Mailqueue

chris — Fri, 19 Jun 2015 10:04:32 GMT

Yesterday a CRINmail newsletter was sent out (see also ticket:14#comment:5) and there are currently 245 emails stuck in the mail queue on Crin2, these are probably undeliverable addresses and there should be a way to mark them so and perhaps disable sending emails to these addresses in the future.

In addition as the queue is greater that 100 it is resulting in Munin sending a alert email every 5 mins, this threshold needs increasing to prevent these emails.

#26: Create ssh and Trac account(s) for Code Positive

chris — Mon, 06 Jul 2015 15:52:34 GMT

Ticket to document the task and the time adding accounts ot the servers for Peter.

#30: Piwik 2.14.2

chris — Tue, 28 Jul 2015 10:48:31 GMT

The Changelog:

What’s new in Piwik 2.14.2?

In this release we have focused on fixing a few regressions reported in the last major release Piwik 2.14.0, as well as 15 other small improvements. As always we – the Piwik team – are very interested to hear your feedback!

15 tickets have been closed by 6 contributors.

(I'll do this now and record the time in August)

#32: PHP error logs not found

mori — Tue, 18 Aug 2015 14:41:56 GMT

I'm having issues on newprod, but I can't find PHP error logs. If they are logged, can you please let me know where they are? If they are not logged, can you start logging them on both CRIN2 and CRIN4? Thanks.

#46: Piwik 2.15.0

chris — Fri, 16 Oct 2015 09:08:19 GMT

The Piwik 2.15.0 changelog:

We are proud to announce Piwik 2.15.0 Release Candidate: a new major release of Piwik!

What’s new?

Piwik 2.15.0 is our Long Term Support release for Piwik 2.X. This release aims to be outstanding from a performance, security, and reliability point of view. It also includes a beautiful new design which lets you focus on your data, and a new awesome search bar to lets you easily navigate to any menu, website or segment in your Piwik dashboards. For administrators and power user, this release includes new diagnostic tools, logging messages, a new Log viewer plugin, and better overall performance and scalability. There are dozens other improvements, see the full list below. Piwik 2.15.0 is built to last.

Security release

This release is rated critical.

We are grateful for Security researchers who disclosed security issues privately to the Piwik Security Response team: Elamaran Venkatraman, Egidio Romano and Dmitriy Shcherbatov. The following vulnerabilities were fixed: XSS, CSRF, possible file inclusion in older PHP versions (low impact), possible Object Injection Vulnerability (low impact).

Database upgrade

This release does not contain any major database upgrade.

Platform Changes

Piwik is an open analytics platform. In an effort to help Piwik developers learn about improvements and changes in the core APIs, we document the changes since the last release.

In this 2.15.0 release there are API deprecations, New features, New APIs, New commmands, Internal API changes.

Read more in Platform Changelog for Developers to see all changes to the platform and APIs (you can also find it in the CHANGELOG.md in the root of your Piwik)

#47: ownCloud 8.1.4.2

chris — Tue, 20 Oct 2015 13:58:02 GMT

From the email list:

Dear ownCloud users!

We're proud to make available the latest ownCloud Server 8.2, which delivers user interface improvements and significant new features for system administrators. This release is deemed stable for home users and small servers. Enterprise users are encouraged to wait for the release of the ownCloud Enterprise subscriptions before upgrading.

A blog post with instructions on upgrading will appear tomorrow on https://owncloud.org/news

Get the latest version on https://owncloud.org/install

You can find links and installation instructions on https://owncloud.org/release-channels and see what's new on https://owncloud.org/changelog

Note that if you use packages from the release channels, it is your responsibility to change your package repositories to a newer version before a release is no longer supported. If you still use ownCloud 6, you are strongly urged to upgrade to the stable 7 series or newer as soon as possible.

They all have received critical stability and security updates and it is highly recommended to upgrade.

Enjoy the latest ownCloud releases!

#55: Migrating Crin2 server to new hardware

chris — Fri, 01 Jan 2016 18:21:49 GMT

In order for the Crin2 server to have it's RAM doubled it needs to be migrated to a different physical server, I would expect this would take between 15 mins and an hour at most, is there any preference when this is done? I'd suggest doing it this weekend before everybody is back at work on Monday would make sense? I have contacted 1984.is about it and they can script it to happen at what ever time is best.

#61: Piwik 2.16.0

chris — Sun, 07 Feb 2016 10:41:42 GMT

From the changelog:

What’s new?

Piwik 2.16.0 is a release which includes more than 250 closed issues. To learn about the most important and visible changes, read our blog post: What’s new in Piwik 2.16.0?

This release is our Long Term Support release.

263 tickets have been closed by more than 30 contributors!

#74: Visit to CRIN office

chris — Sun, 05 Jun 2016 19:04:12 GMT

This ticket is to record the time spent visting the CRIN office on Friday 3rd June 2016.

#83: Piwik 2.16.3

chris — Mon, 03 Oct 2016 10:00:18 GMT

The Changelog contains:

Security release

This release is rated critical.

The Piwik security engineering team has internally identified a critical security issue and has fixed it in Piwik 2.16.3. We recommend all users to upgrade to this latest version.

Database upgrade

Note: This release contains major database upgrades and upgrading your database will take a long time if you have a lot of data in your database.

Please make sure you read the Update Piwik guide for high traffic instances.

#95: Piwik 3.0.0

chris — Mon, 09 Jan 2017 11:16:07 GMT

New version of Piwik, see https://piwik.org/changelog/piwik-3-0-0/

#12: ownCloud Synching

jonas — Mon, 11 May 2015 15:15:54 GMT

Changes are not synching between the desktop app and the browser version. There is no error message showing up at the desktop app.

#14: Blank CRINmail newsletters being sent out

chris — Sat, 16 May 2015 21:12:30 GMT

A newsletter sent via Drupal contained no content, Jenny reported:

We have tried twice today but both times when we send to all it sent out a blank email, with just the subject line. We have tried sending it to a few individual addresses and this works. We have sent other listservs out this week and they have worked as normal.

This is one of the emails:

From: CRIN [mailto:info@crin.org]
Sent: Friday, May 15, 2015 7:04 AM
To: Gertrud Lenzer
Subject: Children in Court CRINmail 48: Special edition on LGBTQI rights
--
*Child Rights International Network - CRIN*
Unit W125-127, Westminster Business Square
1-45 Durham Street
London SE11 5JH
United Kingdom
T: +44 20 7401 2257
 www.crin.org
@CRINwire <https://twitter.com/CRINwire>

#19: PHP Parser Error - node--country.tpl.php

chris — Mon, 15 Jun 2015 10:21:19 GMT

The php-fpm error log at /var/log/fpm-php.www.log contains lots of entries like this:

[15-Jun-2015 11:17:32 Europe/London] PHP Parse error:  syntax error, unexpected end of file in /var/www/drupal/sites/default/themes/crin/templates/node--country.tpl.php on line 522

Line 522 is the last line of the file, it simply contains a closing } bracket, this is the last part of the file:

    <?php
  }
  ?>
  <?php // print render($content['links']); ?>
  <?php // print render($content['comments']); ?>
<?php
  // 'Albania' = 441;
  //$node_load(441);
  /*
  global $user;
  if($user->uid == 1 ){
    if($node->nid == 441){
      //update_country_from_wiki($node);
    }
  }
  */
?>
</article>
<?php
}

This is an issue for the Drupal developers to look at.

#20: Problem adding Drupal user accounts

chris — Mon, 15 Jun 2015 13:58:06 GMT

Jenny has reported:

We are having trouble adding users (see screenshot - it should just be a save button but has an anti spam message) - this was working last week.

#41: CRINmail multipart email contain CSS in the plain text version

chris — Thu, 10 Sep 2015 10:07:17 GMT

CRINmail, which are sent by Drupal, have a HTML and plain text version, the plain text version contains CSS at the top, which takes a while to scroll through and makes it less likely that people with mail user agents set to display the plain text version first will read the newsletter, for example:

Return-Path: <info@crin.org>
Received: from mail.webarch.net ([unix socket])
        by mail.webarch.net (Cyrus v2.3.7-Invoca-RPM-2.3.7-16.el5_11) with LMTPA;
        Thu, 10 Sep 2015 00:19:23 +0100
X-Sieve: CMU Sieve 2.3
Received: from elderberry.rat.burntout.org (mx.webarch.net [81.95.52.71])
        (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits))
        (Client did not present a certificate)
        by mail.webarch.net (Postfix) with ESMTPS id A7BFB100072
        for <chris@webarchitects.co.uk>; Thu, 10 Sep 2015 00:19:23 +0100 (BST)
Received: from crin2.crin.org ([93.95.228.180])
        by elderberry.rat.burntout.org with esmtps (TLS1.2:RSA_AES_128_CBC_SHA1:128)
        (Exim 4.80)
        (envelope-from <info@crin.org>)
        id 1ZZoe5-00056h-Dl
        for chris@webarchitects.co.uk; Thu, 10 Sep 2015 00:19:23 +0100
Received: from www-data by crin2.crin.org with local (Exim 4.84)
        (envelope-from <info@crin.org>)
        id 1ZZoe4-0006TK-78
        for chris@webarchitects.co.uk; Wed, 09 Sep 2015 23:19:16 +0000
To: chris@webarchitects.co.uk
Subject: CRINmail 1445
X-PHP-Originating-Script: 1005:mimemail.module
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="8a276e6c6a6f9459ea8c98217bf4e986"
Content-Transfer-Encoding: 8Bit
X-Mailer: Drupal
Sender: CRIN <info@crin.org>
From: CRIN <info@crin.org>
List-Unsubscribe: <http://www.crin.org/en/newsletter/confirm/remove/XXX>
Precedence: bulk
Message-Id: <E1ZZoe4-0006TK-78@crin2.crin.org>
Date: Wed, 09 Sep 2015 23:19:16 +0000
[-- Attachment #1 --]
[-- Type: multipart/alternative, Encoding: 8bit, Size: 73K --]
Content-Type: multipart/alternative;
    boundary="f5f8c71876cf7825d95e748d55f23268"
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
-------- CRINMAIL 1445 -------------------------------------------------------
#nl_wrapper {background-color:#EFEFEF;}#nl_wrapper table,
#nl_wrapper td  {
   font-family:Verdana, Arial, Helvetica;
   line-height:150%;
   font-size:10px;
   margin: 0;
}
#nl_wrapper td img{
   margin: 0;
}
#nl_wrapper .headerTop {
background-color:#FFCC66; border-top:0px solid #000000; border-bottom:1px
solid #FFFFFF; text-align:center;
}
#nl_wrapper .adminText {
font-size:10px; color:#996600; line-height:200%; font-family:Verdana, Arial,
Helvetica; text-decoration:none;
}
#nl_wrapper .headerBar {
background-color:#FFFFFF; border-top:0px solid #333333; border-bottom:10px
solid #FFFFFF;
}
#nl_wrapper .title {
font-size:20px; font-weight:bold; color:#CC6600; font-family:Verdana, Arial,
Helvetica; line-height:150%;
}
#nl_wrapper .subTitle {
font-size:11px; font-weight:normal; color:#666666; font-style:italic;
font-family:arial;
}
#nl_wrapper td {
font-size:11px; color:#000000; line-height:150%; font-family:Verdana, Arial,
Helvetica;
}
#nl_wrapper .sideColumn {
background-color:#FFFFFF; border-left:1px dashed #CCCCCC; text-align:left;
}
#nl_wrapper .sideColumnText {
font-size:11px; font-weight:normal; color:#AAAAAA; font-family:Verdana,
Arial, Helvetica; line-height:150%;
}
#nl_wrapper .sideColumnTitle {
font-size:15px; font-weight:bold; color:#333333; font-family:Verdana, Arial,
Helvetica; line-height:150%;
}
#nl_wrapper .footerRow {
background-color:#FFFFCC; border-top:10px solid #FFFFFF;
}
#nl_wrapper .footerText {
font-size:10px; color:#996600; line-height:100%; font-family:Verdana, Arial,
Helvetica;
}
#nl_wrapper h1,
#nl_wrapper h2  {
   color:#888888;font-size:20px;font-weight:bold;font-family:Arial,
Helvetica;margin:0 0 10px 0; line-height:150%;
}
#nl_wrapper h2  { font-size:15px; margin:23px 0 10px 0; padding-top:8px;
border-top:1px solid #EFEFEF;}
#nl_wrapper p   {margin:10px 0;
font-size:11px;color:#333333;font-family:Verdana, Arial, Helvetica;
line-height:150%;}
#nl_wrapper a,
#nl_wrapper table td p a,
#nl_wrapper #crintable a,
#nl_wrapper #crintable a:visited { color:#1155cc; /* color:#003a6a;*/ }
#nl_wrapper a:hover,
#nl_wrapper table td p a:hover,
#nl_wrapper #crintable a:hover{color:#000;}
#nl_wrapper ul,
#nl_wrapper li,
#nl_wrapper
#crintable ul   {margin:0; padding:0; list-style:none; list-style-image:none;
list-style-type:none}
#nl_wrapper ul li,
#nl_wrapper #crintable ul li,
#nl_wrapper #CMrightcol ul li
{font-size:11px;color:#333333;background-image:url(http://crin.org/i/crinmail/bg_bullet.gif);
background-repeat:no-repeat; background-position:2px
center;font-family:Verdana, Arial, Helvetica;  margin:0; list-style:none;
list-style-type:none; padding:0 0 0 20px; margin:13px 0;}
#nl_wrapper div.FB,                                                                                     #nl_wrapper div.FB p    { margin:10px 0}
#nl_wrapper div.FB h2   {color:#5e768b; font-size:16px; margin:0;}
#nl_wrapper #crintable table td {
vertical-align: top;
}
#nl_wrapper h3.h3image{
   background-repeat: no-repeat;
   background-position: center right;
}
   http://crin.org
09 September 2015
subscribe http://crin.org/node/41907 | subscribe http://crin.org/node/293 |
submit information http://crin.org/en/home/about/contact-us
======== *CRINMAIL 1445* =====================================================
-------- IN THIS ISSUE: ------------------------------------------------------
*The year in children's rights:*
CRIN annual report 2015 #AR
*Latest news and reports* #news
- Discrimination on school grounds #dis
- Criminalising children for sexting #crim
- Access to bathrooms denied #ac
- States leave migrant children unprotected #sta
- Issues neglected by UN committees #iss
- UN calls for submissions #un
- Contribute to evaluation of CRIN’s work #cont
*Access to justice for children in Ireland* #a2j
*Upcoming events* #events
*Employment* #jbs
/Problems viewing this CRINmail? Click here
https://www.crin.org/en/home/what-we-do/crinmail/crinmail-1445. /
*/CRIN announcement/*
/We are aware that some of our readers may be receiving blank CRINmails. This
is due to our website experiencing some technical issues, which we are
working to resolve. Should you receive a blank or incomplete CRINmail, you
can also access the latest edition via our website here
https://www.crin.org/en/home/what-we-do/crinmail. /

Also note that the list unsub header contains a HTTP URL when this would be better as a HTTPS one for privacy reasons:

List-Unsubscribe: <http://www.crin.org/en/newsletter/confirm/remove/XXX>

#49: Frozen emails on crin4

chris — Thu, 05 Nov 2015 19:22:51 GMT

Munin is sending alert emails every 5 mins because of the number of frozen emails on Crin4.

#53: Blank email: CRINmail 1459

chris — Thu, 17 Dec 2015 13:04:49 GMT

I was sent this last night, with no email body:

To: chris@webarchitects.co.uk
Subject: CRINmail 1459
X-PHP-Originating-Script: 1005:mimemail.module
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8Bit
X-Mailer: Drupal
Sender: CRIN <info@crin.org>
From: CRIN <info@crin.org>
List-Unsubscribe: <https://www.crin.org/en/newsletter/confirm/remove/XXXX>
Precedence: bulk
Message-Id: <E1a9N9t-0001cP-Qt@crin2.crin.org>
Date: Thu, 17 Dec 2015 01:15:05 +0000

This looks like another case of the issue previously raised here: ticket:14

#54: 502 Errors

chris — Tue, 22 Dec 2015 11:06:04 GMT

Last night the wiki:NginxLogWatch script found a lot of 502 errors:

Dec 22     288 502, 0 503 and 0 504 errors from crin2.crin.org
Dec 22    4283 502, 0 503 and 0 504 errors from crin2.crin.org

I'm not sure why yet.

#56: Drupal Caching

chris — Sun, 03 Jan 2016 11:46:35 GMT

This is a ticket to discuss and then implement some additional caching for the https://www.crin.org/ Drupal site.

Possible options:

Generating static content from Drupal and serving these to non-logged in users, for example using Boost or something like it.
Caching PHP generated pages using Nginx.
Using a caching server such as Varnish.

My preference would for 1. and 2., I can start doing some research on how best to do 2. I'd rather not do 3. if possible due to the complication it would add: php5-fpm -> varnish -> nginx.

#57: RSS requests from MediaWiki site to Drupal site

chris — Thu, 07 Jan 2016 13:05:37 GMT

The MediaWiki site requests RSS URL's which are 404's at such a high rate that it is blocked, see ticket:54#comment:6

#60: Advania S3 Account

chris — Thu, 04 Feb 2016 11:36:22 GMT

This ticket is to track the time spent answering Andrew's questions regarding the Advania S3 account which is used for backups by Crin3.

#65: Custom 502 and 503 pages for crin.org

chris — Sun, 28 Feb 2016 14:17:03 GMT

It would be more informative if there were custom 502 and 503 error pages for https://www.crin.org/ rather than the default Nginx ones.

I will write up some details below on the 502 and 503's we seeing getting, see also ticket:54.

I'm happy to create a static 502 and static 503 error page and will look at adding it to the git repo.

#66: Optimise backups, omit some MySQL tables and delete old backups from s3ql

chris — Tue, 08 Mar 2016 10:59:50 GMT

Currently the DumpMysqlTables script doesn't omit or truncate any tables so the backups contain unneeded data, such as cache tables, Drupal tables which can be truncated, see:

https://drupal.stackexchange.com/questions/22041/which-drupal-tables-are-safe-to-clear

The same will be the case for Piwik and ownCloud and MediaWiki.

In addition php session files should be deleted.

Also the S3QLBackup script could be improved so it deleted old backups.

#72: exim paniclog on crin3.crin.org

chris — Tue, 19 Apr 2016 09:47:12 GMT

I'm getting this email every day:

exim paniclog /var/log/exim4/paniclog on crin3.crin.org has non-zero size, mail system might be broken. The last 10 lines are quoted below.

2016-03-31 10:00:15 Warning: No server certificate defined; TLS connections will fail. Suggested action: either install a certificate or change tls_advertise_hosts option

#73: CRIN2 ran out of memory

chris — Sun, 01 May 2016 19:16:38 GMT

The site has been down since Sun, 01 May 2016 18:05:44, it is now 8:15. I'm rebooting the server, it ran out of memory.

#77: Drupal contrib - Highly Critical - Remote code execution PSA-2016-001

chris — Wed, 13 Jul 2016 09:47:24 GMT

At 16:00 UTC (5pm BST) fixes for multiple "highly critical remote code execution vulnerabilities" in unspecified Drupal 7 modules will be released and it is expected that "exploits are expected to be developed within hours/days", see:

https://www.drupal.org/node/2764899

Peter: do you have this in hand? I notice that the dev site is still not running properly:

https://dev.crin.org/

CSS is linked like this:

@import url("https:///sites/all/modules/contrib/domain/domain_nav/domain_nav.css?o4jry7");
@import url("https:///modules/field/theme/field.css?o4jry7");

JS is linked like this:

<script src="https:///sites/default/files/js/js_zzcIWOou_jnX0ZWAIA4sb6Xy_p5a8FZNA0GySvuWjPU.js"></script>

And images are linked like this:

<img typeof="foaf:Image" src="https:///sites/default/files/styles/390x387/public/1.1.jpg?itok=p_PWQReL" width="390" height="387" alt="" />

Is there anything I can do to help?

#81: crin3 backup issue

chris — Fri, 19 Aug 2016 09:36:01 GMT

Munin email alert I'm getting every 5 mins:

Date: Fri, 19 Aug 2016 09:30:16 +0000
From: munin application user <munin@crin1.crin.org>
Subject: crin3.crin.org Munin Alert
crin.org :: crin3.crin.org :: Disk usage in percent
        WARNINGs: / is 97.35 (outside range [:92]).
        OKs: /run is 10.40, /dev/shm is 0.00, /boot is 87.07, /run/lock is 0.00, /sys/fs/cgroup is 0.00, /run/user/1000 is 0.00.

#82: Attempted DDOS?

chris — Sat, 24 Sep 2016 09:20:39 GMT

Yesterday evening there 1k 502 errors caused by one user agent from multiple IP addresses. This isn't an uncommon event, I usually don't write them up, but I thought it might be useful for Mathew if I do with this one.

#89: Broken Links

matthew — Mon, 24 Oct 2016 12:11:44 GMT

https://www.crin.org/en/library/publications/crin-newsletter-15-mainstreaming-child-rights https://www.crin.org/en/library/publications/crin-newsletter-16-children-and-young-peoples-participation

were reported as broken

#92: dev.crin.org needs /en appending the the URL for CSS to load

chris — Tue, 08 Nov 2016 12:35:59 GMT

This came up on ticket:77#comment:10 and it would be good to find the cause and also to solve it.

#93: Piwik 2.17.1

chris — Fri, 02 Dec 2016 14:11:51 GMT

Not urgent, no security updates, see https://piwik.org/changelog/piwik-2-17-1/

#94: Ongoing high load caused by a bot

chris — Thu, 15 Dec 2016 19:23:12 GMT

Ticket to working out what to do with this issue...

#97: Debian Stretch

chris — Mon, 30 Jan 2017 10:59:00 GMT

Debian Stretch will probably be released in May 2017 or so, we need a plan to migrate to it, specifically:

PHP 7.0
MariaDB 10

This has been raised with Russell on Bitbucket.

#99: Let's Encrypt Cert for CRIN.org

chris — Mon, 30 Jan 2017 14:32:43 GMT

The Gandi cert expires in a couple of weeks, in 17th Feb 2017, see:

https://www.ssllabs.com/ssltest/analyze.html?d=www.crin.org&latest

It would make sense to switch to use https://letsencrypt.org/ and the client implementation I suggest we use is this one https://github.com/Neilpang/acme.sh

#100: crin.org site down

chris — Sat, 18 Feb 2017 11:38:48 GMT

Russell, what's up? I'm really sorry I have to go out in 5 mins so I'm not around to help, what has gone wrong?

#102: Security certificate issue

russell — Fri, 10 Mar 2017 12:30:04 GMT

Hi Chris,

I'm having trouble browsing to the stage site today on crin4 - I'm getting:

SEC_ERROR_EXPIRED_CERTIFICATE

.. trying to view stage.crin.org

Are you able to help there?

Many thanks, Russell

#104: Problem with CRIN1 and CRIN2 backups

chris — Tue, 21 Mar 2017 09:32:43 GMT

Errors from the s3ql backup scripts:

Backend reports that file system is still mounted elsewhere. Either the file system has not been unmounted cleanly or the data has not yet propagated through the backend. In the later case, waiting for a while should fix the problem, in the former case you should try to run fsck on the computer where the file system has been mounted most recently. You may also continue and use whatever metadata is available in the backend. However, in that case YOU MAY LOOSE ALL DATA THAT HAS BEEN UPLOADED OR MODIFIED SINCE THE LAST SUCCESSFULL METADATA UPLOAD. Moreover, files and directories that you have deleted since then MAY REAPPEAR WITH SOME OF THEIR CONTENT LOST.

#106: MySQL Error

chris — Mon, 27 Mar 2017 13:37:30 GMT

On Saturday morning the MySQL database on Crin1 was generating errors which prevented the site for working, this was fixed by running this command on the server:

mysqladmin flush-hosts

#111: Backups to Webarchitects server

chris — Thu, 13 Apr 2017 10:11:54 GMT

This is a ticket to setup backups to store1.webarch.net on.

#113: MySQL Performance

chris — Tue, 25 Apr 2017 11:22:11 GMT

MySQL is set to use a lot of resources, see the RAM allocation here:

https://munin.crin.org/munin/crin.org/crin1.crin.org/multips_memory.html

And also a lot of queries are cached via memcache, which also has a lot of RAM allocated, see:

https://munin.crin.org/munin/crin.org/crin2.crin.org/multips_memory.html

What can we do to make it quicker?

See also the work done when the server was setup on ticket:9

#114: Update TLS certs for MySQL

chris — Fri, 05 May 2017 12:20:04 GMT

The TLS certs for MySQL connections between the servers are due to expire on 7th May 2017 so need to be updated.

Peter, Russell, I have added you just so you know this is happening, it'll involve restarting MySQL when the new keys and certs are in place, so there might be a few moments of site downtime, but I'll do this on the dev/staging sever first.

#115: 500 Errors for crin.org

chris — Tue, 16 May 2017 17:00:33 GMT

There have been a lot of 500 errors today and they appear to be related to lines like this in /var/log/php/error.log:

[16-May-2017 17:33:08 Europe/London] PHP Fatal error:  Call to undefined function entity_translation_language_load() in /var/www/prod/docroot/includes/menu.inc on line 593

The section on the file in question:

 538
 539 /**
 540  * Loads objects into the map as defined in the $item['load_functions'].
 541  *
 542  * @param $item
 543  *   A menu router or menu link item
 544  * @param $map
 545  *   An array of path arguments; for example, array('node', '5').
 546  *
 547  * @return
 548  *   Returns TRUE for success, FALSE if an object cannot be loaded.
 549  *   Names of object loading functions are placed in $item['load_functions'].
 550  *   Loaded objects are placed in $map[]; keys are the same as keys in the
 551  *   $item['load_functions'] array.
 553  */
 554 function _menu_load_objects(&$item, &$map) {
 555   if ($load_functions = $item['load_functions']) {
 556     // If someone calls this function twice, then unserialize will fail.
 557     if (!is_array($load_functions)) {
 558       $load_functions = unserialize($load_functions);
 559     }
 560     $path_map = $map;
 561     foreach ($load_functions as $index => $function) {
 562       if ($function) {
 563         $value = isset($path_map[$index]) ? $path_map[$index] : '';
 564         if (is_array($function)) {
 565           // Set up arguments for the load function. These were pulled from
 566           // 'load arguments' in the hook_menu() entry, but they need
 567           // some processing. In this case the $function is the key to the
 568           // load_function array, and the value is the list of arguments.
 569           list($function, $args) = each($function);
 570           $load_functions[$index] = $function;
 571
 572           // Some arguments are placeholders for dynamic items to process.
 573           foreach ($args as $i => $arg) {
 574             if ($arg === '%index') {
 575               // Pass on argument index to the load function, so multiple
 576               // occurrences of the same placeholder can be identified.
 577               $args[$i] = $index;
 578             }
 579             if ($arg === '%map') {
 580               // Pass on menu map by reference. The accepting function must
 581               // also declare this as a reference if it wants to modify
 582               // the map.
 583               $args[$i] = &$map;
 584             }
 585             if (is_int($arg)) {
 586               $args[$i] = isset($path_map[$arg]) ? $path_map[$arg] : '';
 587             }
 588           }
 589           array_unshift($args, $value);
 590           $return = call_user_func_array($function, $args);
 591         }
 592         else {
 593           $return = $function($value);
 594         }
 595         // If callback returned an error or there is no callback, trigger 404.
 596         if ($return === FALSE) {
 597           $item['access'] = FALSE;
 598           $map = FALSE;
 599           return FALSE;
 600         }
 601         $map[$index] = $return;
 602       }
 603     }
 604     $item['load_functions'] = $load_functions;
 605   }
 606   return TRUE;
 607 }
 608

Peter, is the the cause of the 500 errors or is it a symptom, or is it unrelated? Do you have an idea what is going on here?

#116: Limit the number of MySQL connections from Crin4 to Crin1

chris — Wed, 17 May 2017 11:44:18 GMT

Both the dev, Crin4 and live Crin2, servers use the MySQL database on Crin1 and we want to limit the resources available to Crin4 so it reduced the impact on the live site.

Some documentation on this:

#117: Install Newrelic on Crin2

chris — Wed, 17 May 2017 11:51:53 GMT

Peter has requested that Newrelic be installed on Crin2.

Note that last time it was installed we had to remove it due to the resources it was using, see this thread ticket:73#comment:24, so I'd advise against leaving it on the server all the time.