CRIN Trac: Ticket Query

#105: High load and slow responses from Crin2

chris — Mon, 27 Mar 2017 12:15:49 GMT

Trac threw this error:

Warning:

Error with navigation contributor "TimingEstimationAndBillingPage?"
Error with navigation contributor "AdminModule?"
Error with navigation contributor "SearchModule?"
Failed to sync with repository "(default)": database is locked; repository information may be out of date. Look in the Trac log for more information including mitigation strategies.

And in /var/log/php5-fpm.log we have:

[27-Mar-2017 10:20:42] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 10:30:10] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:54:51] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:56:35] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:57:08] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:57:58] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 12:01:48] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 12:02:26] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it

From an initial look at the Nginx logs the problem seesm to be caused but too much agressive crawling of the site by bots.

#75: Load spike and small downtime

chris — Sun, 03 Jul 2016 18:00:34 GMT

Since about 8:10am this morning there has been a noticeable increase in bandwidth usage and load on Crin2 and at 18:33, for 20 seconds, the Webarchitects monitoring server was unable to connect.

#55: Migrating Crin2 server to new hardware

chris — Fri, 01 Jan 2016 18:21:49 GMT

In order for the Crin2 server to have it's RAM doubled it needs to be migrated to a different physical server, I would expect this would take between 15 mins and an hour at most, is there any preference when this is done? I'd suggest doing it this weekend before everybody is back at work on Monday would make sense? I have contacted 1984.is about it and they can script it to happen at what ever time is best.

#51: Number of php5-fpm processes on Crin2

chris — Wed, 09 Dec 2015 12:34:55 GMT

Peter -- a couple of days ago the number of php5-fpm processes on Crin2 jumped from 9 to 39 and this uses upto 5G or RAM and there is only 4G, see the following Munin graphs:

Note: See TracTickets for help on using tickets.

Any idea what changed?

#50: Migrate to INNODB

chris — Wed, 25 Nov 2015 09:50:16 GMT

Email from Peter:

What is involved in changing any of the underlying server resources for Crin.org.

I think we generally have too many CPU's and not enough memory.

To improve the site performance, we need to move the database to INNODB. But we are also already using swap on the DB server, and INNODB wants the entire DB in ram. I think we should switch to at least 8GB ram, but can probably scale down to 4 cpus - and still get a performance increase.

#40: Check Bitbucket documentation

chris — Wed, 09 Sep 2015 13:21:35 GMT

Chris needs to read and check the documentation here: https://bitbucket.org/crin/crin/wiki/Home and update the Trac documentation as need be.

#38: [URGENT] crin.org down

mori — Tue, 08 Sep 2015 15:54:35 GMT

Hi Chris,

Jenny just reached out to us by email reporting crin.org is down. The error message I see on crin.org is:

PDOException: SQLSTATE[28000] [1045] Access denied for user 'newprod'@'crin2' (using password: YES) in lock_may_be_available() (line 167 of /var/www/prod/docroot/includes/lock.inc).

It sounds like it may be something that's related to the issue you mentioned in #36 (https://trac.crin.org.archived.website/trac/ticket/36#comment:7 )

Can you please have a look at this as a matter of urgency?

Thanks,

Mori

#32: PHP error logs not found

mori — Tue, 18 Aug 2015 14:41:56 GMT

I'm having issues on newprod, but I can't find PHP error logs. If they are logged, can you please let me know where they are? If they are not logged, can you start logging them on both CRIN2 and CRIN4? Thanks.

#31: Preparation for deployment to new prod

mori — Mon, 10 Aug 2015 15:32:41 GMT

Hi Chris,

In order to run the CRIN production site under the new codebase, we need to go through a number of steps to set up and test the site. We need your help to accomplish this.

Overview: (please execute each step once we request you to do so through the comment)

Chris to create a new entry for the new prod site on CRIN2
Chris to create a new database for the new prod site on CRIN1
Code+ and CRIN to test the new prod site and sign off
Chris to update the nginx config so the new prod serves as the working prod

Rationale of the go-live process Because of the reasons listed below, deploying the new code to the existing prod results in an extended period of downtime:

repository too large: cloning takes 10+ minutes
changes to directory structure requires the registry to be rebuilt
the new deployment process needs testing

#27: Crin2 certificate access

peter — Mon, 06 Jul 2015 16:29:00 GMT

When attempting to use Drush, when not root, we are not able to access the certificate.

SSL error: Unable to get certificate from '/etc/ssl/cacert/crin1_cert.pem'

How should we manage access to certificates? Should we create a developers group?