CRIN Trac: Ticket Query

#38: [URGENT] crin.org down

mori — Tue, 08 Sep 2015 15:54:35 GMT

Hi Chris,

Jenny just reached out to us by email reporting crin.org is down. The error message I see on crin.org is:

PDOException: SQLSTATE[28000] [1045] Access denied for user 'newprod'@'crin2' (using password: YES) in lock_may_be_available() (line 167 of /var/www/prod/docroot/includes/lock.inc).

It sounds like it may be something that's related to the issue you mentioned in #36 (https://trac.crin.org.archived.website/trac/ticket/36#comment:7 )

Can you please have a look at this as a matter of urgency?

Thanks,

Mori

#27: Crin2 certificate access

peter — Mon, 06 Jul 2015 16:29:00 GMT

When attempting to use Drush, when not root, we are not able to access the certificate.

SSL error: Unable to get certificate from '/etc/ssl/cacert/crin1_cert.pem'

How should we manage access to certificates? Should we create a developers group?

#31: Preparation for deployment to new prod

mori — Mon, 10 Aug 2015 15:32:41 GMT

Hi Chris,

In order to run the CRIN production site under the new codebase, we need to go through a number of steps to set up and test the site. We need your help to accomplish this.

Overview: (please execute each step once we request you to do so through the comment)

Chris to create a new entry for the new prod site on CRIN2
Chris to create a new database for the new prod site on CRIN1
Code+ and CRIN to test the new prod site and sign off
Chris to update the nginx config so the new prod serves as the working prod

Rationale of the go-live process Because of the reasons listed below, deploying the new code to the existing prod results in an extended period of downtime:

repository too large: cloning takes 10+ minutes
changes to directory structure requires the registry to be rebuilt
the new deployment process needs testing

#32: PHP error logs not found

mori — Tue, 18 Aug 2015 14:41:56 GMT

I'm having issues on newprod, but I can't find PHP error logs. If they are logged, can you please let me know where they are? If they are not logged, can you start logging them on both CRIN2 and CRIN4? Thanks.

#40: Check Bitbucket documentation

chris — Wed, 09 Sep 2015 13:21:35 GMT

Chris needs to read and check the documentation here: https://bitbucket.org/crin/crin/wiki/Home and update the Trac documentation as need be.

#50: Migrate to INNODB

chris — Wed, 25 Nov 2015 09:50:16 GMT

Email from Peter:

What is involved in changing any of the underlying server resources for Crin.org.

I think we generally have too many CPU's and not enough memory.

To improve the site performance, we need to move the database to INNODB. But we are also already using swap on the DB server, and INNODB wants the entire DB in ram. I think we should switch to at least 8GB ram, but can probably scale down to 4 cpus - and still get a performance increase.

#51: Number of php5-fpm processes on Crin2

chris — Wed, 09 Dec 2015 12:34:55 GMT

Peter -- a couple of days ago the number of php5-fpm processes on Crin2 jumped from 9 to 39 and this uses upto 5G or RAM and there is only 4G, see the following Munin graphs:

Note: See TracTickets for help on using tickets.

Any idea what changed?

#55: Migrating Crin2 server to new hardware

chris — Fri, 01 Jan 2016 18:21:49 GMT

In order for the Crin2 server to have it's RAM doubled it needs to be migrated to a different physical server, I would expect this would take between 15 mins and an hour at most, is there any preference when this is done? I'd suggest doing it this weekend before everybody is back at work on Monday would make sense? I have contacted 1984.is about it and they can script it to happen at what ever time is best.

#75: Load spike and small downtime

chris — Sun, 03 Jul 2016 18:00:34 GMT

Since about 8:10am this morning there has been a noticeable increase in bandwidth usage and load on Crin2 and at 18:33, for 20 seconds, the Webarchitects monitoring server was unable to connect.

#105: High load and slow responses from Crin2

chris — Mon, 27 Mar 2017 12:15:49 GMT

Trac threw this error:

Warning:

Error with navigation contributor "TimingEstimationAndBillingPage?"
Error with navigation contributor "AdminModule?"
Error with navigation contributor "SearchModule?"
Failed to sync with repository "(default)": database is locked; repository information may be out of date. Look in the Trac log for more information including mitigation strategies.

And in /var/log/php5-fpm.log we have:

[27-Mar-2017 10:20:42] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 10:30:10] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:54:51] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:56:35] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:57:08] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 11:57:58] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 12:01:48] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
[27-Mar-2017 12:02:26] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it

From an initial look at the Nginx logs the problem seesm to be caused but too much agressive crawling of the site by bots.

#100: crin.org site down

chris — Sat, 18 Feb 2017 11:38:48 GMT

Russell, what's up? I'm really sorry I have to go out in 5 mins so I'm not around to help, what has gone wrong?

#73: CRIN2 ran out of memory

chris — Sun, 01 May 2016 19:16:38 GMT

The site has been down since Sun, 01 May 2016 18:05:44, it is now 8:15. I'm rebooting the server, it ran out of memory.

#94: Ongoing high load caused by a bot

chris — Thu, 15 Dec 2016 19:23:12 GMT

Ticket to working out what to do with this issue...

#54: 502 Errors

chris — Tue, 22 Dec 2015 11:06:04 GMT

Last night the wiki:NginxLogWatch script found a lot of 502 errors:

Dec 22     288 502, 0 503 and 0 504 errors from crin2.crin.org
Dec 22    4283 502, 0 503 and 0 504 errors from crin2.crin.org

I'm not sure why yet.

#56: Drupal Caching

chris — Sun, 03 Jan 2016 11:46:35 GMT

This is a ticket to discuss and then implement some additional caching for the https://www.crin.org/ Drupal site.

Possible options:

Generating static content from Drupal and serving these to non-logged in users, for example using Boost or something like it.
Caching PHP generated pages using Nginx.
Using a caching server such as Varnish.

My preference would for 1. and 2., I can start doing some research on how best to do 2. I'd rather not do 3. if possible due to the complication it would add: php5-fpm -> varnish -> nginx.

#97: Debian Stretch

chris — Mon, 30 Jan 2017 10:59:00 GMT

Debian Stretch will probably be released in May 2017 or so, we need a plan to migrate to it, specifically:

PHP 7.0
MariaDB 10

This has been raised with Russell on Bitbucket.

#99: Let's Encrypt Cert for CRIN.org

chris — Mon, 30 Jan 2017 14:32:43 GMT

The Gandi cert expires in a couple of weeks, in 17th Feb 2017, see:

https://www.ssllabs.com/ssltest/analyze.html?d=www.crin.org&latest

It would make sense to switch to use https://letsencrypt.org/ and the client implementation I suggest we use is this one https://github.com/Neilpang/acme.sh

#115: 500 Errors for crin.org

chris — Tue, 16 May 2017 17:00:33 GMT

There have been a lot of 500 errors today and they appear to be related to lines like this in /var/log/php/error.log:

[16-May-2017 17:33:08 Europe/London] PHP Fatal error:  Call to undefined function entity_translation_language_load() in /var/www/prod/docroot/includes/menu.inc on line 593

The section on the file in question:

 538
 539 /**
 540  * Loads objects into the map as defined in the $item['load_functions'].
 541  *
 542  * @param $item
 543  *   A menu router or menu link item
 544  * @param $map
 545  *   An array of path arguments; for example, array('node', '5').
 546  *
 547  * @return
 548  *   Returns TRUE for success, FALSE if an object cannot be loaded.
 549  *   Names of object loading functions are placed in $item['load_functions'].
 550  *   Loaded objects are placed in $map[]; keys are the same as keys in the
 551  *   $item['load_functions'] array.
 553  */
 554 function _menu_load_objects(&$item, &$map) {
 555   if ($load_functions = $item['load_functions']) {
 556     // If someone calls this function twice, then unserialize will fail.
 557     if (!is_array($load_functions)) {
 558       $load_functions = unserialize($load_functions);
 559     }
 560     $path_map = $map;
 561     foreach ($load_functions as $index => $function) {
 562       if ($function) {
 563         $value = isset($path_map[$index]) ? $path_map[$index] : '';
 564         if (is_array($function)) {
 565           // Set up arguments for the load function. These were pulled from
 566           // 'load arguments' in the hook_menu() entry, but they need
 567           // some processing. In this case the $function is the key to the
 568           // load_function array, and the value is the list of arguments.
 569           list($function, $args) = each($function);
 570           $load_functions[$index] = $function;
 571
 572           // Some arguments are placeholders for dynamic items to process.
 573           foreach ($args as $i => $arg) {
 574             if ($arg === '%index') {
 575               // Pass on argument index to the load function, so multiple
 576               // occurrences of the same placeholder can be identified.
 577               $args[$i] = $index;
 578             }
 579             if ($arg === '%map') {
 580               // Pass on menu map by reference. The accepting function must
 581               // also declare this as a reference if it wants to modify
 582               // the map.
 583               $args[$i] = &$map;
 584             }
 585             if (is_int($arg)) {
 586               $args[$i] = isset($path_map[$arg]) ? $path_map[$arg] : '';
 587             }
 588           }
 589           array_unshift($args, $value);
 590           $return = call_user_func_array($function, $args);
 591         }
 592         else {
 593           $return = $function($value);
 594         }
 595         // If callback returned an error or there is no callback, trigger 404.
 596         if ($return === FALSE) {
 597           $item['access'] = FALSE;
 598           $map = FALSE;
 599           return FALSE;
 600         }
 601         $map[$index] = $return;
 602       }
 603     }
 604     $item['load_functions'] = $load_functions;
 605   }
 606   return TRUE;
 607 }
 608

Peter, is the the cause of the 500 errors or is it a symptom, or is it unrelated? Do you have an idea what is going on here?

#117: Install Newrelic on Crin2

chris — Wed, 17 May 2017 11:51:53 GMT

Peter has requested that Newrelic be installed on Crin2.

Note that last time it was installed we had to remove it due to the resources it was using, see this thread ticket:73#comment:24, so I'd advise against leaving it on the server all the time.

#129: Problem with image uploads

chris — Mon, 11 Dec 2017 18:09:39 GMT

unable to upload new documents and images to the website.

#108: Go live

russell — Tue, 04 Apr 2017 16:01:05 GMT

Hi Chris,

I've got a round of work ready to go onto Crin live. All tested and signed off on stage. The code element is very light. I don't expect any problems, but just letting you know. I'm planning to get into this as the working day is finishing today.

Best, Russell

#120: crin2 mail queue

chris — Thu, 15 Jun 2017 18:36:04 GMT

There is a mail queue on Crin2, we haven't had one of these for ages, Munin reports:

crin.org :: crin2.crin.org :: Exim Mailqueue

WARNINGs: frozen mails is 171.00 (outside range 100). OKs: queued mails is 230.00.

Graph here:

https://munin.crin.org/munin/crin.org/crin2.crin.org/exim_mailqueue.html

Anyone know if this is something to be concerned about?