CRIN Trac: Ticket Query

#120: crin2 mail queue

chris — Thu, 15 Jun 2017 18:36:04 GMT

There is a mail queue on Crin2, we haven't had one of these for ages, Munin reports:

crin.org :: crin2.crin.org :: Exim Mailqueue

WARNINGs: frozen mails is 171.00 (outside range 100). OKs: queued mails is 230.00.

Graph here:

https://munin.crin.org/munin/crin.org/crin2.crin.org/exim_mailqueue.html

Anyone know if this is something to be concerned about?

#103: Upgrade CRIN4 to 2GB of RAM

chris — Thu, 16 Mar 2017 09:37:21 GMT

Ticket to track the upgrading of Crin4 from 1GB to 2GB of RAM.

#114: Update TLS certs for MySQL

chris — Fri, 05 May 2017 12:20:04 GMT

The TLS certs for MySQL connections between the servers are due to expire on 7th May 2017 so need to be updated.

Peter, Russell, I have added you just so you know this is happening, it'll involve restarting MySQL when the new keys and certs are in place, so there might be a few moments of site downtime, but I'll do this on the dev/staging sever first.

#19: PHP Parser Error - node--country.tpl.php

chris — Mon, 15 Jun 2015 10:21:19 GMT

The php-fpm error log at /var/log/fpm-php.www.log contains lots of entries like this:

[15-Jun-2015 11:17:32 Europe/London] PHP Parse error:  syntax error, unexpected end of file in /var/www/drupal/sites/default/themes/crin/templates/node--country.tpl.php on line 522

Line 522 is the last line of the file, it simply contains a closing } bracket, this is the last part of the file:

    <?php
  }
  ?>
  <?php // print render($content['links']); ?>
  <?php // print render($content['comments']); ?>
<?php
  // 'Albania' = 441;
  //$node_load(441);
  /*
  global $user;
  if($user->uid == 1 ){
    if($node->nid == 441){
      //update_country_from_wiki($node);
    }
  }
  */
?>
</article>
<?php
}

This is an issue for the Drupal developers to look at.

#113: MySQL Performance

chris — Tue, 25 Apr 2017 11:22:11 GMT

MySQL is set to use a lot of resources, see the RAM allocation here:

https://munin.crin.org/munin/crin.org/crin1.crin.org/multips_memory.html

And also a lot of queries are cached via memcache, which also has a lot of RAM allocated, see:

https://munin.crin.org/munin/crin.org/crin2.crin.org/multips_memory.html

What can we do to make it quicker?

See also the work done when the server was setup on ticket:9

#116: Limit the number of MySQL connections from Crin4 to Crin1

chris — Wed, 17 May 2017 11:44:18 GMT

Both the dev, Crin4 and live Crin2, servers use the MySQL database on Crin1 and we want to limit the resources available to Crin4 so it reduced the impact on the live site.

Some documentation on this:

#117: Install Newrelic on Crin2

chris — Wed, 17 May 2017 11:51:53 GMT

Peter has requested that Newrelic be installed on Crin2.

Note that last time it was installed we had to remove it due to the resources it was using, see this thread ticket:73#comment:24, so I'd advise against leaving it on the server all the time.

#42: How CRINmail deals with email addresses that no longer work

chris — Thu, 10 Sep 2015 10:42:07 GMT

When the CRINmail newsletter is sent out via Drupal it has the following from envelope:

Return-Path: <info@crin.org>

This results in emails that are returned as undeliverable going to that address and the Drupal site doesn't know when a recipient address no longer works so is unable to stop sending emails to address that bounce email back.

One symptom of this problem are the forozen emails that build up after each CRINmail run, see the blue on Munin graphs here:

https://munin.crin.org/munin/crin.org/crin2.crin.org/exim_mailqueue.html

What should happen is that the from envelope (this is different from the From: header users see, that can remain as it is) should be an address that Drupal can check (probably via IMAP but we could set up one of the servers to accept SMPT email if needs be) and then do some thing sensible with the bounces.

See for example the discussion here:

https://stackoverflow.com/q/1011714

#49: Frozen emails on crin4

chris — Thu, 05 Nov 2015 19:22:51 GMT

Munin is sending alert emails every 5 mins because of the number of frozen emails on Crin4.

#77: Drupal contrib - Highly Critical - Remote code execution PSA-2016-001

chris — Wed, 13 Jul 2016 09:47:24 GMT

At 16:00 UTC (5pm BST) fixes for multiple "highly critical remote code execution vulnerabilities" in unspecified Drupal 7 modules will be released and it is expected that "exploits are expected to be developed within hours/days", see:

https://www.drupal.org/node/2764899

Peter: do you have this in hand? I notice that the dev site is still not running properly:

https://dev.crin.org/

CSS is linked like this:

@import url("https:///sites/all/modules/contrib/domain/domain_nav/domain_nav.css?o4jry7");
@import url("https:///modules/field/theme/field.css?o4jry7");

JS is linked like this:

<script src="https:///sites/default/files/js/js_zzcIWOou_jnX0ZWAIA4sb6Xy_p5a8FZNA0GySvuWjPU.js"></script>

And images are linked like this:

<img typeof="foaf:Image" src="https:///sites/default/files/styles/390x387/public/1.1.jpg?itok=p_PWQReL" width="390" height="387" alt="" />

Is there anything I can do to help?

#56: Drupal Caching

chris — Sun, 03 Jan 2016 11:46:35 GMT

This is a ticket to discuss and then implement some additional caching for the https://www.crin.org/ Drupal site.

Possible options:

Generating static content from Drupal and serving these to non-logged in users, for example using Boost or something like it.
Caching PHP generated pages using Nginx.
Using a caching server such as Varnish.

My preference would for 1. and 2., I can start doing some research on how best to do 2. I'd rather not do 3. if possible due to the complication it would add: php5-fpm -> varnish -> nginx.

#65: Custom 502 and 503 pages for crin.org

chris — Sun, 28 Feb 2016 14:17:03 GMT

It would be more informative if there were custom 502 and 503 error pages for https://www.crin.org/ rather than the default Nginx ones.

I will write up some details below on the 502 and 503's we seeing getting, see also ticket:54.

I'm happy to create a static 502 and static 503 error page and will look at adding it to the git repo.

#41: CRINmail multipart email contain CSS in the plain text version

chris — Thu, 10 Sep 2015 10:07:17 GMT

CRINmail, which are sent by Drupal, have a HTML and plain text version, the plain text version contains CSS at the top, which takes a while to scroll through and makes it less likely that people with mail user agents set to display the plain text version first will read the newsletter, for example:

Return-Path: <info@crin.org>
Received: from mail.webarch.net ([unix socket])
        by mail.webarch.net (Cyrus v2.3.7-Invoca-RPM-2.3.7-16.el5_11) with LMTPA;
        Thu, 10 Sep 2015 00:19:23 +0100
X-Sieve: CMU Sieve 2.3
Received: from elderberry.rat.burntout.org (mx.webarch.net [81.95.52.71])
        (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits))
        (Client did not present a certificate)
        by mail.webarch.net (Postfix) with ESMTPS id A7BFB100072
        for <chris@webarchitects.co.uk>; Thu, 10 Sep 2015 00:19:23 +0100 (BST)
Received: from crin2.crin.org ([93.95.228.180])
        by elderberry.rat.burntout.org with esmtps (TLS1.2:RSA_AES_128_CBC_SHA1:128)
        (Exim 4.80)
        (envelope-from <info@crin.org>)
        id 1ZZoe5-00056h-Dl
        for chris@webarchitects.co.uk; Thu, 10 Sep 2015 00:19:23 +0100
Received: from www-data by crin2.crin.org with local (Exim 4.84)
        (envelope-from <info@crin.org>)
        id 1ZZoe4-0006TK-78
        for chris@webarchitects.co.uk; Wed, 09 Sep 2015 23:19:16 +0000
To: chris@webarchitects.co.uk
Subject: CRINmail 1445
X-PHP-Originating-Script: 1005:mimemail.module
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="8a276e6c6a6f9459ea8c98217bf4e986"
Content-Transfer-Encoding: 8Bit
X-Mailer: Drupal
Sender: CRIN <info@crin.org>
From: CRIN <info@crin.org>
List-Unsubscribe: <http://www.crin.org/en/newsletter/confirm/remove/XXX>
Precedence: bulk
Message-Id: <E1ZZoe4-0006TK-78@crin2.crin.org>
Date: Wed, 09 Sep 2015 23:19:16 +0000
[-- Attachment #1 --]
[-- Type: multipart/alternative, Encoding: 8bit, Size: 73K --]
Content-Type: multipart/alternative;
    boundary="f5f8c71876cf7825d95e748d55f23268"
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
-------- CRINMAIL 1445 -------------------------------------------------------
#nl_wrapper {background-color:#EFEFEF;}#nl_wrapper table,
#nl_wrapper td  {
   font-family:Verdana, Arial, Helvetica;
   line-height:150%;
   font-size:10px;
   margin: 0;
}
#nl_wrapper td img{
   margin: 0;
}
#nl_wrapper .headerTop {
background-color:#FFCC66; border-top:0px solid #000000; border-bottom:1px
solid #FFFFFF; text-align:center;
}
#nl_wrapper .adminText {
font-size:10px; color:#996600; line-height:200%; font-family:Verdana, Arial,
Helvetica; text-decoration:none;
}
#nl_wrapper .headerBar {
background-color:#FFFFFF; border-top:0px solid #333333; border-bottom:10px
solid #FFFFFF;
}
#nl_wrapper .title {
font-size:20px; font-weight:bold; color:#CC6600; font-family:Verdana, Arial,
Helvetica; line-height:150%;
}
#nl_wrapper .subTitle {
font-size:11px; font-weight:normal; color:#666666; font-style:italic;
font-family:arial;
}
#nl_wrapper td {
font-size:11px; color:#000000; line-height:150%; font-family:Verdana, Arial,
Helvetica;
}
#nl_wrapper .sideColumn {
background-color:#FFFFFF; border-left:1px dashed #CCCCCC; text-align:left;
}
#nl_wrapper .sideColumnText {
font-size:11px; font-weight:normal; color:#AAAAAA; font-family:Verdana,
Arial, Helvetica; line-height:150%;
}
#nl_wrapper .sideColumnTitle {
font-size:15px; font-weight:bold; color:#333333; font-family:Verdana, Arial,
Helvetica; line-height:150%;
}
#nl_wrapper .footerRow {
background-color:#FFFFCC; border-top:10px solid #FFFFFF;
}
#nl_wrapper .footerText {
font-size:10px; color:#996600; line-height:100%; font-family:Verdana, Arial,
Helvetica;
}
#nl_wrapper h1,
#nl_wrapper h2  {
   color:#888888;font-size:20px;font-weight:bold;font-family:Arial,
Helvetica;margin:0 0 10px 0; line-height:150%;
}
#nl_wrapper h2  { font-size:15px; margin:23px 0 10px 0; padding-top:8px;
border-top:1px solid #EFEFEF;}
#nl_wrapper p   {margin:10px 0;
font-size:11px;color:#333333;font-family:Verdana, Arial, Helvetica;
line-height:150%;}
#nl_wrapper a,
#nl_wrapper table td p a,
#nl_wrapper #crintable a,
#nl_wrapper #crintable a:visited { color:#1155cc; /* color:#003a6a;*/ }
#nl_wrapper a:hover,
#nl_wrapper table td p a:hover,
#nl_wrapper #crintable a:hover{color:#000;}
#nl_wrapper ul,
#nl_wrapper li,
#nl_wrapper
#crintable ul   {margin:0; padding:0; list-style:none; list-style-image:none;
list-style-type:none}
#nl_wrapper ul li,
#nl_wrapper #crintable ul li,
#nl_wrapper #CMrightcol ul li
{font-size:11px;color:#333333;background-image:url(http://crin.org/i/crinmail/bg_bullet.gif);
background-repeat:no-repeat; background-position:2px
center;font-family:Verdana, Arial, Helvetica;  margin:0; list-style:none;
list-style-type:none; padding:0 0 0 20px; margin:13px 0;}
#nl_wrapper div.FB,                                                                                     #nl_wrapper div.FB p    { margin:10px 0}
#nl_wrapper div.FB h2   {color:#5e768b; font-size:16px; margin:0;}
#nl_wrapper #crintable table td {
vertical-align: top;
}
#nl_wrapper h3.h3image{
   background-repeat: no-repeat;
   background-position: center right;
}
   http://crin.org
09 September 2015
subscribe http://crin.org/node/41907 | subscribe http://crin.org/node/293 |
submit information http://crin.org/en/home/about/contact-us
======== *CRINMAIL 1445* =====================================================
-------- IN THIS ISSUE: ------------------------------------------------------
*The year in children's rights:*
CRIN annual report 2015 #AR
*Latest news and reports* #news
- Discrimination on school grounds #dis
- Criminalising children for sexting #crim
- Access to bathrooms denied #ac
- States leave migrant children unprotected #sta
- Issues neglected by UN committees #iss
- UN calls for submissions #un
- Contribute to evaluation of CRIN’s work #cont
*Access to justice for children in Ireland* #a2j
*Upcoming events* #events
*Employment* #jbs
/Problems viewing this CRINmail? Click here
https://www.crin.org/en/home/what-we-do/crinmail/crinmail-1445. /
*/CRIN announcement/*
/We are aware that some of our readers may be receiving blank CRINmails. This
is due to our website experiencing some technical issues, which we are
working to resolve. Should you receive a blank or incomplete CRINmail, you
can also access the latest edition via our website here
https://www.crin.org/en/home/what-we-do/crinmail. /

Also note that the list unsub header contains a HTTP URL when this would be better as a HTTPS one for privacy reasons:

List-Unsubscribe: <http://www.crin.org/en/newsletter/confirm/remove/XXX>

#73: CRIN2 ran out of memory

chris — Sun, 01 May 2016 19:16:38 GMT

The site has been down since Sun, 01 May 2016 18:05:44, it is now 8:15. I'm rebooting the server, it ran out of memory.

#53: Blank email: CRINmail 1459

chris — Thu, 17 Dec 2015 13:04:49 GMT

I was sent this last night, with no email body:

To: chris@webarchitects.co.uk
Subject: CRINmail 1459
X-PHP-Originating-Script: 1005:mimemail.module
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8Bit
X-Mailer: Drupal
Sender: CRIN <info@crin.org>
From: CRIN <info@crin.org>
List-Unsubscribe: <https://www.crin.org/en/newsletter/confirm/remove/XXXX>
Precedence: bulk
Message-Id: <E1a9N9t-0001cP-Qt@crin2.crin.org>
Date: Thu, 17 Dec 2015 01:15:05 +0000

This looks like another case of the issue previously raised here: ticket:14

#14: Blank CRINmail newsletters being sent out

chris — Sat, 16 May 2015 21:12:30 GMT

A newsletter sent via Drupal contained no content, Jenny reported:

We have tried twice today but both times when we send to all it sent out a blank email, with just the subject line. We have tried sending it to a few individual addresses and this works. We have sent other listservs out this week and they have worked as normal.

This is one of the emails:

From: CRIN [mailto:info@crin.org]
Sent: Friday, May 15, 2015 7:04 AM
To: Gertrud Lenzer
Subject: Children in Court CRINmail 48: Special edition on LGBTQI rights
--
*Child Rights International Network - CRIN*
Unit W125-127, Westminster Business Square
1-45 Durham Street
London SE11 5JH
United Kingdom
T: +44 20 7401 2257
 www.crin.org
@CRINwire <https://twitter.com/CRINwire>

#54: 502 Errors

chris — Tue, 22 Dec 2015 11:06:04 GMT

Last night the wiki:NginxLogWatch script found a lot of 502 errors:

Dec 22     288 502, 0 503 and 0 504 errors from crin2.crin.org
Dec 22    4283 502, 0 503 and 0 504 errors from crin2.crin.org

I'm not sure why yet.

#115: 500 Errors for crin.org

chris — Tue, 16 May 2017 17:00:33 GMT

There have been a lot of 500 errors today and they appear to be related to lines like this in /var/log/php/error.log:

[16-May-2017 17:33:08 Europe/London] PHP Fatal error:  Call to undefined function entity_translation_language_load() in /var/www/prod/docroot/includes/menu.inc on line 593

The section on the file in question:

 538
 539 /**
 540  * Loads objects into the map as defined in the $item['load_functions'].
 541  *
 542  * @param $item
 543  *   A menu router or menu link item
 544  * @param $map
 545  *   An array of path arguments; for example, array('node', '5').
 546  *
 547  * @return
 548  *   Returns TRUE for success, FALSE if an object cannot be loaded.
 549  *   Names of object loading functions are placed in $item['load_functions'].
 550  *   Loaded objects are placed in $map[]; keys are the same as keys in the
 551  *   $item['load_functions'] array.
 553  */
 554 function _menu_load_objects(&$item, &$map) {
 555   if ($load_functions = $item['load_functions']) {
 556     // If someone calls this function twice, then unserialize will fail.
 557     if (!is_array($load_functions)) {
 558       $load_functions = unserialize($load_functions);
 559     }
 560     $path_map = $map;
 561     foreach ($load_functions as $index => $function) {
 562       if ($function) {
 563         $value = isset($path_map[$index]) ? $path_map[$index] : '';
 564         if (is_array($function)) {
 565           // Set up arguments for the load function. These were pulled from
 566           // 'load arguments' in the hook_menu() entry, but they need
 567           // some processing. In this case the $function is the key to the
 568           // load_function array, and the value is the list of arguments.
 569           list($function, $args) = each($function);
 570           $load_functions[$index] = $function;
 571
 572           // Some arguments are placeholders for dynamic items to process.
 573           foreach ($args as $i => $arg) {
 574             if ($arg === '%index') {
 575               // Pass on argument index to the load function, so multiple
 576               // occurrences of the same placeholder can be identified.
 577               $args[$i] = $index;
 578             }
 579             if ($arg === '%map') {
 580               // Pass on menu map by reference. The accepting function must
 581               // also declare this as a reference if it wants to modify
 582               // the map.
 583               $args[$i] = &$map;
 584             }
 585             if (is_int($arg)) {
 586               $args[$i] = isset($path_map[$arg]) ? $path_map[$arg] : '';
 587             }
 588           }
 589           array_unshift($args, $value);
 590           $return = call_user_func_array($function, $args);
 591         }
 592         else {
 593           $return = $function($value);
 594         }
 595         // If callback returned an error or there is no callback, trigger 404.
 596         if ($return === FALSE) {
 597           $item['access'] = FALSE;
 598           $map = FALSE;
 599           return FALSE;
 600         }
 601         $map[$index] = $return;
 602       }
 603     }
 604     $item['load_functions'] = $load_functions;
 605   }
 606   return TRUE;
 607 }
 608

Peter, is the the cause of the 500 errors or is it a symptom, or is it unrelated? Do you have an idea what is going on here?

#127: 1984.is Server Rebuild

chris — Tue, 21 Nov 2017 19:26:16 GMT

Due to critical hardware failure at 1984.is all three servers needed rebuilding from scratch.