CRIN Trac: Ticket #6: Migrate Drupal site from GreenQloud

type changed

chris — Wed, 29 Apr 2015 08:59:01 GMT

type changed from defect to task

milestone changed

chris — Wed, 29 Apr 2015 09:00:06 GMT

milestone changed from Install and configure crin1 to Install and configure crin2

hours changed; totalhours set

chris — Sun, 03 May 2015 11:54:55 GMT

hours changed from 0 to 0.25
totalhours set to 0.25

Adding ssh keys and doing an initial sync of the site, on crin2, generate a key pair:

sudo -i
ssh-keygen -t rsa -b 4096

Add the public key to web1 and set it to allow allow connections from web1, ~/.ssh/authorized_keys:

from="93.95.228.180" ssh-rsa AAAA...

Create ~/.ssh/config containing:

Host web1
  User root
  Hostname web1.crin.org
Host web2
  User root
  Hostname web2.crin.org
Host db1
  User root
  Hostname db1.crin.org

Create a directory for the site on crin1, install and rsync the site (checking the ssh fingerprint):

mkdir -p /var/www/drupal
aptitude install rsync
rsync -av web1:/var/www/crin/ /var/www/drupal/

The site is 20G so it'll take some time to copy:

du -h --max-depth=1
19G     ./sites
672K    ./themes
381M    ./.git
127M    ./owncloud_old
452K    ./stats
16K     ./.idea
2.3M    ./includes
108K    ./profiles
84K     ./scripts
11M     ./modules
912K    ./misc
20G     .

Jonas: are these things needed on the new server?

https://www.crin.org/owncloud_old/ (a non-working ownCloud site)
https://www.crin.org/stats/ (inaccessible web stats from 2014)

hours, totalhours changed

chris — Wed, 06 May 2015 13:58:31 GMT

hours changed from 0 to 1.1
totalhours changed from 0.25 to 1.35

PHP packages installed on the current live server:

aptitude search php | grep ^i
i A libphp5-embed                   - HTML-embedded scripting language (Embedded
i   php-apc                         - APC (Alternative PHP Cache) module for PHP
i A php-console-table               - PHP PEAR module to make it easy to build c
i   php-pear                        - PEAR - PHP Extension and Application Repos
i   php-xml-parser                  - PHP PEAR module for parsing XML
i A php5                            - server-side, HTML-embedded scripting langu
i A php5-cli                        - command-line interpreter for the php5 scri
i A php5-common                     - Common files for packages built from the p
i   php5-curl                       - CURL module for php5
i   php5-dev                        - Files for PHP5 module development
i   php5-fpm                        - server-side, HTML-embedded scripting langu
i   php5-gd                         - GD module for php5
i A php5-imagick                    - ImageMagick module for php5
i   php5-intl                       - internationalisation module for php5
i A php5-mcrypt                     - MCrypt module for php5
i   php5-memcached                  - memcached extension module for PHP5, uses
i   php5-mysql                      - MySQL module for php5
i A php5-pgsql                      - PostgreSQL module for php5
i   php5-sqlite                     - SQLite module for php5
i   phpmyadmin                      - MySQL web administration tool

And nginx:

aptitude search nginx | grep ^i
i A nginx-common                    - small, powerful, scalable web/proxy server
i   nginx-extras                    - nginx web/proxy server (extended version)

And memcache:

i A libmemcached10                  - C and C++ client library to the memcached
i   memcached                       - A high-performance memory object caching s
i   php5-memcached                  - memcached extension module for PHP5, uses

Not sure we need the same PHP packages (for example phpmyadmin is running on Crin1 and not needed on Crin2) so installing these packages to start with:

aptitude install nginx-common nginx-extras php5 php5-fpm php-pear php5-mysql php5-intl php5-imagick php5-memcached memcached drush

Copy the key Nginx config files from the live server:

cd /etc/nginx/sites-available
scp web1:/etc/nginx/sites-available/crin.com .
scp web1:/etc/nginx/sites-available/crin.org .
scp web1:/etc/nginx/sites-available/enoc.crin.org .

Copy the SSL/TLS key and cert:

cd /etc/ssl
mkdir gandi
chmod 700 gandi
rsync -av web1:/etc/ssl/gandi/ /etc/ssl/gandi/

Symlink and test Nginx:

cd /etc/nginx/sites-enabled
rm default
ln -s ../sites-available/crin.org 00-crin.org
service nginx configtest
  [FAIL] Testing nginx configuration: failed!

This is the error in the log:

2015/05/06 13:01:30 [info] 19112#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:63
2015/05/06 13:14:22 [emerg] 19856#0: open() "/etc/nginx/gzip" failed (2: No such file or directory) in /etc/nginx/sites-enabled/00-crin.org:13

So:

cd /etc/nginx
scp web1:/etc/nginx/gzip .
service nginx configtest
  [FAIL] Testing nginx configuration: failed!

The error this time:

2015/05/06 13:17:10 [emerg] 20015#0: could not build the server_names_hash, you should increase server_names_hash_bucket_size: 32

So these settings were copied into the main /etc/ngin/nginx.conf file from the live server:

        server_names_hash_max_size 2048;
        client_max_body_size 50M;

And we still have :

service nginx configtest
  [FAIL] Testing nginx configuration: failed!

2015/05/06 13:19:56 [info] 20066#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:65
2015/05/06 13:19:56 [emerg] 20066#0: could not build the server_names_hash, you should increase server_names_hash_bucket_size: 32

So this was changed:

        server_names_hash_bucket_size 128;

And now the error is:

2015/05/06 13:21:46 [emerg] 20101#0: zero size shared memory zone "uploads"

So the upload_progress.conf file was copied over:

cd /etc/nginx/conf.d
scp web1:/etc/nginx/conf.d/upload_progress.conf .

And success:

service nginx configtest
  [ ok ] Testing nginx configuration:.

Start the services:

service php5-fpm start
service nginx start

And as expected we now has a database error as that needs copying and the settings.php will need editing.

On Crin1 create a database and user:

mysql mysql
  mysql> CREATE DATABASE drupal;
  mysql> GRANT ALL ON drupal.* to 'drupal'@'crin2' identified by 'XXX' REQUIRE SSL;
  mysql> FLUSH PRIVILEGES;

On Crin1 create a script to dump the database on db1 and rscnc and import it after enabling ssh access to db1:

#!/bin/bash
# sync files
rsync -av  --exclude "settings.php" web1:/var/www/crin/ /var/www/drupal/
# dump and copy the database
ssh db1 "mysqldump -uroot -pXXX crin > /root/crin.sql"
scp db1:crin.sql /root/
# import the database
cat /root/crin.sql | mysql drupal

The script takes a while to run, it's a 1.4G database file and a 20G site...

I'll continue with this tomorrow.

hours, estimatedhours, totalhours changed

chris — Thu, 07 May 2015 11:39:19 GMT

hours changed from 0 to 0.05
estimatedhours changed from 0 to 5
totalhours changed from 1.35 to 1.4

Outstanding things that will need doing on this ticket:

PHP configuration
Setting up Memcache
Setting up Solr
Testing the copy of the site
Document the setup at wiki:Drupal
Final data sync
Update DNS

So far 1.35 hours have been spent on it, I estimate that it might end up being 6 in total.

After the server goes live there will a need to monitor and adjust server settings, see ticket:9.

hours, totalhours changed

chris — Thu, 07 May 2015 12:58:50 GMT

hours changed from 0 to 0.5
totalhours changed from 1.4 to 1.9

Note the ~/bin/drupal-db-sync script on Crin1 mentioned in ticket:6#comment:4 has the rsync of the files commented out, this command is in ~/bin/drupal-sync on Crin2:

#!/bin/bash
# sync files
rsync -av  --exclude "settings.php" web1:/var/www/crin/ /var/www/drupal/

The two sync script were run and settings.php was edited on Crin2:

$databases = array (
  'default' =>
  array (
    'default' =>
    array (
      'database' => 'drupal',
      'username' => 'drupal',
      'password' => 'XXX',
      'host' => 'crin1',
      'port' => '',
      'driver' => 'mysql',
      'prefix' => '',
      'pdo' => array(
           PDO::MYSQL_ATTR_SSL_KEY => '/etc/ssl/cacert/crin1_yassl_privatekey.pem',
           PDO::MYSQL_ATTR_SSL_CERT => '/etc/ssl/cacert/crin1_cert.pem',
           PDO::MYSQL_ATTR_SSL_CA => '/etc/ssl/cacert/cacert.pem',
        ),
    ),
  ),
);

And the site, http://crin2.crin.org/ was checked and we have an error:

PDOException: SQLSTATE[HY000] [2026] SSL connection error: Unable to get certificate in lock_may_be_available() (line 167 of /var/www/drupal/includes/lock.inc).

So, check that we can connect via MySQL on the command line, edit /root/.mysql to:

[client]
host=crin1
user = root
password = XXX
ssl-cipher=DHE-RSA-AES256-SHA
ssl-ca=/etc/ssl/cacert/cacert.pem
ssl-cert=/etc/ssl/cacert/crin1_cert.pem
ssl-key=/etc/ssl/cacert/crin1_yassl_privatekey.pem

On Crin1 allow access from Crin2:

mysql mysql
  mysql> GRANT ALL ON *.* to 'root'@'crin2' identified by 'XXX' REQUIRE SSL;
  mysql> FLUSH PRIVILEGES;
  mysql> exit;

Test on Crin2:

mysql mysql
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 174
Server version: 5.5.43-0+deb8u1 (Debian)
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>

Create a /var/www/drupal/.my.cnf on Crin2:

[client]
host=crin1
ssl-cipher=DHE-RSA-AES256-SHA
ssl-ca=/etc/ssl/cacert/cacert.pem
ssl-cert=/etc/ssl/cacert/crin1_cert.pem
ssl-key=/etc/ssl/cacert/crin1_yassl_privatekey.pem

The issue could be that the www-data user doesn't have access to the CAcert cert and key, so on Crin2

sudo -i
chown -R root:www-data /etc/ssl/cacert/
chmod 770 /etc/ssl/cacert/
chmod 660 /etc/ssl/cacert/*.pem

And the site is now up and running: http://crin2.crin.org/ -- Jonas can you login and see if things appear to be working OK?

https://crin2.crin.org/en/user/login

I'll also do some testing.

hours, totalhours changed

chris — Thu, 07 May 2015 13:55:58 GMT

hours changed from 0 to 1
totalhours changed from 1.9 to 2.9

Looked at setting up drush on Crin2 but have hit this issue:

https://github.com/drush-ops/drush/issues/965

Checking the documentation for the GreenQloud servers and I found that I wrote:

The Debian packaged version of Drush doesn't support MySQL SSL connections so a new version has been installed from github.com in /usr/local/src/drush-6.x and symliked from /usr/local/sbin/

So, download the latest version of Drush on Crin2:

sudo -i
cd /usr/local/src
https://github.com/drush-ops/drush/archive/master.zip
aptitude install unzip
unzip master.zip
cd /usr/local/bin/
ln -s ../src/drush-master/drush

Set up /root/.drush/drushrc.php and /var/www/drushrc.php:

mkdir /root/.drush
cp /usr/local/src/drush-master/examples/example.drushrc.php /root/.drush/drushrc.php
mkdir /var/www/.drush
cp /usr/local/src/drush-master/examples/example.drushrc.php /var/www/.drush/drushrc.php
chown -R www-data:www-data /var/www/.drush

Edit both files to change:

// Specify a particular multisite.
# $options['l'] = 'http://example.com/subdir';
$options['l'] = 'http://crin1.crin.org/';
// Specify your Drupal core base directory (useful if you use symlinks).
# $options['r'] = '/home/USER/workspace/drupal-6';
$options['r'] = '/var/www/drupal';

Note this will need changing when the site is made live.

Test:

sudo -i
su - www-data -s /bin/bash
cd drupal/
drush uli
  Unable to load autoload.php. Drush now requires Composer in order to install its dependencies and autoload classes. Please see README.md

So, following https://getcomposer.org/doc/00-intro.md#globally

sudo -i
cd /usr/local/src
aptitude install curl
curl -sS https://getcomposer.org/installer | php
  #!/usr/bin/env php
  All settings correct for using Composer
  Downloading...
  Composer successfully installed to: /usr/local/src/composer.phar
  Use it: php composer.phar
mv composer.phar /usr/local/bin/composer

Try drush again:

sudo -i
su - www-data -s /bin/bash
drush --version
  Unable to load autoload.php. Drush now requires Composer in order to install its dependencies and autoload classes. Please see README.md

So:

sudo -i
cd /usr/local/src/drush-master
composer install
  Loading composer repositories with package information
  Installing dependencies (including require-dev) from lock file
    - Installing d11wtq/boris (v1.0.8)
      Downloading: Connecting...    Failed to download d11wtq/boris from dist: The "https://api.github.com/repos/d11wtq/boris/zipball/125dd4e5752639af7678a22ea597115646d89c6e" file could not be downloaded (HTTP/1.1 404 Not Found)
      Now trying to download from source
    - Installing d11wtq/boris (v1.0.8)
      Cloning 125dd4e5752639af7678a22ea597115646d89c6e
    [RuntimeException]
    Failed to clone git@github.com:d11wtq/boris.git, git was not found, check that it is installed and in your PATH env.
    sh: 1: git: not found
  install [--prefer-source] [--prefer-dist] [--dry-run] [--dev] [--no-dev] [--no-plugins] [--no-custom-installers] [--no-autoloader] [--no-scripts] [--no-progress] [-v|vv|vvv|--verbose] [-o|--optimize-autoloader] [--ignore-platform-reqs] [packages1] ... [packagesN]
aptitude install git
composer install
  Loading composer repositories with package information
  Installing dependencies (including require-dev) from lock file
    - Installing d11wtq/boris (v1.0.8)
      Downloading: Connecting...    Failed to download d11wtq/boris from dist: The "https://api.github.com/repos/d11wtq/boris/zipball/125dd4e5752639af7678a22ea597115646d89c6e" file could not be downloaded (HTTP/1.1 404 Not Found)
      Now trying to download from source
    - Installing d11wtq/boris (v1.0.8)
      Cloning 125dd4e5752639af7678a22ea597115646d89c6e
    - Installing pear/console_table (1.2.1)
      Downloading: 100%
    - Installing symfony/var-dumper (v2.6.3)
      Downloading: 100%
    - Installing phpunit/php-token-stream (1.4.0)
      Downloading: 100%
    - Installing symfony/yaml (v2.6.3)
      Downloading: 100%
    - Installing sebastian/version (1.0.4)
      Downloading: 100%
    - Installing sebastian/recursion-context (1.0.0)
      Downloading: 100%
    - Installing sebastian/global-state (1.0.0)
      Downloading: 100%
    - Installing sebastian/exporter (1.2.0)
      Downloading: 100%
    - Installing sebastian/environment (1.2.1)
      Downloading: 100%
    - Installing sebastian/diff (1.2.0)
      Downloading: 100%
    - Installing sebastian/comparator (1.1.1)
      Downloading: 100%
    - Installing phpunit/php-text-template (1.2.0)
      Downloading: 100%
    - Installing doctrine/instantiator (1.0.4)
      Downloading: 100%
    - Installing phpunit/phpunit-mock-objects (2.3.0)
      Downloading: 100%
    - Installing phpunit/php-timer (1.0.5)
      Downloading: 100%
    - Installing phpunit/php-file-iterator (1.3.4)
      Downloading: 100%
    - Installing phpunit/php-code-coverage (2.0.15)
      Downloading: 100%
    - Installing phpunit/phpunit (4.4.5)
      Downloading: 100%
    - Installing symfony/process (v2.4.5)
      Downloading: 100%
  pear/console_table suggests installing pear/Console_Color2 (>=0.1.2)
  symfony/var-dumper suggests installing ext-symfony_debug ()
  sebastian/global-state suggests installing ext-uopz (*)
  phpunit/php-code-coverage suggests installing ext-xdebug (>=2.2.1)
  phpunit/phpunit suggests installing phpunit/php-invoker (~1.1)
  Generating autoload files

And now Drush runs:

drush --version
 Drush Version   :  7.0-dev
drush status
 Drupal version         :  7.24
 Site URI               :  http://crin1.crin.org/
 Database driver        :  mysql
 Database hostname      :  crin1
 Database port          :
 Database username      :  drupal
 Database name          :  drupal
 PHP executable         :  /usr/bin/php
 PHP configuration      :  /etc/php5/cli/php.ini
 PHP OS                 :  Linux
 Drush script           :  /usr/local/src/drush-master/drush.php
 Drush version          :  7.0-dev
 Drush temp directory   :  /tmp
 Drush configuration    :  /var/www/.drush/drushrc.php
 Drush alias files      :
 Drupal root            :  /var/www/drupal
 Site path              :  sites/default

But we need to set up memcache:

drush uli
  PHP Fatal error:  Class 'MemCacheDrupal' not found in /var/www/drupal/includes/cache.inc on line 31
  Drush command terminated abnormally due to an unrecoverable error.                                      [error]
  Error: Class 'MemCacheDrupal' not found in /var/www/drupal/includes/cache.inc, line 31

hours, totalhours changed

chris — Mon, 11 May 2015 10:48:11 GMT

hours changed from 0 to 1.61
totalhours changed from 2.9 to 4.51

Replying to chris:

But we need to set up memcache:

drush uli
  PHP Fatal error:  Class 'MemCacheDrupal' not found in /var/www/drupal/includes/cache.inc on line 31
  Drush command terminated abnormally due to an unrecoverable error.                                      [error]
  Error: Class 'MemCacheDrupal' not found in /var/www/drupal/includes/cache.inc, line 31

This bug says:

In fact, there is a way to specify memcache cache handler in settings.php, using $conf['cache_backends'] variable, like this:
$conf['cache_backends'][] = 'sites/all/modules/memcache/memcache.inc';

We have:

 $conf['cache_backends'][] = 'sites/all/modules/memcache/memcache.inc';
   // The 'cache_form' bin must be assigned no non-volatile storage.
 $conf['cache_class_cache_form'] = 'DrupalDatabaseCache';
 $conf['cache_default_class'] = 'MemCacheDrupal';
 $conf['memcache_key_prefix'] = 'crin_';

Following comment #29, I have changed it to:

include_once DRUPAL_ROOT . '/includes/cache.inc';
include_once DRUPAL_ROOT . '/sites/all/modules/memcache/memcache.inc';
$conf['cache_default_class'] = 'MemCacheDrupal';

And this seem to work, drush runs without a problem.

Checking https://crin2.crin.org/en/admin/reports/status there are these issues:

Required PHP extension not found. Install the memcache (recommended) or memcached extension.

So:

aptitude install php5-memcache
/etc/init.d/php5-fpm restart

Other packages needed:

aptitude install php5-memcache php5-curl

We have this:

Multibyte string input conversion in PHP is active and must be disabled. Check the php.ini mbstring.http_input setting.

However according to the docs we should:

PHP 5.6 and later users should leave this empty and set default_charset instead.

And we have:

default_charset = "UTF-8"

On the live server we have no default charset set and:

mbstring.http_input = pass
mbstring.http_output = pass

So these settings were copied to Crin2.

Setting up a crontab for the www-data user:

crontab -e -u www-data

58 * * * * /usr/bin/env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin COLUMNS=72 /usr/local/bin/drush --root=/var/www/drupal --uri=crin1.crin.org --quiet cron

TODO: Note the above will need updating when the site is made live.

We have:

GD library rotate and desaturate effects	Not installed

Checking what the live server has:

aptitude search gd | grep ^i
i A libgd2-xpm                      - GD Graphics Library version 2
i   libgdbm3                        - GNU dbm database routines (runtime version
i A libgdk-pixbuf2.0-0              - GDK Pixbuf library
i A libgdk-pixbuf2.0-common         - GDK Pixbuf library - data files
i   php5-gd                         - GD module for php5

And Crin2 has:

aptitude search gd | grep ^i
i A libgd3                          - GD Graphics Library
i   libgdbm3                        - GNU dbm database routines (runtime version
i A libgdk-pixbuf2.0-0              - GDK Pixbuf library
i A libgdk-pixbuf2.0-common         - GDK Pixbuf library - data files

So:

aptitude install php5-gd
service php5-fpm restart

Now the only outstanding issue on the status page is:

Apache Solr Your site was unable to contact the Apache Solr server. Default environment url: http://web1:8080/solr

On the live server we have:

aptitude search solr | grep ^i
i A libsolr-java                    - Enterprise search server based on Lucene -
i A solr-common                     - Enterprise search server based on Lucene3
i   solr-tomcat                     - Enterprise search server based on Lucene3

So:

aptitude install libsolr-java solr-tomcat solr-common

Check the settings:

The "Solr server URL" was changed to: http://localhost:8080/solr.

TODO: this will need doing again after the final data sync, when the site on Crin2 is made live.

On the live server we have a password protected Nginx reverse proxy to the Solr admin interface, so the following was added to /etc/nginx/sites-available/solr.crin.org:

# default virtual server
server {
        # listen for ipv4
        # http://nginx.org/en/docs/http/ngx_http_core_module.html#listen
        listen   80;
        # server name and server aliases
        # http://nginx.org/en/docs/http/ngx_http_core_module.html#server_name
        server_name solr.crin.org;
        # Prevent access to any files starting with a dot, like .htaccess
        # or text editor temp files
        location ~ /\. {
                access_log off;
                log_not_found off;
                deny all;
        }
        # Prevent access to tmp files created by vim
        location ~ .~$ {
                return 403;
        }
        location / {
                rewrite ^/(.*)$ https://solr.crin.org/$1? permanent;
        }
}
# HTTPS server
#
server {
        #listen   4430;
        listen   443;
        server_name solr.crin.org;
        access_log  /var/log/nginx/solr.crin.org.ssl_access.log;
        error_log   /var/log/nginx/solr.crin.org.ssl_error.log notice;
        ssl  on;
        ssl_certificate  /etc/ssl/cacert/crin2_cert.chained.pem;
        ssl_certificate_key  /etc/ssl/cacert/crin2_privatekey.pem;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;
        ssl_prefer_server_ciphers on;
        #add_header Strict-Transport-Security max-age=31536000;
        # https://developer.mozilla.org/en-US/docs/HTTP/X-Frame-Options
        add_header X-Frame-Options SAMEORIGIN;
        location / {
                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Server $host;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_pass http://127.0.0.1:8080/;
                satisfy any;
                deny all;
                auth_basic "Solr Admin";
                auth_basic_user_file /var/www/.htpasswd;
        }
}

And the site was enabled and tested:

cd /etc/nginx/sites-enabled
ln -s ../sites-available/solr.crin.org 20-solr.crin.org
service nginx configtest
  [ ok ] Testing nginx configuration:.
service nginx restart

And a sub-domain was created at a DNS level.

Create a username / passwd for the Solr admin interface:

cd /var/www/
aptitude install apache2-utils
htpasswd -c .htpasswd crin

So, now the site is fully ready for testing: https://crin2.crin.org/

hours, totalhours changed

chris — Mon, 11 May 2015 10:59:34 GMT

hours changed from 0 to 0.1
totalhours changed from 4.51 to 4.61

Current status of the Drupal code in terms of available updates:

sudo -i
su - www-data -s /bin/bash
cd drupal/
 drush status
 Drupal version         :  7.24
 Site URI               :  http://crin1.crin.org/
 Database driver        :  mysql
 Database hostname      :  crin1
 Database port          :
 Database username      :  drupal
 Database name          :  drupal
 PHP executable         :  /usr/bin/php
 PHP configuration      :  /etc/php5/cli/php.ini
 PHP OS                 :  Linux
 Drush script           :  /usr/local/src/drush-master/drush.php
 Drush version          :  7.0-dev
 Drush temp directory   :  /tmp
 Drush configuration    :  /var/www/.drush/drushrc.php
 Drush alias files      :
 Drupal root            :  /var/www/drupal
 Site path              :  sites/default
drush up -n | grep available
 Drupal                                                     7.24               7.37              SECURITY UPDATE available
 Views Bulk Operations (views_bulk_operations)              7.x-3.1            7.x-3.2           Update available
 Administration menu (admin_menu)                           7.x-3.0-rc4        7.x-3.0-rc5       Update available
 Apache Solr framework (apachesolr)                         7.x-1.6            7.x-1.7           Update available
 Apache Solr Multisite Search (apachesolr_multisitesearch)  7.x-1.0            7.x-1.1           Update available
 Autocomplete Deluxe (autocomplete_deluxe)                  7.x-2.0-beta3      7.x-2.1           Update available
 Better Exposed Filters (better_exposed_filters)            7.x-3.0-beta3      7.x-3.2           Update available
 Chaos tools (ctools)                                       7.x-1.3            7.x-1.7           SECURITY UPDATE available
 CAPTCHA (captcha)                                          7.x-1.1            7.x-1.3           Update available
 CKEditor (ckeditor)                                        7.x-1.13           7.x-1.16          SECURITY UPDATE available
 Facet API (facetapi)                                       7.x-1.3            7.x-1.5           Update available
 Date (date)                                                7.x-2.6            7.x-2.8           SECURITY UPDATE available
 Devel (devel)                                              7.x-1.3            7.x-1.5           Update available
 Domain Access (domain)                                     7.x-3.10           7.x-3.11          Update available
 Entity API (entity)                                        7.x-1.2            7.x-1.6           SECURITY UPDATE available
 Entityforms (entityform)                                   7.x-2.0-beta4      7.x-2.0-rc1       Update available
 Field collection (field_collection)                        7.x-1.0-beta5      7.x-1.0-beta8     Update available
 Fieldgroup (field_group)                                   7.x-1.3            7.x-1.4           Update available
 Google Map Field (google_map_field)                        7.x-2.4            7.x-2.13          Update available
 Internationalization (i18n)                                7.x-1.10           7.x-1.13          SECURITY UPDATE available
 IMCE (imce)                                                7.x-1.7            7.x-1.9           Update available
 jQuery Update (jquery_update)                              7.x-2.3            7.x-2.5           Update available
 Localization update (l10n_update)                          7.x-1.0-beta3      7.x-1.1           Update available
 Libraries (libraries)                                      7.x-2.1            7.x-2.2           Update available
 LoginToboggan (logintoboggan)                              7.x-1.3            7.x-1.5           SECURITY UPDATE available
 Memcache (memcache)                                        7.x-1.2            7.x-1.5           Update available
 Menu attributes (menu_attributes)                          7.x-1.0-rc2        7.x-1.0-rc3       Update available
 Menu Block (menu_block)                                    7.x-2.3            7.x-2.5           Update available
 Menu Node Views (menu_node_views)                          7.x-1.x-dev        7.x-1.x-dev       Update available
 Mime Mail (mimemail)                                       7.x-1.0-beta1      7.x-1.0-beta3     SECURITY UPDATE available
 MultiBlock (multiblock)                                    7.x-1.1            7.x-1.2           Update available
 OAuth (oauth)                                              7.x-3.1            7.x-3.2           Update available
 reCAPTCHA (recaptcha)                                      7.x-1.11           7.x-1.12          Update available
 Rules (rules)                                              7.x-2.6            7.x-2.9           Update available
 Search API (search_api)                                    7.x-1.6            7.x-1.14          Update available
 Database search (search_api_db)                            7.x-1.2            7.x-1.4           Update available
 Taxonomy menu (taxonomy_menu)                              7.x-1.4            7.x-1.5           Update available
 Token (token)                                              7.x-1.5            7.x-1.6           Update available
 Transliteration (transliteration)                          7.x-3.1            7.x-3.2           Update available
 Variable (variable)                                        7.x-2.3            7.x-2.5           Update available
 Views (views)                                              7.x-3.7            7.x-3.11          SECURITY UPDATE available
 Views Autocomplete Filters (views_autocomplete_filters)    7.x-1.0            7.x-1.2           Update available
NOTE: A security update for the Drupal core is available.
Cancelled.                                                                                                                                 [cancel]
Multibyte string input conversion in PHP is active and must be disabled. Check the php.ini mbstring.http_input setting. Please refer to the[warning]
PHP mbstring documentation for more information. (Currently using Unicode library Error)
No database updates required                                                                                                               [success]
'all' cache was cleared.                                                                                                                   [success]
Finished performing updates.                                                                                                               [ok]

So it looks like there is perhaps still an issue with "Multibyte string input conversion in PHP".

Jonas, the security updates listed above might contain issues which apply to CRIN, can check about these with the web developers?

jonas — Mon, 11 May 2015 11:00:07 GMT

I tested crin2.crin.org and the site is a bit slow. I was able to log in to the CMS, but apart from that I haven't done much.

jonas — Mon, 11 May 2015 11:02:17 GMT

The problem is that currently, we don't have any web developers, only people who are assessing the work that Effusion did. However, they work on a tight deadline and won't have time to check this, also it's not what my colleagues want them to do. So I think we will have to wait until we have new web developer, but then it will be too late, right?

chris — Wed, 13 May 2015 08:21:58 GMT

Replying to jonas:

I tested crin2.crin.org and the site is a bit slow.

OK, I'll see what can be done to speed things up by adjusting the MySQL / PHP-FPM and Nginx memory usage etc.

I was able to log in to the CMS, but apart from that I haven't done much.

Could you test uploading a file?

chris — Wed, 13 May 2015 08:27:25 GMT

Replying to jonas:

So I think we will have to wait until we have new web developer, but then it will be too late, right?

I don't know if any of the security issues listed in ticket:6#comment:9 apply to your site, if you would like I could spend some time reading up on them to see if it is clear if any do, however I wouldn't feel confident that I could do any Drupal updates without the potential for breaking things (this is why there are lot of Drupal sites in situations like this -- updating things often breaks them).

attachment set

chris — Wed, 13 May 2015 13:34:38 GMT

attachment set to db1.my.cnf

hours, totalhours changed

chris — Wed, 13 May 2015 13:42:46 GMT

hours changed from 0 to 0.28
totalhours changed from 4.61 to 4.89

Testing using pingdom.com we have:

https://crin2.crin.org/ - Load time: 3.42s
https://www.crin.org/ - Load time: 2.36s

So, lets see what we MySQL settings can be changed, on Crin1:

aptitude install mysqltuner
mysqltuner
 >>  MySQLTuner 1.3.0 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering
[OK] Logged in using credentials from debian maintenance account.
[OK] Currently running supported MySQL version 5.5.43-0+deb8u1
[OK] Operating on 64-bit architecture
-------- Storage Engine Statistics -------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MRG_MYISAM
[--] Data in InnoDB tables: 718M (Tables: 506)
[--] Data in MEMORY tables: 0B (Tables: 1)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[--] Data in MyISAM tables: 953M (Tables: 79)
[!!] Total fragmented tables: 518
-------- Security Recommendations  -------------------------------------------
[OK] All database users have passwords assigned
-------- Performance Metrics -------------------------------------------------
[--] Up for: 5d 1h 0m 15s (133K q [0.306 qps], 1K conn, TX: 5B, RX: 172M)
[--] Reads / Writes: 41% / 59%
[--] Total buffers: 192.0M global + 2.7M per thread (151 max threads)
[OK] Maximum possible memory usage: 597.8M (15% of installed RAM)
[OK] Slow queries: 0% (16/133K)
[OK] Highest usage of available connections: 2% (4/151)
[OK] Key buffer size / total MyISAM indexes: 16.0M/238.9M
[OK] Key buffer hit rate: 96.9% (24M cached / 757K reads)
[OK] Query cache efficiency: 73.5% (67K cached / 91K selects)
[!!] Query cache prunes per day: 2228
[OK] Sorts requiring temporary tables: 0% (3 temp sorts / 2K sorts)
[OK] Temporary tables created on disk: 10% (74 on disk / 713 total)
[OK] Thread cache hit rate: 99% (4 created / 1K connections)
[!!] Table cache hit rate: 5% (400 open / 7K opened)
[OK] Open file limit used: 12% (128/1K)
[OK] Table locks acquired immediately: 99% (63K immediate / 63K locks)
[!!] InnoDB  buffer pool / data size: 128.0M/718.5M
[OK] InnoDB log waits: 0
-------- Recommendations -----------------------------------------------------
General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    Enable the slow query log to troubleshoot bad queries
    Increase table_open_cache gradually to avoid file descriptor limits
    Read this before increasing table_open_cache over 64: http://bit.ly/1mi7c4C
Variables to adjust:
    query_cache_size (> 16M)
    table_open_cache (> 400)
    innodb_buffer_pool_size (>= 718M)

Based on the above edit /etc/mysql/my.cnf and change:

#query_cache_size        = 16M
query_cache_size        = 32M
innodb_buffer_pool_size        = 1024M

Restart MySQL:

service mysql restart

Now the results of mysqltuner can be checked again tomorrow (the server needs to be running a while to generate stats).

For future reference the existing db1.crin.org /etc/mysql/my.cnf has been attached, I'm not simply copying this over since the existing db1.crin.org GreenQloud server has more resources available to it -- it doesn't host web sites.

db1.my.cnf

Note that db1.crin.org uses a RAM disk for the tmpdir.

attachment set

chris — Wed, 13 May 2015 13:59:56 GMT

attachment set to web1.php.ini

hours, totalhours changed

chris — Wed, 13 May 2015 15:27:56 GMT

hours changed from 0 to 0.25
totalhours changed from 4.89 to 5.14

Attached is the /etc/php5/fpm/php.ini from the existing live GreenQloud server, web1.crin.org:

web1.php.ini

Comparing the configurations these things were changed in /etc/php5/fpm/php.ini on Crin2:

;post_max_size = 8M
post_max_size = 20M
;upload_max_filesize = 2M
upload_max_filesize = 20M
;max_file_uploads = 20
max_file_uploads = 60
extension=uploadprogress.so

Following Install PECL uploadprogress on Debian 7 Wheezy:

aptitude install make php5-dev php-pear
pecl install uploadprogress
service php5-fpm restart

hours, totalhours changed

chris — Wed, 13 May 2015 16:08:50 GMT

hours changed from 0 to 0.25
totalhours changed from 5.14 to 5.39

To enable spdy lines like this were changed in the Nginx config:

listen 443 ssl spdy default_server;

And this was tested at http://spdycheck.org/

Note that the spdycheck.org site incorrectly resported that HSTS isn't in use, but is is, see the SSLLabs report:

https://www.ssllabs.com/ssltest/analyze.html?d=crin2.crin.org

This seems to have helped with the speed of the site -- it now loads in less than 2.5 seconds via the Pingdom full page test from Sweden and less than 2 seconds from Amsterdam.

Also memcached was added to the Munin memory graph -- currently it is set to use 64M of RAM, this could be increased if needs be.

Jonas -- does the site seem quicker now? Do file uploads report their progress (I'm wondering if I should have installed a Nginx specific version of upload progress)?

I noticed that Google Analytics, as well a Piwik, is enabled on this site.

chris — Thu, 14 May 2015 00:45:47 GMT

Benchmarking the front page from another server at 1984.is, requesting the front page 10k times with 20 concurrent requests:

ab -n 10000 -c20 http://crin2.crin.org/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking crin2.crin.org (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
Server Software:        nginx/1.6.2
Server Hostname:        crin2.crin.org
Server Port:            80
Document Path:          /
Document Length:        74044 bytes
Concurrency Level:      20
Time taken for tests:   1050.591 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      745820000 bytes
HTML transferred:       740440000 bytes
Requests per second:    9.52 [#/sec] (mean)
Time per request:       2101.182 [ms] (mean)
Time per request:       105.059 [ms] (mean, across all concurrent requests)
Transfer rate:          693.27 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    7  53.1      2    1032
Processing:   596 2093 679.7   1937    5671
Waiting:      559 2066 672.3   1915    5637
Total:        598 2099 683.3   1942    5680
Percentage of the requests served within a certain time (ms)
  50%   1942
  66%   2264
  75%   2474
  80%   2616
  90%   3012
  95%   3447
  98%   3963
  99%   4242
 100%   5680 (longest request)

So, Requests per second: 9.52 and Time per request: 2 sec, and looking at the Munin graphs for the servers they didn't really break a sweat, but the number of requests per second could probably be higher with some tweaks, another test:

ab -n 100 -c40 https://crin2.crin.org/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking crin2.crin.org (be patient).....done
Server Software:        nginx/1.6.2
Server Hostname:        crin2.crin.org
Server Port:            443
SSL/TLS Protocol:       TLSv1/SSLv3,ECDHE-RSA-AES128-GCM-SHA256,2048,128
Document Path:          /
Document Length:        74087 bytes
Concurrency Level:      40
Time taken for tests:   28.608 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      7467605 bytes
HTML transferred:       7408700 bytes
Requests per second:    3.50 [#/sec] (mean)
Time per request:       11443.296 [ms] (mean)
Time per request:       286.082 [ms] (mean, across all concurrent requests)
Transfer rate:          254.91 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       15   91  81.8     46     260
Processing:  4439 6857 2554.0   6115   28492
Waiting:     4228 6581 1352.8   6061   10188
Total:       4535 6948 2586.8   6188   28606
Percentage of the requests served within a certain time (ms)
  50%   6188
  66%   7293
  75%   7667
  80%   7941
  90%   9243
  95%   9866
  98%  10487
  99%  28606
 100%  28606 (longest request)

It's slower with HTTPS by the looks of this, without it:

ab -n 100 -c40 http://crin2.crin.org/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking crin2.crin.org (be patient).....done
Server Software:        nginx/1.6.2
Server Hostname:        crin2.crin.org
Server Port:            80
Document Path:          /
Document Length:        74044 bytes
Concurrency Level:      40
Time taken for tests:   11.985 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      7458200 bytes
HTML transferred:       7404400 bytes
Requests per second:    8.34 [#/sec] (mean)
Time per request:       4793.849 [ms] (mean)
Time per request:       119.846 [ms] (mean, across all concurrent requests)
Transfer rate:          607.73 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1   14  99.4      4     998
Processing:   574 4007 1096.9   4473    5386
Waiting:      555 3983 1100.9   4451    5371
Total:        581 4021 1104.9   4474    5543
Percentage of the requests served within a certain time (ms)
  50%   4474
  66%   4614
  75%   4692
  80%   4738
  90%   4925
  95%   5133
  98%   5390
  99%   5543
 100%   5543 (longest request)

8 requests per second compared with 3.5, quite a difference.

On the GreenQloud server in /etc/nginx/nginx.conf we have:

#worker_processes 4;
worker_processes 127;

And on Crin2 we have:

worker_processes 4;

Doubling this and testing what it does for the benchmark:

ab -n 100 -c40 http://crin2.crin.org/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking crin2.crin.org (be patient).....done
Server Software:        nginx/1.6.2
Server Hostname:        crin2.crin.org
Server Port:            80
Document Path:          /
Document Length:        74044 bytes
Concurrency Level:      40
Time taken for tests:   7.715 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      7458200 bytes
HTML transferred:       7404400 bytes
Requests per second:    12.96 [#/sec] (mean)
Time per request:       3085.845 [ms] (mean)
Time per request:       77.146 [ms] (mean, across all concurrent requests)
Transfer rate:          944.10 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   1.7      2       6
Processing:   277 2468 820.1   2795    3431
Waiting:      268 2455 819.0   2788    3417
Total:        279 2471 819.5   2800    3432
Percentage of the requests served within a certain time (ms)
  50%   2800
  66%   2923
  75%   3011
  80%   3043
  90%   3159
  95%   3249
  98%   3386
  99%   3432
 100%   3432 (longest request)

12 requests per second rather than 8 and no fails, doubling the number of Nginx processes again and testing:

 ab -n 100 -c40 http://crin2.crin.org/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking crin2.crin.org (be patient).....done
Server Software:        nginx/1.6.2
Server Hostname:        crin2.crin.org
Server Port:            80
Document Path:          /
Document Length:        74044 bytes
Concurrency Level:      40
Time taken for tests:   9.192 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      7458200 bytes
HTML transferred:       7404400 bytes
Requests per second:    10.88 [#/sec] (mean)
Time per request:       3676.632 [ms] (mean)
Time per request:       91.916 [ms] (mean, across all concurrent requests)
Transfer rate:          792.40 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    3   3.4      2      26
Processing:   336 3150 995.5   3499    4361
Waiting:      322 3131 1000.3   3488    4344
Total:        338 3153 996.2   3500    4365
Percentage of the requests served within a certain time (ms)
  50%   3500
  66%   3746
  75%   3874
  80%   3910
  90%   4056
  95%   4196
  98%   4269
  99%   4365
 100%   4365 (longest request)

Slower, so setting the worker_processes to 8 for now.

In /etc/php5/fpm/pool.d/www.conf we have:

pm.max_children = 5

And on the live GreenQloud server we have:

pm.max_children = 127

Another test :

crin2.crin.org (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software:        nginx/1.6.2
Server Hostname:        crin2.crin.org
Server Port:            80
Document Path:          /
Document Length:        74044 bytes
Concurrency Level:      80
Time taken for tests:   123.974 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      74582000 bytes
HTML transferred:       74044000 bytes
Requests per second:    8.07 [#/sec] (mean)
Time per request:       9917.929 [ms] (mean)
Time per request:       123.974 [ms] (mean, across all concurrent requests)
Transfer rate:          587.49 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    6  31.8      3     998
Processing:   506 9654 2600.3   9653   16149
Waiting:      497 9627 2594.4   9629   16108
Total:        510 9659 2601.9   9659   16152
Percentage of the requests served within a certain time (ms)
  50%   9659
  66%  10598
  75%  11121
  80%  11789
  90%  12971
  95%  14076
  98%  15243
  99%  15664
 100%  16152 (longest request)

8 requests per second, so this isn't hlping, it seems, testing with the live server:

www.crin.org (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software:        nginx/1.2.1
Server Hostname:        www.crin.org
Server Port:            80
Document Path:          /
Document Length:        74026 bytes
Concurrency Level:      80
Time taken for tests:   15.556 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      74599000 bytes
HTML transferred:       74026000 bytes
Requests per second:    64.28 [#/sec] (mean)
Time per request:       1244.470 [ms] (mean)
Time per request:       15.556 [ms] (mean, across all concurrent requests)
Transfer rate:          4683.15 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       40   41   2.5     41      58
Processing:   298 1170 166.2   1180    1796
Waiting:      172 1045 165.7   1056    1674
Total:        339 1212 166.1   1221    1847
Percentage of the requests served within a certain time (ms)
  50%   1221
  66%   1264
  75%   1292
  80%   1305
  90%   1376
  95%   1458
  98%   1565
  99%   1665
 100%   1847 (longest request)

64 requests per second, something must be missing? With HTTPS:

 ab -n 1000 -c80 https://www.crin.org/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking www.crin.org (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software:        nginx/1.2.1
Server Hostname:        www.crin.org
Server Port:            443
SSL/TLS Protocol:       TLSv1/SSLv3,ECDHE-RSA-AES128-GCM-SHA256,2048,128
Document Path:          /
Document Length:        74069 bytes
Concurrency Level:      80
Time taken for tests:   15.981 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      74693000 bytes
HTML transferred:       74069000 bytes
Requests per second:    62.57 [#/sec] (mean)
Time per request:       1278.502 [ms] (mean)
Time per request:       15.981 [ms] (mean, across all concurrent requests)
Transfer rate:          4564.24 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      165  222  97.3    191     594
Processing:   305 1024 167.6   1017    1605
Waiting:      237  912 162.9    899    1485
Total:        632 1246 182.6   1229    2199
Percentage of the requests served within a certain time (ms)
  50%   1229
  66%   1293
  75%   1321
  80%   1343
  90%   1415
  95%   1536
  98%   1868
  99%   2067
 100%   2199 (longest request)

16 requests per second, the 1984.is servers:

ab -n 1000 -c80 https://crin2.crin.org/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking crin2.crin.org (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software:        nginx/1.6.2
Server Hostname:        crin2.crin.org
Server Port:            443
SSL/TLS Protocol:       TLSv1/SSLv3,ECDHE-RSA-AES128-GCM-SHA256,2048,128
Document Path:          /
Document Length:        74087 bytes
Concurrency Level:      80
Time taken for tests:   114.247 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      74676000 bytes
HTML transferred:       74087000 bytes
Requests per second:    8.75 [#/sec] (mean)
Time per request:       9139.735 [ms] (mean)
Time per request:       114.247 [ms] (mean, across all concurrent requests)
Transfer rate:          638.32 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       11   75 190.4     27    1274
Processing:   535 8487 2552.3   7933   15406
Waiting:      516 8465 2548.7   7917   15386
Total:        650 8562 2548.7   7960   15911
Percentage of the requests served within a certain time (ms)
  50%   7960
  66%   8800
  75%   9588
  80%  10074
  90%  12444
  95%  14361
  98%  14994
  99%  15243
 100%  15911 (longest request)

9 requests per second, close but slower. Looking at the spikes on the Munin graphs for both servers there is still a huge amount of unused CPU and RAM so with tweaking these figures should be able to be improved.

hours, totalhours changed

chris — Thu, 14 May 2015 18:38:51 GMT

hours changed from 0 to 0.25
totalhours changed from 5.39 to 5.64

Email was configured on Crin2, the following was added to /etc/aliases:

root: chris@webarchitects.co.uk,admin@crin.org

And newaliases was run and also dpkg-reconfigure exim4-config and logwatch and metche were installed and email was tested by sending email to root.

In addition both server have been rebooted due to venom.

hours, totalhours changed

chris — Thu, 14 May 2015 21:01:35 GMT

hours changed from 0 to 0.75
totalhours changed from 5.64 to 6.39

The current live GreenQloud server has APC installed:

aptitude search apc | grep ^i
  i   php-apc                         - APC (Alternative PHP Cache) module for PHP

This could be a cause for the slowness of the Drupal site, so on Crin2:

aptitude search apc | grep php
  p   php-apc                         - APC User Cache for PHP 5 (transitional pac
  p   php5-apcu                       - APC User Cache for PHP 5
aptitude install php5-apcu
logchange "php5-apcu : installed"

On the GreenQloud server /etc/php5/mods-available/apc.ini contains:

extension=apc.so
apc.shm_size="512M"
apc.max_file_size="2M"

And we might want to add these settings, but to know what they can best be set to we need some Munin stats, so this munin-php-apc plugin looks like it will do the job, so:

cd /usr/local/share/munin/plugins/
wget https://github.com/geerlingguy/munin-php-apc/archive/master.zip
unzip master.zip
  Archive:  master.zip
  bc7f1ad0103bbf500e8d36e342cfaa53733fbc7d
     creating: munin-php-apc-master/
    inflating: munin-php-apc-master/CHANGELOG.txt
    inflating: munin-php-apc-master/README.md
    inflating: munin-php-apc-master/apc_info.php
    inflating: munin-php-apc-master/php_apc_

Make a directory and move the apc_info.php file:

mkdir /var/www/localhost
mv munin-php-apc-master/apc_info.php /var/www/localhost/apc-info.php

Edit /etc/nginx/sites-available/localhost to add:

 	root /var/www/localhost;
        location ~ /apc-info\.php$ {
                fastcgi_pass unix:/var/run/php5-fpm.sock;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_intercept_errors on;
                include fastcgi_params;
                access_log off;
                allow 127.0.0.1;
                deny all;
        }

Test and restart and test:

service nginx configtest
  [ ok ] Testing nginx configuration:.
service nginx restart
lynx -dump http://localhost/apc-info.php
   size: 33554296 used: 32992 free: 33521304 hits: 0.00 misses: 0.00
   request_rate: 0.00 hit_rate: 0.00 miss_rate: 0.00 insert_rate: 0.00
   entries: 0 inserts: 0 purges: purge_rate: 100.00 fragment_percentage:
   0.00 fragmented: 0.00 fragment_segments: 0 optcode_size: 0 user_size: 0
   user_hits: 0.00 user_misses: 0.00 user_request_rate: 0.00
   user_hit_rate: 0.00 user_miss_rate: 0.00 user_insert_rate: 0.00
   user_entries: 0 user_inserts: 0 user_purges: user_purge_rate: 0.00

So now to enable the munin plugin:

cd /etc/munin/plugins/
ln -s /usr/local/share/munin/plugins/munin-php-apc-master/php_apc_ php_apc_files
ln -s /usr/local/share/munin/plugins/munin-php-apc-master/php_apc_ php_apc_fragmentation
ln -s /usr/local/share/munin/plugins/munin-php-apc-master/php_apc_ php_apc_hit_miss
ln -s /usr/local/share/munin/plugins/munin-php-apc-master/php_apc_ php_apc_purge
ln -s /usr/local/share/munin/plugins/munin-php-apc-master/php_apc_ php_apc_rates
ln -s /usr/local/share/munin/plugins/munin-php-apc-master/php_apc_ php_apc_usage
ln -s /usr/local/share/munin/plugins/munin-php-apc-master/php_apc_ php_apc_mem_size
ln -s /usr/local/share/munin/plugins/munin-php-apc-master/php_apc_ php_apc_user_hit_miss
ln -s /usr/local/share/munin/plugins/munin-php-apc-master/php_apc_ php_apc_user_entries
ln -s /usr/local/share/munin/plugins/munin-php-apc-master/php_apc_ php_apc_user_rates

Add the following to /etc/munin/plugin-conf.d/munin-node:

[php_apc_*]
user root
env.url http://localhost/apc-info.php?auto

Test and restart:

cd /etc/munin/plugins/
munin-run php_apc_files
  used.value 32992
  free.value 33521304
  hits.value 0.00
  misses.value 0.00
  request_rate.value 0.00
  hit_rate.value 0.00
  miss_rate.value 0.00
  insert_rate.value 0.00
  entries.value 0
  inserts.value 0
  purges.value U
  purge_rate.value 100.00
  fragmented.value 0.00
  fragment_segments.value 0
  fragment_percentage.value 0.00
  optcode_size.value 0
  user_size.value 0
  user_hits.value 0.00
  user_misses.value 0.00
  user_request_rate.value 0.00
  user_hit_rate.value 0.00
  user_miss_rate.value 0.00
  user_insert_rate.value 0.00
  user_entries.value 0
  user_inserts.value 0
  user_purges.value U
  user_purge_rate.value 0.00
service munin-node restart

And we should soon have some APC stats here:

https://munin.crin.org/munin/crin.org/crin2.crin.org/#php-apc

Testing to see if there is noticable speed increase:

ab -n 1000 -c80 https://crin2.crin.org/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking crin2.crin.org (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software:        nginx/1.6.2
Server Hostname:        crin2.crin.org
Server Port:            443
SSL/TLS Protocol:       TLSv1/SSLv3,ECDHE-RSA-AES128-GCM-SHA256,2048,128
Document Path:          /
Document Length:        74087 bytes
Concurrency Level:      80
Time taken for tests:   67.240 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      74676000 bytes
HTML transferred:       74087000 bytes
Requests per second:    14.87 [#/sec] (mean)
Time per request:       5379.193 [ms] (mean)
Time per request:       67.240 [ms] (mean, across all concurrent requests)
Transfer rate:          1084.56 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        8   35  78.0     12    1012
Processing:   161 5141 874.1   5278    6769
Waiting:      156 5136 874.0   5270    6766
Total:        305 5177 820.7   5293    6782
Percentage of the requests served within a certain time (ms)
  50%   5293
  66%   5392
  75%   5487
  80%   5556
  90%   5708
  95%   5822
  98%   5932
  99%   6014
 100%   6782 (longest request)

15 requests per second, that is an improvement on the 9 requests per second we had with the same test last night.

attachment set

chris — Thu, 14 May 2015 21:16:44 GMT

attachment set to crin2.crin.org-speed-test.2015-05-14.png

hours, totalhours changed

chris — Thu, 14 May 2015 21:22:29 GMT

hours changed from 0 to 0.1
totalhours changed from 6.39 to 6.49

The front page of the site loads in less than 2 seconds when tested via http://tools.pingdom.com/fpt using the Amsterdam test server.

chris — Thu, 14 May 2015 21:47:37 GMT

Looking at the php-apc stats it doesn't appear to be doing anything, the config file is /etc/php5/fpm/conf.d/20-apcu.ini and it contains:

extension=apcu.so

The apc config documentation lists all the config options, to start with these were added:

apc.enabled=1
apc.shm_size=128M
apc.max_file_size=1M

And service php5-fpm restart was run, and some more benchmarking was done using ab:

ab -n 1000 -c80 https://crin2.crin.org/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking crin2.crin.org (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software:        nginx/1.6.2
Server Hostname:        crin2.crin.org
Server Port:            443
SSL/TLS Protocol:       TLSv1/SSLv3,ECDHE-RSA-AES128-GCM-SHA256,2048,128
Document Path:          /
Document Length:        74087 bytes
Concurrency Level:      80
Time taken for tests:   57.527 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      74676000 bytes
HTML transferred:       74087000 bytes
Requests per second:    17.38 [#/sec] (mean)
Time per request:       4602.125 [ms] (mean)
Time per request:       57.527 [ms] (mean, across all concurrent requests)
Transfer rate:          1267.69 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        8   33  76.1     12    1006
Processing:   312 4397 683.8   4459    6235
Waiting:      307 4390 683.6   4454    6222
Total:        432 4430 664.3   4479    6346
Percentage of the requests served within a certain time (ms)
  50%   4479
  66%   4625
  75%   4739
  80%   4807
  90%   5045
  95%   5209
  98%   5369
  99%   5555
 100%   6346 (longest request)

17 pages a second and another Pingdom test came in at 1.85s for the front page.

hours, totalhours changed

chris — Fri, 12 Jun 2015 09:35:05 GMT

hours changed from 0 to 0.25
totalhours changed from 6.49 to 6.74

Doing a final Drupal database sync on Crin1:

bash /root/bin/drupal-db-sync

And doing a final sync of the files on Crin2:

bash /root/bin/drupal-sync

chris — Fri, 12 Jun 2015 10:06:32 GMT

The MySQL sync has run but the sync of the files is still running.

chris — Fri, 12 Jun 2015 11:13:11 GMT

Copying the files is taking a long time, a 214M file is taking 30 mins to copy...

chris — Fri, 12 Jun 2015 13:39:48 GMT

The files have finally synced, but I'm going to be away from a computer between 3pm and 4pm so I don't want to do the DNS update right now, I'll do it at 4pm. Sorry for the delay, I wasn't expecting it to take so long to sync the files that have changed over the last few weeks.

chris — Fri, 12 Jun 2015 15:24:32 GMT

I'm going to update the DNS now.

hours, totalhours changed

chris — Fri, 12 Jun 2015 15:35:22 GMT

hours changed from 0 to 0.2
totalhours changed from 6.74 to 6.94

I have changed these lines in the DNS for crin.org:

@ 900 IN A 93.95.228.180
enoc 900 IN A 93.95.228.180
www 900 IN A 93.95.228.180

And these lines for crin.com:

@ 900 IN A 93.95.228.180
www 900 IN A 93.95.228.180

Checking the DNS servers:

dig @NS0.1984.IS crin.org +short
46.149.19.215
dig @NS0.1984.IS crin.com +short
46.149.19.215

These DNS servers should update soon and then others will follow.

hours, totalhours changed

chris — Fri, 12 Jun 2015 16:01:40 GMT

hours changed from 0 to 0.4
totalhours changed from 6.94 to 7.34

The enoc.crin.org Nginx config file was copied from crin-web1.

The 1984.is DNS servers have updated:

dig @NS0.1984.IS crin.com +short
93.95.228.180
dig @NS0.1984.IS crin.org +short
93.95.228.180
dig @NS0.1984.IS enoc.crin.org +short
93.95.228.180

The main site at https://www.crin.org/ seems fine, the SSL settings get a A+ rating:

https://www.ssllabs.com/ssltest/analyze.html?d=crin.org

There is an issue with http://enoc.crin.org/ - it displays this message:

You are not authorized to access this page.

But the GreenQloud server does the same thing...

Everything seems to be working OK, I'll check the Munin graphs in a while to see what is happening load wise:

hours, totalhours changed

chris — Fri, 12 Jun 2015 16:38:33 GMT

hours changed from 0 to 0.3
totalhours changed from 7.34 to 7.64

Update Drupal crontab as per TODO note above:

crontab -e -u www-data

It now contains:

58 * * * * /usr/bin/env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin COLUMNS=72 /usr/local/bin/drush --root=/var/www/drupal --uri=www.crin.org --quiet cron

Test it:

su - www-data -s /bin/bash
/usr/local/bin/drush --root=/var/www/drupal --uri=www.crin.org --quiet cron
-su: /usr/local/bin/drush: Permission denied

So:

chmod 755 /usr/local/src/drush-master/drush

And try again:

/usr/local/bin/drush --root=/var/www/drupal --uri=www.crin.org --quiet cron
Command core-cron needs a higher bootstrap level to run - you will need to invoke drush from a more functional Drupal environment [error]
to run this command.
The drush command 'cron' could not be executed. [error]
Drush was not able to start (bootstrap) the Drupal database. [error]
Hint: This may occur when Drush is trying to:
* bootstrap a site that has not been installed or does not have a configured database. In this case you can select another site
with a working database setup by specifying the URI to use with the --uri parameter on the command line. See `drush topic
docs-aliases` for details.
* connect the database through a socket. The socket file may be wrong or the php-cli may have no access to it in a jailed shell.
See http://drupal.org/node/1428638 for details.
Drush was attempting to connect to:
Drupal version : 7.24
Site URI : www.crin.org
Database driver : mysql
Database hostname : crin1
Database port :
Database username : drupal
Database name : drupal
PHP executable : /usr/bin/php
PHP configuration : /etc/php5/cli/php.ini
PHP OS : Linux
Drush script : /usr/local/src/drush-master/drush.php
Drush version : 7.0-dev
Drush temp directory : /tmp
Drush configuration : /var/www/.drush/drushrc.php
Drush alias files :
Drupal root : /var/www/drupal
Site path : sites/default

Testing on the GreenQloud server:

su - www-data -s /bin/bash
/usr/bin/env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin COLUMNS=72 /usr/local/sbin/drush --root=/var/www/crin --uri=www.crin.org --quiet cron
PHP Fatal error:  Class 'MemCacheDrupal' not found in /var/www/crin/includes/cache.inc on line 31
Drush command terminated abnormally due to an unrecoverable  [error]
error.
Error: Class 'MemCacheDrupal' not found in
/var/www/crin/includes/cache.inc, line 31

So it hasn't been working on the old server either, I have created a new ticket for this, ticket:18.

cc, status changed; resolution set

chris — Fri, 12 Jun 2015 17:50:37 GMT

cc jenny gillian added; jonas removed
status changed from new to closed
resolution set to fixed

Closing this ticket now -- the main Drupal site, https://www.crin.org/ is now running on the 1984.is servers, follow up can be done on ticket:9.

hours, totalhours changed

chris — Sat, 13 Jun 2015 11:24:02 GMT

hours changed from 0 to 0.15
totalhours changed from 7.64 to 7.79

Jenny asked:

Are ok to upload info on the site now?

Yes you should be, the GreenCloud servers are still seeing some traffic so not everybodies DNS has updated yet, though all the recent requests are from robots, eg:

Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

The Googlebot doesn't appear to be a fake Googlebot, it has reverse DNS set up:

dig -x 66.249.64.189 +short
crawl-66-249-64-189.googlebot.com.

I think best leave the old servers running for a day or two.