CRIN Trac: Ticket #37: Site slowdown on 8th Sept 2015

attachment set

chris — Tue, 08 Sep 2015 13:15:23 GMT

attachment set to crin-pingdom-2015-09-08.png

attachment set

chris — Tue, 08 Sep 2015 13:15:41 GMT

attachment set to crin2_2015-09-08_multips-day.png

attachment set

chris — Tue, 08 Sep 2015 13:16:01 GMT

attachment set to crin2_2015-09-08_multips_memory-day.png

attachment set

chris — Tue, 08 Sep 2015 13:16:28 GMT

attachment set to crin2_2015-09-08_phpfpm_memory-day.png

attachment set

chris — Tue, 08 Sep 2015 13:16:41 GMT

attachment set to crin2_2015-09-08_phpfpm_status-day.png

attachment set

chris — Tue, 08 Sep 2015 13:16:54 GMT

attachment set to crin2_2015-09-08_php_opcache_memoryusage-day.png

hours changed; totalhours set

chris — Tue, 08 Sep 2015 13:21:12 GMT

hours changed from 0 to 0.25
totalhours set to 0.25

Testing via Pingdom we have a 30 seconds load time for the front page (this should be 2 seconds or so):

These Munin graphs illustrate a massive increase in the number of php-fpm5 processes and a corresponding memory usage:

chris — Tue, 08 Sep 2015 13:28:51 GMT

Mori did you restart php5-fpm on Crin2 after deploying the new code today? If not that could explain some of the issues.

mori — Tue, 08 Sep 2015 14:27:50 GMT

Thanks for picking this up Chris. No I didn't restart php5-fpm, as I was not aware it was required.

Can you let me know what command I should run? I'll add it to the wiki and may improve the deployment commands to restart it automatically.

attachment set

chris — Wed, 09 Sep 2015 13:10:29 GMT

attachment set to crin-pingdom-2015-09-09.png

hours, status, totalhours changed; resolution set

chris — Wed, 09 Sep 2015 13:28:57 GMT

hours changed from 0 to 0.5
status changed from new to closed
resolution set to fixed
totalhours changed from 0.25 to 0.75

When new code is deployed the Drupal caches need flushing and memcache and php-5-fpm need restarting:

sudo -i
service php5-fpm restart
service memcache restart

The reason php5-fpm need restarting is because the opcache was set to only check that .php files have changed the first time they are access (see ticket:#comment:17 ) however at some point it must have been changed as it is now set to check every 5 mins based on the file time stamp, in /etc/php5/fpm/php.ini on Crin2:

; When disabled, you must reset the OPcache manually or restart the
; webserver for changes to the filesystem to take effect.
;opcache.validate_timestamps=1
opcache.validate_timestamps=1
; How often (in seconds) to check file timestamps for changes to the shared
; memory storage allocation. ("1" means validate once per second, but only
; once per request. "0" means always validate)
;opcache.revalidate_freq=2
opcache.revalidate_freq=300

I think it would be safe to change opcache.validate_timestamps to zero again if a php5-fpm restart is always done after deploying updated code.

The speed of the site issue is now sorted:

Though at some point it might be worth looking at reducing the number of css and js files.

The Munin stats were showing a dramatic reduction in MySQL traffic, I expect this is due to more requests being cached by memcache, the memcache memory limit was also reached so I have increased it from 0.5G to 0.75G and also reduced the default number of php5-fpm processes started as there were a lot of idle ones almost all the time.

I have opened a new ticket for me to check the Bitbucket documentation, ticket:9#comment:17, closing this ticket.