Site slowdown on 8th Sept 2015

[chris]

The site is really slow and the number of php5-fpm processes has gone up dramatically and so has the memory usage.

[chris]

Testing via ​Pingdom we have a 30 seconds load time for the front page (this should be 2 seconds or so):

These ​Munin graphs illustrate a massive increase in the number of php-fpm5 processes and a corresponding memory usage:

[chris]

Mori did you restart php5-fpm on Crin2 after deploying the new code today? If not that could explain some of the issues.

[mori]

Thanks for picking this up Chris. No I didn't restart php5-fpm, as I was not aware it was required.

Can you let me know what command I should run? I'll add it to the wiki and may improve the deployment commands to restart it automatically.

[chris]

When new code is deployed the Drupal caches need flushing and memcache and php-5-fpm need restarting:

sudo -i
service php5-fpm restart
service memcache restart

The reason php5-fpm need restarting is because the opcache was set to only check that .php files have changed the first time they are access (see ticket:#comment:17 ) however at some point it mush have been changed as it is now set to check every 5 mins based on the file time stamp, in /etc/php5/fpm/php.ini on Crin2:

; When disabled, you must reset the OPcache manually or restart the
; webserver for changes to the filesystem to take effect.
;opcache.validate_timestamps=1
opcache.validate_timestamps=1

; How often (in seconds) to check file timestamps for changes to the shared
; memory storage allocation. ("1" means validate once per second, but only
; once per request. "0" means always validate)
;opcache.revalidate_freq=2
opcache.revalidate_freq=300

I think it would be safe to change opcache.validate_timestamps to zero again if a php5-fpm restart is always done after deploying updated code.

The speed of the site issue is now sorted:

Though at some point it might be worth looking at reducing the number of css and js files.

The Munin stats were showing a dramatic reduction in MySQL traffic, I expect this is due to more requests being cached by memcache, the memcache memory limit was also reached so I have increased it from 0.5G to 0.75G and also reduced the default number of php5-fpm processes started as there were a lot of idle ones almost all the time.

I have opened a new ticket for me to check the Bitbucket documentation, ticket:9#comment:17, closinng this ticket.