Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#37 closed defect (fixed)

Site slowdown on 8th Sept 2015

Reported by: chris Owned by: chris
Priority: critical Milestone: Maintenance
Component: drupal Version:
Keywords: Cc: mori, jenny
Estimated Number of Hours: 0 Add Hours to Ticket: 0
Billable?: yes Total Hours: 0.75

Description

The site is really slow and the number of php5-fpm processes has gone up dramatically and so has the memory usage.

Attachments (7)

crin-pingdom-2015-09-08.png (48.9 KB) - added by chris 3 years ago.
crin2_2015-09-08_multips-day.png (17.4 KB) - added by chris 3 years ago.
crin2_2015-09-08_multips_memory-day.png (21.0 KB) - added by chris 3 years ago.
crin2_2015-09-08_phpfpm_memory-day.png (18.7 KB) - added by chris 3 years ago.
crin2_2015-09-08_phpfpm_status-day.png (24.3 KB) - added by chris 3 years ago.
crin2_2015-09-08_php_opcache_memoryusage-day.png (18.1 KB) - added by chris 3 years ago.
crin-pingdom-2015-09-09.png (49.0 KB) - added by chris 3 years ago.

Download all attachments as: .zip

Change History (11)

Changed 3 years ago by chris

Changed 3 years ago by chris

Changed 3 years ago by chris

Changed 3 years ago by chris

Changed 3 years ago by chris

comment:1 Changed 3 years ago by chris

  • Add Hours to Ticket changed from 0 to 0.25
  • Total Hours set to 0.25

Testing via Pingdom we have a 30 seconds load time for the front page (this should be 2 seconds or so):


These Munin graphs illustrate a massive increase in the number of php-fpm5 processes and a corresponding memory usage:






comment:2 Changed 3 years ago by chris

Mori did you restart php5-fpm on Crin2 after deploying the new code today? If not that could explain some of the issues.

comment:3 Changed 3 years ago by mori

Thanks for picking this up Chris. No I didn't restart php5-fpm, as I was not aware it was required.

Can you let me know what command I should run? I'll add it to the wiki and may improve the deployment commands to restart it automatically.

Changed 3 years ago by chris

comment:4 Changed 3 years ago by chris

  • Add Hours to Ticket changed from 0 to 0.5
  • Resolution set to fixed
  • Status changed from new to closed
  • Total Hours changed from 0.25 to 0.75

When new code is deployed the Drupal caches need flushing and memcache and php-5-fpm need restarting:

sudo -i
service php5-fpm restart
service memcache restart

The reason php5-fpm need restarting is because the opcache was set to only check that .php files have changed the first time they are access (see ticket:#comment:17 ) however at some point it must have been changed as it is now set to check every 5 mins based on the file time stamp, in /etc/php5/fpm/php.ini on Crin2:

; When disabled, you must reset the OPcache manually or restart the
; webserver for changes to the filesystem to take effect.
;opcache.validate_timestamps=1
opcache.validate_timestamps=1

; How often (in seconds) to check file timestamps for changes to the shared
; memory storage allocation. ("1" means validate once per second, but only
; once per request. "0" means always validate)
;opcache.revalidate_freq=2
opcache.revalidate_freq=300

I think it would be safe to change opcache.validate_timestamps to zero again if a php5-fpm restart is always done after deploying updated code.

The speed of the site issue is now sorted:


Though at some point it might be worth looking at reducing the number of css and js files.

The Munin stats were showing a dramatic reduction in MySQL traffic, I expect this is due to more requests being cached by memcache, the memcache memory limit was also reached so I have increased it from 0.5G to 0.75G and also reduced the default number of php5-fpm processes started as there were a lot of idle ones almost all the time.

I have opened a new ticket for me to check the Bitbucket documentation, ticket:9#comment:17, closinng this ticket.

Version 1, edited 3 years ago by chris (previous) (next) (diff)
Note: See TracTickets for help on using tickets.