CRIN Trac: Ticket #36: Piwik stats stopped on 31st August 2015

attachment set

chris — Tue, 08 Sep 2015 11:14:32 GMT

attachment set to crin-piwik-2015-09-08.png

description changed

chris — Tue, 08 Sep 2015 11:15:27 GMT

description modified (diff)

hours changed; totalhours set

chris — Tue, 08 Sep 2015 11:41:20 GMT

hours changed from 0 to 0.25
totalhours set to 0.25

I have just used drush uli to login to the live Drupal site and noticed that it very slow.

I think this module can probably be removed:

Cookie Control for Google Analytics

I guess the GA code is hard coded into the templates? What is the URL for the project on Bitbucket?

The Piwik module needs adding back, I can sort the settings needed by this module on the dev/staging site if you wish, it would perhaps make sense to have a separate Piwik account for the dev/staging sites so that we can do testing without an impact on the live site stats.

mori — Tue, 08 Sep 2015 12:02:05 GMT

Thanks for the heads-up. The Piwik module has been added back to Dev, Stage and Prod.

Whether we want the module enabled on non-production environment depends on how CRIN would like to use it. The module's status changes every time the database sync between environments happen so switching to respective account needs to happen automatically. We can handle this as part of a maintenance task in the future.

On Drupal sites, usually Google Analytics code is added through modules. The module however does not exist so I'd assume the code was hard-coded. This can also be handled as part of ongoing maintenance.

hours, totalhours changed

chris — Tue, 08 Sep 2015 12:28:37 GMT

hours changed from 0 to 0.35
totalhours changed from 0.25 to 0.6

Replying to mori:

Thanks for the heads-up. The Piwik module has been added back to Dev, Stage and Prod.

Thanks, I'm struggling to find the form to configure it, the modules list links to a list of system settings pages rather than the specific form fields for the Piwik plugin. On a Drupal 6 site I would expect these settings to be at https://www.crin.org/en/admin/settings/piwik but that URL doesn't work either. Do you know what the URL to configure the Piwik module is?

Whether we want the module enabled on non-production environment depends on how CRIN would like to use it. The module's status changes every time the database sync between environments happen so switching to respective account needs to happen automatically. We can handle this as part of a maintenance task in the future.

OK, so it would be far simpler using the same account on all 3 sites I guess? In any case developers can opt out in various ways from being tracked (whitelisting IPs in Piwik, setting DNT in Firefox, not loading JS from stats.crin.org etc)

On Drupal sites, usually Google Analytics code is added through modules. The module however does not exist so I'd assume the code was hard-coded. This can also be handled as part of ongoing maintenance.

Great.

mori — Tue, 08 Sep 2015 13:03:46 GMT

Replying to chris:

Thanks, I'm struggling to find the form to configure it

I had to disable and re-enable the module to get the module to work properly.

https://stage.crin.org/en/admin/config/system/piwik

The module is now working on Stage and Prod. Will take care of Dev as soon as it responds.

Whether we want the module enabled on non-production environment depends on how CRIN would like to use it. The module's status changes every time the database sync between environments happen so switching to respective account needs to happen automatically. We can handle this as part of a maintenance task in the future.

OK, so it would be far simpler using the same account on all 3 sites I guess?

Yes. If CRIN can configure Piwik to filter out data from non-prod, that would be the simplest solution

Just so you know, there seems to be an outstanding performance issue on the site. I've noticed that there are some modules that are marked as enabled in systems table (this is how Drupal decides to inlcude PHP files), but the actual files are not present. Drupal 7 has a performance bug which keeps scanning for missing files and it drastically reduces the site's performance. We will handle this issue in due course.

When new developers (who are not from Code Positive) get involved in the development of the site, can you please point them to our wiki* and make sure they manage the codebase via git and never add/remove files on the server?

just added you to the Crin project: https://bitbucket.org/crin/crin/wiki/Home

hours, totalhours changed

chris — Tue, 08 Sep 2015 13:13:08 GMT

hours changed from 0 to 0.25
totalhours changed from 0.6 to 0.85

Replying to mori:

https://stage.crin.org/en/admin/config/system/piwik

Thanks I'll get it set up.

Yes. If CRIN can configure Piwik to filter out data from non-prod, that would be the simplest solution

OK.

Just so you know, there seems to be an outstanding performance issue on the site. I've noticed that there are some modules that are marked as enabled in systems table (this is how Drupal decides to inlcude PHP files), but the actual files are not present. Drupal 7 has a performance bug which keeps scanning for missing files and it drastically reduces the site's performance. We will handle this issue in due course.

Yes, it's really slow, I'll open a separate ticket on this.

When new developers (who are not from Code Positive) get involved in the development of the site, can you please point them to our wiki* and make sure they manage the codebase via git and never add/remove files on the server?

just added you to the Crin project: https://bitbucket.org/crin/crin/wiki/Home

Thanks, I'll check that out.

hours, totalhours changed

chris — Tue, 08 Sep 2015 13:50:43 GMT

hours changed from 0 to 0.35
totalhours changed from 0.85 to 1.2

Getting a login URL via Drush doesn't work for the staging site on Crin4:

su - bitbucket -s /bin/bash
cd /var/www/stage/docroot/
drush uli
exception 'PDOException' with message 'SQLSTATE[28000] [1045] Access denied for user 'newprod'@'crin4' (using password: YES)' in             [error]
/var/www/stage/docroot/includes/database/database.inc:304
Stack trace:
#0 /var/www/stage/docroot/includes/database/database.inc(304): PDO->__construct('mysql:host=crin...', 'newprod', 'XXXX', Array)
#1 /var/www/stage/docroot/includes/database/mysql/database.inc(51): DatabaseConnection->__construct('mysql:host=crin...', 'newprod',
'XXXX', Array)
#2 /var/www/stage/docroot/includes/database/database.inc(1686): DatabaseConnection_mysql->__construct(Array)
#3 /var/www/stage/docroot/includes/database/database.inc(1476): Database::openConnection('default', 'default')
#4 /var/www/stage/docroot/includes/database/database.inc(2347): Database::getConnection('default')
#5 /var/www/stage/docroot/includes/bootstrap.inc(1915): db_query('SELECT 1 FROM {...', Array)
#6 /var/www/stage/docroot/includes/bootstrap.inc(1928): drupal_is_denied('127.0.0.1')
#7 /var/www/stage/docroot/includes/bootstrap.inc(2378): drupal_block_denied('127.0.0.1')
#8 /var/www/stage/docroot/includes/bootstrap.inc(2233): _drupal_bootstrap_page_cache()
#9 /var/www/stage/docroot/sites/all/modules/contrib/domain/domain.bootstrap.inc(108): drupal_bootstrap(2, true)
#10 /var/www/stage/docroot/sites/all/modules/contrib/domain/domain.bootstrap.inc(70): _domain_bootstrap(0)
#11 /var/www/stage/docroot/sites/all/modules/contrib/domain/settings.inc(21): domain_bootstrap()
#12 /var/www/stage/docroot/sites/default/settings.php(490): include('/var/www/stage/...')
#13 /var/www/stage/docroot/includes/bootstrap.inc(716): include_once('/var/www/stage/...')
#14 /var/www/stage/docroot/includes/bootstrap.inc(2355): drupal_settings_initialize()
#15 /var/www/stage/docroot/includes/bootstrap.inc(2229): _drupal_bootstrap_configuration()
#16 /usr/local/src/drush-master/lib/Drush/Boot/DrupalBoot7.php(62): drupal_bootstrap(0)
#17 /usr/local/src/drush-master/includes/bootstrap.inc(314): Drush\Boot\DrupalBoot7->bootstrap_drupal_configuration()
#18 /usr/local/src/drush-master/commands/user/user.drush.inc(388): drush_bootstrap(5)
#19 [internal function]: drush_user_login()
#20 /usr/local/src/drush-master/includes/command.inc(359): call_user_func_array('drush_user_logi...', Array)
#21 /usr/local/src/drush-master/includes/command.inc(210): _drush_invoke_hooks(Array, Array)
#22 [internal function]: drush_command()
#23 /usr/local/src/drush-master/includes/command.inc(178): call_user_func_array('drush_command', Array)
#24 /usr/local/src/drush-master/lib/Drush/Boot/BaseBoot.php(62): drush_dispatch(Array)
#25 /usr/local/src/drush-master/drush.php(70): Drush\Boot\BaseBoot->bootstrap_and_dispatch()
#26 /usr/local/src/drush-master/drush.php(11): drush_main()
#27 {main}

I created a ~/.my.cnf for this user containing:

[client]
host=crin1
ssl-cipher=DHE-RSA-AES256-SHA
ssl-ca=/etc/ssl/cacert/cacert.pem
ssl-cert=/etc/ssl/cacert/crin1_cert.pem
ssl-key=/etc/ssl/cacert/crin1_yassl_privatekey.pem

But that didn't solve it -- there is the same error on the command line:

mysql -unewprod -p -hcrin1 newprod
Enter password:
ERROR 1045 (28000): Access denied for user 'newprod'@'crin4' (using password: YES)

One error was with the permissions on Crin1:

mysql> SELECT Host from user where User="newprod";
+-------+
| Host  |
+-------+
| crin2 |
+-------+
1 row in set (0.00 sec)
mysql> UPDATE user SET Host="crin4" WHERE User="newprod";
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.15 sec)

But that also hasn't solved it, I'll have to continue with this tomorrow.

mori — Tue, 08 Sep 2015 14:04:44 GMT

It works for me. The problem you had is caused by lack of specificity so Drush to work out which site to access.

Run drush uli within the site-specific directory (docroot/sites/[site]/ . The dir default is for the production), or use a drush alias @crin.[env].

You can see the list of environments by executing drush site-alias. An explanation on the aliases is found here: https://bitbucket.org/crin/crin/wiki/CRIN%20Drush%20aliases%20explained

Below are the examples on how to run drush uli:

mori@CRIN4:/var/www/stage/docroot/sites/stage.crin.org$ drush uli
http://stage.crin.org/en/user/reset/1/1441720321'''/[hash]'''/login
mori@CRIN4:/var/www/stage/docroot/sites$ drush @crin.stage uli
http://stage.crin.org/en/user/reset/1/1441720340/'''[hash]'''/login

===

Regarding the issue you had when logging into mysql, I also experienced it when attempting to edit the prod database (newprod). Can you look into this as well?

hours, status, totalhours changed; resolution set

chris — Wed, 09 Sep 2015 12:14:05 GMT

hours changed from 0 to 0.55
status changed from new to closed
resolution set to fixed
totalhours changed from 1.2 to 1.75

Replying to mori:

The problem you had is caused by lack of specificity so Drush to work out which site to access.

Yes, I'm sorry, that was stupid of me, thanks for pointing my mistake out.

The Piwik settings here:

https://stage.crin.org/en/admin/config/system/piwik

Look like they are unchanged, so are probably fine, on dev and prod I was just getting the list of system settings pages:

Until I ran "Flush all caches" and then the settings appeared on dev, but not on the prod site, I had to restart memcache and php5-fpm to get the tracking code to appear on the site and to get the Piwik settings form to be available in admin:

service php5-fpm restart
service memcached restart

So I think the lesson here is that when new code is deployed we need to do a drush cc to clear all the caches and restart these services, I'll look at where this could be added to the bitbucket documentation.

The Piwik stats are being collected again now, so closing this ticket as fixed, thanks for you help on this.