CRIN Trac: Ticket #10: Install Munin server and clients

hours changed; totalhours set

chris — Mon, 11 May 2015 12:26:31 GMT

hours changed from 0 to 0.86
totalhours set to 0.86

Install the client and server on Crin1:

aptitude install munin munin-node munin-plugins-extra munin-plugins-java munin-plugins-java

On Crin2:

aptitude install munin-node

A sub-domain, munin.crin.org was added to the DNS servers.

An Apache config files was created at /etc/apache2/sites-available/munin.conf containing:

<VirtualHost *:80>
        <IfModule mpm_itk_module>
                AssignUserID munin munin
                MaxClientsVHost 60
        </IfModule>
        ServerName munin.crin.org
        ServerAlias munin.crin1.crin.org
        ServerAdmin chris@webarchitects.co.uk
        <If "%{HTTP_HOST} == 'munin.crin1.crin.org'">
                Redirect / https://munin.crin1.crin.org/
        </If>
        Redirect / https://munin.crin.org/
</VirtualHost>
<IfModule mod_ssl.c>
        <VirtualHost *:443>
        <IfModule mpm_itk_module>
                AssignUserID munin munin
                MaxClientsVHost 60
        </IfModule>
        ServerName munin.crin.org
        ServerAlias munin.crin1.crin.org
        ServerAdmin chris@webarchitects.co.uk
        SSLEngine on
        SSLCertificateFile    /etc/ssl/cacert/crin1_cert.pem
        SSLCertificateKeyFile /etc/ssl/cacert/crin1_privatekey.pem
        SSLCACertificateFile  /etc/ssl/cacert/cacert.pem
Alias /munin /var/cache/munin/www
<Directory /var/cache/munin/www>
        Require all granted
        Options None
</Directory>
ScriptAlias /munin-cgi-graph /usr/lib/munin/cgi/munin-cgi-graph
<Location /munin-cgi-graph>
        Require all granted
        <IfModule mod_fcgid.c>
            SetHandler fcgid-script
        </IfModule>
        <IfModule !mod_fcgid.c>
            SetHandler cgi-script
        </IfModule>
</Location>
        RedirectMatch ^/$ https://munin.crin.org/munin
        Alias /munin/static /var/cache/munin/www/static
        ScriptAlias /munin-cgi-graph /usr/lib/munin/cgi/munin-cgi-graph
        ScriptAlias /munin /usr/lib/munin/cgi/munin-cgi-html
        LogLevel error
        ErrorLog ${APACHE_LOG_DIR}/munin_ssl_error.log
        CustomLog ${APACHE_LOG_DIR}/munin_ssl_access.log combined

Enabled, tested and restarted:

cd /etc/apache2/sites-enabled
ln -s ../sites-available/munin.conf 50-munin.conf
a2enmod cgi
apache2ctl configtest
  Syntax OK
apache2ctl restart

And at https://munin.crin1.crin.org/munin we have:

Can't open /var/log/munin/munin-cgi-html.log (Permission denied) at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 124.

So:

chown -R munin:munin /var/log/munin

Next to configure the Munin server and clients, on Crin1 edit /etc/munin/munin.conf and add:

#graph_strategy cron
graph_strategy cgi
#cgiurl_graph /munin-cgi/munin-cgi-graph
cgiurl_graph /munin-cgi-graph
html_strategy cgi
[crin1.crin.org]
     address 127.0.0.1
     use_node_name yes
[crin2.crin.org]
     address 93.95.228.180
     use_node_name yes

And we now have some stats at https://munin.crin.org/munin/crin.org/crin1.crin.org/index.html

On Crin2 open the firewall for Munin, add the following to /etc/iptables/rules.v4:

# munin allow 93.95.228.179
-A INPUT -p tcp -s 93.95.228.179 --dport 4949 -j ACCEPT
-A INPUT -p tcp --dport 4949 -j DROP

Reload firewall:

iptables-restore < /etc/iptables/rules.v4
iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
fail2ban-ssh  tcp  --  anywhere             anywhere             multiport dports ssh
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             loopback/8           reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     tcp  --  crin1                anywhere             tcp dpt:munin
DROP       tcp  --  anywhere             anywhere             tcp dpt:munin
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
LOG        all  --  anywhere             anywhere             limit: avg 5/min burst 5 LOG level debug prefix "iptables denied: "
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
Chain fail2ban-ssh (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

On Crin2 edit /etc/munin/munin-node.conf and add:

allow ^93\.95\.228\.179$

Restart the client:

service munin-node restart

However Crin2 isn't showing up at https://munin.crin.org/munin/ and also the graphs that are generated for Crin1 have numbers but no graphical representation so some more work is needed on this, in addition to the enabling of modules for things like MySQL.

hours, totalhours changed

chris — Mon, 11 May 2015 13:21:52 GMT

hours changed from 0 to 0.52
totalhours changed from 0.86 to 1.38

Checking the Munin install on Crin1:

sudo -i
su - munin -s /bin/bash
cd /etc/munin/plugins
munin-check *
  check /var/cache/munin/www
  check /var/lib/munin/cgi-tmp
  ls: cannot access /var/lib/munin/cgi-tmp/*: No such file or directory
  # /var/lib/munin/cgi-tmp/* : Wrong owner ( != munin)
  check /var/lib/munin/crin.org
  check /var/lib/munin/datafile
  check /var/lib/munin/datafile.storable
  check /var/lib/munin/graphs
  check /var/lib/munin/htmlconf.storable
  check /var/lib/munin/limits
  check /var/lib/munin/limits.storable
  check /var/lib/munin/localdomain
  check /var/lib/munin/munin-graph.stats
  check /var/lib/munin/munin-update.stats
  check /var/lib/munin/state-crin.org-crin1.crin.org.storable
  check /var/lib/munin/state-crin.org-crin2.crin.org.storable
  check /var/lib/munin/state-localdomain-localhost.localdomain.storable
  check miscellaneous
  # /var/lib/munin-node/plugin-state : Wrong owner (root != nobody)
  # /var/lib/munin-node/plugin-state : Wrong permissions (755 != 775)
  # /etc/munin/plugin-conf.d : Wrong permissions (750 != 755)
  Check done.  Please note that this script only checks most things,
  not all things.
  Please also note that this script may be buggy.

So change a few things:

chown -R munin:munin /var/lib/munin/cgi-tmp/
chown munin:munin /var/lib/munin-node/plugin-state
chmod 755 /var/lib/munin-node/plugin-state

Still no joy, things work on the command line:

cd /etc/munin/plugins
  munin-run cpu
  user.value 334992
  nice.value 3496
  system.value 103674
  idle.value 1201065630
  iowait.value 169340
  irq.value 6
  softirq.value 7112
  steal.value 6535
  guest.value 0

But no graph: https://munin.crin.org/munin/crin.org/crin1.crin.org/cpu.html

Testing with telnet, as per this suggestion:

telnet localhost 4949
  Trying ::1...
  Connected to localhost.
  Escape character is '^]'.
  # munin node at crin1
nodes
  crin1
  .
list crin1
  apache_accesses apache_processes apache_volume cpu df df_inode entropy exim_mailqueue exim_mailstats fail2ban forks fw_conntrack fw_forwarded_local fw_packets http_loadtime if_err_eth0 if_eth0 interrupts ip_93.95.228.180 irqstats load memory munin_stats netstat nfs4_client nfs_client nfsd nfsd4 ntp_kernel_err ntp_kernel_pll_freq ntp_kernel_pll_off ntp_offset open_files open_inodes proc_pri processes swap threads uptime users vmstat
fetch df
  _dev_dm_0.value 5.65480745635344
  _run.value 6.09743476010941
  _dev_shm.value 0
  _run_lock.value 0
  _sys_fs_cgroup.value 0
  _dev_sda1.value 14.5380714213827
  .
quit
Connection closed by foreign host.
telnet 93.95.228.180 4949
  Trying 93.95.228.180...
  Connected to 93.95.228.180.
  Escape character is '^]'.
  # munin node at crin2
nodes
  crin2
  .
list crin2
  cpu df df_inode entropy exim_mailqueue exim_mailstats fail2ban forks fw_conntrack fw_forwarded_local fw_packets http_loadtime if_err_eth0 if_eth0 interrupts irqstats load memory netstat nfs4_client nfs_client nfsd nfsd4 ntp_kernel_err ntp_kernel_pll_freq ntp_kernel_pll_off ntp_offset open_files open_inodes proc_pri processes swap threads uptime users vmstat
fetch df
  _dev_dm_0.value 18.1259598359727
  _run.value 4.07777038515561
  _dev_shm.value 0
  _run_lock.value 0
  _sys_fs_cgroup.value 0
  _dev_sda1.value 14.5363211117966
.
quit
Connection closed by foreign host.

Trying this suggestion:

su - munin -s /bin/bash
/usr/share/munin/munin-update --nofork --debug

That generates lots of output but nothing that helps.

Checking the crontab at /etc/cron.d/munin, it contains:

#
# cron-jobs for munin
#
MAILTO=root
*/5 * * * *     munin if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi
14 10 * * *     munin if [ -x /usr/share/munin/munin-limits ]; then /usr/share/munin/munin-limits --force --contact nagios --contact old-nagios; fi

I'm at a bit of a loss, I'll continue trying to track down the problem(s) tomorrow.

hours, totalhours changed

chris — Tue, 12 May 2015 14:10:17 GMT

hours changed from 0 to 1
totalhours changed from 1.38 to 2.38

I removed, purged and reinstalled the munin clients and servers on both Crin1 and Crin2 and now static graphs are being generated but not dynamic ones.

Apache mod fast cgi was installed:

aptitude install libapache2-mod-fcgid

Still don't have dynamic graphs, very sprry that this is taking a lot longer than expected to configure.

Next thing to try is reading the documentation:

http://munin-monitoring.org/wiki/MuninConfigurationMasterCGI

hours, totalhours changed

chris — Wed, 13 May 2015 10:10:20 GMT

hours changed from 0 to 0.5
totalhours changed from 2.38 to 2.88

So after enabling Munin to be served over HTTP and chaning the Apache log level to debug I found the issue:

[Wed May 13 10:00:19.865357 2015] [fcgid:debug] [pid 8342] fcgid_proc_unix.c(542): (13)Permission denied: [client XX.XX.XX.XX:53224] mod_fcgid: can't connect unix domain socket: /var/lib/apache2/fcgid/sock/8309.17, referer: http://munin.crin.org/

So the VirtualHost was changed to run as www-data rather than munin, this is the final Apache configuration:

<VirtualHost *:80>
        <IfModule mpm_itk_module>
                AssignUserID www-data www-data
                MaxClientsVHost 60
        </IfModule>
        ServerName munin.crin.org
        ServerAlias munin.crin1.crin.org
        ServerAdmin chris@webarchitects.co.uk
        <If "%{HTTP_HOST} == 'munin.crin1.crin.org'">
                Redirect / https://munin.crin1.crin.org/
        </If>
        Redirect / https://munin.crin.org/
        LogLevel error
        ErrorLog ${APACHE_LOG_DIR}/munin_error.log
        CustomLog ${APACHE_LOG_DIR}/munin_access.log combined
</VirtualHost>
<IfModule mod_ssl.c>
        <VirtualHost *:443>
        <IfModule mpm_itk_module>
                AssignUserID www-data www-data
                MaxClientsVHost 60
        </IfModule>
        ServerName munin.crin.org
        ServerAlias munin.crin1.crin.org
        ServerAdmin chris@webarchitects.co.uk
        SSLEngine on
        SSLCertificateFile    /etc/ssl/cacert/crin1_cert.pem
        SSLCertificateKeyFile /etc/ssl/cacert/crin1_privatekey.pem
        SSLCACertificateFile  /etc/ssl/cacert/cacert.pem
        RedirectMatch ^/$ https://munin.crin.org/munin
        Alias /munin/static /etc/munin/static
        <Directory /etc/munin/static/>
                AllowOverride None
                Options -Indexes -ExecCGI -MultiViews
                Require all granted
        </Directory>
        ScriptAlias /munin /usr/lib/munin/cgi/munin-cgi-html
        ScriptAlias /munin-cgi-graph /usr/lib/munin/cgi/munin-cgi-graph
        <Directory /usr/lib/munin/cgi>
                AllowOverride None
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                SSLOptions +StdEnvVars
                Require all granted
                <IfModule mod_fcgid.c>
                        SetHandler fcgid-script
                </IfModule>
         </Directory>
        <Directory /var/cache/munin/www/static>
                AllowOverride None
                Options -Indexes -ExecCGI -MultiViews
                Require all granted
                <IfModule mod_expires.c>
                        ExpiresActive On
                        ExpiresDefault M310
                </IfModule>
        </Directory>
        LogLevel error
        ErrorLog ${APACHE_LOG_DIR}/munin_ssl_error.log
        CustomLog ${APACHE_LOG_DIR}/munin_ssl_access.log combined
</VirtualHost>

And for munin.conf:

graph_strategy cgi
cgiurl_graph /munin-cgi-graph
html_strategy cgi

The remaining task on this ticket is to disable plugins we don't need and enable ones we need but are not yet using.

hours, totalhours changed

chris — Wed, 13 May 2015 13:04:19 GMT

hours changed from 0 to 0.5
totalhours changed from 2.88 to 3.38

Looking at the Munin graphs we have, these plugins were automatically configured, some we don't want, some extra ones are wanted:

We don't need NFS, (NFS graphs are being generated), so on both servers:

aptitude remove nfs-common rpcbind
rm /etc/munin/plugins/nfs*

On Crin1 we want MySQL graphs, so using the new mysql_ plugin:

cd /etc/munin/plugins
/usr/share/munin/plugins/mysql_ suggest
  Missing dependency Cache::Cache at /usr/share/munin/plugins/mysql_ line 728.
aptitude install libcache-cache-perl
/usr/share/munin/plugins/mysql_ suggest
  bin_relay_log
  commands
  connections
  files_tables
  innodb_bpool
  innodb_bpool_act
  innodb_insert_buf
  innodb_io
  innodb_io_pend
  innodb_log
  innodb_rows
  innodb_semaphores
  innodb_tnx
  myisam_indexes
  network_traffic
  qcache
  qcache_mem
  replication
  select_types
  slow
  sorts
  table_locks
  tmp_tables
ln -s /usr/share/munin/plugins/mysql_ mysql_bin_relay_log
ln -s /usr/share/munin/plugins/mysql_ mysql_commands
ln -s /usr/share/munin/plugins/mysql_ mysql_connections
ln -s /usr/share/munin/plugins/mysql_ mysql_files_tables
ln -s /usr/share/munin/plugins/mysql_ mysql_innodb_bpool
ln -s /usr/share/munin/plugins/mysql_ mysql_innodb_bpool_act
ln -s /usr/share/munin/plugins/mysql_ mysql_innodb_insert_buf
ln -s /usr/share/munin/plugins/mysql_ mysql_innodb_io
ln -s /usr/share/munin/plugins/mysql_ mysql_innodb_io_pend
ln -s /usr/share/munin/plugins/mysql_ mysql_innodb_log
ln -s /usr/share/munin/plugins/mysql_ mysql_innodb_rows
ln -s /usr/share/munin/plugins/mysql_ mysql_innodb_semaphores
ln -s /usr/share/munin/plugins/mysql_ mysql_innodb_tnx
ln -s /usr/share/munin/plugins/mysql_ mysql_myisam_indexes
ln -s /usr/share/munin/plugins/mysql_ mysql_network_traffic
ln -s /usr/share/munin/plugins/mysql_ mysql_qcache
ln -s /usr/share/munin/plugins/mysql_ mysql_qcache_mem
ln -s /usr/share/munin/plugins/mysql_ mysql_replication
ln -s /usr/share/munin/plugins/mysql_ mysql_select_types
ln -s /usr/share/munin/plugins/mysql_ mysql_slow
ln -s /usr/share/munin/plugins/mysql_ mysql_sorts
ln -s /usr/share/munin/plugins/mysql_ mysql_table_locks
ln -s /usr/share/munin/plugins/mysql_ mysql_tmp_tables
service munin-node restart

The MySQL graphs:

https://munin.crin.org/munin/crin.org/crin1.crin.org/index.html#mysql2

We want to track the number of and memory use of key processes, as apache2 and mysqld so enable multips and multips_memory, on Crin1:

cd /etc/munin/plugins
ln -s /usr/share/munin/plugins/multips
ln -s /usr/share/munin/plugins/multips_memory

Add the following to /etc/munin/plugin-conf.d/munin-node:

[multips]
env.names apache2 mysqld
[multips_memory]
env.names apache2 mysqld

On Crin2:

[multips]
env.names php5-fpm java nginx
[multips_memory]
env.names php5-fpm java nginx

And the graphs:

That is probably enough plugin configuration for now.

hours, status, totalhours changed; resolution set

chris — Wed, 13 May 2015 13:16:44 GMT

hours changed from 0 to 0.14
status changed from new to closed
resolution set to fixed
totalhours changed from 3.38 to 3.52

I have created Munin and CAcert pages to document the Munin install, closing this ticket now.

hours, totalhours changed

chris — Wed, 13 May 2015 21:48:41 GMT

hours changed from 0 to 0.25
totalhours changed from 3.52 to 3.77

Forgot to add Nginx graphs for Crin2, so on that machine:

cd /etc/munin/plugins
ln -s /usr/share/munin/plugins/nginx_status
ln -s /usr/share/munin/plugins/nginx_request
munin-run nginx_status
  total.value U
  reading.value U
  writing.value U
  waiting.value U
munin-run nginx_request
  request.value U

Reading the scripts we have:

This shows the default configuration of this plugin.  You can override
the status URL.
  [nginx*]
      env.url http://localhost/nginx_status
Nginx must also be configured.  Firstly the stub-status module must be
compiled, and secondly it must be configured like this:
  server {
        listen 127.0.0.1;
        server_name localhost;
        location /nginx_status {
                stub_status on;
                access_log   off;
                allow 127.0.0.1;
                deny all;
        }
  }

So /etc/nginx/sites-available/localhost was created containing the above. It was symlinked and tested:

cd /etc/nginx/sites-enabled/
ln -s ../sites-available/localhost 30-localhost
service nginx configtest
  [ ok ] Testing nginx configuration:.
service nginx restart
lynx -dump http://localhost/nginx_status
  Active connections: 1
  server accepts handled requests
   1 1 1
  Reading: 0 Writing: 1 Waiting: 0
cd /etc/munin/plugins/
munin-run nginx_request
  request.value 3
munin-run nginx_status
  total.value 1
  reading.value 0
  writing.value 1
  waiting.value 0
service munin-node restart

It's working so we should soon have Nginx stats here:

https://munin.crin.org/munin/crin.org/crin2.crin.org/index.html

hours, totalhours changed

chris — Wed, 13 May 2015 23:02:53 GMT

hours changed from 0 to 0.4
totalhours changed from 3.77 to 4.17

Install the memcache plugin, on Crin2:

cd  /etc/munin/plugins
ln -s /usr/share/munin/plugins/memcached_ memcached_rates
ln -s /usr/share/munin/plugins/memcached_ memcached_bytes
ln -s /usr/share/munin/plugins/memcached_ memcached_counters
munin-run memcached_rates
  no (Cache::Memcached not found)
aptitude install libcache-memcached-perl
  munin-run memcached_rates
  memcache_cache_hits.value 41588
  memcache_cache_misses.value 3746
  memcache_cmd_get.value 45334
  memcache_cmd_set.value 11077
  memcache_total_connections.value 197
  memcache_total_items.value 11077
memcached_bytes
  memcache_bytes_read.value 15691427
  memcache_bytes_written.value 49907613
memcached_counters
  memcache_bytes_allocated.value 1931023
  memcache_curr_connections.value 5
  memcache_curr_items.value 947
service munin-node restart

So we should soon have memcache stats.

Next php-fpm, there seem to be several options available, trying tjstein's:

mkdir -p /usr/local/share/munin/plugins
cd /usr/local/share/munin/plugins
git clone git://github.com/tjstein/php5-fpm-munin-plugins.git
  Cloning into 'php5-fpm-munin-plugins'...
  remote: Counting objects: 191, done.
  remote: Total 191 (delta 0), reused 0 (delta 0), pack-reused 191
  Receiving objects: 100% (191/191), 25.50 KiB | 0 bytes/s, done.
  Resolving deltas: 100% (105/105), done.
  Checking connectivity... done.
chmod +x php5-fpm-munin-plugins/phpfpm_*
chown -R root:root php5-fpm-munin-plugins/
cd /etc/munin/plugins/
ln -s /usr/local/share/munin/plugins/php5-fpm-munin-plugins/phpfpm_average
ln -s /usr/local/share/munin/plugins/php5-fpm-munin-plugins/phpfpm_connections
ln -s /usr/local/share/munin/plugins/php5-fpm-munin-plugins/phpfpm_memory
ln -s /usr/local/share/munin/plugins/php5-fpm-munin-plugins/phpfpm_status
ln -s /usr/local/share/munin/plugins/php5-fpm-munin-plugins/phpfpm_processes

Edit /etc/php5/fpm/pool.d/www.conf changing:

;pm.status_path = /status
pm.status_path = /status

Edit /etc/nginx/sites-available/localhost adding:

        location ~ ^/(status|ping)$ {
                fastcgi_pass unix:/var/run/php5-fpm.sock;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_intercept_errors on;
                include fastcgi_params;
                access_log off;
                allow 127.0.0.1;
                deny all;
        }

Test and restart services:

service nginx configtest
  [ ok ] Testing nginx configuration:.
service php5-fpm restart
service munin-node restart

Test:

lynx -dump http://localhost/status
  pool:                 www
  process manager:      dynamic
  start time:           13/May/2015:22:46:56 +0000
  start since:          375
  accepted conn:        1
  listen queue:         0
  max listen queue:     0
  listen queue len:     0
  idle processes:       1
  active processes:     1
  total processes:      2
  max active processes: 1
  max children reached: 0
  slow requests:        0

So, the new stats:

I'm sorry this has taken longer to set up than estimated, but it should pay off in the long run because by being able to monitor the state of the MySQL, Nginx, php-fpm and Memcache processes will enable the tuning of applications better to ensure that the amount of RAM needed is as little as possible to deliver the site as fast as required -- the new servers currently have around 50% of the RAM of the current, live GreenQloud servers.

chris — Wed, 13 May 2015 23:34:32 GMT

The php-fpm munin plugins needed editing like this:

#PHP_BIN=${phpbin-"php5-fpm"}
PHP_BIN=${phpbin-"php-fpm"}

hours, totalhours changed

chris — Thu, 14 May 2015 21:04:13 GMT

hours changed from 0 to 0.05
totalhours changed from 4.17 to 4.22

Munin email alerts were enabled by adding the following to /etc/munin/munin.conf:

contact.me.command mail -s "${var:host} Munin Alert" root@localhost
contact.me.always_send warning critical

hours, totalhours changed

chris — Thu, 14 May 2015 21:27:20 GMT

hours changed from 0 to 0.05
totalhours changed from 4.22 to 4.27

PHP-APC Munin plugins were installed on ticket:6#comment:19 and the php_apc_purge graph was generating an email every 5 mins:

Date: Thu, 14 May 2015 21:20:16 +0000
From: munin application user <munin@crin1.crin.org>
To: root@localhost
Subject: crin2.crin.org Munin Alert
crin.org :: crin2.crin.org :: Purge rate
        WARNINGs: Optcode Cache is 100.00 (outside range [:10]).
        OKs: User Cache is 0.00.

So it was disabled:

cd /etc/munin/plugins
rm php_apc_purge
service munin-node restart

cc changed

chris — Thu, 18 Jun 2015 11:59:50 GMT

cc jenny gillian added; jonas removed

CCs changed.