CRIN Trac: Ticket #3: Migrate ownCloud from GreenQloud

hours changed; totalhours set

chris — Fri, 01 May 2015 14:08:57 GMT

hours changed from 0 to 1.6
totalhours set to 1.6

Following the install instructions:

For Debian 8.0 run the following as root:
echo 'deb http://download.opensuse.org/repositories/isv:/ownCloud:/community/Debian_8.0/ /' >> /etc/apt/sources.list.d/owncloud.list
apt-get update
apt-get install owncloud
You can add the repository key to apt. Keep in mind that the owner of the key may distribute updates, packages and repositories that your system will trust (more information). To add the key, run:
wget http://download.opensuse.org/repositories/isv:ownCloud:community/Debian_8.0/Release.key
apt-key add - < Release.key

So /etc/apt/sources.list.d/owncloud.list was created containing:

# See https://trac.crin.org/trac/ticket/3
deb http://download.opensuse.org/repositories/isv:/ownCloud:/community/Debian_8.0/ /

And the GPG key was installed:

wget http://download.opensuse.org/repositories/isv:ownCloud:community/Debian_8.0/Release.key -O opensuse.asc
cat opensuse.asc | apt-key add -
  OK

These ownCloud packges are now available:

aptitude upadate
aptitude search owncloud
p   libowncloudsync-dev                                 - development files for owncloudsync
p   libowncloudsync0                                    - ownCloudSync folder synchronization - libraries
p   owncloud                                            - ownCloud Server - Private file sync and share server
p   owncloud-3rdparty                                   - The ownCloud app owncloud-3rdparty
p   owncloud-app-activity                               - The ownCloud app owncloud-app-activity
p   owncloud-app-external                               - The ownCloud app owncloud-app-external
p   owncloud-app-files                                  - The ownCloud app owncloud-app-files
p   owncloud-app-files-encryption                       - The ownCloud app owncloud-app-files-encryption
p   owncloud-app-files-external                         - The ownCloud app owncloud-app-files-external
p   owncloud-app-files-locking                          - The ownCloud app owncloud-app-files-locking
p   owncloud-app-files-pdfviewer                        - The ownCloud app owncloud-app-files-pdfviewer
p   owncloud-app-files-sharing                          - The ownCloud app owncloud-app-files-sharing
p   owncloud-app-files-texteditor                       - The ownCloud app owncloud-app-files-texteditor
p   owncloud-app-files-trashbin                         - The ownCloud app owncloud-app-files-trashbin
p   owncloud-app-files-versions                         - The ownCloud app owncloud-app-files-versions
p   owncloud-app-files-videoviewer                      - The ownCloud app owncloud-app-files-videoviewer
p   owncloud-app-firstrunwizard                         - The ownCloud app owncloud-app-firstrunwizard
p   owncloud-app-gallery                                - The ownCloud app owncloud-app-gallery
p   owncloud-app-provisioning-api                       - The ownCloud app owncloud-app-provisioning-api
p   owncloud-app-templateeditor                         - The ownCloud app owncloud-app-templateeditor
p   owncloud-app-updater                                - The ownCloud app owncloud-app-updater
p   owncloud-app-user-external                          - The ownCloud app owncloud-app-user-external
p   owncloud-app-user-ldap                              - The ownCloud app owncloud-app-user-ldap
p   owncloud-app-user-webdavauth                        - The ownCloud app owncloud-app-user-webdavauth
p   owncloud-apps                                       - third party applications for ownCloud
p   owncloud-client                                     - folder synchronization with an ownCloud server - GUI
p   owncloud-client-cmd                                 - folder synchronization with an ownCloud server - cmd client
p   owncloud-client-doc                                 - ownCloudSync folder synchronization - documentation
p   owncloud-client-l10n                                - ownCloudSync folder synchronization - localization
p   owncloud-config-apache                              - The ownCloud configuration for the apache web server
p   owncloud-doc                                        - documentation for ownCloud
p   owncloud-server                                     - The owncloud-server package
v   owncloud-server-core                                -

The packages installed on web2.crin.org are:

aptitude search owncloud | grep ^i
i   owncloud                        - ownCloud Server - Private file sync and sh
i A owncloud-3rdparty               - The ownCloud app owncloud-3rdparty
i A owncloud-app-activity           - The ownCloud app owncloud-app-activity
i A owncloud-app-external           - The ownCloud app owncloud-app-external
i A owncloud-app-files              - The ownCloud app owncloud-app-files
i A owncloud-app-files-encryption   - The ownCloud app owncloud-app-files-encryp
i A owncloud-app-files-external     - The ownCloud app owncloud-app-files-extern
i A owncloud-app-files-locking      - The ownCloud app owncloud-app-files-lockin
i A owncloud-app-files-pdfviewer    - The ownCloud app owncloud-app-files-pdfvie
i A owncloud-app-files-sharing      - The ownCloud app owncloud-app-files-sharin
i A owncloud-app-files-texteditor   - The ownCloud app owncloud-app-files-texted
i A owncloud-app-files-trashbin     - The ownCloud app owncloud-app-files-trashb
i A owncloud-app-files-versions     - The ownCloud app owncloud-app-files-versio
i A owncloud-app-files-videoviewer  - The ownCloud app owncloud-app-files-videov
i A owncloud-app-firstrunwizard     - The ownCloud app owncloud-app-firstrunwiza
i A owncloud-app-gallery            - The ownCloud app owncloud-app-gallery
i A owncloud-app-provisioning-api   - The ownCloud app owncloud-app-provisioning
i A owncloud-app-templateeditor     - The ownCloud app owncloud-app-templateedit
i A owncloud-app-user-external      - The ownCloud app owncloud-app-user-externa
i A owncloud-app-user-ldap          - The ownCloud app owncloud-app-user-ldap
i A owncloud-app-user-webdavauth    - The ownCloud app owncloud-app-user-webdava
i A owncloud-config-apache          - The ownCloud configuration for the apache
i A owncloud-server                 - The owncloud-server package

So these were installed:

aptitude install owncloud owncloud-3rdparty owncloud-app-activity owncloud-app-external owncloud-app-files owncloud-app-files-encryption owncloud-app-files-external owncloud-app-files-locking owncloud-app-files-pdfviewer owncloud-app-files-sharing owncloud-app-files-texteditor owncloud-app-files-trashbin owncloud-app-files-versions owncloud-app-files-videoviewer owncloud-app-firstrunwizard owncloud-app-gallery owncloud-app-provisioning-api owncloud-app-templateeditor owncloud-app-user-external owncloud-app-user-ldap owncloud-app-user-webdavauth owncloud-config-apache owncloud-server
  The following NEW packages will be installed:
    curl{a} libmcrypt4{a} libpq5{a} mysql-server{a} owncloud owncloud-3rdparty owncloud-app-activity
    owncloud-app-external owncloud-app-files owncloud-app-files-encryption owncloud-app-files-external
    owncloud-app-files-locking owncloud-app-files-pdfviewer owncloud-app-files-sharing owncloud-app-files-texteditor
    owncloud-app-files-trashbin owncloud-app-files-versions owncloud-app-files-videoviewer
    owncloud-app-firstrunwizard owncloud-app-gallery owncloud-app-provisioning-api owncloud-app-templateeditor
    owncloud-app-user-external owncloud-app-user-ldap owncloud-app-user-webdavauth owncloud-config-apache
    owncloud-server php5{a} php5-mcrypt{a} php5-pgsql{a} php5-sqlite{a}
  0 packages upgraded, 31 newly installed, 0 to remove and 0 not upgraded.
  Need to get 17.8 MB of archives. After unpacking 73.1 MB will be used.
  Do you want to continue? [Y/n/?] y

The install has enabled /etc/apache/conf-available/owncloud.conf via the /etc/apache/conf-enabled/owncloud.conf symlink, we don't want /owncloud to work for all domains on the server so that symlink was deleted, the file contains:

Alias /owncloud "/var/www/owncloud/"
<Directory "/var/www/owncloud">
    Options +FollowSymLinks
    AllowOverride All
</Directory>

An apache config was created at /etc/apache2/sites-available/owncloud.conf containing:

<VirtualHost *:80>
        <IfModule mpm_itk_module>
                AssignUserID owncloud owncloud
                MaxClientsVHost 60
        </IfModule>
        ServerName cloud.crin.org
        ServerAlias www.cloud.crin.org
        ServerAlias cloud.crin1.crin.org
        <If "%{HTTP_HOST} == 'cloud.crin.org'">
                Redirect / https://cloud.crin.org/
        </If>
        <If "%{HTTP_HOST} == 'www.cloud.crin.org'">
                Redirect / https://cloud.crin.org/
        </If>
        <If "%{HTTP_HOST} == 'cloud.crin1.crin.org'">
                Redirect / https://cloud.crin1.crin.org/
        </If>
</VirtualHost>
<IfModule mod_ssl.c>
<VirtualHost *:443>
        <IfModule mpm_itk_module>
                AssignUserID owncloud owncloud
                MaxClientsVHost 60
        </IfModule>
        ServerAdmin chris@webarchitects.co.uk
        ServerName cloud.crin.org
        ServerAlias www.cloud.crin.org
        ServerAlias cloud.crin1.crin.org
        SSLEngine on
        SSLCertificateFile    /etc/ssl/gandi/cloud.crt.pem
        SSLCertificateKeyFile /etc/ssl/gandi/cloud.key.pem
        SSLCACertificateFile  /etc/ssl/gandi/root.pem
        SetEnv TMPDIR /var/www/owncloud/tmp
        DocumentRoot /var/www/owncloud
        <Directory />
                Options FollowSymLinks
                AllowOverride None
                Require all denied
        </Directory>
        <Directory /var/www/owncloud>
                Options Indexes FollowSymLinks
                AllowOverride All
                Require all granted
        </Directory>
        <IfModule mod_php5.c>
                php_admin_value open_basedir /var/www/owncloud
                php_admin_value upload_tmp_dir /var/www/owncloud/tmp
                php_admin_value session.save_path /var/www/owncloud/tmp
                php_admin_value file_uploads 1
        </IfModule>
        ErrorLog ${APACHE_LOG_DIR}/owncloud.error.log
        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn
        CustomLog ${APACHE_LOG_DIR}/owncloud.ssl_access.log combined
        <IfModule headers_module>
                # Use HTTP Strict Transport Security to force client to use secure connections only
                #Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
                Header always set Strict-Transport-Security "max-age=31536000"
                # mitigate TIME attack
                Header always append X-Frame-Options "sameorigin"
        </IfModule>
        <FilesMatch "\.(cgi|shtml|phtml|php)$">
                SSLOptions +StdEnvVars
        </FilesMatch>
        <Directory /usr/lib/cgi-bin>
                SSLOptions +StdEnvVars
        </Directory>
        BrowserMatch "MSIE [2-6]" \
                nokeepalive ssl-unclean-shutdown \
                downgrade-1.0 force-response-1.0
        # MSIE 7 and newer should be able to use keepalive
        BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
</VirtualHost>
</IfModule>

And then this was symlinked and tested:

cd /etc/apache2/sites-enabled
ln -s ../sites-available/owncloud.conf 20-owncloud.conf
apache2ctl configtest
  AH00543: apache2: bad user name owncloud
  Action 'configtest' failed.
  The Apache error log may have more information.

So the owncloud user was created:

adduser --system --disabled-password --disabled-login --group --home=/var/www/owncloud owncloud
  Warning: The home dir /var/www/owncloud you specified already exists.
  Adding system user `owncloud' (UID 113) ...
  Adding new group `owncloud' (GID 118) ...
  Adding new user `owncloud' (UID 113) with group `owncloud' ...
  The home directory `/var/www/owncloud' already exists.  Not copying from `/etc/skel'.
  adduser: Warning: The home directory `/var/www/owncloud' does not belong to the user you are currently creating.
apache2ctl configtest
  Syntax OK

Creat a TMPDIR:

mkdir /var/www/owncloud/tmp
chown owncloud:owncloud /var/www/owncloud/tmp/

Setup vim for the owncloud user and enable history:

touch /var/www/owncloud/.vimrc ;chown owncloud:owncloud /var/www/owncloud/.vimrc
echo "syntax on" > /var/www/owncloud/.vimrc ; chown owncloud:owncloud /var/www/owncloud/.vimrc
touch /var/www/owncloud/.bash_history ;chown owncloud:owncloud /var/www/owncloud/.bash_history

So now sync the data and create a script for future data syncs that will be needed before the DNS update.

First create a MySQL database for the site:

mysql mysql
 mysql> CREATE DATABASE owncloud;
 mysql> GRANT ALL ON owncloud.* to 'owncloud'@'localhost' identified by 'XXX';
 mysql> FLUSH PRIVILEGES;

Initial sync of files:

rsync -av web2:/var/www/owncloud/apps/ /var/www/owncloud/apps/
rsync -av web2:/var/www/owncloud/config/ /var/www/owncloud/config/
rsync -av web2:/var/www/owncloud/data/ /var/www/owncloud/data/

Update the /var/www/owncloud/config/config.php with the correct MySQL details.

Create a script for future sycncs, /root/bin/owncloud-sync:

#!/bin/bash
rsync -av web2:/var/www/owncloud/apps/ /var/www/owncloud/apps/
rsync -av web2:/var/www/owncloud/config/ /var/www/owncloud/config/
rsync -av web2:/var/www/owncloud/data/ /var/www/owncloud/data/
chown -R owncloud:owncloud  /var/www/owncloud/apps/
chown -R owncloud:owncloud  /var/www/owncloud/config/
chown -R owncloud:owncloud  /var/www/owncloud/data/
ssh web2 "mysqldump -uowncloud -pXXX cloud > /root/owncloud.sql"
scp web2:owncloud.sql /root/
cat /root/owncloud.sql | mysql owncloud

And run it.

Check for database updates:

su - owncloud -s /bin/bash
php console.php upgrade
  PHP is configured to populate raw post data. Since PHP 5.6 this will lead to PHP throwing notices for perfectly valid code.
  To fix this issue set <code>always_populate_raw_post_data</code> to <code>-1</code> in your php.ini

So that was added to the Apache config for the site and also to /etc/php5/cli/php.ini, try again:

su - owncloud -s /bin/bash
php console.php upgrade
  ownCloud is already latest version

Restart Apache and test at https://cloud.crin1.crin.org/

And the login doesn't work, no error, it just doesn't log one in.

This variables was changed in /var/www/owncloud/config/config.php just in case that helped:

  'overwrite.cli.url' => 'https://cloud.crin1.crin.org',

But it don't help.

I think I need to look up documentation on migrating sites / and or how to change the admin password to solve this.

hours, totalhours changed

chris — Sat, 02 May 2015 10:38:14 GMT

hours changed from 0 to 0.5
totalhours changed from 1.6 to 2.1

There is a document Migrating to a Different Server which only differs in as much as it advises that the site should first be put into "maintenance mode".

The site can be put into "maintenance mode" using the occ comand line interface, see Using the occ Command.

So trying that, on the live server:

sudo -i
su - cloud -s /bin/bash
php occ maintenance:mode --on
Maintenance mode enabled

Check the status at https://cloud.crin.org/ then run the data sync command on Crin1:

sudo -i
bash bin/owncloud-sync

Now the site at https://cloud.crin1.crin.org/ works (no idea why it wasn't yesterday) but isn't in "maintenance mode"...

Migration script updated to add these steps:

#!/bin/bash
# enable maintainance mode on remote server
ssh web2 "sudo -u cloud php /var/www/owncloud/occ maintenance:mode --on"
rsync -av web2:/var/www/owncloud/apps/ /var/www/owncloud/apps/
#rsync -av web2:/var/www/owncloud/config/ /var/www/owncloud/config/
rsync -av web2:/var/www/owncloud/data/ /var/www/owncloud/data/
chown -R owncloud:owncloud  /var/www/owncloud/apps/
chown -R owncloud:owncloud  /var/www/owncloud/config/
chown -R owncloud:owncloud  /var/www/owncloud/data/
ssh web2 "mysqldump -uowncloud -pXXX owncloud > /root/owncloud.sql"
scp web2:owncloud.sql /root/
cat /root/owncloud.sql | mysql owncloud
# disable maintainance mode on remote server
ssh web2 "sudo -u cloud php /var/www/owncloud/occ maintenance:mode --on"

Maintainance mode on the live server was switched off.

The admin interface displays:

Configuration Checks No problems found

So I think this application is now ready to be migrated.

A start has been made on documenting ownCloud at wiki:OwnCloud.

hours, estimatedhours, totalhours changed

chris — Thu, 07 May 2015 10:57:53 GMT

hours changed from 0 to 0.05
estimatedhours changed from 0 to 2.5
totalhours changed from 2.1 to 2.15

2h 6m have been spent on this ticket so far, it's now ready for testing at https://cloud.crin1.crin.org/ and if everything is good it can be quickly updated prior to a DNS switch using the /root/bin/owncloud-sync script on Crin1, I'm adding a total estimated time to this ticket of 2.5 hours.

jonas — Thu, 07 May 2015 11:23:16 GMT

I tried ownCloud and it opens and displays all the files. However, opening files is very slow. Could have to do with my connection as well though..

hours, totalhours changed

chris — Fri, 05 Jun 2015 17:12:17 GMT

hours changed from 0 to 0.25
totalhours changed from 2.15 to 2.4

Doing a final sync before the DNS update on Crin1:

sudo -i
bash bin/owncloud-sync

Update ownCloud and other Debian packages:

a-up
su - owncloud -s /bin/bash
php console.php upgrade
  Turned on maintenance mode
  Checked database schema update
  Checked database schema update for apps
  Updated database
  Disabled 3rd-party app: documents
  Turned off maintenance mode
  Update successful

Tried to update the DNS but I don't have the right username and password, so will come back to this later.

cc changed

chris — Fri, 05 Jun 2015 17:13:14 GMT

cc jenny gillian added; jonas removed

Added Jenny and Gillian as CCs for this ticket.

hours, status, totalhours changed; resolution set

chris — Fri, 05 Jun 2015 18:39:31 GMT

hours changed from 0 to 0.15
status changed from new to closed
resolution set to fixed
totalhours changed from 2.4 to 2.55

I have updated the dns for ownCloud so in a while accessing https://cloud.crin.org/ should result in people getting the Crin1 server. The 1984.is DNS have updated:

dig @NS0.1984.IS cloud.crin.org +short
  93.95.228.179

The site on the GreenCloud server is in maintenance mode, so this ticket is now being closed, if there are any problems new tickets can be opened.