CRIN Trac: Ticket #70: s3ql backup error

description changed

chris — Mon, 11 Apr 2016 10:10:21 GMT

description modified (diff)

hours changed; totalhours set

chris — Mon, 11 Apr 2016 10:53:53 GMT

hours changed from 0 to 0.75
totalhours set to 0.75

I missed that s3ql was being upgraded with the other updates on ticket:17#comment:77.

s3qladm upgrade s3c://s.qstack.advania.com:443/crin4
Getting file system parameters..
Using cached metadata.
File system is damaged, need to run fsck before upgrade.
ERROR: Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/bin/s3qladm", line 9, in <module>
    load_entry_point('s3ql==2.17.1+hg2', 'console_scripts', 's3qladm')()
  File "/usr/lib/s3ql/s3ql/adm.py", line 94, in main
    return upgrade(options)
  File "/usr/lib/s3ql/s3ql/common.py", line 514, in wrapper
    return fn(*a, **kw)
  File "/usr/lib/s3ql/s3ql/adm.py", line 264, in upgrade
    print(get_old_rev_msg(param['revision'], 'fsck.s3ql'))
  File "/usr/lib/s3ql/s3ql/adm.py", line 209, in get_old_rev_msg
    ''' % { 'version': REV_VER_MAP[rev],
KeyError: 22
fsck.s3ql s3c://s.qstack.advania.com:443/crin4
Starting fsck of s3c://s.qstack.advania.com:443/crin4/
Using cached metadata.
WARNING: Remote metadata is outdated.
ERROR: File system revision too old, please run `s3qladm upgrade` first.

Not sure what to try next, spent some time reading the list archives and have sent a email to the list.

hours, totalhours changed

chris — Mon, 11 Apr 2016 11:33:17 GMT

hours changed from 0 to 0.23
totalhours changed from 0.75 to 0.98

Checking through the emails sent by the backup jobs and the last good backup appears to have been made on 3rd March 2016 for Crin1 and 4th March 2016 for Crin2 and Crin4, very sorry not to have noticed the errors sooner.

hours, totalhours changed

chris — Tue, 12 Apr 2016 09:19:07 GMT

hours changed from 0 to 0.5
totalhours changed from 0.98 to 1.48

I spent some time last night emailing the list, see the thread, Nikolaus has advised:

Go back to the S3QL version that you used before, and then go forward one minor version after another until you reach 2.17.1. (not every minor version will require a file system update though, for most updates you'll just get a "file system revision is up-to-date" message).

hours, totalhours changed

chris — Tue, 12 Apr 2016 11:20:59 GMT

hours changed from 0 to 0.5
totalhours changed from 1.48 to 1.98

Downgrading s3ql:

sudo -i
aptitude remove s3ql
wget http://snapshot.debian.org/archive/debian/20160229T214851Z/pool/main/s/s3ql/s3ql_2.16%2Bdfsg-1_amd64.deb
shasum s3ql_2.16+dfsg-1_amd64.deb | grep 0fcba074d3f8c69e1841285fee059d2ad6a7793a
aptitude install python3-apsw python3-crypto python3-crypto python3-defusedxml python3-dugong  python3-llfuse python3-requests libjs-sphinxdoc python3-pkg-resources
dpkg -i /root/s3ql_2.16+dfsg-1_amd64.deb

Mounting the file systems:

mnt-s3ql crin1
mnt-s3ql crin2
mnt-s3ql crin4

And waiting for the results...

hours, totalhours changed

chris — Tue, 12 Apr 2016 13:04:09 GMT

hours changed from 0 to 0.5
totalhours changed from 1.98 to 2.48

The filesystems have now mounted OK:

df -h
s3c://s.qstack.advania.com:443/crin4/  1.0T   21G 1004G   2% /media/s3ql/crin4
s3c://s.qstack.advania.com:443/crin1/  1.0T  242G  783G  24% /media/s3ql/crin1
s3c://s.qstack.advania.com:443/crin2/  1.0T   76G  949G   8% /media/s3ql/crin2

So upgrading s3ql and everything else installed the following:

2016-04-12      chris
        *       apt:amd64/stretch apt-listchanges:all/stretch apt-utils:amd64/stretch bind9-host:amd64/stretch console-setup:all/stretch console-setup-linux:all/stretch dnsutils:amd64/stretch gnupg:amd64/stretch gnupg-curl:amd64/stretch gpgv:amd64/stretch host:all/stretch isc-dhcp-client:amd64/stretch isc-dhcp-common:amd64/stretch iso-codes:all/stretch keyboard-configuration:all/stretch kmod:amd64/stretch libapt-inst2.0:amd64/stretch libapt-pkg5.0:amd64/stretch libatk1.0-0:amd64/stretch libatk1.0-data:all/stretch libdebconfclient0:amd64/stretch libgraphite2-3:amd64/stretch libio-socket-ssl-perl:all/stretch libkmod2:amd64/stretch libnghttp2-14:amd64/stretch libpng12-0:amd64/stretch libseccomp2:amd64/stretch libxapian22v5:amd64/stretch s3ql:amd64/stretch : updated

Upgrading the filesystems:

s3qladm upgrade s3c://s.qstack.advania.com:443/crin4
  ERROR: Can not work on mounted file system.
umnt-s3ql crin4
s3qladm upgrade s3c://s.qstack.advania.com:443/crin4
Getting file system parameters..
Using cached metadata.
File system is damaged, need to run fsck before upgrade.
ERROR: Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/bin/s3qladm", line 9, in <module>
    load_entry_point('s3ql==2.17.1+hg2', 'console_scripts', 's3qladm')()
  File "/usr/lib/s3ql/s3ql/adm.py", line 94, in main
    return upgrade(options)
  File "/usr/lib/s3ql/s3ql/common.py", line 514, in wrapper
    return fn(*a, **kw)
  File "/usr/lib/s3ql/s3ql/adm.py", line 264, in upgrade
    print(get_old_rev_msg(param['revision'], 'fsck.s3ql'))
  File "/usr/lib/s3ql/s3ql/adm.py", line 209, in get_old_rev_msg
    ''' % { 'version': REV_VER_MAP[rev],
KeyError: 22

So reverting again:

cd
aptitude remove s3ql
shasum s3ql_2.16+dfsg-1_amd64.deb | grep 0fcba074d3f8c69e1841285fee059d2ad6a7793a
aptitude install python3-apsw python3-crypto python3-crypto python3-defusedxml python3-dugong  python3-llfuse python3-requests libjs-sphinxdoc python3-pkg-resources
dpkg -i /root/s3ql_2.16+dfsg-1_amd64.deb

And fsck'ing:

fsck.s3ql s3c://s.qstack.advania.com:443/crin1
fsck.s3ql s3c://s.qstack.advania.com:443/crin2
fsck.s3ql s3c://s.qstack.advania.com:443/crin4

hours, totalhours changed

chris — Wed, 13 Apr 2016 14:31:49 GMT

hours changed from 0 to 1
totalhours changed from 2.48 to 3.48

I have run fsck.s3ql --force several times until for each bucket it has run without this error:

Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 3)...

All the filesystems are marked as clean:

fsck.s3ql s3c://s.qstack.advania.com:443/crin4
Starting fsck of s3c://s.qstack.advania.com:443/crin4/
Using cached metadata.
File system is marked as clean. Use --force to force checking.
fsck of s3c://s.qstack.advania.com:443/crin1/
root@crin3:~# fsck.s3ql s3c://s.qstack.advania.com:443/crin1
Starting fsck of s3c://s.qstack.advania.com:443/crin1/
Using cached metadata.
File system is marked as clean. Use --force to force checking.
fsck.s3ql s3c://s.qstack.advania.com:443/crin2
Starting fsck of s3c://s.qstack.advania.com:443/crin2/
Using cached metadata.
File system is marked as clean. Use --force to force checking.

So upgrading s3ql again and then upgrading the filesystems:

s3qladm upgrade s3c://s.qstack.advania.com:443/crin4
Getting file system parameters..
Using cached metadata.
I am about to update the file system to the newest revision.
You will not be able to access the file system with any older version
of S3QL after this operation.
You should make very sure that this command is not interrupted and
that no one else tries to mount, fsck or upgrade the file system at
the same time.
Please enter "yes" to continue.
> yes
Upgrading from revision 22 to 23...
Dumping metadata...
..objects..
..blocks..
..inodes..
..inode_blocks..
..symlink_targets..
..names..
..contents..
..ext_attributes..
Compressing and uploading metadata...
Wrote 26.8 MiB of compressed metadata.
Cycling metadata backups...
Backing up old metadata...
File system upgrade complete.
s3qladm upgrade s3c://s.qstack.advania.com:443/crin1
Getting file system parameters..
Using cached metadata.
I am about to update the file system to the newest revision.
You will not be able to access the file system with any older version
of S3QL after this operation.
You should make very sure that this command is not interrupted and
that no one else tries to mount, fsck or upgrade the file system at
the same time.
Please enter "yes" to continue.
> yes
Upgrading from revision 22 to 23...
Dumping metadata...
..objects..
..blocks..
..inodes..
..inode_blocks..
..symlink_targets..
..names..
..contents..
..ext_attributes..
Compressing and uploading metadata...
Wrote 101 MiB of compressed metadata.
Cycling metadata backups...
Backing up old metadata...
Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 3)...
File system upgrade complete.
upgrade s3c://s.qstack.advania.com:443/crin2
Getting file system parameters..
Using cached metadata.
I am about to update the file system to the newest revision.
You will not be able to access the file system with any older version
of S3QL after this operation.
You should make very sure that this command is not interrupted and
that no one else tries to mount, fsck or upgrade the file system at
the same time.
Please enter "yes" to continue.
> yes
Upgrading from revision 22 to 23...
Dumping metadata...
..objects..
..blocks..
..inodes..
..inode_blocks..
..symlink_targets..
..names..
..contents..
..ext_attributes..
Compressing and uploading metadata...
Wrote 111 MiB of compressed metadata.
Cycling metadata backups...
Backing up old metadata...
Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 3)...
File system upgrade complete.

I'm now manually running the backups, all looks good so far,

hours, status, totalhours changed; resolution set

chris — Thu, 14 Apr 2016 09:37:26 GMT

hours changed from 0 to 0.25
status changed from new to closed
resolution set to fixed
totalhours changed from 3.48 to 3.73

The backups ran fine last night, the gap in backups is between 2016-03-03 and 2016-04-13 for Crin1 and between 2016-03-04 and 2016-04-13 for Crin2. Added a note to ticket:67 and closing this ticket.