Opened 2 years ago

Closed 2 years ago

#70 closed defect (fixed)

s3ql backup error

Reported by: chris Owned by: chris
Priority: major Milestone: Maintenance
Component: backups Version:
Keywords: Cc:
Estimated Number of Hours: 0 Add Hours to Ticket: 0
Billable?: yes Total Hours: 3.73

Description (last modified by chris)

Email from Crin3:

Starting fsck of s3c://s.qstack.advania.com:443/crin4/
Using cached metadata.
WARNING: Remote metadata is outdated.
ERROR: File system revision too old, please run `s3qladm upgrade` first.

Change History (8)

comment:1 Changed 2 years ago by chris

  • Description modified (diff)

comment:2 Changed 2 years ago by chris

  • Add Hours to Ticket changed from 0 to 0.75
  • Total Hours set to 0.75

I missed that s3ql was being upgraded with the other updates on ticket:17#comment:77.

s3qladm upgrade s3c://s.qstack.advania.com:443/crin4
Getting file system parameters..
Using cached metadata.
File system is damaged, need to run fsck before upgrade.
ERROR: Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/bin/s3qladm", line 9, in <module>
    load_entry_point('s3ql==2.17.1+hg2', 'console_scripts', 's3qladm')()
  File "/usr/lib/s3ql/s3ql/adm.py", line 94, in main
    return upgrade(options)
  File "/usr/lib/s3ql/s3ql/common.py", line 514, in wrapper
    return fn(*a, **kw)
  File "/usr/lib/s3ql/s3ql/adm.py", line 264, in upgrade
    print(get_old_rev_msg(param['revision'], 'fsck.s3ql'))
  File "/usr/lib/s3ql/s3ql/adm.py", line 209, in get_old_rev_msg
    ''' % { 'version': REV_VER_MAP[rev],
KeyError: 22

fsck.s3ql s3c://s.qstack.advania.com:443/crin4
Starting fsck of s3c://s.qstack.advania.com:443/crin4/
Using cached metadata.
WARNING: Remote metadata is outdated.
ERROR: File system revision too old, please run `s3qladm upgrade` first.

Not sure what to try next, spent some time reading the list archives and have ​sent a email to the list.

comment:3 Changed 2 years ago by chris

  • Add Hours to Ticket changed from 0 to 0.23
  • Total Hours changed from 0.75 to 0.98

Checking through the emails sent by the backup jobs and the last good backup appears to have been made on 3rd March 2016 for Crin1 and 4th March 2016 for Crin2 and Crin4, very sorry not to have noticed the errors sooner.

comment:4 Changed 2 years ago by chris

  • Add Hours to Ticket changed from 0 to 0.5
  • Total Hours changed from 0.98 to 1.48

I spent some time last night emailing the list, see ​the thread, Nikolaus has advised:

Go back to the S3QL version that you used before, and then go forward
one minor version after another until you reach 2.17.1. (not every minor
version will require a file system update though, for most updates
you'll just get a "file system revision is up-to-date" message).

comment:5 Changed 2 years ago by chris

  • Add Hours to Ticket changed from 0 to 0.5
  • Total Hours changed from 1.48 to 1.98

Downgrading s3ql:

sudo -i
aptitude remove s3ql
wget http://snapshot.debian.org/archive/debian/20160229T214851Z/pool/main/s/s3ql/s3ql_2.16%2Bdfsg-1_amd64.deb
shasum s3ql_2.16+dfsg-1_amd64.deb | grep 0fcba074d3f8c69e1841285fee059d2ad6a7793a
dpkg -i /root/s3ql_2.16+dfsg-1_amd64.deb 
aptitude install s3ql_2.16+dfsg-1_amd64.deb

Mounting the file systems:

mnt-s3ql crin1
mnt-s3ql crin2
mnt-s3ql crin4

And waiting for the results...

Version 0, edited 2 years ago by chris (next)

comment:6 Changed 2 years ago by chris

  • Add Hours to Ticket changed from 0 to 0.5
  • Total Hours changed from 1.98 to 2.48

The filesystems have now mounted OK:

df -h
s3c://s.qstack.advania.com:443/crin4/  1.0T   21G 1004G   2% /media/s3ql/crin4
s3c://s.qstack.advania.com:443/crin1/  1.0T  242G  783G  24% /media/s3ql/crin1
s3c://s.qstack.advania.com:443/crin2/  1.0T   76G  949G   8% /media/s3ql/crin2

So upgrading s3ql and everything else installed the following:

2016-04-12      chris
        *       apt:amd64/stretch apt-listchanges:all/stretch apt-utils:amd64/stretch bind9-host:amd64/stretch console-setup:all/stretch console-setup-linux:all/stretch dnsutils:amd64/stretch gnupg:amd64/stretch gnupg-curl:amd64/stretch gpgv:amd64/stretch host:all/stretch isc-dhcp-client:amd64/stretch isc-dhcp-common:amd64/stretch iso-codes:all/stretch keyboard-configuration:all/stretch kmod:amd64/stretch libapt-inst2.0:amd64/stretch libapt-pkg5.0:amd64/stretch libatk1.0-0:amd64/stretch libatk1.0-data:all/stretch libdebconfclient0:amd64/stretch libgraphite2-3:amd64/stretch libio-socket-ssl-perl:all/stretch libkmod2:amd64/stretch libnghttp2-14:amd64/stretch libpng12-0:amd64/stretch libseccomp2:amd64/stretch libxapian22v5:amd64/stretch s3ql:amd64/stretch : updated

Upgrading the filesystems:

s3qladm upgrade s3c://s.qstack.advania.com:443/crin4
  ERROR: Can not work on mounted file system.
umnt-s3ql crin4
s3qladm upgrade s3c://s.qstack.advania.com:443/crin4
Getting file system parameters..
Using cached metadata.
File system is damaged, need to run fsck before upgrade.
ERROR: Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/bin/s3qladm", line 9, in <module>
    load_entry_point('s3ql==2.17.1+hg2', 'console_scripts', 's3qladm')()
  File "/usr/lib/s3ql/s3ql/adm.py", line 94, in main
    return upgrade(options)
  File "/usr/lib/s3ql/s3ql/common.py", line 514, in wrapper
    return fn(*a, **kw)
  File "/usr/lib/s3ql/s3ql/adm.py", line 264, in upgrade
    print(get_old_rev_msg(param['revision'], 'fsck.s3ql'))
  File "/usr/lib/s3ql/s3ql/adm.py", line 209, in get_old_rev_msg
    ''' % { 'version': REV_VER_MAP[rev],
KeyError: 22

So reverting again:

cd
aptitude remove s3ql
shasum s3ql_2.16+dfsg-1_amd64.deb | grep 0fcba074d3f8c69e1841285fee059d2ad6a7793a 
aptitude install python3-apsw python3-crypto python3-crypto python3-defusedxml python3-dugong  python3-llfuse python3-requests libjs-sphinxdoc python3-pkg-resources
dpkg -i /root/s3ql_2.16+dfsg-1_amd64.deb

And fsck'ing:

fsck.s3ql s3c://s.qstack.advania.com:443/crin1
fsck.s3ql s3c://s.qstack.advania.com:443/crin2
fsck.s3ql s3c://s.qstack.advania.com:443/crin4

comment:7 Changed 2 years ago by chris

  • Add Hours to Ticket changed from 0 to 1
  • Total Hours changed from 2.48 to 3.48

I have run fsck.s3ql --force several times until for each bucket it has run without this error:

Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 3)...

All the filesystems are marked as clean:

fsck.s3ql s3c://s.qstack.advania.com:443/crin4 
Starting fsck of s3c://s.qstack.advania.com:443/crin4/
Using cached metadata.
File system is marked as clean. Use --force to force checking.

fsck of s3c://s.qstack.advania.com:443/crin1/
root@crin3:~# fsck.s3ql s3c://s.qstack.advania.com:443/crin1 
Starting fsck of s3c://s.qstack.advania.com:443/crin1/
Using cached metadata.
File system is marked as clean. Use --force to force checking.

fsck.s3ql s3c://s.qstack.advania.com:443/crin2 
Starting fsck of s3c://s.qstack.advania.com:443/crin2/
Using cached metadata.
File system is marked as clean. Use --force to force checking.

So upgrading s3ql again and then upgrading the filesystems:

s3qladm upgrade s3c://s.qstack.advania.com:443/crin4
Getting file system parameters..
Using cached metadata.

I am about to update the file system to the newest revision.
You will not be able to access the file system with any older version
of S3QL after this operation.

You should make very sure that this command is not interrupted and
that no one else tries to mount, fsck or upgrade the file system at
the same time.


Please enter "yes" to continue.
> yes
Upgrading from revision 22 to 23...
Dumping metadata...
..objects..
..blocks..
..inodes..
..inode_blocks..
..symlink_targets..
..names..
..contents..
..ext_attributes..
Compressing and uploading metadata...
Wrote 26.8 MiB of compressed metadata.
Cycling metadata backups...
Backing up old metadata...
File system upgrade complete.

s3qladm upgrade s3c://s.qstack.advania.com:443/crin1
Getting file system parameters..
Using cached metadata.

I am about to update the file system to the newest revision.
You will not be able to access the file system with any older version
of S3QL after this operation.

You should make very sure that this command is not interrupted and
that no one else tries to mount, fsck or upgrade the file system at
the same time.


Please enter "yes" to continue.
> yes
Upgrading from revision 22 to 23...
Dumping metadata...
..objects..
..blocks..
..inodes..
..inode_blocks..
..symlink_targets..
..names..
..contents..
..ext_attributes..
Compressing and uploading metadata...
Wrote 101 MiB of compressed metadata.
Cycling metadata backups...
Backing up old metadata...
Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 3)...
File system upgrade complete.

 upgrade s3c://s.qstack.advania.com:443/crin2
Getting file system parameters..
Using cached metadata.

I am about to update the file system to the newest revision.
You will not be able to access the file system with any older version
of S3QL after this operation.

You should make very sure that this command is not interrupted and
that no one else tries to mount, fsck or upgrade the file system at
the same time.


Please enter "yes" to continue.
> yes
Upgrading from revision 22 to 23...
Dumping metadata...
..objects..
..blocks..
..inodes..
..inode_blocks..
..symlink_targets..
..names..
..contents..
..ext_attributes..
Compressing and uploading metadata...
Wrote 111 MiB of compressed metadata.
Cycling metadata backups...
Backing up old metadata...
Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 3)...
File system upgrade complete.

I'm now manually running the backups, all looks good so far,

comment:8 Changed 2 years ago by chris

  • Add Hours to Ticket changed from 0 to 0.25
  • Resolution set to fixed
  • Status changed from new to closed
  • Total Hours changed from 3.48 to 3.73

The backups ran fine last night, the gap in backups is between 2016-03-03 and 2016-04-13 for Crin1 and between 2016-03-04 and 2016-04-13 for Crin2. Added a note to ticket:67 and closing this ticket.

Note: See TracTickets for help on using tickets.