Opened 2 years ago
Closed 2 years ago
#70 closed defect (fixed)
s3ql backup error
| Reported by: | chris | Owned by: | chris |
|---|---|---|---|
| Priority: | major | Milestone: | Maintenance |
| Component: | backups | Version: | |
| Keywords: | Cc: | ||
| Estimated Number of Hours: | 0 | Add Hours to Ticket: | 0 |
| Billable?: | yes | Total Hours: | 3.73 |
Description (last modified by chris)
Email from Crin3:
Starting fsck of s3c://s.qstack.advania.com:443/crin4/ Using cached metadata. WARNING: Remote metadata is outdated. ERROR: File system revision too old, please run `s3qladm upgrade` first.
Change History (8)
comment:1 Changed 2 years ago by chris
- Description modified (diff)
comment:2 Changed 2 years ago by chris
- Add Hours to Ticket changed from 0 to 0.75
- Total Hours set to 0.75
comment:3 Changed 2 years ago by chris
- Add Hours to Ticket changed from 0 to 0.23
- Total Hours changed from 0.75 to 0.98
comment:4 Changed 2 years ago by chris
- Add Hours to Ticket changed from 0 to 0.5
- Total Hours changed from 0.98 to 1.48
I spent some time last night emailing the list, see the thread, Nikolaus has advised:
Go back to the S3QL version that you used before, and then go forward
one minor version after another until you reach 2.17.1. (not every minor
version will require a file system update though, for most updates
you'll just get a "file system revision is up-to-date" message).
comment:5 Changed 2 years ago by chris
- Add Hours to Ticket changed from 0 to 0.5
- Total Hours changed from 1.48 to 1.98
Downgrading s3ql:
sudo -i aptitude remove s3ql wget http://snapshot.debian.org/archive/debian/20160229T214851Z/pool/main/s/s3ql/s3ql_2.16%2Bdfsg-1_amd64.deb shasum s3ql_2.16+dfsg-1_amd64.deb | grep 0fcba074d3f8c69e1841285fee059d2ad6a7793a aptitude install python3-apsw python3-crypto python3-crypto python3-defusedxml python3-dugong python3-llfuse python3-requests libjs-sphinxdoc python3-pkg-resources dpkg -i /root/s3ql_2.16+dfsg-1_amd64.deb
Mounting the file systems:
mnt-s3ql crin1 mnt-s3ql crin2 mnt-s3ql crin4
And waiting for the results...
comment:6 Changed 2 years ago by chris
- Add Hours to Ticket changed from 0 to 0.5
- Total Hours changed from 1.98 to 2.48
The filesystems have now mounted OK:
df -h s3c://s.qstack.advania.com:443/crin4/ 1.0T 21G 1004G 2% /media/s3ql/crin4 s3c://s.qstack.advania.com:443/crin1/ 1.0T 242G 783G 24% /media/s3ql/crin1 s3c://s.qstack.advania.com:443/crin2/ 1.0T 76G 949G 8% /media/s3ql/crin2
So upgrading s3ql and everything else installed the following:
2016-04-12 chris
* apt:amd64/stretch apt-listchanges:all/stretch apt-utils:amd64/stretch bind9-host:amd64/stretch console-setup:all/stretch console-setup-linux:all/stretch dnsutils:amd64/stretch gnupg:amd64/stretch gnupg-curl:amd64/stretch gpgv:amd64/stretch host:all/stretch isc-dhcp-client:amd64/stretch isc-dhcp-common:amd64/stretch iso-codes:all/stretch keyboard-configuration:all/stretch kmod:amd64/stretch libapt-inst2.0:amd64/stretch libapt-pkg5.0:amd64/stretch libatk1.0-0:amd64/stretch libatk1.0-data:all/stretch libdebconfclient0:amd64/stretch libgraphite2-3:amd64/stretch libio-socket-ssl-perl:all/stretch libkmod2:amd64/stretch libnghttp2-14:amd64/stretch libpng12-0:amd64/stretch libseccomp2:amd64/stretch libxapian22v5:amd64/stretch s3ql:amd64/stretch : updated
Upgrading the filesystems:
s3qladm upgrade s3c://s.qstack.advania.com:443/crin4
ERROR: Can not work on mounted file system.
umnt-s3ql crin4
s3qladm upgrade s3c://s.qstack.advania.com:443/crin4
Getting file system parameters..
Using cached metadata.
File system is damaged, need to run fsck before upgrade.
ERROR: Uncaught top-level exception:
Traceback (most recent call last):
File "/usr/bin/s3qladm", line 9, in <module>
load_entry_point('s3ql==2.17.1+hg2', 'console_scripts', 's3qladm')()
File "/usr/lib/s3ql/s3ql/adm.py", line 94, in main
return upgrade(options)
File "/usr/lib/s3ql/s3ql/common.py", line 514, in wrapper
return fn(*a, **kw)
File "/usr/lib/s3ql/s3ql/adm.py", line 264, in upgrade
print(get_old_rev_msg(param['revision'], 'fsck.s3ql'))
File "/usr/lib/s3ql/s3ql/adm.py", line 209, in get_old_rev_msg
''' % { 'version': REV_VER_MAP[rev],
KeyError: 22
So reverting again:
cd aptitude remove s3ql shasum s3ql_2.16+dfsg-1_amd64.deb | grep 0fcba074d3f8c69e1841285fee059d2ad6a7793a aptitude install python3-apsw python3-crypto python3-crypto python3-defusedxml python3-dugong python3-llfuse python3-requests libjs-sphinxdoc python3-pkg-resources dpkg -i /root/s3ql_2.16+dfsg-1_amd64.deb
And fsck'ing:
fsck.s3ql s3c://s.qstack.advania.com:443/crin1 fsck.s3ql s3c://s.qstack.advania.com:443/crin2 fsck.s3ql s3c://s.qstack.advania.com:443/crin4
comment:7 Changed 2 years ago by chris
- Add Hours to Ticket changed from 0 to 1
- Total Hours changed from 2.48 to 3.48
I have run fsck.s3ql --force several times until for each bucket it has run without this error:
Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 3)...
All the filesystems are marked as clean:
fsck.s3ql s3c://s.qstack.advania.com:443/crin4 Starting fsck of s3c://s.qstack.advania.com:443/crin4/ Using cached metadata. File system is marked as clean. Use --force to force checking. fsck of s3c://s.qstack.advania.com:443/crin1/ root@crin3:~# fsck.s3ql s3c://s.qstack.advania.com:443/crin1 Starting fsck of s3c://s.qstack.advania.com:443/crin1/ Using cached metadata. File system is marked as clean. Use --force to force checking. fsck.s3ql s3c://s.qstack.advania.com:443/crin2 Starting fsck of s3c://s.qstack.advania.com:443/crin2/ Using cached metadata. File system is marked as clean. Use --force to force checking.
So upgrading s3ql again and then upgrading the filesystems:
s3qladm upgrade s3c://s.qstack.advania.com:443/crin4 Getting file system parameters.. Using cached metadata. I am about to update the file system to the newest revision. You will not be able to access the file system with any older version of S3QL after this operation. You should make very sure that this command is not interrupted and that no one else tries to mount, fsck or upgrade the file system at the same time. Please enter "yes" to continue. > yes Upgrading from revision 22 to 23... Dumping metadata... ..objects.. ..blocks.. ..inodes.. ..inode_blocks.. ..symlink_targets.. ..names.. ..contents.. ..ext_attributes.. Compressing and uploading metadata... Wrote 26.8 MiB of compressed metadata. Cycling metadata backups... Backing up old metadata... File system upgrade complete. s3qladm upgrade s3c://s.qstack.advania.com:443/crin1 Getting file system parameters.. Using cached metadata. I am about to update the file system to the newest revision. You will not be able to access the file system with any older version of S3QL after this operation. You should make very sure that this command is not interrupted and that no one else tries to mount, fsck or upgrade the file system at the same time. Please enter "yes" to continue. > yes Upgrading from revision 22 to 23... Dumping metadata... ..objects.. ..blocks.. ..inodes.. ..inode_blocks.. ..symlink_targets.. ..names.. ..contents.. ..ext_attributes.. Compressing and uploading metadata... Wrote 101 MiB of compressed metadata. Cycling metadata backups... Backing up old metadata... Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 3)... File system upgrade complete. upgrade s3c://s.qstack.advania.com:443/crin2 Getting file system parameters.. Using cached metadata. I am about to update the file system to the newest revision. You will not be able to access the file system with any older version of S3QL after this operation. You should make very sure that this command is not interrupted and that no one else tries to mount, fsck or upgrade the file system at the same time. Please enter "yes" to continue. > yes Upgrading from revision 22 to 23... Dumping metadata... ..objects.. ..blocks.. ..inodes.. ..inode_blocks.. ..symlink_targets.. ..names.. ..contents.. ..ext_attributes.. Compressing and uploading metadata... Wrote 111 MiB of compressed metadata. Cycling metadata backups... Backing up old metadata... Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 3)... File system upgrade complete.
I'm now manually running the backups, all looks good so far,
comment:8 Changed 2 years ago by chris
- Add Hours to Ticket changed from 0 to 0.25
- Resolution set to fixed
- Status changed from new to closed
- Total Hours changed from 3.48 to 3.73
Note: See
TracTickets for help on using
tickets.
I missed that s3ql was being upgraded with the other updates on ticket:17#comment:77.
s3qladm upgrade s3c://s.qstack.advania.com:443/crin4 Getting file system parameters.. Using cached metadata. File system is damaged, need to run fsck before upgrade. ERROR: Uncaught top-level exception: Traceback (most recent call last): File "/usr/bin/s3qladm", line 9, in <module> load_entry_point('s3ql==2.17.1+hg2', 'console_scripts', 's3qladm')() File "/usr/lib/s3ql/s3ql/adm.py", line 94, in main return upgrade(options) File "/usr/lib/s3ql/s3ql/common.py", line 514, in wrapper return fn(*a, **kw) File "/usr/lib/s3ql/s3ql/adm.py", line 264, in upgrade print(get_old_rev_msg(param['revision'], 'fsck.s3ql')) File "/usr/lib/s3ql/s3ql/adm.py", line 209, in get_old_rev_msg ''' % { 'version': REV_VER_MAP[rev], KeyError: 22 fsck.s3ql s3c://s.qstack.advania.com:443/crin4 Starting fsck of s3c://s.qstack.advania.com:443/crin4/ Using cached metadata. WARNING: Remote metadata is outdated. ERROR: File system revision too old, please run `s3qladm upgrade` first.Not sure what to try next, spent some time reading the list archives and have sent a email to the list.