CRIN Trac: Ticket #60: Advania S3 Account

hours changed; totalhours set

chris — Thu, 04 Feb 2016 11:46:26 GMT

hours changed from 0 to 0.25
totalhours set to 0.25

I have just checked the backup emails and everything appears to be working OK, I'm now manually mounting the S3QL storage to check it is all OK, I'll post an update here when all the backup space has been mounted, currently the Crin1 space is being mounted:

Starting fsck of s3c://s.qstack.advania.com:443/crin1/
Using cached metadata.
Remote metadata is outdated.
Checking DB integrity...

hours, totalhours changed

chris — Thu, 04 Feb 2016 13:08:14 GMT

hours changed from 0 to 0.15
totalhours changed from 0.25 to 0.4

Oh dear, when running the fsck on the crin1 bucket:

Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/bin/fsck.s3ql", line 9, in <module>
    load_entry_point('s3ql==2.15', 'console_scripts', 'fsck.s3ql')()
  File "/usr/lib/s3ql/s3ql/fsck.py", line 1285, in main
    dump_and_upload_metadata(backend, db, param)
  File "/usr/lib/s3ql/s3ql/metadata.py", line 312, in dump_and_upload_metadata
    upload_metadata(backend, fh, param)
  File "/usr/lib/s3ql/s3ql/metadata.py", line 326, in upload_metadata
    cycle_metadata(backend)
  File "/usr/lib/s3ql/s3ql/metadata.py", line 125, in cycle_metadata
    cycle_fn("s3ql_metadata_bak_%d" % i, "s3ql_metadata_bak_%d" % (i + 1))
  File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 312, in copy
    self._copy_or_rename(src, dest, rename=False, metadata=metadata)
  File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 346, in _copy_or_rename
    self.backend.copy(src, dest, metadata=meta_raw)
  File "/usr/lib/s3ql/s3ql/backends/common.py", line 107, in wrapped
    return method(*a, **kw)
  File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 398, in copy
    resp = self._do_request('PUT', '/%s%s' % (self.prefix, dest), headers=headers)
  File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 510, in _do_request
    self._parse_error_response(resp)
  File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 544, in _parse_error_response
    raise get_S3Error(tree.findtext('Code'), tree.findtext('Message'), resp.headers)
s3ql.backends.s3c.S3Error: ServiceUnavailable: Please reduce your request rate.

hours, totalhours changed

chris — Thu, 04 Feb 2016 14:11:56 GMT

hours changed from 0 to 0.15
totalhours changed from 0.4 to 0.55

Just got that error again:

Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/bin/fsck.s3ql", line 9, in <module>
    load_entry_point('s3ql==2.15', 'console_scripts', 'fsck.s3ql')()
  File "/usr/lib/s3ql/s3ql/fsck.py", line 1285, in main
    dump_and_upload_metadata(backend, db, param)
  File "/usr/lib/s3ql/s3ql/metadata.py", line 312, in dump_and_upload_metadata
    upload_metadata(backend, fh, param)
  File "/usr/lib/s3ql/s3ql/metadata.py", line 326, in upload_metadata
    cycle_metadata(backend)
  File "/usr/lib/s3ql/s3ql/metadata.py", line 125, in cycle_metadata
    cycle_fn("s3ql_metadata_bak_%d" % i, "s3ql_metadata_bak_%d" % (i + 1))
  File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 312, in copy
    self._copy_or_rename(src, dest, rename=False, metadata=metadata)
  File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 346, in _copy_or_rename
    self.backend.copy(src, dest, metadata=meta_raw)
  File "/usr/lib/s3ql/s3ql/backends/common.py", line 107, in wrapped
    return method(*a, **kw)
  File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 398, in copy
    resp = self._do_request('PUT', '/%s%s' % (self.prefix, dest), headers=headers)
  File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 510, in _do_request
    self._parse_error_response(resp)
  File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 544, in _parse_error_response
    raise get_S3Error(tree.findtext('Code'), tree.findtext('Message'), resp.headers)
s3ql.backends.s3c.S3Error: ServiceUnavailable: Please reduce your request rate.

hours, totalhours changed

chris — Thu, 04 Feb 2016 17:22:32 GMT

hours changed from 0 to 0.35
totalhours changed from 0.55 to 0.9

The issues with the Crin1 backup have finally been resolved, the backups of all three servers have been mounted OK and space wise we are using:

Filesystem                             Size  Used Avail Use% Mounted on
s3c://s.qstack.advania.com:443/crin1/  1.0T  206G  819G  21% /media/s3ql/crin1
s3c://s.qstack.advania.com:443/crin2/  1.0T   67G  958G   7% /media/s3ql/crin2
s3c://s.qstack.advania.com:443/crin4/  1.0T   15G 1010G   2% /media/s3ql/crin4

So, 206+67+15=288 -- 288G in total.

I could take some time to delete old backups and / or automate the deletion of old backups.

I can't find where the Advania S3 prices are.

I have also edited the root crontab on Crin3 so the backups run at these times:

# m h  dom mon dow   command
00 10 * * * /usr/local/bin/s3ql_backup crin4
01 00 * * * /usr/local/bin/s3ql_backup crin2
03 00 * * * /usr/local/bin/s3ql_backup crin1