Advania S3 Account

[chris]

This ticket is to track the time spent answering Andrew's questions regarding the Advania S3 account which is used for backups by Crin3.

[chris]

I have just checked the backup emails and everything appears to be working OK, I'm now manually mounting the S3QL storage to check it is all OK, I'll post an update here when all the backup space has been mounted, currently the Crin1 space is being mounted:

Starting fsck of s3c://s.qstack.advania.com:443/crin1/
Using cached metadata.
Remote metadata is outdated.
Checking DB integrity...

[chris]

Oh dear, when running the fsck on the crin1 bucket:

Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/bin/fsck.s3ql", line 9, in <module>
    load_entry_point('s3ql==2.15', 'console_scripts', 'fsck.s3ql')()
  File "/usr/lib/s3ql/s3ql/fsck.py", line 1285, in main
    dump_and_upload_metadata(backend, db, param)
  File "/usr/lib/s3ql/s3ql/metadata.py", line 312, in dump_and_upload_metadata
    upload_metadata(backend, fh, param)
  File "/usr/lib/s3ql/s3ql/metadata.py", line 326, in upload_metadata
    cycle_metadata(backend)
  File "/usr/lib/s3ql/s3ql/metadata.py", line 125, in cycle_metadata
    cycle_fn("s3ql_metadata_bak_%d" % i, "s3ql_metadata_bak_%d" % (i + 1))
  File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 312, in copy
    self._copy_or_rename(src, dest, rename=False, metadata=metadata)
  File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 346, in _copy_or_rename
    self.backend.copy(src, dest, metadata=meta_raw)
  File "/usr/lib/s3ql/s3ql/backends/common.py", line 107, in wrapped
    return method(*a, **kw)
  File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 398, in copy
    resp = self._do_request('PUT', '/%s%s' % (self.prefix, dest), headers=headers)
  File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 510, in _do_request
    self._parse_error_response(resp)
  File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 544, in _parse_error_response
    raise get_S3Error(tree.findtext('Code'), tree.findtext('Message'), resp.headers)
s3ql.backends.s3c.S3Error: ServiceUnavailable: Please reduce your request rate.

[chris]

Just got that error again:

Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/bin/fsck.s3ql", line 9, in <module>
    load_entry_point('s3ql==2.15', 'console_scripts', 'fsck.s3ql')()
  File "/usr/lib/s3ql/s3ql/fsck.py", line 1285, in main
    dump_and_upload_metadata(backend, db, param)
  File "/usr/lib/s3ql/s3ql/metadata.py", line 312, in dump_and_upload_metadata
    upload_metadata(backend, fh, param)
  File "/usr/lib/s3ql/s3ql/metadata.py", line 326, in upload_metadata
    cycle_metadata(backend)
  File "/usr/lib/s3ql/s3ql/metadata.py", line 125, in cycle_metadata
    cycle_fn("s3ql_metadata_bak_%d" % i, "s3ql_metadata_bak_%d" % (i + 1))
  File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 312, in copy
    self._copy_or_rename(src, dest, rename=False, metadata=metadata)
  File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 346, in _copy_or_rename
    self.backend.copy(src, dest, metadata=meta_raw)
  File "/usr/lib/s3ql/s3ql/backends/common.py", line 107, in wrapped
    return method(*a, **kw)
  File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 398, in copy
    resp = self._do_request('PUT', '/%s%s' % (self.prefix, dest), headers=headers)
  File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 510, in _do_request
    self._parse_error_response(resp)
  File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 544, in _parse_error_response
    raise get_S3Error(tree.findtext('Code'), tree.findtext('Message'), resp.headers)
s3ql.backends.s3c.S3Error: ServiceUnavailable: Please reduce your request rate.

[chris]

The issues with the Crin1 backup have finally been resolved, the backups of all three servers have been mounted OK and space wise we are using:

Filesystem                             Size  Used Avail Use% Mounted on

s3c://s.qstack.advania.com:443/crin1/  1.0T  206G  819G  21% /media/s3ql/crin1
s3c://s.qstack.advania.com:443/crin2/  1.0T   67G  958G   7% /media/s3ql/crin2
s3c://s.qstack.advania.com:443/crin4/  1.0T   15G 1010G   2% /media/s3ql/crin4

So, 206+67+15=288 -- 288G in total.

I could take some time to delete old backups and / or automate the deletion of old backups.

I can't find where the Advania S3 prices are.

I have also edited the root crontab on Crin3 so the backups run at these times:

# m h  dom mon dow   command
00 10 * * * /usr/local/bin/s3ql_backup crin4
01 00 * * * /usr/local/bin/s3ql_backup crin2 
03 00 * * * /usr/local/bin/s3ql_backup crin1