CRIN Trac: Ticket #90: Cost of backup is estimated

hours changed; totalhours set

chris — Tue, 25 Oct 2016 09:40:14 GMT

hours changed from 0 to 0.1
totalhours set to 0.1

I have started to mount the s3ql filesystems on Crin3 so we can get an idea of the space usage, but I don't think this will be accurate as the data will be compressed and de-duplicated on the server, what about logging into Advania and checking if they display space used?

There is also this issue, ticket:66 -- in the past it has been really time consuming to delete old backups so this hasn't been done manually as often as it should have and also it hasn't been automated, sorting this issue out should make a big difference.

matthew — Tue, 25 Oct 2016 09:48:34 GMT

thank you

hours, totalhours changed

chris — Tue, 25 Oct 2016 11:03:23 GMT

hours changed from 0 to 0.1
totalhours changed from 0.1 to 0.2

The Crin1 s3ql filesystem has mounted, the Crin2 one is not there yet:

Filesystem                             Size  Used Avail Use% Mounted on
s3c://s.qstack.advania.com:443/crin1/  1.0T  307G  718G  30% /media/s3ql/crin1

hours, totalhours changed

chris — Tue, 25 Oct 2016 13:06:10 GMT

hours changed from 0 to 0.1
totalhours changed from 0.2 to 0.3

No joy in mounting the Crin2 s3ql backups yet, this is where it is getting stuck:

Backing up old metadata...
Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 3)...
Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 4)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 5)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 6)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 7)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 8)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 9)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 10)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 11)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 12)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 13)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 14)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 15)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 16)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 17)...

hours, totalhours changed

chris — Tue, 25 Oct 2016 19:28:51 GMT

hours changed from 0 to 0.1
totalhours changed from 0.3 to 0.4

I'm afraid that the attempts to mount the Crin2 s3ql backups are on-going...

WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 66)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 67)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 68)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 69)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 70)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 71)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 72)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 73)...

attachment set

chris — Wed, 26 Oct 2016 09:47:41 GMT

attachment set to Storage Cloud.png

hours, totalhours changed

chris — Wed, 26 Oct 2016 09:50:04 GMT

hours changed from 0 to 0.15
totalhours changed from 0.4 to 0.55

I'm afraid the Crin2 s3ql backup filesystem still hasn't mounted:

WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 57)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 58)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 59)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 60)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 61)...

The Advania storage prices are here:

But I don't know which rate is used.

hours, totalhours changed

chris — Thu, 27 Oct 2016 11:44:13 GMT

hours changed from 0 to 0.15
totalhours changed from 0.55 to 0.7

It finally got there but then failed for another reason (Backend reports that fs is still mounted elsewhere?!):

WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 59)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 60)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 61)...
Cleaning up local metadata...
Completed fsck of s3c://s.qstack.advania.com:443/crin2/
Filesystem check of s3c://s.qstack.advania.com:443/crin2 success
Autodetected 65501 file descriptors available for cache entries
Ignoring locally cached metadata (outdated).
ERROR: Backend reports that fs is still mounted elsewhere, aborting.
Mounting s3c://s.qstack.advania.com:443/crin2 on /media/s3ql/crin2 failed

So trying again:

mnt-s3ql crin2
No problem with disconnected endpoints
Starting fsck of s3c://s.qstack.advania.com:443/crin2/
Ignoring locally cached metadata (outdated).
File system is marked as clean. Use --force to force checking.
Filesystem check of s3c://s.qstack.advania.com:443/crin2 success
Autodetected 65501 file descriptors available for cache entries
Ignoring locally cached metadata (outdated).
Downloading and decompressing metadata
...

Mounting Crin4:

mnt-s3ql crin4
No problem with disconnected endpoints
Starting fsck of s3c://s.qstack.advania.com:443/crin4/
Using cached metadata.
File system is marked as clean. Use --force to force checking.
Filesystem check of s3c://s.qstack.advania.com:443/crin4 success
Autodetected 65501 file descriptors available for cache entries
Using cached metadata.
Setting cache size to 4839 MB
Mounting s3c://s.qstack.advania.com:443/crin4/ at /media/s3ql/crin4...
Mounting s3c://s.qstack.advania.com:443/crin4 on /media/s3ql/crin4 success

And the space used:

df -h
Filesystem                             Size  Used Avail Use% Mounted on
s3c://s.qstack.advania.com:443/crin4/  1.0T   59G  966G   6% /media/s3ql/crin4

hours, totalhours changed

chris — Thu, 27 Oct 2016 11:56:05 GMT

hours changed from 0 to 0.15
totalhours changed from 0.7 to 0.85

And success!

Ignoring locally cached metadata (outdated).
Downloading and decompressing metadata...
Reading metadata...
..objects..
..blocks..
..inodes..
..inode_blocks..
..symlink_targets..
..names..
..contents..
..ext_attributes..
Setting cache size to 3001 MB
Mounting s3c://s.qstack.advania.com:443/crin2/ at /media/s3ql/crin2...
Mounting s3c://s.qstack.advania.com:443/crin2 on /media/s3ql/crin2 success

Space wise:

df -h
Filesystem                             Size  Used Avail Use% Mounted on
s3c://s.qstack.advania.com:443/crin2/  1.0T  127G  898G  13% /media/s3ql/crin2

So the total:

307+127+59
493

But remember that this 493G is how big the filesystem appears without deduplication and without compression -- it will hopefully be using a lot less space on the server.

hours, totalhours changed

chris — Thu, 27 Oct 2016 12:01:56 GMT

hours changed from 0 to 0.1
totalhours changed from 0.85 to 0.95

And with all the filsystems mounted on Crin3 this is how it looks:

df -h
Filesystem                             Size  Used Avail Use% Mounted on
udev                                   489M     0  489M   0% /dev
tmpfs                                  100M   11M   90M  11% /run
/dev/mapper/CRIN3--vg-root              15G  9.6G  4.5G  69% /
tmpfs                                  499M     0  499M   0% /dev/shm
tmpfs                                  5.0M     0  5.0M   0% /run/lock
tmpfs                                  499M     0  499M   0% /sys/fs/cgroup
/dev/sda1                              236M   60M  165M  27% /boot
tmpfs                                  100M     0  100M   0% /run/user/1000
crin2:/                                121G   25G   91G  22% /media/sshfs/crin2
s3c://s.qstack.advania.com:443/crin4/  1.0T   59G  966G   6% /media/s3ql/crin4
s3c://s.qstack.advania.com:443/crin2/  1.0T  127G  898G  13% /media/s3ql/crin2
s3c://s.qstack.advania.com:443/crin1/  1.0T  309G  716G  31% /media/s3ql/crin1
crin1:/                                121G   27G   88G  24% /media/sshfs/crin1
crin4:/                                 15G  6.3G  7.8G  45% /media/sshfs/crin4

I suggest we should do some follow up on ticket:66.

hours, totalhours changed

chris — Thu, 27 Oct 2016 13:36:10 GMT

hours changed from 0 to 0.15
totalhours changed from 0.95 to 1.1

Based on the prices on ticket:90#comment:6 the cost for the lead tier in US dollars per month:

493 / 40 * 2.25 = 27.73125

And the the platinum tier in US dollars per month:

493 / 40 * 8.98 = 110.6785

However we don't know how much disk space we are actually using and we don't know which tier we are on... so the only thing I think we could say with confidence is that they are probably invoicing for something in the region of £20 and £100 per month?