Opened 23 months ago

Last modified 23 months ago

#90 new task

Cost of backup is estimated

Reported by: matthew Owned by: chris
Priority: major Milestone: Maintenance
Component: backups Version:
Keywords: Cc:
Estimated Number of Hours: 0.5 Add Hours to Ticket: 0
Billable?: yes Total Hours: 1.1

Description

It would be useful to estimate the expected cost of back up, by estimated the size and correlating this with the advertised price page.

Attachments (1)

Storage Cloud.png (24.4 KB) - added by chris 23 months ago.

Download all attachments as: .zip

Change History (11)

comment:1 Changed 23 months ago by chris

  • Add Hours to Ticket changed from 0 to 0.1
  • Total Hours set to 0.1

I have started to mount the s3ql filesystems on Crin3 so we can get an idea of the space usage, but I don't think this will be accurate as the data will be compressed and de-duplicated on the server, what about logging into Advania and checking if they display space used?

There is also this issue, ticket:66 -- in the past it has been really time consuming to delete old backups so this hasn't been done manually as often as it should have and also it hasn't been automated, sorting this issue out should make a big difference.

comment:2 Changed 23 months ago by matthew

thank you

comment:3 Changed 23 months ago by chris

  • Add Hours to Ticket changed from 0 to 0.1
  • Total Hours changed from 0.1 to 0.2

The Crin1] s3ql filesystem has mounted, the [[Crin2? one is not there yet:

Filesystem                             Size  Used Avail Use% Mounted on
s3c://s.qstack.advania.com:443/crin1/  1.0T  307G  718G  30% /media/s3ql/crin1
Version 0, edited 23 months ago by chris (next)

comment:4 Changed 23 months ago by chris

  • Add Hours to Ticket changed from 0 to 0.1
  • Total Hours changed from 0.2 to 0.3

No joy in mounting the Crin2 s3ql backups yet, this is where it is getting stuck:

Backing up old metadata...
Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 3)...
Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 4)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 5)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 6)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 7)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 8)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 9)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 10)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 11)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 12)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 13)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 14)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 15)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 16)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 17)...

comment:5 Changed 23 months ago by chris

  • Add Hours to Ticket changed from 0 to 0.1
  • Total Hours changed from 0.3 to 0.4

I'm afraid that the attempts to mount the Crin2 s3ql backups are on-going...

WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 66)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 67)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 68)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 69)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 70)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 71)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 72)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 73)...

Changed 23 months ago by chris

comment:6 Changed 23 months ago by chris

  • Add Hours to Ticket changed from 0 to 0.15
  • Total Hours changed from 0.4 to 0.55

I'm afraid the Crin2 s3ql backup filesystem still hasn't mounted:

WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 57)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 58)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 59)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 60)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 61)...

The Advania storage prices are here:


But I don't know which rate is used.

comment:7 Changed 23 months ago by chris

  • Add Hours to Ticket changed from 0 to 0.15
  • Total Hours changed from 0.55 to 0.7

It finally got there but then failed for another reason (Backend reports that fs is still mounted elsewhere?!):

WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 59)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 60)...
WARNING: Encountered ConnectionTimedOut (send/recv timeout exceeded), retrying Backend.copy (attempt 61)...
Cleaning up local metadata...
Completed fsck of s3c://s.qstack.advania.com:443/crin2/
Filesystem check of s3c://s.qstack.advania.com:443/crin2 success
Autodetected 65501 file descriptors available for cache entries
Ignoring locally cached metadata (outdated).
ERROR: Backend reports that fs is still mounted elsewhere, aborting.
Mounting s3c://s.qstack.advania.com:443/crin2 on /media/s3ql/crin2 failed

So trying again:

mnt-s3ql crin2
No problem with disconnected endpoints
Starting fsck of s3c://s.qstack.advania.com:443/crin2/
Ignoring locally cached metadata (outdated).
File system is marked as clean. Use --force to force checking.
Filesystem check of s3c://s.qstack.advania.com:443/crin2 success
Autodetected 65501 file descriptors available for cache entries
Ignoring locally cached metadata (outdated).
Downloading and decompressing metadata
...

Mounting Crin4:

mnt-s3ql crin4
No problem with disconnected endpoints
Starting fsck of s3c://s.qstack.advania.com:443/crin4/
Using cached metadata.
File system is marked as clean. Use --force to force checking.
Filesystem check of s3c://s.qstack.advania.com:443/crin4 success
Autodetected 65501 file descriptors available for cache entries
Using cached metadata.
Setting cache size to 4839 MB
Mounting s3c://s.qstack.advania.com:443/crin4/ at /media/s3ql/crin4...
Mounting s3c://s.qstack.advania.com:443/crin4 on /media/s3ql/crin4 success

And the space used:

df -h
Filesystem                             Size  Used Avail Use% Mounted on
s3c://s.qstack.advania.com:443/crin4/  1.0T   59G  966G   6% /media/s3ql/crin4

comment:8 Changed 23 months ago by chris

  • Add Hours to Ticket changed from 0 to 0.15
  • Total Hours changed from 0.7 to 0.85

And success!

Ignoring locally cached metadata (outdated).
Downloading and decompressing metadata...
Reading metadata...
..objects..
..blocks..
..inodes..
..inode_blocks..
..symlink_targets..
..names..
..contents..
..ext_attributes..
Setting cache size to 3001 MB
Mounting s3c://s.qstack.advania.com:443/crin2/ at /media/s3ql/crin2...
Mounting s3c://s.qstack.advania.com:443/crin2 on /media/s3ql/crin2 success

Space wise:

df -h
Filesystem                             Size  Used Avail Use% Mounted on
s3c://s.qstack.advania.com:443/crin2/  1.0T  127G  898G  13% /media/s3ql/crin2

So the total:

307+127+59
493

But remember that this 493G is how big the filesystem appears without deduplication and without compression -- it will hopefully be using a lot less space on the server.

Last edited 23 months ago by chris (previous) (diff)

comment:9 Changed 23 months ago by chris

  • Add Hours to Ticket changed from 0 to 0.1
  • Total Hours changed from 0.85 to 0.95

And with all the filsystems mounted on Crin3 this is how it looks:

df -h
Filesystem                             Size  Used Avail Use% Mounted on
udev                                   489M     0  489M   0% /dev
tmpfs                                  100M   11M   90M  11% /run
/dev/mapper/CRIN3--vg-root              15G  9.6G  4.5G  69% /
tmpfs                                  499M     0  499M   0% /dev/shm
tmpfs                                  5.0M     0  5.0M   0% /run/lock
tmpfs                                  499M     0  499M   0% /sys/fs/cgroup
/dev/sda1                              236M   60M  165M  27% /boot
tmpfs                                  100M     0  100M   0% /run/user/1000
crin2:/                                121G   25G   91G  22% /media/sshfs/crin2
s3c://s.qstack.advania.com:443/crin4/  1.0T   59G  966G   6% /media/s3ql/crin4
s3c://s.qstack.advania.com:443/crin2/  1.0T  127G  898G  13% /media/s3ql/crin2
s3c://s.qstack.advania.com:443/crin1/  1.0T  309G  716G  31% /media/s3ql/crin1
crin1:/                                121G   27G   88G  24% /media/sshfs/crin1
crin4:/                                 15G  6.3G  7.8G  45% /media/sshfs/crin4

I suggest we should do some follow up on ticket:66.

comment:10 Changed 23 months ago by chris

  • Add Hours to Ticket changed from 0 to 0.15
  • Total Hours changed from 0.95 to 1.1

Based on the prices on ticket:90#comment:6 the cost for the lead tier in US dollars per month:

493 / 40 * 2.25 = 27.73125

And the the platinum tier in US dollars per month:

493 / 40 * 8.98 = 110.6785

However we don't know how much disk space we are actually using and we don't know which tier we are on... so the only thing I think we could say with confidence is that they are probably invoicing for something in the region of £20 and £100 per month?

Note: See TracTickets for help on using tickets.