Opened 2 years ago
Last modified 2 years ago
#66 new defect
Optimise backups, omit some MySQL tables and delete old backups from s3ql
| Reported by: | chris | Owned by: | chris |
|---|---|---|---|
| Priority: | major | Milestone: | Maintenance |
| Component: | backups | Version: | |
| Keywords: | Cc: | ||
| Estimated Number of Hours: | 0 | Add Hours to Ticket: | 0 |
| Billable?: | yes | Total Hours: | 3.15 |
Description
Currently the DumpMysqlTables script doesn't omit or truncate any tables so the backups contain unneeded data, such as cache tables, Drupal tables which can be truncated, see:
The same will be the case for Piwik and ownCloud and MediaWiki.
In addition php session files should be deleted.
Also the S3QLBackup script could be improved so it deleted old backups.
Change History (7)
comment:1 Changed 2 years ago by chris
- Add Hours to Ticket changed from 0 to 0.1
- Total Hours set to 0.1
comment:2 Changed 2 years ago by chris
- Add Hours to Ticket changed from 0 to 2
- Total Hours changed from 0.1 to 2.1
comment:3 Changed 2 years ago by chris
- Add Hours to Ticket changed from 0 to 0.25
- Total Hours changed from 2.1 to 2.35
The Crin1 deleting of old backups failed after deleting 5 old backups with "file system appears to have crashed errors" so running it again...
comment:4 Changed 2 years ago by chris
- Add Hours to Ticket changed from 0 to 0.15
- Total Hours changed from 2.35 to 2.5
That failed again with:
ERROR: File system appears to have crashed.
So I unmounted it and am now remounting it...
comment:5 Changed 2 years ago by chris
- Add Hours to Ticket changed from 0 to 0.15
- Total Hours changed from 2.5 to 2.65
The filesystem has been fsck'd and is now remounted and I'm trying to delete some old backups again, just two months worth this time.
comment:6 Changed 2 years ago by chris
- Add Hours to Ticket changed from 0 to 0.25
- Total Hours changed from 2.65 to 2.9
Trying to delete old backups again.
comment:7 Changed 2 years ago by chris
- Add Hours to Ticket changed from 0 to 0.25
- Total Hours changed from 2.9 to 3.15
That failed again with:
ERROR: File system appears to have crashed.
And I'm afraid I have hit the 10 hours per month for April so this will have to be picked up in May. Note that the nightly backups are running fine.
Looking at the expire_backups.py script.
To keep the latest 30 backups and one in 30 going back 300 backup cycles the command would be:
Running this in the /media/s3ql/crin4 directory generates:
So running it and generating the state file:
And again to get a list of what it would do:
And the /tmp/dry-run.txt contains:
So that is no good as it would delete all the backups because it does this:
So I'm not sure this script is usable with the backups we have.
I started writing a script which would:
But this is complicated so I haven't got it working and a quick fix for now is to delete everything form last year apart from the last backup of each month, for Crin4 this means deleting these backups:
So now we have these backups left:
And I'm doing the same for Crin1 and Crin2.