menuimage

Description

Usage

Options

Why_timedicer-verify

Dependencies

License

Changelog

Download

My_Other_Sites

My_Programs

Comments

timedicer-verify v2.60 [24 May 2017] by Dominic

Description

Verifies integrity of rdiff-backup repositories at /home/*/[here] and /home/*/*/[here]. Unless options -f or -d are used it just verifies the most recent session in each repository (this session is held in the clear in the repository). With -f or -d options it may take a long time!

If run by root (or with sudo) timedicer-verify.sh checks repositories for all users (as restricted by -u, -n and -x options), otherwise it checks only the current user's repositories. A non-zero exit code is returned if one or more problems were found.

Each successful verification for a repository session is saved in that repository's rdiff-backup-data/verified.log file, on a subsequent run that session is skipped for verification unless -r option is used. This file therefore provides a linear record of successful verifications in the form [datetime-of-original-session],[datetime-when-verified],[rdiff-backup exit code].

You can override problem detection (if you have a known reported problem which is not in fact such or you do not consider fatal) by adding the error text to /opt/timedicer-verify.sh-ignore.txt, any whole line matches for such text will be ignored on future runs.

Normally timedicer-verify.sh will exit immediately if rdiff-backup reports an error (unless the error text is matched in /opt/timedicer-verify.sh-ignore.txt), but you can override this behaviour with -i option.

Also with -b option it fixes ownership of restore.log files (which rdiffWeb may have created and set to root ownership, thus causing warning errors upon user trying to retrieve files).

timedicer-verify.sh requires a small modification to rdiff-backup in compare.py - this is applied automatically (unless already present) and a warning printed if it fails. The modification ensures that if a verification fails rdiff-backup exits with a warning text and exit code 2; so timedicer-verify.sh can distinguish between such cases (which may, subject to the contents of /opt/timedicer-verify.sh-ignore.txt, be ok), and an aborted or crashed rdiff-backup session (exit code 1).

timedicer-verify.sh is part of the TimeDicer Server software suite.

Usage

timedicer-verify.sh [options]

Options

-a [datetime] - stop (abort) verifications at datetime (with exit code 0) *
-b - fix ownership of any rdiffweb-created restore.log files
-c [number] - maximum number of concurrent verify sessions (default 1) - specifying a higher number may speed up operations provided your system does not become cpu/memory/io bound in which case it may be slower (it also increases the risk of running out of temporary disk space)
-d [datetime] - verify backup sessions back to session on, or next before, datetime *
-e - debug mode (unexpected things may happen)
-f - verify all sessions in each repository (slowest)
-h - show this help and quit
-i - continue after rdiff-backup reports an error (aka 'ignore')
-l - show changelog and quit
-m - create and use lvm snapshot of source data (discarded at end)
-n [name] - check only repositories containing text 'name' in their pathname (cf. -x)
-q - quiet (text output only on error)
-r - don't skip any repository sessions previously verified as ok
-s - list backup sessions and date/time of the corresponding previous verification session (if any), then quit
-t [tmpdir] - use tmpdir for temporary storage
-u [user] - only for named user (ignored unless run by root)
-v - verbose output
-x [name] - skip any repositories containing text 'name' in their pathname (cf. -n)
-z [name] - process any repositories with path containing text 'name' last
-1 - verify only the earliest session for each repository that falls within the specification (cf. -4,-5)
-2 - skip verification of the most recent session for each repository
-3 - skip any verifications that are in progress elsewhere, normally in a different session on the same machine, but with an additional script can also detect sessions in progress on another (i.e. mirror) machine
-4 - skip all verifications that would be performed for each repository except the earliest and instead mark them in that repo's verified.log as having been performed but with a special 'fake' flag. Sessions thus marked will be skipped on any subsequent run of timedicer-verify.sh for this repository unless -6 or -r is specified. In effect this is the same as running timedicer-verify.sh -1, or running rdiff-backup --verify for the earliest verification date, but it affects future runs of timedicer-verify.sh. Marking of fake sessions in verified.log only occurs if and after successful verification of the earliest session, or confirmation of a previous successful verification of the earliest session.
-5 [datetime] - as -4 but perform an actual verification for the earliest session even if it was previously successfully verified but such verification was performed before [datetime] *
-6 - don't skip over 'fake' former verifications (see -4,-5)

* datetime can be but does not have to be in the format used by rdiff-backup; it just needs to be understood by the 'date' command e.g. yesterday, "3 July 2012", "2 weeks ago", "2012-05-03 14:26". See 'man date' for more details.

Why timedicer-verify

rdiff-backup provides the --verify-at-time option to verify a single repository session. However this does not record the successful verification nor, more importantly, does it verify any intermediate sessions. For instance, a file that did *not* exist at the time of the earlier session, *did* exist at the time of an intermediate session, but has subsequently been deleted, could prove irrecoverable even when the earlier session had been verified. (This is not a likely scenario but it is possible.)

The only way to be confident about all repository sessions is to use rdiff-backup's --verify-at-time option to verify each session individually. This is what timedicer-verify.sh accomplishes, and although it may be a slow process, by running verification sessions concurrently and by keeping a record of successful previous verifications and not (unless required by -r or -5 options) rechecking them, it becomes manageable. A suggested use is as a weekly cron job with -d "one month ago" option (sessions earlier than a week ago will normally be skipped automatically because they were verified previously).

Dependencies

awk bash column coreutils diff findutils grep lvm(optional) lvm-delete-snapshot.sh(optional) rdiff-backup sed

License

Copyright © 2017 Dominic Raferd. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Changelog

2.60 [24 May 2017] - serious bugfix - was not patching rdiff_backup/compare.py
2.54 [29 Mar 2016] - serious bugfix - v2.25-2.53 were failing to detect many verification failures
2.53 [25 Feb 2016] - bugfix unnecessary re-verification of single session repos with -d
2.52 [12 Feb 2016] - bugfix -5 option
2.51 [29 Jan 2016] - add -5 option (renumber previous -5 as -6)
2.50 [13 Jan 2016] - add -4 and -5 options ('fake' verifications)
2.49 [17 Dec 2015] - add automatic patching of rdiff_backup/compare.py
2.48 [16 Dec 2015] - bugfix -z option
2.47 [15 Dec 2015] - add -z option
2.46 [15 Dec 2015] - bugfix -3 option
2.45 [14 Dec 2015] - add -3 option
2.44 [12 Dec 2015] - bugfix -a option
2.43 [09 Dec 2015] - add -a option
2.42 [30 Nov 2015] - bugfix for error detection, depends on modified compare.py, bugfix for -1 option
2.41 [19 Nov 2015] - bugfix for -1 option
2.40 [16 Nov 2015] - treat non-zero exitcode <> 2 from rdiff-backup as fatal (fix bug whereby killing/aborting process - exit code 1 - set a success 'flag' for the, in fact incomplete, verification action)
2.30 [08 Nov 2015] - add -1 option
2.26 [29 Oct 2015] - default concurrent sessions = 1
2.25 [21 Oct 2015] - add -e debug mode, bugfix use of /opt/timedicer-verify.sh-ignore.txt
2.24 [04 Oct 2015] - abort if an instance of timedicer-verify.sh is already running as a different process
2.23 [14 Jan 2015] - don't touch verified.log file(s) unless need to be updated
2.22 [08 Dec 2014] - bugfix, was not using exit code >0 when error found in repository (except the last)
2.21 [11 Oct 2014] - bugfix, was not counting errors that occurred with rdiff-backup --list-increments
2.20 [27 Mar 2014] - use rdiff-backup --list-increments to obtain list - avoid checking null increments
2.15 [10 Oct 2013] - tweak snapshot settings to allow concurrent runs of timedicer-verify.sh
2.14 [09 Oct 2013] - changes to text output, reassign -o option to -x
2.13 [19 Sep 2013] - add (-o ->) -x option
2.12 [19 Feb 2013] - web help layout fix, reassign -w to -b
2.10 [09 Jul 2012] - first public release
2.04 [10 Jun 2012] - improved concurrency
2.03 [09 Jun 2012] - add -c, -m, -n, -t options
2.02 [06 Jun 2012] - bugfixes
2.0105 [05 Jan 2012] - hide lvremove non-error text on removing snapshot
1.1230 [30 Dec 2011] - (create and) use /home/tmp as TMPDIR, add -l option to show this changelog
1.0818 [18 Aug 2011] - add use of LVM snapshot if available (for root only), add -u and -d options
1.0817 [17 Aug 2011] - add non-root usage, add -s option

Download timedicer-verify.sh

My Other Sites

My Programs

Here is a selection of some (other) programs I have written, most of which run from the command line (CLI), are freely available and can be obtained by clicking on the links. Dependencies are shown and while in most cases written for a conventional Linux server, they should run even on a Raspberry Pi, and many can run under Windows using Cygwin. Email me if you have problems or questions, or if you think I could help with a programming requirement.

Backup Utilities

Debian/Ubuntu kernel and LVM Utilities

Dellmont / Three / Giffgaff / Vodafone - VoIP and Mobile Phone Account Utilities

Miscellaneous Programs

Comments

No comments yet
*Name:
Email:
Hide my email
*Text:
 
Powered by Scriptsmill Comments Script