menuimage

Description

Usage

Example

Options

Details

TimeDicer_BaseID

Warning

Dependencies

License

Changelog

Download

My_Other_Sites

My_Programs

Comments

timedicer-mirror v6.2 [23 Nov 2016] by Dominic

Description

timedicer-mirror.sh mirrors (synchronises) critical data from a local (source) machine to a remote (destination) machine using rsync and other tools. It is part of the TimeDicer suite (http://www.timedicer.co.uk); its purpose is to update the rdiff-backup repositories, and other TimeDicer-specific setings, of a Mirror TimeDicer to be the same as those on a Primary TimeDicer Server. It is always run from the sending machine (Primary).

On the remote (destination) machine timedicer-mirror.sh creates any missing Linux users (ID>1000) and groups, it also synchronises Rdiffweb users and the directories /opt and /home. (It does not mirror the operating system.) Copies of /etc/crontab and /etc/rc.local, appended with a dot and the machine's BaseID, are placed in source machine's /opt before it is mirrored to remote /opt.

The drawn-out part of the task is mirroring /home. timedicer-mirror.sh ensures that all new /home data is successfully transferred to the remote machine before any of the old data there is replaced. Only after successful transfer of all new data is the old /home data on the remote machine replaced with the new. Failed sessions, say caused by internet connection problems, may delay the updating of the remote machine but should not corrupt its data.

Tested under Ubuntu Server 14.04LTS and 16.04LTS, and designed for use as a nightly cron job with TimeDicer Server.

Usage

timedicer-mirror.sh [options] dest_address

Example

sudo /opt/timedicer-mirror.sh -v 192.168.100.130

Options

-a: active progress mode - shows file transfer progress, implies verbose
-c: determine if file backup is needed by checksum rather than comparing file date, time & size, and also compare usage of /home on source and destination (much slower)
-e: force no snapshot - backup directly from /home not from a snapshot created using btrfs or LVM, even if such is possible
-f: fast mode - don't check space on destination machine and don't backup /home (see -o)
-h: show help and exit
-i: check space on destination machine as normal, but ignore results (see also -o)
-k num: specify remote WOL UDP port 'num' (default 9; see also -m)
-l: show changelog and exit
-m macid: specify MAC address of destination machine - if asleep a WOL 'magic packet' will be sent to wake it (see also -k)
-n: poweroff destination machine at end ('night night')
-o: don't check space on destination machine - faster (see -i, -f)
-p num: specify destination SSH TCP port 'num' (default 22)
-q: quiet mode - no output unless there is a problem
-r: restart i.e. stop any prior instance, then continue
-s: stop any prior running instance of this program
-t: test mode - no changes on destination (see also -f)
-u: use pre-existing mounted source snapshot (e.g. as left behind after a previous incomplete run of timedicer-mirror.sh)
-v: verbose mode
-z: use compression when transferring files (rsync -zz is used if rsync version >=3.1.1)

Details

Before timedicer-mirror.sh can work, you must have added the root public key of source machine (/root/.ssh/id_rsa.pub) to destination machine in file /root/.ssh/authorized_keys.

If a prior instance of timedicer-mirror.sh is already running, a new instance will abort unless called with -r.

If source /home is on a btrfs or LVM volume, timedicer-mirror.sh will create and use a temporary snapshot of such volume for the operation - this can be overridden with -e option.

TimeDicer BaseID

Every Timedicer Server machine has a BaseID - a single digit which is normally 1. This default value is overridden based on the last character (digit) in file /opt/baseid or, if there is no such file, the last character (digit) of the machine's hostname. timedicer-mirror.sh will only backup /home subdirectories that are homes for users with uids in the range (1000xBaseID)+1 to (1000xBaseID)+999 - default 1001-1999 - these are considered the 'local TimeDicer users'.

Users created through the TimeDicer web interface will have uids and gids which conform to this specification, and it is required for mutual mirroring (bi-directional) or if you run a TimeDicer Server Pool. If you are setting up more than one TimeDicer Server and you might one day want to extend this to have more than two such servers - or where the second machine might have its own local TimeDicer users and you might want to run timedicer-mirror.sh back from there to the first, as well as vice-versa - you should ensure that they have different BaseIDs e.g. by naming the first 'timedicer1', the second 'timedicer2', the third 'timedicer3' etc.

Warning

Because of the major changes which it makes to the destination machine, you should not run timedicer-mirror.sh to a machine which has any other purpose than being a TimeDicer Server, either as a mirror for your source machine or with its own local TimeDicer users in which case it must be configured with a different BaseID to the source machine (see above). timedicer-mirror.sh may alter users and groups on the destination machine as well as overwriting some /home subdirectories and /opt. You have been warned!

Dependencies

[m/g]awk bash coreutils diffutils grep net-tools(hostname) iputils(ping) openssh ps rsync sed util-linux [wakeonlan]

License

Copyright 2016 Dominic Raferd. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Changelog

6.2 [23 Nov 2016] - install sqlite3 if required and missing
6.1 [16 Nov 2016] - add error handling on retrieving destination users/groups
6.0825 [25 Aug 2016] - allow more time to umount snapshot; update help text
6.0808 [08 Aug 2016] - small tweaks
6.0525 [25 May 2016] - small text and code tweaks removing source LVM snapshot
6.0430 [30 Apr 2016] - few small code tweaks
6.0427 [27 Apr 2016] - make source/destination user-checking compatible with TimeDicer Server Pool concept, remove redundant -b option
6.0426 [26 Apr 2016] - update destination rdiffweb database in consistent way for TimeDicer Server Pool (kudos: Grant Emsley), also allocate 80% of free VG space for snapshot instead of fixed 4 Gibibytes
6.0425 [25 Apr 2016] - fix timeout error creating snapshot ('special device does not exist')
6.0424 [24 Apr 2016] - really fix fatal error (since 6.0422)
6.0423 [23 Apr 2016] - bugfix fatal error
6.0422 [22 Apr 2016] - add -b switch, only mirror data for local timedicer users based upon the source machine's BaseID
6.0115 [15 Jan 2016] - if rsync operation fails, check whether source snapshot has failed and abort if it has
5.1226 [26 Dec 2015] - bugfix copying of /etc/rdiffweb/rdw.db if destination directory doesn't exist
5.1210 [10 Dec 2015] - bugfix naming of LV (kudos bug report: Alex Racov)
5.1123 [23 Nov 2015] - increase timeout from 2 mins to 30 mins, use new rsync compression (-zz), use --append-verify (later removed)
5.0922 [22 Sep 2015] - fix bug identifying LV
5.0908 [08 Sep 2015] - updated help info
5.0813 [13 Aug 2015] - do not delete extraneous files from destination's /opt
5.0513 [13 May 2015] - minor improvement to LV identification
5.0331 [31 Mar 2015] - improved stop/restart code
5.0314 [14 Mar 2015] - speed optimisation (combined some remote ssh calls)
5.0313 [13 Mar 2015] - make compatible with btrfs and/or non-LVM-mapped filesystems, add -e (force-no-snapshot) option
5.0217 [17 Feb 2015] - minor tweaks
5.0130 [30 Jan 2015] - minor text changes and code tidying
5.0127 [27 Jan 2015] - minor text changes and bugfix
5.0123 [23 Jan 2015] - minor text changes and code tidying
5.0119 [19 Jan 2015] - move mac address to new -m option
5.0118 [18 Jan 2015] - fix bugs in file exclusions
5.0113 [13 Jan 2015] - code tidying
4.1230 [30 Dec 2014] - abort on error when obtaining destination backup space
4.1224 [24 Dec 2014] - remove compression if retrying after timeout (rsync bug 7757)
4.1220 [20 Dec 2014] - add -u option, change -n option to put destination to sleep, add 120s timeout to rsync, fix a bug where it failed to detect rsync failure in verbose or active progress modes
4.1211 [11 Dec 2014] - faster -f fast mode
4.1121 [21 Nov 2014] - add -i option and change -o option
4.0822 [22 Aug 2014] - improved/bugfixed comparing source and destination users
4.0814 [14 Aug 2014] - more bugfixes for large source and destination sizes
4.0813 [13 Aug 2014] - bugfixes for root LVM and large destination size, allow MAC addresses with dashes
4.0808 [08 Aug 2014] - test destination /var/lib/sudo/timedicer/0 is accessible
4.0730 [30 Jul 2014] - bugfix counting space used by destination /home/backup
4.0409 [09 Apr 2014] - resolve destination address explicitly (uses ping)
4.0324 [24 Mar 2014] - bugfix comparing users on source and destination
4.0123 [23 Jan 2014] - use specified SSH port for rsync too
4.0115 [15 Jan 2014] - re-open remote machine if powered off during -c checking
3.1011 [10 Oct 2013] - quit if any ongoing rdiff-backup verification processes
3.0918 [18 Sep 2013] - minor text changes
3.0916 [16 Sep 2013] - bugfix calculation of destination /home/backup space
3.0828 [28 Aug 2013] - test that destination /home is writeable
3.0803 [03 Aug 2013] - bugfix calculation of destination space
3.0530 [30 May 2013] - minor text changes
3.0523 [23 May 2013] - remove dg834g_src option
3.0514 [14 May 2013] - correct help text description of -o option
2.1117 [17 Nov 2012] - change option -o to check for enough space on destination instead of *not* checking (default behaviour is now *not* to check)
2.1107 [07 Nov 2012] - fix bug preventing creation of users and groups on destination
2.0929 [29 Sep 2012] - remove erroneous fail message on local ping
2.0821 [21 Aug 2012] - local ping test informational instead of critical, use /tmp instead of /var/tmp, deprecate dg834g_src, change previous -P (ssh port) switch to -p and previous -p switch ('live progress') to -a
2.0818 [18 Aug 2012] - add -k option to specify remote wakeonlan port
2.0721 [21 Jul 2012] - add -P option to specify remote SSH port, remove warnings about RSA host key not in list of known hosts
2.0715 [15 Jul 2012] - add -o option to skip remote machine space checking
2.0627 [27 Jun 2012] - fix for machine with hyphen in name and re-enable backup of /opt (n/w since 2.0317)
2.0607 [07 Jun 2012] - allow concurrent rdiff-backup verify/restore operations
2.0514 [14 May 2012] - improved help text
2.0412 [12 Apr 2012] - add code (for checksum mode) comparing usage of /home
2.0404 [04 Apr 2012] - further minor changes and bugfixes
2.0330 [30 Mar 2012] - bug fix for recording size of files
2.0325 [25 Mar 2012] - add recording of size of files transferred so that on subsequent runs the required space on destination can be better estimated
2.0317 [17 Mar 2012] - add explicit -z compression option, transfers no longer compressed by default
2.0313 [13 Mar 2012] - add -l 'changelog' option, change previous -l (i.e. 'live') option to be -p (i.e. 'progress')
2.0308 [08 Mar 2012] - renamed program from 'timedicer-mirror' to 'timedicer-mirror.sh', improved help
1.1023 [23 Oct 2011] - exclude /home/tmp/*
1.1012a [12 Oct 2011] - copy .ssh and other hidden folders in user directories on fast copy (disabled in 4.1211)
1.1011 [11 Oct 2011] - further fixes and variable name changes
1.1010 [10 Oct 2011] - further fixes for base filesystem search
1.1009 [09 Oct 2011] - modify search for base filesystem
1.0818 [18 Aug 2011] - remove temporary files on successful completion

Download timedicer-mirror.sh

My Other Sites

My Programs

Here is a selection of some (other) programs I have written, most of which run from the command line (CLI), are freely available and can be obtained by clicking on the links. Dependencies are shown and while in most cases written for a conventional Linux server, they should run even on a Raspberry Pi, and many can run under Windows using Cygwin. Email me if you have problems or questions, or if you think I could help with a programming requirement.

Backup Utilities

Debian/Ubuntu kernel and LVM Utilities

Dellmont / Three / Giffgaff / Vodafone - VoIP and Mobile Phone Account Utilities

Miscellaneous Programs

Comments

No comments yet
*Name:
Email:
Hide my email
*Text:
 
Powered by Scriptsmill Comments Script