Date last modified: Tue Jun 24 2014 1:50 PM
TimeDicer is a free file backup and recovery solution (using rdiff-backup) for Windows (Windows 8.1, Windows 8, Windows 7, Windows Vista, Windows XP, Windows 2008 R2, Windows 2008, Windows 2003 - for 32-bit x86 and 64-bit x64). It allows you to backup and then later recover your files to/from a separate machine, with an option for offsite mirroring. Even if you have subsequently altered or deleted files, and backed up your erroneous changes with TimeDicer, you can still recover versions that you previously backed up.
Under development since 2008 and with latest version dated 2014-03-24, TimeDicer backs up files to your own TimeDicer Server (which can be real or virtual), and is normally run as an automated daily routine for one or more computers located on the same local network (though it can be used in other configurations). TimeDicer is not in itself a system recovery tool, but if your system has been lost - and you have previously been using TimeDicer to backup your data - it will enable you to recover your data into a repaired or new system. It also works easily for recovering earlier versions of individual files.
TimeDicer carries out a push backup - that is, the backup is started by the source machine (the client) and data is pushed to the destination (the server). TimeDicer allows you to revert to different 'timeslices'. You could say it was dicing (that is, chopping or slicing) time - hence the name.
Not only does it give a backup which is a snapshot of files as they were at one time, but also earlier versions of these files, and files that were long ago deleted from your operating computer; so it adds the 4th dimension (time) to a backup archive, and does so elegantly, optimising both transfer (for speed) and storage (for space).
TimeDicer consists of:
- TimeDicer Client: a Windows command script (batch file) which can be run as a regular (daily) Scheduled Task to backup data from one or many Windows computers to the primary backup server. It allows backup of locked files. The latest version of TimeDicer Client is 4.0324 (dated 2014-03-24), available here (changelog here). TimeDicer Client can backup to any Linux machine which has rdiff-backup, but for best integration I recommend a dedicated TimeDicer Server. Instructions for installing and configuring TimeDicer Client are found below.
- TimeDicer Server: based on Ubuntu Server OS, for onsite backup - simple setup instructions below. It can be a virtual machine.
- TimeDicer Mirror Server: an optional machine like the TimeDicer Server, but offering secondary (e.g. offsite) backup
- No cloud services - all data remains on your own machines under your ownership and control
- All backup actions use secure encryption for data transfer
- Windows Volume Shadow Services - backup from a snapshot, no locked file issues
- Backups are stored on your own standalone TimeDicer Server - independent of your operational machine(s)
- Easy to implement for multiple machines, each with independent secure repositories on your TimeDicer Server
- TimeDicer Client has no TSR - consumes no system resources when not in use
- TimeDicer Server uses LVM, hence easy to expand and easy to backup/mirror
- TimeDicer Server can be a virtual machine, minimizing power and hardware requirements...
- ... or it can run on some of your old dust-gathering hardware
- TimeDicer Mirror Server (mirrored from Primary) provides a very robust offsite backup solution
TimeDicer is based on and uses a number of open source and free projects and software, in particular rdiff-backup, my thanks to all the brilliant people who made and make these possible. Please send any questions about TimeDicer to email@example.com.
TimeDicer Client can backup to any suitably configured GNU/Linux-based machine (i.e. server) which has rdiff-backup installed - the server can be virtual or physical. However the supported configuration uses a single-purpose TimeDicer Server running Ubuntu 12.04LTS Server (64-bit or 32-bit), obtainable here, and created as described below. Earlier versions of Ubuntu Server back to 10.04LTS should work fine, too.
I recommend that your Primary TimeDicer Server (which can be a virtual machine - see VM Server Setup) should be located on your local area network (LAN), not at an offsite or remote (internet) location. In order to back up to an offsite location you can create a Mirror TimeDicer Server (see below).
Having a single-purpose TimeDicer Server, rather than using a machine which has other functions, is strongly recommended because it makes it easy to duplicate. You can create a secondary ‘mirror’ server offsite, have this machine synchronized daily with the primary server and then, if the primary server was lost, you could easily recreate it and then mirror the data back to it from the secondary.
Running this setup will take about 20 minutes. See the separate section below if you are setting up TimeDicer Server as a VMware® virtual machine.
- Install Ubuntu Server 12.04LTS: Boot your machine with the Ubuntu Server CD. For installation choose to install Ubuntu Server, and except as specified here follow the default settings:
- For hostname I recommend 'timedicer1' [for your primary server, or 'timedicer2' for your secondary or mirror server] [a minute or two may now elapse]
- 'Full name for the new user' - e.g. 'timedicer'
- 'Username for your account' - e.g. 'timedicer'
- 'Choose a password for the new user' - specify your own, don't forget it! [a minute or two may now elapse]
- 'Encrypt your home directory?' - I strongly recommend <No>
- For partitioning method select 'Guided - use entire disk and set up LVM'
- 'Write the changes to disks and configure LVM' - select <Yes>
- 'Amount of volume group to use for guided partitioning' - I strongly recommend you select less than the full disk, say 80% - you can expand it later, and leaving some spare gives you future flexibility
- 'Write the changes to disks?' - select <Yes> [after this the base system is installed - takes say 4 minutes]
- 'Configure the package manager' - usually blank [after this packages are downloaded and configured - takes a minute or two]
- 'How do you want to manage upgrades on this system?' - I recommend 'Install security updates automatically'
- 'Choose software to install' - select: OpenSSH server [package configuration continues - takes say 6 minutes]
- 'Install the GRUB boot loader to master boot record' - usually <Yes>
- 'Installation complete' - <Continue> for reboot
- First Boot of TimeDicer Server: You can find the IP address of the machine by logging in at the local console and doing ifconfig, or by examining your local router's admin web page; once you have this then you can log in to the machine remotely with putty from Windows or with ssh from Linux. A big advantage of remote login is that you can copy and paste from here.
- Download and run timedicer-server-setup: Substituting your own email address for the one given in the 4th line below, and entering only the text after the '$' signs, do:
$ cd /optYou may have to answer some questions about your timezone and about postfix (email) configuration:
$ sudo wget -O timedicer-server-setup.sh http://www.timedicer.co.uk/server/timedicer-server-setup.sh
$ sudo chmod 744 ./timedicer-server-setup.sh
$ sudo ./timedicer-server-setup.sh -f firstname.lastname@example.org
- 'General type of mail configuration' - select Internet Site
- 'System mail name' - this is the domain name part of the source email address for emails sent by your TimeDicer Server (e.g. mydomain.com in email@example.com) - use your real domain name or at least a name that appears to be a real domain name e.g. mytimedicer.org
- Check for email: the system should have sent you an email to report that it is up and running; if you don't receive this check the syslog with tail -n 100 /var/log/syslog|grep postfix - if there are problems check the section below about Email Relaying.
- Reboot the computer: because of the many updates that may have been introduced to the machine, it is advisable to reboot now by typingsudo reboot
- Test your webserver: you should now be able to reach your new TimeDicer Server's web page from any other computer located on your lan, by using a browser and pointing it at the TimeDicer Server's ip address e.g. http://192.168.100.35.
- Secure your webserver: there is also a web interface at http://[ip-address]:8080 which is used by rdiffWeb (see below). You should log in here with your username and change the default password from 'password'.
You can create TimeDicer Server as a virtual machine (aka ‘Virtual Appliance’). This is especially appropriate if you will also (as advised) establish a secondary mirror backup server which will be in a different physical machine and location. You could even create a VM while still installing TimeDicer Server on its own hard disk so that you retain the option to move it later to a physical installation.
The instructions below are for VMware® Player and with a few adaptations should work for other VMware products. Other virtual configurations are possible but presently unsupported - for instance I know of success with OpenVZ.
The approach below allows the automated Ubuntu/Vmware Tools installation to proceed on a relatively small virtual drive without LVM, and then adds a second drive which becomes your 'home' under LVM and the storage point for your backups. If you later fill up this second drive you can use the power of LVM to add drives (real or virtual) to make more space - easily.
- If you haven't already done so, download and install your preferred VMware® product, instructions here are for the free VMware® Player (this may require a reboot).
- Obtain and place in a suitable location ubuntu-12.04[.n]-server-i386.iso or ubuntu-12.04[.n]-server-amd64.iso which you can download here.
- From the VMWare® Player window:
- choose to 'Create a New Virtual Machine'
- point the installer disc image file at the ubuntu-server iso file, click 'Next'
- supply the primary user’s Full name 'TimeDicer', User name 'timedicer', Password (and password again under Confirm) & click 'Next'
- call the VM 'TimeDicer1' (or if it is a mirror machine, 'TimeDicer2'), and select the location for the VM files, click 'Next'
- set Maximum disk size to 20GB (this is to allow plenty of room for expansion of the /tmp directory but in practice the vmdk disk file is unlikely ever to get anywhere near this size) and store as a single file, click 'Next'
- click on 'Customize Hardware...', alter 'Network Adapter'/'Network Connection' to 'Bridged', click 'OK', then 'Finish' and let the new machine start up
- The automated Ubuntu setup runs, and then the VMware Easy Install. Eventually, but without any user intervention, the machine returns to a screen showing 'ubuntu login:' prompt. Login!
- Alter the internal name of the TimeDicer Server to something more suitable than 'ubuntu', such as 'timedicer1' for instance, and then to get the change to take full effect, logout and log back in again: sudo hostname timedicer1
sudo sed -i 's/ubuntu/timedicer1/' /etc/hostname /etc/hosts
- Run timedicer-server-setup as shown above.
- sudo poweroff to shutdown the machine, then open it, unstarted, in VMware® Player and from the top bar choose 'Virtual Machine / Virtual Machine Settings'; in the 'Hardware' tab choose 'Add..' and add a second Hard Disk of whatever size you like (this is the big one that will hold your data). (This disk can be real or virtual according to your taste.)
- Start up ('Play') the TimeDicer virtual machine and login with Putty or ssh, so that you can copy/paste commands from here.
- You can find the new 2nd disk with sudo fdisk -l - it will probably be /dev/sdb
- Configure the new 2nd disk (say /dev/sdb) to create a single LVM partition. Instructions for fdisk (italics are purely descriptive) are: sudo fdisk /dev/sdb, then: n[enter] (new partition), p[enter] (primary type), 1[enter] (partition 1), [enter] (default start), [enter] (default end), t[enter] (partition type), 8e[enter] (Linux LVM type), w[enter] (write and save configuration, then exit fdisk)
- Run the make-home-lv script to create your 'home' LVM logical volume and map it to '/home' (alter the partition from sdb1 if required):
sudo /opt/make-home-lv.sh /dev/sdb1
- reboot as requested at the end.
- After rebooting, sudo df -H should show that you have lots of space in /home, which can now be used for your TimeDicer backups.
- Download the timedicer.zip file (link above), extract the contents to a temporary location, then run install.bat with administrative privileges (e.g. right-click on install.bat and choose 'Run as Administrator'). Enter the ip address of your TimeDicer Server when requested.
- Use Puttygen which is started by the install routine (or you can find in the TimeDicer folder) to generate a new public/private key pair. Under 'Parameters' you should have 'SSH-2 RSA' selected and 'Number of bits in a generated key' set to 1024. Click 'Generate' and move your mouse around a bit as advised. Save the private key as privatekey.ppk at %APPDATA%\TimeDicer (despite the warning message, don't set a 'Key passphrase'), and install the public key into your TimeDicer Server by visiting its website: to do this, select and copy all the text from Puttygen's 'Public key' box (beginning 'ssh-rsa') and paste it in to the webpage - and follow the instructions on the Server website.
- On your TimeDicer Server's website you also specify a password, this is not required to make a backup but it will be needed later when you use rdiffWeb to recover files.
- Check and if necessary modify your configuration file %APPDATA%\TimeDicer\timedicer.txt, which has been created and is opened automatically by the install routine, as required. This could be as simple as these lines:
SET server=192.168.1.35This tells TimeDicer Client that the TimeDicer Server is at 192.168.1.35, to exclude files based on the names listed in exclude-filelist.txt, and to backup your userbase (%USERPROFILE%, which contains your Desktop, Documents, Pictures, Music etc) to an archive on the Server called 'userbase'.
- Now is a good time to consider what you really want to backup! Your first backup might take a long time to complete and you want to avoid backing up unnecessary data. Have a look at the usage of your hard disk with a program like WinDirStat. TimeDicer Client should be configured according to the instructions that you can can see here. Set the line(s) in your TimeDicer configuration file, and in your excludelist, accordingly.
- Run TimeDicer Client from your start menu by right-clicking on TimeDicer.cmd and choosing 'Run as Administrator' (right-click not required for Windows XP).
- When it has completed, have a look at the log file (also available from your start menu), and also visit your TimeDicer Server's rdiffWeb webpage and login as user for this machine (see Recovering Files with TimeDicer); here you can see and recover backed up files from the server.
- If errors have occurred the log file will be automatically displayed and you can also take a look at the output in the TimeDicer command prompt window which will pause at the end of the run (only if there is an error - to prevent pausing use /x switch). If you can't solve the problem, post a message at the bottom of this page.
- Once you are happy with your backup, add it to Task Scheduler to run (say) daily; use /x switch to prevent TimeDicer from pausing on error (log file will still be automatically displayed). Unless you have switched off VSS (see below), or are using Windows XP, make sure to set it to 'Run with highest privileges'.
- Note that to run TimeDicer using Volume Shadow Services (which is the best and default behaviour) it will require administrative privileges. If these are not available to you, disable VSS by adding the command 'SET vss=n' to your configuration file.
- If at a later time you want to remove TimeDicer from your client machine, just delete the 2 folders that it uses - one at %ProgramFiles% or %ProgramFiles(x86)%, and the other at %APPDATA%.
You can also make backups from your Linux, Mac or FreeBSD computers to TimeDicer Server by running rdiff-backup. If you need help with this, send an email.
There are two ways to view or recover files, including earlier versions:
- rdiffWeb - for everyday purposes this is the recommended way to recover files because it works easily through a web interface. You login in to rdiffWeb at http://timedicer_server_ip:8080, supplying the username and the password that was set when this user was set up via your TimeDicer Server's web page. The first time that you login to rdiffWeb as this user, you need to update the Backup Locations by clicking on 'Preferences', then on 'Find and Update Backup Locations', and then on 'rdiffWeb'.
- rdiff-backup command line (--restore option) - this is the more powerful but less user-friendly way to recover files. Study the rdiff-backup manual.
It’s a good idea to update TimeDicer Server regularly because new features might be added or bugs fixed. The easy way to do this is to rerun the setup script without the 'force' option, substituting your own email address for the one given here:
Just press enter to skip all questions except the first two:
- Download and extract key scripts from www.timedicer.co.uk - y
- Download and install latest updates for distro - y
Having a second or mirror TimeDicer Server is optional but strongly recommended. Your Primary TimeDicer Server needs to be on your local LAN, so that your source machines can reach it easily, fast, and with a stable connection; but this means it probably has the same physical vulnerability as the source machines. Having a remote off-site mirror protects against a catastrophe (earthquake or whatever) that might affect both the source machines and your Primary TimeDicer Server. The mirror is essentially a copy of the Primary TimeDicer Server, updated regularly.
Why, you might wonder, does the Primary Timedicer Server have to be on the LAN? The overriding reason is that rdiff-backup, which is used by TimeDicer Client to communicate with the primary server, requires a stable network - if the connection breaks during a backup session, the data on the server can be damaged. Although rdiff-backup should correct this on the next run, repeated instability might result in irrecoverable damage to the backup archives. By contrast, rsync - used (by timedicer-mirror) to synchronize data between the Primary and Mirror TimeDicer Servers - copes well with the breaks in a session which can occur when sending data over the internet.
To create a mirror TimeDicer Server, you follow the same procedure as above, except call it 'timedicer2' instead of 'timedicer1'. The mirroring of the setup and data from your Primary TimeDicer Server is handled by the timedicer-mirror program (located in your Primary's /opt folder). Before running this for the first time, you need to add the public key of the root user of your primary machine to the list of authorised users on your mirror (secondary) machine. To do this, log in to your primary machine with putty, show the root public key sudo cat /root/.ssh/id_rsa.pub (this was created by the timedicer setup program), highlight this to copy it, then log in to your mirror machine with putty and do sudo nano /root/.ssh/authorized_keys and then right-click in this window to paste into here, then CTRL+X and answer 'y' to save this change.
The basic operation is to run, on your primary server, /opt/timedicer-mirror.sh to mirror its contents to the secondary machine. An additional feature is that if it is supplied with the mac address of the secondary server, the program can 'wake' it up and then put it to 'sleep' after completion (though this may not work over the internet if your mirror is behind a remote router because the router may refuse to admit broadcast packets such as wake-on-lan magic packets, please contact me if you have problems with this).
I suggest that for the initial tests and the first full run of timedicer-mirror you locate the secondary machine on the same local lan as the primary - it is easier for testing and much faster. First run it with -tn, which tests all connections without transferring data, then try it with -fn option which does everything except mirroring /home (and is still pretty fast), and when this is working fine, run with -l which does a full backup and provides verbose live feedback - this might take a long time (many hours even).
With this first full backup complete, subsequent runs will be much quicker and so the secondary machine could now be placed at a remote location and the backup done over the internet. (It is quite secure because it uses ssh.) Typically it is run at night when plenty of bandwidth is available. Once it is running smoothly add it, without -l option, to /etc/crontab to run each night.
The timedicer-mirror program mirrors directories /opt and /home, and the file /etc/rdiffweb/rdw.db, from the primary to the mirror. It also creates a copy of /etc/crontab and saves this in /opt on both the primary and mirror machines, and it creates users and groups on the mirror to match those on the primary, except for the root user (UID=0) and the primary user (UID=1000) which are not altered. Warning: because of the major changes which it makes to the destination machine, you should not run timedicer-mirror to a machine which has any other purpose than being a mirror for your primary.
- Managing space on TimeDicer Server is easy! You will be emailed if space is running low or you can check anytime as shown below. Login to the server with putty and:
- Refer to the email or do sudo /opt/lvm-usage.sh and look at the 'Free' space for your PV(s), to make sure you have sufficient physical storage space in the volume group. If you are short of physical space, add some as described below under 'Adding space to TimeDicer Server'.
- Refer again to the email to check the name of your logical volume (LV) e.g. /dev/timedicer/home.
- Here is a one-line example to use when you have at least 20G showing as 'Free' in your PV and in your VG. This will extend a /dev/timedicer1/root LV by 20G and then resize the filesystem inside it to fill the new space, assuming ext4 or ext3 or ext2 formatting. The LV name appears twice (here '/dev/timedicer1/root') and may be different in your case. (Note that making LVM volumes smaller is not so easy):
sudo lvextend -L+20G /dev/timedicer1/root && sudo resize2fs /dev/timedicer1/root
- Adding space to TimeDicer Server is easy too! If you are running out of physical storage space:
- Add a physical hard disk to your system (or if you are using a VM you can add a new virtual disk).
- Login in to the server with putty. sudo pvs shows drive partition(s) (in 'PV' column) that is/are already part of your LVM system and the name of your volumegroup or VG (in 'VG' column). Now find the new disk by listing all your disks and partitions with sudo fdisk -l - the new disk will not have appeared in the pvs list. Let’s assume here that the new disk is /dev/sdb...
- Partition the new disk with sudo cfdisk /dev/sdb. Create 1 primary partition for the whole disk and set this as type Linux LVM [8E], leaving it not bootable, and then write the changes.
- Add the new drive partition (same name as the new disk, with a '1' at the end) to your existing VG with vgextend. Assuming your existing VG is 'timedicer1', for example: sudo vgextend timedicer1 /dev/sdb1
- The new drive is now part of your Volume Group (VG) as an additional Physical Volume (PV) (you can confirm this with sudo pvs). To make it available to your filesystem just follow the instructions above on 'Managing space on TimeDicer Server'
- In principle each user's backup on your TimeDicer Server cannot be accessed by other users, so that one person's backed-up documents cannot be read by another person. This is because all backups are done by restricted users who do not have rights to view documents in others users' directories. But:
- rdiffWeb provides a way round this if one user knows the rdiffWeb username and password of a different user.
- The administrator can gain such access using sudo, and it is also easy for her to log in as any of the restricted users (by adding her public key to their authorized_keys files).
- It is possible to backup securely from the TimeDicer Server to an untrusted location by using encfs; at the cost of more than doubling the space consumed on the Primary TimeDicer Server you could copy the contents of /home to an encfs-protected mountpoint and then you could copy (using rsync) the underlying encrypted directory to the insecure destination.
- A slightly different approach though still using encfs can be seen here.
- Using the ecryptfs system for backups (bundled with Ubuntu) might protect against someone with physical access to the data but without administrative user rights, but not against a malicious administrator.
- TimeDicer Client communicates with TimeDicer Server using ssh with a private/public key pair and this is extremely secure, provided you keep the private key safe. Rdiffweb however does not use secure communication and so the possibility of man-in-the-middle interception while recovering data using the rdiffWeb interface cannot be excluded. You could remove this security risk by recovering data using the command line program rdiff-backup.exe.
- Lastly, please note that neither TimeDicer Client nor TimeDicer Server 'phone home', so no information about your installation (not even that it exists) is passed back to us at TimeDicer or indeed to any outside party.
- Verifying backups: The utility timedicer-verify is provided in the TimeDicer Server's /opt directory, see the help information here.
- What username should I have? When you 'Add New TimeDicer User' from the TimeDicer Server's homepage, the Server creates a Linux user, and normally you should (as advised on the homepage) set the username to the Windows machine's own 'username'-'domain', which in Windows parlance is %USERNAME%-%USERDOMAIN%, in lower case and with any spaces stripped out; for instance jimjones-widgetoffice. You can download a little batch file from the homepage to give you this so you don't have to work it out. If you choose to set something different you should also set 'user' appropriately in the TimeDicer configuration file.
- Backing up from more than one local partition: TimeDicer takes a snapshot of only one local partition, usually C:. This means that if you specify a range of datasets covering more than one local partition (e.g. C: and E:), then only one of them will be backed up from a snapshot and the other(s) will be backed up directly from an active drive, with possible file-locking issues. If you need to backup from snapshots of each local drive, create separate configuration files for each local drive and do separate runs of TimeDicer for each configuration file.
- Foreign Language Characters: If you are backing up files with unusual characters (letters) in the filenames, it is necessary that Windows fully supports these characters from the command line. If they can't be viewed correctly from the command line then they can't be backed up correctly by TimeDicer. The solution is to add the relevant language for 'non-Unicode programs' (e.g. Chinese) under the Administrative tab in Region and Language setting of Control Panel (for Windows 7, this is found under 'Clock, Language, and Region').
- Where are backups stored? Each 'archive' is stored in a physical location called a 'repository' at /home/'username'-'domain' - in the example case above this would be /home/jimjones-widgetoffice. If you specify a 'basearchive' in your configuration file then this introduces an additional level (or levels) to the repository storage - for instance if basearchive is set to 'archives' then the archives will be found at /home/jimjones-widgetoffice/archives/. For each archive, there is a sub-directory 'rdiff-backup-data' which holds, in complex and compressed form, all the data about previous versions of files. You should never normally need to look here and if you change anything in it you might break your archive.
- Relaying Emails with Postfix with/without authentication: If you need to relay your emails through an open smtp mail server (for instance, to prevent outgoing emails being blocked as spam by other mail servers) just add the line relayhost = mail.ispserver.com to /etc/postfix/main.cf, then do sudo postfix reload. But if the relaying server requires authentication for your outgoing emails then it is a bit more complicated:
- install cyrus SASL with sudo apt-get install sasl2-bin
- edit /etc/default/saslauthd to set START=yes
- start SASL with /etc/init.d/saslauthd start
- Type sudo nano /etc/postfix/sasl_passwd to set up a passwords file as follows:
mail.ispserver.com username:password(substituting your own relayhost, username and password, of course), exit (CTRL+X) and save the file (y)
- To change the ownership and permissions on this file, do
sudo chown root:root /etc/postfix/sasl_passwd
sudo chmod 600 /etc/postfix/sasl_passwd
- To load the password information from this file into the database, dosudo postmap /etc/postfix/sasl_passwd
- Do sudo nano /etc/postfix/main.cf and add or change lines as follows:
relayhost = mail.ispserver.com (i.e. your relayhost)
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
- Now exit and save the file, then do sudo postfix reload
- Test with e.g. echo This is a test|sendmail firstname.lastname@example.org
- Permissions / ACLs: There are unconfirmed reports that rdiff-backup 1.2.8, the engine upon which TimeDicer is based, is unable successfully to backup, or at least to restore, Windows ACLs. You are therefore advised to use the option setting --no-acls (set automatically for new Timedicer installations since 1 Oct 2012). This means that advanced file permissions will not be recoverable from a backup. For most users this will not be a problem for data files.
- Changing a username / Deleting a user: If you reconfigure a source machine (for instance, system upgrade) you may have a different %USERNAME%-%USERDOMAIN% than previously. To continue using the same backup identity you could add a line beginning 'SET user=' to the configuration file, but as an alternative you can change the name on the TimeDicer Server using the timedicer-rename-user script with sudo /opt/timedicer-rename-user.sh oldusername newusername. If you are using a TimeDicer Mirror Server you must also run the same script there. The same script can be used with -x option to delete a user account.
- Networking: If you move your TimeDicer Server to different hardware, or from a virtual to a real platform, the network configuration will probably not work 'out of the box'. Type ifconfig -a, the interface that is not working (e.g. 'eth1') will have no 'inet addr:' line. This is probably because the old network interface is eth0 (for instance) and the new interface is eth1 which is not configured. To fix this, do sudo rm /etc/udev/rules.d/70-persistent-net.rules; sudo udevadm trigger --subsystem-match=net; sudo udevadm settle; sudo /etc/init.d/networking restart. Now do ifconfig and hopefully it will show an external inet addr beginning with 192.168.
- Why LVM? Really the only non-standard aspect of the setup of TimeDicer Server is the use of LVM for storage. LVM permits easy expansion of the backup area by adding disks, and allows snapshots for copying data to the Mirror Server. For an introduction to LVM see here.
- EFS: Files on EFS encrypted file system (available in Professional versions of Windows) cannot be backed up using VSS. The workaround is to disable use of VSS by TimeDicer. The same may be true (untested) of data on drives which use Microsoft's BitLocker.
- rdiff-backup - the underlying backup technology used in TimeDicer
- Ascot Hotel - boutique hotel on the sunny island of Buyukada just off the coast of Istanbul, Turkey
- Ascot House - superb holiday apartments in Sultanahmet, Istanbul
- Thalia Villa - elegant 3-bedroom villa in Coral Bay, Cyprus
- Villa Chez Max - luxurious 4-bedroom villa in the Peyia Hills, Cyprus
- Free'n'Easy Windows File Server - using Devil-Linux with Samba for network storage
- Finding a 4D Backup Solution
- Web Scraping How To - extracting data from web sites
Here is a selection of some (other) programs I have written, most of which run from the command line (CLI), are freely available and can be obtained by clicking on the links. Dependencies are shown and while in most cases written for a conventional Linux server, they should run even on a Raspberry Pi, and many can run under Windows using Cygwin. Email me if you have problems or questions, or if you think I could help with a programming requirement.
- rdiff-backup-install - GNU/Linux script to install rdiff-backup. [GNU/Linux: 2012]
- rdiffweb-install - GNU/Linux script to install rdiffWeb, fixing various bugs that otherwise prevent it working correctly. [GNU/Linux: 2011-14]
- rdiff-backup-regress - GNU/Linux script to regress an rdiff-backup archive. [GNU/Linux: 2012-14]
Kernel, Boot and Device Utilities
- remove-kernel - GNU/Linux script to list the installed GNU/Linux kernels in a Debian-based distro (e.g. Ubuntu), and can be used to remove an unwanted kernel and related packages, updating grub appropriately. (Ubuntu Tweak can do the same but remove-kernel.sh is a command-line script so does not require GUI.) [GNU/Linux-Debian/Ubuntu: 2010-12]
- disk-wiper - GNU/Linux script to wipe a disk drive comprehensively, especially one connected by USB, and also check it for bad blocks. For use on a surplus drive before passing to a third party. [GNU/Linux: 2011]
- lvm-usage - GNU/Linux script to show available disk space and how it is used; run as cron job to warn if usage is above a set percentage. Provides additional information if LVM is in use. [GNU/Linux-Debian/Ubuntu: 2012-14]
- lvm-delete-snapshot - GNU/Linux script to remove LVM snapshot that has been left over by another process. [GNU/Linux-Debian/Ubuntu: 2012-13]
Dellmont / Three / Vodafone - VoIP and Mobile Phone Account Utilities
- dellmont-credit-checker - GNU/Linux script to check credit balance on many Dellmont / Finarea / Betamax portals such as voicetrading.com and voipdiscount.com. [GNU/Linux: 2008-13]
- get-vt-cdrs - GNU/Linux script to download CDRs (call detail records) from Dellmont’s voicetrading.com or voippro.com. [GNU/Linux: 2010-14]
- saynoto0870 - For people in UK, a GNU/Linux script which performs automated lookup of the www.saynoto0870.com database, finding cheap or free geographic number replacements for expensive non-geographic (087* or 084*) numbers. [GNU/Linux: 2012]
- three-credit-checker - GNU/Linux script which checks credit/calls/text/data remaining on a mobile phone account with Three.co.uk. [GNU/Linux: 2014]
- vodafone-compile-bills - GNU/Linux script which reprocesses downloaded call record 'csv' files from vodafone.co.uk so that they can be easily analysed via spreadsheet - including analysis of bundled minutes which even Vodafone do not seem able to perform! [GNU/Linux: 2012]
- sleepwalker - Windows© program which can be run from a remote machine to 'wake up' a Windows© machine behind a router, wait for it to start and then initiate Remote Desktop session. [MS Windows©: 2008-13]
- wake-thru-vigor - GNU/Linux script to wake up a machine behind a remote Draytek Vigor router. [GNU/Linux: 2012]
- huawei-fix-wol - GNU/Linux and Windows (Cygwin) script to fix local Huawei HG533 or HG532 router (e.g. from TalkTalk) so you can send WOL/wakeonlan packets from internet to a machine behind the router. [GNU/Linux & MS Windows©: 2012-14]
- huawei-ip-upload - GNU/Linux and Windows (Cygwin) script to obtain ip from local Huawei HG533 router (e.g. from TalkTalk) and upload it to remote site/file by ftp. [GNU/Linux & MS Windows©: 2014]
- form-extractor - GNU/Linux script to extract form tags from a web page or downloaded file. [GNU/Linux: 2012-14]
- tiny-device-monitor - GNU/Linux program to test webpages (including password-protected) or machines to check they are live; use as a cron job for your own websites, for hardware presenting a webpage, or for any machines with a presence on your local LAN or on the internet. [GNU/Linux: 2009-2014]
- dutree - GNU/Linux program to show a tree-style list of files and directories at the specified location and greater than the specified size (default 1GB). [GNU/Linux: 2012-2013]
- man2text - GNU/Linux one-liner program to convert man page output to straightforward text. [GNU/Linux: 2012]
- Accounts - Multi-business multi-currency accounting software, uses Access [MS Windows©: 1996-2014]
- Rents Program - Residential lettings/landlord front office program, with many special features for UK market [MS Windows©: 1991-2014]
One request. Could you produce a how-to on manually installing this on a general distro?
I have an Arch linux system all ready setup with rdiff and would like to just add or "convert" it to timedicer. I've looked through your install script and honestly my bash skills are a little wanting.
Thanks for your ingenuity and skills!
The Arch equivalent command would be pacman -S --noconfirm <package name>.
The packages your install script grabs are all in the Arch repositories although under different names.
wol, apache, php, rdiff-backup, and sshfs that I can see.
But the differences in layout might be a tough can of worms. I'm pretty sure the web root is elsewhere for instance, and Arch uses BSD style init scripts instead of sysvinit. It looks like this would clobber a machine that already had Apache and a site setup on it rather than just placing the new files in a webroot subfolder for instance.
But I have a test box I can play with and see what breaks (and maybe even learn more Linux in the process :)
We can take this to my sporadically checked email if you like.
If you use an existing machine as TimeDicer Server you have the problem of how to backup that machine - whereas a standalone TS can backup *all* your existing machines, then you have an offsite mirror to backup TS and so all your data on all your machines is safe.
I think it is important that setup of TS is pretty easy and standard. Say your existing primary TS fails, you just create a new one, run timedicer-mirror to mirror contents from your mirror TS to the new primary and you now have your running primary TS again. If the TS configuraton is bespoke (because the machine is doing other stuff) you would need a different (bespoke) approach to recover a bust TS.
So, we could look at modifying the config script to work for a clean install of Arch, or alternatively I could look at putting some instructions on this page as to how anyone could retrofit TS onto an existing machine (any Linux distro), but these would describe the required steps (e.g. install rdiff-backup, install rdiffweb), not how to do them.
This is a very nice utility. I've been able to incorporate into a nightly backup process for server servers, and it is working well.
In the log file, I notice that VSHADOW.EXE provides several writers and components, such as the IIS Metabase Writer, with the \IISMETABASE component. Can these components be used as sources in config file? For example, could I specific \IISMETABASE (or even \Registry) as a source and have rdiff-backup copy the contents to my backup server?
Your idea is interesting but I can't see a way to do it. TimeDicer uses the vshadow utility which creates a snapshot from a given source (in this case, a drive) and maps this to a share, typically \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1.
I can't see how to achieve this for data from the other VSS writers you have identified. Let me know if you find out more.
first of all thank you for all this, great job!
Amazed that I found your page so "late", as I have been looking for similar backup solutions for quite a while and never came across yours.
The link you mention,
"Ubuntu 10.04LTS Server (64-bit or 32-bit), obtainable here."
leads to a 404.
Would the one searched via google "Ubuntu 10.04LTS Server" be OK, or could we use the newer version?
Also, is there a way to use your software without any SSH?
On one hand traffic is local and does not need SSH.
On the other, mirroring via remote location, it goes through a VPN tunnel so again, SSH is not needed.
It would be nice to be able to exclude that by choice, as I think that SSH through a shaky connection adds to the lack of reliability of those connections.
However a VPN tunnel is rock solid.
I had the same issues with rsync & rdiff-backup.
I also do strongly recommend to include date stamps to you changelog, it will give visaitors a sense of when what was daone, if it is up to date at all etc.
Also, is the source available? Does your program "phone home"?
I found that a few do, and that is quite disturbing, as it would give the writer of the software an opportunity to get into your system.
Again, thanks for all your efforts!
From West LA, CA
The link to Ubuntu Server works fine, maybe there was a temporary problem with their website? I recommend you stick with 10.04LTS because the instructions here are tailored for that version. If you use a later version it may be fine but some steps may be different.
I'm not convinced of the value of connecting without ssh, I have never heard before of it contributing to connection problems. I very strongly recommend that you only run backups from TimeDicer Client to TimeDicer Server over a reliable local connection, not over internet (even VPN), but this is because rdiff-backup has limited recovery capability if the connection gets broken. rsync is good at recovering from broken connections and so this is used for mirroring the Primary TimeDicer Server to an offsite mirror server.
Allowing users not to use ssh would open up potential security problems, although I accept that over a VPN it may not have much practical value (except against internal snooping). But you could look into it yourself if you like, TimeDicer Client uses plink to connect to the Server and you could modify the code lines (2 in all I think, one to test the connection [which can be skipped], and one to run the backup).
Regarding the changelog, the dates are there really in the version numbers: [year mod 100].[2-digit month][2-digit day]
The source is all available here, and no the software does not 'phone home'.
When running timedicer-server-setup it ends with an error: line 253: mail: command not found
Then I installed mailutils and run server-setup again. No error and I got an email with setup-results.
In most of the places, where I plan to use Timedicer, they cannot / don't want to have a local server. You very strongly recommend only to run backups from TimeDicer Client to a server over a reliable local connection, not over internet.
Is the rdiff-backup's limited recovery capability (if the connection gets broken), on the current data or only on the old data-sets ?
Let's say, the most valuable data, a family have, is their digital photos of children growing up. There is very little change in those photos, only new ones coming in, so they can live with uncertanty about rolling back to old versions.
If there is a failed rdiff-backup session and as result the archive is not marked as 'clean' then on the next session rdiff-backup should automatically 'regress' the archive to its previous clean state. This does usually work, but if there is a breakdown while this regression is happening you might then get irrecoverable damage, but still you can probably retrieve manually all or nearly all of the latest versions of current files.
I have to say that you are not the first to request a non-local primary backup, and there is no problem with doing it technically, but a local primary backup (e.g. on VM) with a secondary offsite backup is more robust I think.
On my windows 7 laptop I have a directory
that I do not want to backup.
How do you put a directory in exclude-filelist.txt ?
If only the latter works, let me know and I will fix it so the former works too.
Might be a problem with network-drives or systems with data on a D:-partitions from the factory.
The workaround for now is, in exclude-filelist.txt, not to use the drive letter for directories in C: drive.
Do you have any ideas about, how to install Timedicer as a service, so it can run at night witout any user logged in on the server?
I know how to change port in plink.exe, when I write by own bat-files, but cannot find a way to pass a port-number through Timedicers advanced cmd-file.
Is it possible to change port in Timedicer ?
I think it should be possible to set TimeDicer as a Scheduled Task for a user and it would still run when user is not logged on? But I haven't tried this.
which sets the port (default 22).
Existing Servers can be updated by following the instructions at 'Updating Server' above; to avoid email message about deprecated code when you add a user, choose to reinstall rdiffWeb.
We find that a bit lang, and want to change it to just "longcomputername".
On the client we can do:
But on the server we cannot add a new TimeDicer User. It ends with:
there is already a user with this name. Aborting...
Is it impossible to reuse the last part of a username ?
No new directory under /home an no new .ssh. No new line in /etc/passwd.
1) I added as a task to Task Scheduler with Action
of "C:\Program Files (x86)\TimeDicer\TimeDicer.cmd"
When the script finishes the Task Scheduler is unware. When I look at the Task Manager, cmd.exe is still running. I know TimeDicer is finished because it finished writing its output log.
2) I specified a Backup path of
and rdiff-backup fails. If I change that to
C:\Users\myusername\Documents,Documents it works.
Yet using %USERPROFILE%\Desktop works fine.
"Once you are happy with your backup you should add it as a task to Task Scheduler to run (say) daily. [When invoking from the Task Scheduler, use the /x option]"
Regarding your second problem, the reason is that Windows 7 has a 'Documents' folder not 'My Documents'. So this should work for you:
I think that environment would be much more comfortable (for me at least) than fragile .cmd scripts. Have you considered that?
My first backup last night showed a few lines of
'The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.'
in the logs.
However, one of the directories to be backed up was not even attempted. It appears in the summary of the html log, but there is no mention of it in the rest of the log and it did not run. However, the lastchanges*.txt file was updated as if it worked. Very scary.
Other changes that I think are essential are
o The ability of the backup process to first run a --remove-older-than 30D
o The ability to email the rdiff output when finished rather than remembering to check a log file on several machines.
Regarding working in python, that is beyond me, it sounds like a good new project if you want to try it?!
Regarding the messages you saw, these all relate to the problem of rdiff-backup.exe hanging on to the log file after it has finished. Please try the new version of TimeDicer.cmd (2.0101) which may resolve this problem.
Your other suggestions can I think be handled by you calling TimeDicer.cmd from another batch script which can perform what you want.
I also notice that even though I added the /x command so that the TimeDicer.cmd exits, it must still pause or something if there is some sort of error. This causes a cmd.exe to be left running and the task scheduler to never say it is finished and no future attempts to backup will occur.
For the second problem, the latest version 2.0102 should address this, please let me know if you still experience it.
However, considering the limitations and quirks, as well as trying to decipher Windows batch file programming, I'll think I'll move on to writing my own using python.
It should be fairly simple and way fewer lines of code to accomplish what the Windows batch file is trying to do. And doing it in something like python will make it readable and more easily maintainable.
When I have something available, I could drop you an email if you would be interested in trying/testing it out. Since it will use vshadow and rdiff-backup, if should be compatible with your backend server setup.
The html client log is now working pretty well. The client program no longer pauses by default, only if it has hit an error (pausing can as before be overridden with /x switch).
Email me if you are aware of any bugs or have any feature requests.
I have a problem with EFS (encrypted files) not getting backed up, they are just ignored. I was hoping they would be backed up in their encrypted state. I'd guess that the issue is with rdiff-backup. Any idea how I can get this working?
A workaround may be to disable VSS by adding this line in your configuration file:
I'm using Linux for the first time in order to get your application up and running in a test environment. First of all I'd like to thank you for the great job you've done both building TimeDicer and writing this step-by-step installation and configuration manager.
I'm kind of stuck in the middle of the server setup process. I've just reached step three - downloading and running the timedicer-server-setup, but when I get to the
part, I get the following message:
timedicer-server-setup: Permission denied
Cannot write to 'timedicer-server-setup' (Permission denied)
I realize that I'm on the absolute beginner level as far as Linux is concerned and I'd like to apologize for wasting your time on that issue... yet, your help will be highly appreciated!
sudo wget http://www.timedicer.co.uk/server/timedicer-server-setup
sudo chmod 744 timedicer-server-setup
The 'sudo' command means that the actual command following is run not as the normal user but as 'root' (you may have to enter your password) - root has full permissions to go anywhere, do anything, unlike the normal user which is restricted. So by using 'sudo' the command should now work.
I have updated the web page too with this advice. Let me know how you get on.
The fix worked for me. Now I got a bit further. When I get to the "Download and extract key scripts from www.timedicer.co.uk" part I receive the following message:
/opt/timedicer-server-setup: line 183: syntax error near unexpected token 'fi'
/opt/timedicer-server-setup: line 183: 'fi'
sudo wget http://www.timedicer.co.uk/server/timedicer-server-setup
(If you provide your email address and tick the option to 'notify me about new comments on this page' you will hear immediately when a comment is added, you can always unsubscribe later...)
Trying to install on Ubuntu Server 12.04 get the following error:
sed: can't read /usr/lib/python2.7/dist-packages/rdiffWeb/db_sqlite.py: No such file or directory
An error occurred modifying /usr/lib/python2.7/dist-packages/rdiffWeb/db_sqlite.py, aborting...
Install CherryPy and rdiffWeb [FAIL]
I can see this is coming from rdiffWeb which I also tried manually downloading and installing with the same error.
Thanks for your very well implemented TimeDicer wrapper and setup, and particularly the documentation you've provided. I've been wanting a decent backup solution and after a long search was concluding that rdiff-backup was the best option for me, and with TimeDicer it is even better.
FYI I have set TimeDicer up on a Windows 7 x64 client for backup to my Excito B3 Linux server (Debian Squeeze). This was fairly straightforward, but required some manual tweaks. There's only one problem (non-critical), and not with TimeDicer.
With both rdiff-backup and TimeDicer I have a problem with the first full backup failing after around 45-50G and no info as to why. It is not the same file. It isn't to do with access problems.
Doing manual rdiff-backup - I tried increasing -vX levels in rdiff-backup, but this wasn't greatly helpful:
- if the level is too high (e.g. -v9) the backup aborts after a few seconds!
- just right (-v7) and the backup completes but is way slower!
- too low (-v6) and it fails around 45G-50G, but will complete if repeated.
Caveat: These conclusions are based on only a few runs, so not necessarily 100% true.
So far, if I just redo the backup command it completes ok, "topping up" the failed backup with the remaining files (up to 160G).
1) I wonder if you or anyone else has experience with very large backups using rdiff-backup, and any reasons why it should choke in this way after a certain amount, but then be able to complete even though the remaining backup is even bigger!?
2) Also, is there any reason NOT to just repeat the backup command after a failure like this?
3) Do you have a mailing list or any way to be kept informed of updates?
4) Setting up for B3 Linux, I encountered some problems. One was failure of "Setup user 'admin' private/public key". Am I right to think this is only needed for mirror to another server? (I completed the process manually, but would like to know what this is for in case I have not yet got it setup correctly).
FYI: Your Windows setup script failed to start PuTTY at any time. I think there might have been some other minor glitches, but the information it provided made it easy to complete the process anyway (e.g. by starting PuTTY manually).
For example, I had some trouble getting ssh to work without prompting for a password (as usual!), and this issue was not caught by timedicer.cmd. Instead it alternated between telling me it couldn't connect to the server, or issuing endless "Wrong passphrase" messages which was rather confusing.
I hacked timedicer.cmd to see what was going on and then used plink.exe manually to debug this area. The issue was that I hadn't set up the public key properly on the server, which I find is often tricky, solved by copying the public key from the PuTTYgen dialog instead of copying the from one of the saved key files. (I think when you do the latter the newlines that break the key over several lines mess things up.)
I'd like to suggest a security tip regarding the public keys: in the authorised_keys file (on the server) it is possible to restrict use of each key to an IP address range, and to a particular command. Worth considering!
I would really appreciate your response to my questions. Let me know if you would like a new web host because as amongst other things I operate a hosting service, and would be delighted to host TimeDicer et al.
I wonder about an option to turn off use of ssh as I think this probably slows down large backups significantly, and on a local network the risk of a man in the middle attack is very low.
What do you think?
BTW - Thanks for a really good implementation of rdiff-backup for windows users!
I haven't implemented this feature but you should be able to run it successfully from the command prompt (e.g. putty) on the server. You would need to specify the base archive correctly, perhaps this was your problem?
If you need to 'regress' your archive, that is remove the most recent, rather than the oldest, backups, then I have a script to do this. It can be helpful if a recent backup has bloated your archive because it inadvertently included some big files you don't want (and which you have subsequently excluded from backup through the excludelist).
your project looks very promising to me, and it's
great you provide this kind of 'out-of-the-box-solution'.
But nevertheless I'd like to use your TimeDicer-
client to backup to rdiff-backup on an existing server.
I think I don't need the web frontend for a start
(no user access - managed by admin only ;),
so what do I need to configure on the server to use
rdiff-backup with your TimeDicer-Client?
I probably need the accounts and homes it uses,
and I probably need to put the public key into the .ssh subdirectory of the user on the server side?
I didn't try it yet, but am I missing someting,
or do you think I can start this way?
Best regards and thanks for TimeDicer - in advance
Thank you so much for TimeDicer. I know that once I get it working it will do exactly what I need.
I believe I have followed your instructions meticulously and everything has gone like clockwork until I try to do a backup using timedicer.cmd. What I get is the following:
Session Sat 11/09/2013 13:52
using configuration 'C:\Users\Robert\AppData\Roaming\TimeDicer\timedicer.txt'
FATAL ERROR: Network error: Connection refused
Cannot make connection to email@example.com
TimeDicer failed with error 9 - cannot connect to Server
Log at C:\Users\Robert\AppData\Roaming\TimeDicer\timedicer-log....
Press any key to continue . . .
I have used Putty to login to my timedicer server using the public key which exists as authorized_keys in the .ssh/ directory and it works perfectly, but when I try to run the cmd file whether as a user or administrator I get the error above.
I have also tested using the default timedicer.txt file and one where I commented out all but the SET server=192.168.1.5
Any suggestion where I should next look would be much appreciated.
Poisoned Pen Press--Discover Mystery
6962 E. 1st Ave. #103, Scottsdale, AZ 85251
I can't thank you enough for all the help you've provided in making this backup solution available and helping me to get it working. I'm a complete Linux newbie and your instructions were perfect. It appears to me that the few problems I had were entirely due to my inability to read carefully enough.
For a small office solution to the problem of reliable, restorable, safe, and easy-to-access backups Timedicer is superb. And equally appealing was the ability to take an older machine which was no longer in service, put in a new hard drive, and have Timedicer Server running quickly.
Poisoned Pen Press
I do think I've run into a problem with long file paths (more than 255 characters). I consistently get the following exception with certain files:
Exception '[Error 3] The system cannot find the path specified: 'W:/a/long/path/name/*.*'' raised of class '':
File "rdiff_backup\robust.pyc", line 32, in check_common_error
File "rdiff_backup\rpath.pyc", line 998, in listdir
They backup fine after reducing the path length to 255 characters or less, but that is not always an option.
If you don't need backup of the folders that are beyond the 255 character limit, you could try adding them to your exclude list, though I don't know if this would work.
A rather horrible workaround is to move that portion of the source which is beyond the 255 character limit to a different location where the pathname would be shorter. I appreciate this may not be possible in your case...
Also it has been suggested that Long Path Tool can help (find it with google), I haven't tried it and I am not sure if you have to pay to use it. If you do give it a go I would be very interested to know if it works.
There were no problems manipulating the folders through windows explorer (win7 x64, enterprise edition), so I didn't need the long path tool.
Using subst could be a neat workaround, but you might have to put 'set VSS=n' in your configuration file i.e. don't backup from a snapshot. I suspect snapshots will not be possible from a virtual drive letter as allocated by subst.
TimeDicer concept, implementation and documentation by Dominic firstname.lastname@example.org.