When configuring a File Replication job, the user will be able to select folders and files to be backed up using a tree-like interface, similar to the existing “Files and folders” settings tab of NTBackup jobs.
There are 4 main factors in the design of the File Replication Engine:
- Envisaged applications — data backup, VM backup, etc.
- Mapping of the source path to the destination path
- Backup History
- Media Rotation (for portable media, like USB HDD)
1. Envisaged Applications
We anticipated that the File Replication Engine will be very useful for:
- Data backup — thanks to the single instance store architecture, it’s possible to store tens or hundreds of revisions on a single medium. For example, if you have 100GB of data and approximately 1GB changes each day, you will be able to store around 60 “snapshots” of your data on one 160GB USB HDD! This feature will far exceed the typical capabilities of VSS snapshots. To get your data back — just copy the entire snapshotted directory back to your server, or individual files if you prefer.
- VM backup — to backup client VMs, simply use the File Replication Engine to replicate the entire VM directories to a USB HDD. If your server goes down, simply plug the USB HDD into a different machine, install your VM software (eg. VMware Server) and run the virtual machines — this reduces your downtime to virtually zero!
- Exchange backup — backup your Exchange database files by a VSS-compliant file-copy operation, enabling you to forklift the database easily
… if you have other scenarios, we’d like to hear about them to make sure we cater for them all!
2. Mapping of the Source Path to the Destination Path
A single destination folder may be selected for each job. The destination folder may be on a local drive, NAS or removeable media. There are two options for how source files and folders are mapped to files and folders within the destination folder. The user interface will allow selection of one of these options, e.g. using a checkbox.
In the discussion below, it is assumed that the destination folder is X:\Backups.
Option 1: Full path
The full path to each source file will be encoded within the destination folder.
Example
| Source File | Destination File |
|---|---|
| C:\Foo\bar.doc | X:\Backup\C\Foo\bar.doc |
Option 2: Relative to common ancestor
Rather than encoding the full path for each file, use only a sub path relative to the lowest common ancestor of all selected files and folders.
Example 1. One source folder selected
| Source File | Destination File |
|---|---|
| C:\Foo\bar.doc | X:\Backup\bar.doc |
| C:\Foo\baz.doc | X:\Backup\baz.doc |
Common ancestor is C:\Foo so backup file paths are relative to this folder.
Example 2. Multiple source folders
| Source File | Destination File |
|---|---|
| C:\Foo\Bar\baz.doc | X:\Backup\Bar\baz.doc |
| C:\Foo\Qux\quux.doc | X:\Backup\Qux\quux.doc |
Common ancestor is C:\Foo so backup file paths are relative to this folder.
Example 3. Multiple source drives
| Source File | Destination File |
|---|---|
| C:\Foo\Bar\baz.doc | X:\Backup\C\Foo\Bar\baz.doc |
| D:\Qux\quux.doc | X:\Backup\D\Qux\quux.doc |
Selections span multiple drives so there is no common ancestor and the full path to each file, including drive letter, is used in the backup.
3. Backup History
A number of backup schemes will be provided to allow historic backup to be retained. Historic backup stored on the same volume will use a single instance store to ensure that only one copy of unchanged files is stored on the disk. When a scheme with backup history is in use, there will be an extra level of folders within the backup destination folder to store historical backups.
Example (assuming “Full path” option chosen)
| Source File | Destination File |
|---|---|
| C:\Foo\bar.doc | X:\Backup\2008-05-13\C\Foo\bar.doc |
| X:\Backup\2008-05-12\C\Foo\bar.doc | |
| … |
This scheme has daily backups stored under destination folder\date.
4. Media Rotation
For portable media, such as USB disk drives, the user will be able to set up media rotation schemes to allow regular backups to be rotated among different volumes. Each volume will hold a set of complete (i.e. not incremental) backups. The single instance store architecture will ensure that identical files are shared between backups within a volume, but not across volumes. For example, if a user has 3 USB HDDs and chooses to rotate them one after the other, the list of backups on each drive might be:
- Drive 1:
- 2008-05-01
- 2008-05-06
- Drive 2:
- 2008-05-02
- 2008-05-07
- Drive 3:
- 2008-05-05
- 2008-05-08
Similarly, if the user has 5 drives (one for each day of the week) then each drive would contain backups of data for every week, and the backup history goes back in one week increments.
Feedback
If you have any comments or suggestions about this proposal, or scenarios for which you think the File Replication Engine would be useful, please let us know by leaving a comment.
8 thoughts on “File Replication Engine UI proposal”
I love the idea of file copy backup with deduplication.
Please provide for:
encryption of files on backup disk
multiple backups on ONE disk
disk rotation on GFS, but one disk for weekdays, one for weekends, one for Month, but switch to DVD for yearly backups.
Backup Assist is a wonderful program!
I hope in your new version you have a feature where only files of specific dates are backed up ?
Hi there i’m happy to hear that we are going to make this fantastic product even better . I may suggest we use technology similar to ter copy which works well with large files.
At the moment i’m using script to move a 140gb backup file to removable hds it works well this way because BA backups to the always attached drive first then it copies to the removable drive. but because of the system I lose features like media management (how many copy’s I keep in the removable drives ) and reporting, so it will be fantastic to have this new feature.
Also important is to keep BA using the internal HD as backup destination first and then copying to the destination.
Thanks fro you effort
Aruna
Hi , been a while, hope you are well.
We definitely have this as an issue and would love to see the replication engine
Most common requirements –
Replication of image repository to multiple external drive
Replication of Images from NAS to NAS
Happy to open a dialogue.
Cheers
Steve
Dear Linus,
I’m very interested in your new File Replication Engine. With ever increasing amounts of data to backup and large removable disks, incremental replication makes much more sense than full backups – especially where combined with single instance storage.
I support a number of clients who are using BackupAssist and like its simplicity and excellent reporting. However, some are moving towards MS DPM because of its efficiency in the backup traffic and superfast restore times. But it is complex and its tape management is a nightmare.
So I see a good but simpler file replication system would have great appeal. Given the lower overall traffic it is quite feasible to replicate a number of systems to a server/disk system in a nearby building via 1Gb Ethernet and so improve security and protection of the backed up data. The DPM model is to then stream this backed up store to tape at predefined intervals which can of course be done during the day without impact on the main systems.
Of course for single remote servers with a USB disk for backup the replication model also works well – so long as it can cope with a swap of the USB disk on regular intervals. One client is doing this with Robocopy and as all the USB disks in his rotation are named the same the server doesn’t seem to notice the change. But it would be great to have better control over the whole process though.
DPM of course understands SQL, Exchange and Virtual Machines data, but I know you do too so incorporating them (perhaps as an add-in) would be helpful but not essential to the basic product.
I hope this helps and look forward to hearing more about your development.
Regards
Mark
Over the last 2 days, I’ve received a fair amount of feedback via email. I’ve posted this feedback here, so that we can all share the ideas and hopefully build an awesome product!
I’d like to be able to take a backup at a client site and put it on a RAID 5 array at my location and just have incremental, single instance backup sent from client site to my site (or a service) to have an off-site backup with minimal traffic.
I’d like to have the ability to right click on the backup file and mount it as a drive letter to just drag and drop files that need to be recovered.
Are you having a beta program for this? I may be able to get you some volunteers for it from the DFW-SBS group. I’m not sure what kind of traction SBS 2008 is going to get right out of the gate.
This single instance storage capability sounds like a great thing for small businesses.
Thank you for the opportunity to participate in the design process.
All of these improvements look fantastic and we are really looking forward to this as extended times of existing full backups are thrashing servers nowdays and mirroring is the answer. Having users be able to logon to backup data while it is restoring back to new hard disk would be good feature. Like relpication software can do. Also my big thing this year is trying to inform users/ companies what is being backed up and where etc. so the monitoring alerting and reporting of backups mirroring etc. could really be beefed up to give various reports with maybe graphs etc. This would help us place some burden for the backups back towards the clients who after all should be involved in their backups which we administer for them.
Also would like to see some deduplification tools appearing to help guide our clients on keeping data sizes down.
Nick