Subscribe

Receive updates via email:

 Updates via RSS

Tag Cloud

Blog Archive

Monday, May 19, 2008

Fixed! ROBOCOPY Usage on Linux EXT2 / EXT 3 File Systems

For some time, I have been a fan of the robust copy feature that has been available for several Windows releases including Windows XP, Windows 2003, and of course Windows Vista. In fact, I changed from using programs like SecureCopy or built previously built in utilities like cp (copy) or xcopy.

In fact my primary to secondary disk backup method for some time has been to use the following command syntax to mirror directories to the backup drives:

ROBOCOPY "C:\My Music" "B:\Backup\My Music" /MIR

With the above syntax, it creates an exact mirrored copy of the data in the directory "C:\My Music" and copies (purging deleted files and directories) the entire tree to "B:\Backup\My Music". This works like a charm on NTFS files systems.

Now enter into the picture my new NAS device which is running a Linux based EXT2 / EXT3 file system and we start to see some problems. I first noticed the symptoms I saw is that when I was testing out the backup copies to the new location on that Linux based network attached storage device. Specifically speaking, I would see the identical file being copied over and over again.

Thinking that it most likely had to do with either an archive bit or similar type backup issue, I first tried adding the /M switch to the command line to forcibly set the archive bit with every file.

That new syntax looked like: ROBOCOPY "C:\My Music" "B:\Backup\My Music" /MIR /M

However, that was not much of a help -- so my attention turned to the Linux EXT2 / EXT3 file system structure. Some research by entering "robocopy ext2" into Google revealed some information buried ironically within a NetGear message board. The posts revealed the following information about inodes that I was not too familiar with:

The standard ext2/ext3 inode has 3 time fields, last inode change, last file data change and deletion time. These timestamp fields have a resolution of 1 second. The only time field that maps from NTFS to ext2/ext3 is the file data modification time.

That revealed the issue -- the NTFS timestamp was more precise than the resolution for the EXT2 / EXT3 inode file systems. Therefore, a different option had to be tweaked with ROBOCOPY to allow for a less precise timestamp measurement.

Fortunately, out of the box there is a /FFT switch for ROBOCOPY which specifically accomplishes this task by assuming FAT File Times which translates to a more forgiving 2-second granularity. This is more than sufficient to account for the one-second granularity that the EXT2 / EXT3 file systems support.

So, I modified my backup command one more time by adding '/FFT' to the end of the command:
ROBOCOPY "C:\My Music" "B:\Backup\My Music" /MIR /FFT

I gave the command a couple of tests and bingo, that solved the problem of the same files copy over and over again. If you are encountering a similar issue, then most likely the /FFT switch may resolve issues that you have copying files between NTFS and other non-NTFS systems including Linux and others.

Good luck!

24 comments:

Claudia & Marc said...

Great blog entry, was having the same issue for some time, your fix works perfectly fine! Thanks Marc

Ken Hanscom said...

Good to hear Mark, it has been smooth sailing since I made that change!

-Ken

Woongbin Kang said...

Thanks a lot! I was wondering why robocopy shows me a bunch of older/newer, which was a total nonsense to me.

Anonymous said...

Was having the same issue on my Western Digital MyBook World NAS. This worked perfectly. Thanks!

Anonymous said...

I also thank you! This solved my backup problem with my new Buffalo NAS.

Anonymous said...

Thankyou Thankyou Thankyou Thankyou Thankyou Thankyou Thankyou !

I use perl with robocopy, and just migrating to ext3 server noticed uber slow copies, where it should have only been deltas..

Anonymous said...

Thanks, Ken. I was getting this exact issue with copying to/from an EMC NAS system.

Your blog is much appreciated!

tristof said...

Thank you very much! I had the same problem too when using robocopy to backup my data to a Western Digital network drive. I tried a lot of options for robocopy but without any success. Your solution worked perfectly ;-)

Anonymous said...

Man, you saved my life with this trick!

Anonymous said...

Thanks very much, I had a similar problem when mapping a drive with webdav and the /FFT parameter solved the problem

Anonymous said...

Nice post this worked like a champ

Anonymous said...

Whee, tanks from Finland! Our small office backups run a lot smoother now, since it knows to skip most of the old files. How could one know the meaning of all those switches :D

Anonymous said...

Awesome tip! I switched to rsync because of this problem with my Dlink DNS-323. But it was so freaking slow! Probably 50-60 times slower. /fft on robocopy works great!

GorillaD said...

Rocking tip! We have an Intel NAS device and were experiencing the same behavior. You were on of the top google hits, simple solution, and most importantly it works! :)

Thanks
--Andrew Duey, MCSE

Anonymous said...

Thank you, this works great. I was pulling my hair out over why a backup to my Dlink 323 was taking days!

Anonymous said...

Your fix for me worked fantastic for my new NAS devices. I backup at least 3TB a night and could not figure out why it was pulling files that were the same date. Thank you very much.

Will

Anonymous said...

thx: EXACTLY what I wanted to know!

Anonymous said...

Thanks....just spent the last week working on the Mirror prior to the NAS being attached....

Attached the EMC NAS and it ran and ran and ran....

Then some testing and reailzed the issue.. I'm with Will above...lots of data being redone over and over. Thanks.

Anonymous said...

Thanks !!!

Wes Weston said...

Firstly, thank You!

I saw this switch but was unable to think it could solve my iPhone sync problem. But now, if I know that the target or source drive is a ext2 or ext3 filesystem I could use the famous FFT switch and all is running smooth.

Anonymous said...

Wow! After years of having this problem its finally working correctly!

Windows 7 64-bit to Synology DS410 NAS server

Anonymous said...

Thank you very much! You saved me!!!!

Anonymous said...

The /fft switch does help a lot but I still have a few files that have not changed that still copy every time robocopy is run. I can live with it since its just a few and doesn't slow the process much but I would still really like to understand why this is happening. Anyone else see this too?

Anonymous said...

Thanx!
My DLINK DNS-325 will have an easier life after reading this.