Saturday, January 27, 2007

Image a drive with dd

I ended up removing Linux from the laptop cause I was running out of space on the Windows partition, and wanted to re-install Windows, since I was having problems with it (Windows, not Linux). So I installed XP with SP2. I then patched it up fully, installed Firefox, Zone Alarm, and Clamwin. The used portion of the drive was about 3GB.

To create an image of this setup, I decided to try the dd command from a Linux LiveCD. First step was to move the data to the beginning of the disk. Running Windows defrag a few times is supposedly all you need. From my experience, the visual after the defrag shows some data is left in the middle of the drive. If you have an idea about this, please post a comment.

Next I booted up to Kanotix, (my favorite LiveCD), and ran the commands:
su
mkdir /mnt/smb
mount -t smbfs -o lfs,rw,username=jbess //10.0.0.5/Backups /mnt/smb

lfs is needed because your image may exceed the 2GB default limit. Unless you want to make small chunks of your image, this is what you must use. The 10.0.0.5 is my Mandriva server running Samba and ProFTPd.

Next, mount the drive to be imaged (for me I just clicked on the drive on the desktop that Kanotix automatically put there), and then run df to check the used drive space. I had 3013344KB used, so I rounded to 3100MB, or 3.1GB. After you get the numbers, be sure to unmount the drive, or the next step may have problems.

To compress the image, I used gzip. I typed:
dd if=/dev/hdc bs=1M count=3100 | gzip > /mnt/smb/laptop_SP2.img.gz

Be sure to replace the count with how many megabytes are used on the drive. I ended up with a 1.8GB file on my server. That's about a 42% compression, which is less than what the man page for gzip states. It took 70 minutes at 766KB/s (according to the stats). Quite slow. Too slow actually. I thought it was samba being slow. So I tried ftp.

From the ftp> prompt, I typed:
put "|dd if=/dev/hdc bs=1M count=3100 | gzip" laptop_SP2.img.gz

The | tells ftp to run a shell command, in this case, dd. This was a little better at around 1.6MB/s, more than twice as fast. During both samba and ftp I was monitoring the transfers with KNemo traffic plotter. I noticed there was dips or lulls in traffic being sent every few seconds, and lasted for several seconds. I thought that maybe running it through gzip was slowing the stream down, and with no buffer in place, the data transfered stopped, which increased the time. So I then took out gzip. Wow, I was impressed!

The samba share command was:
dd if=/dev/hdc bs=1M count=3100> /mnt/smb/laptop_SP2.img
Took 15 minutes at 3.5MB/s on average. I tried again with FTP.

With FTP, the command was:
put "|dd if=/dev/hdc bs=1M count=3100" laptop_SP2.img
That finished in 8.5 minutes, with a 6.4MB/s transfer rate.

Curious, I googled and found this article about zip program comparisons:
http://www.linuxjournal.com/article/8051

According to the article rzip give the fastest speed and compression. I checked and rzip is NOT installed on the Kanotix LiveCD, but the next best one was, lzop. I ran my tests again.

smb (samba):
dd if=/dev/hdc bs=1M count=3100 | lzop > /mnt/smb/laptop_SP2.img
Time - 23 Minutes, Rate - 2.3MB/s

ftp:
put "|dd if=/dev/hdc bs=1M count=3100 | lzop" laptop_SP2.img.lzo
Time - 20 Minutes, Rate - 2.7MB/s

Using lzop got me a lower compression, at 33%, but at two-thirds faster speed compared to gzip when using samba, and just under two times faster with FTP. Here are the full comparisons:

SMB - No Compression, 3.5 MB/s
SMB - gzip (42% compression), 766 KB/s
SMB - lzop (33% compression), 2.3 MB/s
FTP - No Compression, 6.4 MB/s
FTP - gzip (42% compression), 1.6 MB/s
FTP - lzop (33% compression), 2.7 MB/s

My conclusions? Faster is better in my situation. I want an image quickly. I can always zip the image later if I need the space. If I do need to zip it, I will use gzip, or install rzip and use that. For my setup, FTP is king in the transfer speed. If you do want to compress the image file, and have a decent transfer speed, I suggest the lzop methods, and obviously FTP is faster. My preferred method: FTP with no compression.

This is the results of my setup. Your numbers will probably vary. Do some testing yourself. Post a comment if you do with your results. I will write a post about restoring an image later on. There are lots of articles already if you google it, but basically you just reverse the commands.

Edit: upon attempting to restore this image, I found some problems, mostly boot record stuff. I am sure this works, just need to play with it some more when I get a chance.

1 comment:

Mikey said...

Hi there!

There seems to be a lot of random hidden things you can do with the dd command. I haven't really explored that but reading this makes me want to now.

Anyways, I wanted to mention about the defrag showing data in the middle of the drive. From what I understand, this happens when a file gets fragmented across two or more platters. When that happens, the file is not considered fragmented since there is not a gap in the file within the disk. A file is only considered to be fragmented when there is a gab between the blocks of a file.

--fluffymikey