Downloading Large Files

I need to upgrade my VMware cluster from 6.5 to 7.0 U1 in order to continue to be supported by the VMUG organization. It’s something I should do anyway just to keep my VMware chops current plus continue to learn new stuff.

Problem though. The download site has an expiration tag. Since we’re on High Speed WiFi up here in the mountains, my speeds can fluctuate from 100 Megabits per second (Mb/s; 100 Megabytes per second would be MB/s) up to 1,800 Mb/s. The download is 5 Gigabytes in size so at the maximum speed of 1,800 Megabits per second, it would take about 30 minutes if it could sustain 1,800 Mb/s. So it estimates an hour and constantly jumps to 6 hours and down to 45 minutes as the speed changes.

This means I’m not getting the file as the expiration also means I don’t have access to the file any more.

I do have a remote server and after some thought, I was able to retrieve the file to that server which is on a higher speed connection. However it was still timing out afterwards when I was downloading the file from Miami to here.

Well, not the best solution but it worked, I used the Unix tools, uuencode/uudecode and split, old methods of transferring files from back in Usenet days. The uuencode utility converts the binary file into their text equivalents. This is key because split lets me split up files into some number of lines. Since uuencode creates an ~80 character line, I can count the number of lines and split them up into easily downloadable files. The positive aspect here is if the connection times out, I can resend the small chunk that failed and not have to try and transfer the 5 Gigabyte file again.

I encoded the 5 Gigabyte file into a uuencoded file of about 8 Gigabytes, and t hen used uuencode to create about 70 text chunks; about 180 Megabytes for each file.

I was able to transfer them a file at a time and a few times, the connection timed out and I had to break it, back up the files that were successful, and start again with the file that failed. Even that wouldn’t work twice for some reason but eventually they were all copied down.

I did a quick visual comparison of the transferred files, all looked correct. I concatenated them into a single uuencoded file and did a visual comparison between the two uuencoded files. Then used the uudecode command to convert it back into a binary file and visually compare the original and new binary files. Finally I used the md5sum command, which creates a fingerprint and I compared the original file’s fingerprint with the transferred one and it was a successful transfer and reassembly.

Once that was done, I can move forward with upgrading my cluster.

This entry was posted in Computers and tagged , , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *