Large shutil copies are slow!

[chris.eastwood]

Hey All,

This is my first post, so I'll do my best...

I'm writing a Python app (using wxPython for the GUI) to copy large amounts of files. Each file is about 8-15 MB and there could be as many as 150,000 files. I'm currently using shutil (either .copy(), .copy2() or .copyfiles() ) to manage the copies but I've noticed they are taking about twice as long as a normal finder copy (I'm working on a MAC). Is this just the Nature of shutil, or is the strange way i've set up my copy... maybe something else I'm missing?

Here's the code for the copy, it's a function that is calls itself to recursively copy subfolders and files. (I purposefully didn't use os.walk() or shutil.copytree() )

def CopyDir(self, srcPath, dstPath, qItem, filesCopied=0):
'''
Function to recursively copy all files and subfolders within a given
srcPath to the dstPath. qItem is the reference to the queue object
for feed back of the progress of the copy.
'''
        
        dirAndFileList = os.listdir(srcPath)

        dirList = []
        fileList = []

        for i in dirAndFileList:
            if os.path.isdir(os.path.join(srcPath, i)):
                dirList.append(i)
            elif os.path.isfile(os.path.join(srcPath, i)):
                fileList.append(i)

        dirList.sort()
        fileList.sort()

        for directory in dirList:
            if qItem.HasCancelled(): #If user has cancelled  the copy, quit
                break
            os.mkdir(os.path.join(dstPath, directory))
            newSrc = os.path.join(srcPath, directory)
            newDst = os.path.join(dstPath, directory)
            #Call itself to do move into the newly created subfolder and repeat...
            self.CopyDir(newSrc, newDst, qItem, filesCopied)

        for file in fileList:
            if qItem.HasCancelled(): #If user has cancelled the copy, quit
                break
            # Copy the file
            shutil.copyfile(os.path.join(srcPath, file), os.path.join(dstPath, file))
            self.filesCopied += 1
            #Feedback to the progress bar of the queue Item
            if self.filesCopied % 10 == 0:
                try:
                    qItem.UpdateProgressBar(self.filesCopied)
                except:
                    continue

Thanks in advance!
Chris

[jlm699]

I too have found shutil copies to be very time consuming, especially when copying between a hard-drive and a thumb-drive.

If you are writing a MAC-only utility you could simply use a system call to the recursive copy function, ie (in Linux):

import os
#
src_dir = '/home/usr/test_source'
dst_dir = '/home/usr/test_dest'
#
r = os.system( 'cp -fr %s %s' % (src_dir, dst_dir) )
if r != 0:
    print 'An error occurred!'

In the past I've replaced shutil with my own file copy method:

def CopyFile(self, old, new):
    """ This function is a blind and fast copy operation.
          old and new are absolute file paths. """
    fsrc = None
    fdst = None
    keepGoing = False
    max = os.stat(old).st_size
    try:
        fsrc = open(old, 'rb')
        fdst = open(new, 'wb')
        dlg = wx.ProgressDialog("File Copy Progress",
                "Copied 0 bytes of " + str(max) + " bytes.",
                maximum = max, parent=self, style = wx.PD_CAN_ABORT
                | wx.PD_APP_MODAL | wx.PD_ELAPSED_TIME | wx.PD_REMAINING_TIME
        )
        keepGoing = True
        count = 0
        #
        while keepGoing:
            # Read blocks of size 2**20 = 1048576
            # Depending on system may require smaller
            #  or could go larger... 
            #  check your fs's max buffer size
            buf = fsrc.read(2**20)
            if not buf:
                break
            fdst.write(buf)
            count += len(buf)
            (keepGoing, skip) = dlg.Update(count, "Copied " + \
                str(count) + " bytes of " + str(max) + " bytes.")
        dlg.Destroy()
    finally:
        if fdst:
            fdst.close()
        if fsrc:
            fsrc.close()
    #
    return keepGoing

Note that the above function was developed under wxPython and used a progress dialog box to display the status, so remove that bit if this is a simple script...

**EDIT: I also found portions of this function on the internets and adapted it for my usage; however it's been so long that I forget the source.

[chris.eastwood]

Thanks!

I'm actually developing it to run on multiple platforms so I may use your second code example to start and make my own copy file method. We'll see what happens...