File size is increased after pickle

Expand Post »

I'm reading in a file and sending the data (once encrypted) to a dictionary, with a hash of the data before and after encryption. I then pickle the dictionary but find the file size is massive compared to the source file size. If I write the encrypted data straight to a file the size is identical to the source. Any idea why my pickled file is so large?

Python Syntax (Toggle Plain Text)
#Encrypt data and get hashes        
    def encryptAndExportFile(self, key, inFile, outFile):
 
        openInFile = open(inFile,"rb")
        inFileSize = os.path.getsize(inFile)
        inFileData = openInFile.readlines()
        openInFile.close()
 
        """ initialise cipher """
 
        cipher = AES.new(key, AES.MODE_CFB)
 
        """ initialise MD5 """
 
        m = hashlib.md5() #hash
        h = hashlib.md5() #hash of encrypted dataq
 
        encryptedData = []
 
        for data in inFileData:
 
            m.update(data) 
            encData = cipher.encrypt(data)
            h.update(encData)
            encryptedData.append(encData)
 
 
        hashResult = m.digest()
        encHashResult = h.digest()
 
        return hashResult, encryptedData, encHashResult
#Encrypt data and get hashes        
    def encryptAndExportFile(self, key, inFile, outFile):
        
        openInFile = open(inFile,"rb")
        inFileSize = os.path.getsize(inFile)
        inFileData = openInFile.readlines()
        openInFile.close()
        
        """ initialise cipher """
        
        cipher = AES.new(key, AES.MODE_CFB)
        
        """ initialise MD5 """
        
        m = hashlib.md5() #hash
        h = hashlib.md5() #hash of encrypted dataq

        encryptedData = []
        
        for data in inFileData:
            
            m.update(data) 
            encData = cipher.encrypt(data)
            h.update(encData)
            encryptedData.append(encData)
            

        hashResult = m.digest()
        encHashResult = h.digest()
        
        return hashResult, encryptedData, encHashResult

Python Syntax (Toggle Plain Text)
def storeEncryptedObject(self, obj, path):
 
        outFile = open(path, 'wb')
        pickle.dump(obj, outFile)
        outFile.close()
def storeEncryptedObject(self, obj, path):
        
        outFile = open(path, 'wb')
        pickle.dump(obj, outFile)
        outFile.close()

Similar Threads

file size in C++
File size in C (Win32) in C

zyrus001

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

14 posts
since Sep 2009

Permalink

Nov 3rd, 2009

Re: File size is increased after pickle

Using protocol 2 greatly improved file size, which only increases by 5% in some cases.

Python Syntax (Toggle Plain Text)
def storeEncryptedObject(self, obj, path):
 
        outFile = open(path, 'wb')
        pickle.dump(obj, outFile, protocol = 2)
        outFile.close()
def storeEncryptedObject(self, obj, path):
        
        outFile = open(path, 'wb')
        pickle.dump(obj, outFile, protocol = 2)
        outFile.close()

zyrus001

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

14 posts
since Sep 2009

Permalink

Nov 3rd, 2009

Re: File size is increased after pickle

Thanks for letting the rest of us know!
Most folks use protocol=-1 as the highest possible protocol.

vegaseat

Moderator

Reputation Points: 1333

Solved Threads: 1403

DaniWeb's Hypocrite

Offline

5,792 posts
since Oct 2004

This thread is solved

Either the thread starter or a moderator has marked this thread as solved. You can most likely trust the responses and answers given. There is most likely no reason for any further responses to be posted here. If you have a related question, please start a new thread in this forum instead.

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.

Previous Thread in Python Forum Timeline: Removing a substring from string

Next Thread in Python Forum Timeline: Ceasar Cipher

DaniWeb Message

Cancel Changes

User Name
Password