Memory Error

Please support our Python advertiser: Programming Forums - DaniWeb Sister Site
Reply

Join Date: Apr 2008
Posts: 8
Reputation: Edwards8 is an unknown quantity at this point 
Solved Threads: 0
Edwards8 Edwards8 is offline Offline
Newbie Poster

Memory Error

 
0
  #1
Jul 7th, 2008
So, I wrote a little script to convert tifs into jpegs. The tifs are about 600mb in size, I reduce the size, and then convert. Here is the code in its entirety:

  1. ## First we import the image library, Image object, and garbage
  2. ## collector, as well as os
  3. from PIL import Image
  4. import gc, os
  5. ## Get the path for the directory to be converted, then change to that path
  6. print "Please enter the path to the files that you wish to convert."
  7. path = raw_input()
  8. os.chdir(path)
  9. ## For each file in every subdirectory, see if it's a tif file
  10. for root, dir, files in os.walk(path):
  11. for name in files:
  12. if name[-4:] == ".tif":
  13. print 'Opening ' + name
  14. os.chdir(path)
  15. im = Image.open(root + '/' + name)
  16. x, y = im.size
  17. ## Resize the tiff tile at a 2/3 scale. Make a new directory to
  18. ## mimic the file heiarchy of the original file, only on the C
  19. ## drive instead of wherever it was to begin with.
  20. print 'Resizing'
  21. im2 = im.resize((int(x*.66), int(y*.66)), Image.ANTIALIAS)
  22. n = 'c' + root[1:]
  23. savedfile = n +"/jpegs/"
  24. try:
  25. os.makedirs(savedfile)
  26. os.chdir(savedfile)
  27. except WindowsError:
  28. os.chdir(savedfile)
  29. savedfile = name[:-4] + ".jpg"
  30. ## Save the file as a jpg, with a high quality
  31. print 'Saving'
  32. im2.save(savedfile, quality=85)
  33. del im
  34. del im2
  35. ## Force a memory dump. Otherwise memory will get cluttered
  36. ## up, very quickly too.
  37. gc.collect()
  38. print 'Memory Wiped'

Due to space limitations in the workplace, I need it to copy the directory structure onto the c drive of the computer the script is running off of, hence the funky directory dance. I'm still kinda new to all of this, and I thought the garbage collector would help me out. But, it'll wipe the memory at first, but things will still accumulate. After converting about 30-40 images (and I need to convert literally tens of thousands) the program crashed with a MemoryError. Page File usage was at about 2G. There is about 3 gigs of ram on this PC, which is running XP. Any tips or advice?
Reply With Quote Quick reply to this message  
Join Date: Jul 2006
Posts: 608
Reputation: jrcagle is on a distinguished road 
Solved Threads: 150
jrcagle jrcagle is offline Offline
Practically a Master Poster

Re: Memory Error

 
0
  #2
Jul 8th, 2008
Well, a quick tangential note: your method for extracting the extension

name[-4:]

is buggy. What if the file extension is ".jpeg"?

Better:

  1. filename, ext = os.path.splitext(name)
  2. if ext == ".tif":
  3. do stuff ...
  4. savedfile = filename+".jpg"
  5. do more stuff...
Reply With Quote Quick reply to this message  
Join Date: Jul 2006
Posts: 608
Reputation: jrcagle is on a distinguished road 
Solved Threads: 150
jrcagle jrcagle is offline Offline
Practically a Master Poster

Re: Memory Error

 
0
  #3
Jul 8th, 2008
WRT to the main problem, the memory error, I agree that there's something wrong, and it doesn't appear to be your code.

You might try e-mailing the PIL support at

image-sig@python.org

Jeff
Reply With Quote Quick reply to this message  
Join Date: Apr 2008
Posts: 8
Reputation: Edwards8 is an unknown quantity at this point 
Solved Threads: 0
Edwards8 Edwards8 is offline Offline
Newbie Poster

Re: Memory Error

 
0
  #4
Jul 8th, 2008
If it's anything but a tif, it'll be ignored, as I don't need to do anything with it. However, I think that the method you showed me is much more practical than what I've got (I wasn't aware of that tool to grab the extension name). So, I'll be modifying it. Thanks for the tip. And I'll shoot them an email. Thanks for the help, really. Happy to know that I at least had the code right.
Reply With Quote Quick reply to this message  
Join Date: Jun 2005
Posts: 146
Reputation: G-Do is an unknown quantity at this point 
Solved Threads: 28
G-Do's Avatar
G-Do G-Do is offline Offline
Junior Poster

Re: Memory Error

 
0
  #5
Jul 11th, 2008
Hi Edwards8,

It turns out that PIL will let you read pieces of an image file in sequence without loading the entire thing into memory. Fred Lundh has a post about it on a python mailing list here. I don't know if this will work for TIFs, but you can always give it a shot.

It sounds like some piece of the file is being kept resident in memory, despite your deleting and garbage collection - and from the sizes you quoted, it looks to be about a tenth of each file. Maybe PIL is reading the files in pieces, and hanging onto the last piece of each? Or maybe the file is being kept open in some way, even though you delete the image object? One thing you could try is to create a file handle on the image file, then use Image.open() on that, instead of the file name. Then, when you are finished, close the original file handle. This may convince PIL to let go of whatever data it's holding onto.

Unfortunately, I don't know a whole lot about PIL, so I'm shooting in the dark, here.

Hope this helps!
Vi veri veniversum vivus vici
Reply With Quote Quick reply to this message  
Join Date: Dec 2006
Posts: 1,056
Reputation: woooee is a jewel in the rough woooee is a jewel in the rough woooee is a jewel in the rough 
Solved Threads: 298
woooee woooee is online now Online
Veteran Poster

Re: Memory Error

 
0
  #6
Jul 12th, 2008
Try running everything after "if name[-4:] == ".tif":" in a separate function. The memory should be garbage collected when the function exits.
Reply With Quote Quick reply to this message  
Reply

This thread is more than three months old.
Perhaps start a new thread instead?
Message:


Thread Tools Search this Thread



Tag cloud for Python
About Us | Contact Us | Advertise | DaniWeb | Acceptable Use Policy | RSS Feed

©2003 - 2009 DaniWeb® LLC