 |  |  |  |  |  |  |  |
Opening 100+ files at once in python
Please support our Python advertiser: Programming Forums - DaniWeb Sister Site
Thread Solved |
•
•
Join Date: May 2009
Posts: 70
Reputation: 
Solved Threads: 2
Junior Poster in Training
Hey guys,
A while ago, with your help, I was able to create a code which scans a data file for names, and when a new name is found, it appends it to a list. For each entry is the list, the code then opens a new file for each entry. Using the same code, I am trying to parse huge data files, with over 100 names each. Python doesn't like to open such a large number of files and is complaining:
I was wondering if any of you knew a way to bypass this, whether it be an import or otherwise? I'd prefer to not completely restructure my code if there is a simple import that can allow python to open more files.
A while ago, with your help, I was able to create a code which scans a data file for names, and when a new name is found, it appends it to a list. For each entry is the list, the code then opens a new file for each entry. Using the same code, I am trying to parse huge data files, with over 100 names each. Python doesn't like to open such a large number of files and is complaining:
Python Syntax (Toggle Plain Text)
I was wondering if any of you knew a way to bypass this, whether it be an import or otherwise? I'd prefer to not completely restructure my code if there is a simple import that can allow python to open more files.
•
•
Join Date: Jul 2007
Posts: 489
Reputation:
Solved Threads: 76
Posting Pro in Training
Can you post some of your code? I'm confused as to why you need to hold so many files open at one time... could you just cycle them one by one and not have them all open?
Last edited by shadwickman; Jun 24th, 2009 at 4:41 pm.
"Two good old boys in a fire-apple red convertible. Stoned. Ripped. Twisted. Good people."
- Hunter S. Thompson
my photography
- Hunter S. Thompson
my photography
•
•
Join Date: Dec 2005
Posts: 5,850
Reputation: 
Solved Threads: 749
The maximum number of open files a process is allowed to have comes from the Operating System.
It's usually possible to change this (but details vary), but doing so almost certainly requires "admin/root" privileges.
It's also something which shouldn't be done lightly to save a bit of inconvenience on your part.
Just consider it in your future designs that you don't have an unlimited number of open files to play with.
It's usually possible to change this (but details vary), but doing so almost certainly requires "admin/root" privileges.
It's also something which shouldn't be done lightly to save a bit of inconvenience on your part.
Just consider it in your future designs that you don't have an unlimited number of open files to play with.
My question : Do we really need this? Why?
If you are needing to open 100+ files at once, you should perhaps re factor your code and design.
BTW, if you are on *inx (Unix,Linux), you can run the following command to know what is the maximum number of open file descriptors:
Note that $ is my shell prompt.
Hence, there can be at most 95141 possible file descriptors opened at once.
To change this use: where 104854 is max number which you want.
To check how many of file descriptor are been used;
The first number is total allocated file descriptors (the number of file descriptors allocated since boot)
If you are needing to open 100+ files at once, you should perhaps re factor your code and design.
BTW, if you are on *inx (Unix,Linux), you can run the following command to know what is the maximum number of open file descriptors:
Note that $ is my shell prompt.
Python Syntax (Toggle Plain Text)
To change this use: where 104854 is max number which you want.
Python Syntax (Toggle Plain Text)
To check how many of file descriptor are been used;
Python Syntax (Toggle Plain Text)
Siddhant Sanyam
(Not posting much)
Migrate to Standard C++ :When to tell your C++ Code is Non-Standard.
Please Read before posting: How To Ask Questions The Smart Way
(Not posting much)
Migrate to Standard C++ :When to tell your C++ Code is Non-Standard.
Please Read before posting: How To Ask Questions The Smart Way
•
•
Join Date: Jul 2008
Posts: 1,054
Reputation:
Solved Threads: 265
I agree with everyone else here. There should never ever be a reason to open this many files at once. Ever.
You should explain what you're actually trying to do so that we can help you come up with an intelligent solution.
You should explain what you're actually trying to do so that we can help you come up with an intelligent solution.
•
•
Join Date: May 2009
Posts: 70
Reputation:
Solved Threads: 2
Junior Poster in Training
Ok, here is what I am doing:
First my code scans a large data file. In column 10, the genetic data is listed by "name", and there are about 110 different names total. Whenever it gets to a new name, it stores it in a list, so I have something like:
For each name in the name list, I want to store a file of the same name, eg:
The program will rescan the data file, and for each name, that line gets written into the appropriate file.
Because the raw data file is so large, I can't just open up one file for a name in the list, scan the whole data file, append entries for only the first name (AluSp.bed) and then close. Doing so would require a 110 scans of a 50 million line file. What I am doing is scanning the raw data file once, while all the 110 name files remain open, and for each line in the data file, that line gets written to the appropriate name file.
Is that clear?
The code is in place, so if you'd prefer to look at it, I can post it.
First my code scans a large data file. In column 10, the genetic data is listed by "name", and there are about 110 different names total. Whenever it gets to a new name, it stores it in a list, so I have something like:
Python Syntax (Toggle Plain Text)
For each name in the name list, I want to store a file of the same name, eg:
Python Syntax (Toggle Plain Text)
The program will rescan the data file, and for each name, that line gets written into the appropriate file.
Because the raw data file is so large, I can't just open up one file for a name in the list, scan the whole data file, append entries for only the first name (AluSp.bed) and then close. Doing so would require a 110 scans of a 50 million line file. What I am doing is scanning the raw data file once, while all the 110 name files remain open, and for each line in the data file, that line gets written to the appropriate name file.
Is that clear?
The code is in place, so if you'd prefer to look at it, I can post it.
•
•
Join Date: Oct 2006
Posts: 2,279
Reputation:
Solved Threads: 176
•
•
•
•
Originally Posted by Salem 
The maximum number of open files a process is allowed to have comes from the Operating System.
...
python Syntax (Toggle Plain Text)
No one died when Clinton lied.
•
•
•
•
Originally Posted by shoemoodoshaloo
Ok, here is what I am doing:
First my code scans a large data file. In column 10, the genetic data is listed by "name", and there are about 110 different names total. Whenever it gets to a new name, it stores it in a list, so I have something like:
Python Syntax (Toggle Plain Text)
For each name in the name list, I want to store a file of the same name, eg:
Python Syntax (Toggle Plain Text)
The program will rescan the data file, and for each name, that line gets written into the appropriate file.
Because the raw data file is so large, I can't just open up one file for a name in the list, scan the whole data file, append entries for only the first name (AluSp.bed) and then close. Doing so would require a 110 scans of a 50 million line file. What I am doing is scanning the raw data file once, while all the 110 name files remain open, and for each line in the data file, that line gets written to the appropriate name file.
Is that clear?
The code is in place, so if you'd prefer to look at it, I can post it.
•
•
Join Date: May 2009
Posts: 70
Reputation:
Solved Threads: 2
Junior Poster in Training
•
•
•
•
Originally Posted by The_Kernel
There's no need to keep all the files open at the same time though. Instead of having a list of 100 file handles replace it with a list of the filenames, then each time you want to write to one of them just open the file, write to it, then close it.
•
•
•
•
Originally Posted by shoemoodoshaloo
Won't that the process of opening and closing the file cause it to get overwritten, instead of the line being appended?
 |
Similar Threads
- Matlab To Python (Python)
- What are the .apy files in Python (Python)
- C:\Program Files\Common keeps opening on boot (Windows NT / 2000 / XP)
- Export multiple csv files with Python (Python)
- How to set default directory for saving files in Python IDLE (Python)
- Searching for Files through a Directory -- PLEASE HELP!!! (Python)
- freevo uses python (Python)
- Several problem with video files (Windows Software)
- Working with array of files (C++)
Other Threads in the Python Forum
- Previous Thread: matplotlib legend goes opaque when saved as eps
- Next Thread: WX python button trouble
| Thread Tools | Search this Thread |
You need to join our community in order to build your list of favorite forums. You can visit the forum index for a listing of all forums.
Latest Python Posts
- Today's Posts
- Unanswered Threads
Related Forum Features
accessdenied apache application argv array beginner book builtin change chmod converter countpasswordentry curved dan08 dictionary dynamic edit enter examples file filename float format function gui homework import inches input java keyboard lapse library line lines linux list lists loop microphone mouse movingimageswithpygame mysql mysqlquery newb number numbers numeric output parameters parsing path phonebook plugin port prime programming projects py2exe pygame pyopengl pysimplewizard python random recursion redirect remote reverse scrolledtext session simple smtp software sprite statictext string strings syntax table tennis terminal text textarea thread threading time tkinter tlapse trick tuple tutorial ubuntu unicode unit urllib urllib2 variable windows wordgame wxpython
